2024-09-10T20:27:42.732 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T20:27:42.755 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:27:42.844 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622 branch: wip-hemanth-testing-2024-09-10-0723 description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} email: hyelloji@redhat.com first_in_suite: false job_id: '7899622' kernel: client: sha1: distro flavor: default kdb: 1 sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: pid file: '' global: ms die on skipped message: false mgr: client mount timeout: 30 debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6190 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 479/1024 suite: fs suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHd80x6m2B/v65Fp3xZZwWlBYopdV0S+7+h+Yw4RPGGn0R3bBxlIHSIrjkKWVKxVNAK2PCp3K51oXAdmXWRbtJg= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJGNyHwlIkvDvh/X2kKAL3uV77D/4ACogTrSKXEu+FxRh+lQteFZPjyPoUK7AluuY8Q8F++HAJPdmb5L7hEawM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestMisc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_16:59:47 tube: smithi user: hyelloji verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T20:27:42.844 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-10T20:27:42.845 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-10T20:27:42.845 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T20:27:42.847 INFO:teuthology.task.internal:Checking packages... 2024-09-10T20:27:42.869 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8f219c476cb4f26fc5719c661c0323719a45cb30' 2024-09-10T20:27:42.869 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T20:27:42.869 INFO:teuthology.packaging:ref: None 2024-09-10T20:27:42.869 INFO:teuthology.packaging:tag: None 2024-09-10T20:27:42.869 INFO:teuthology.packaging:branch: wip-hemanth-testing-2024-09-10-0723 2024-09-10T20:27:42.869 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-10T20:27:42.870 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-hemanth-testing-2024-09-10-0723 2024-09-10T20:27:43.016 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4854.g8f219c47 2024-09-10T20:27:43.018 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T20:27:43.028 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T20:27:43.028 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T20:27:43.074 INFO:teuthology.task.internal:Saving configuration 2024-09-10T20:27:43.090 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T20:27:43.098 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T20:27:43.118 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 20:22:44.800299', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHd80x6m2B/v65Fp3xZZwWlBYopdV0S+7+h+Yw4RPGGn0R3bBxlIHSIrjkKWVKxVNAK2PCp3K51oXAdmXWRbtJg='} 2024-09-10T20:27:43.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 20:22:44.798992', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJGNyHwlIkvDvh/X2kKAL3uV77D/4ACogTrSKXEu+FxRh+lQteFZPjyPoUK7AluuY8Q8F++HAJPdmb5L7hEawM='} 2024-09-10T20:27:43.140 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T20:27:43.146 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-09-10T20:27:43.146 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-09-10T20:27:43.146 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T20:27:43.228 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb6eb0bfbe0>, signals=[15]) 2024-09-10T20:27:43.228 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T20:27:43.233 INFO:teuthology.task.internal:Opening connections... 2024-09-10T20:27:43.233 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:27:43.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:27:43.317 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:27:43.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:27:43.394 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T20:27:43.401 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-09-10T20:27:43.419 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-09-10T20:27:43.419 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-09-10T20:27:43.477 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-09-10T20:27:43.477 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-09-10T20:27:43.477 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T20:27:43.478 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T20:27:43.479 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-09-10T20:27:43.500 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-09-10T20:27:43.518 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-09-10T20:27:43.519 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-09-10T20:27:43.576 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-09-10T20:27:43.576 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9" 2024-09-10T20:27:43.576 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-09-10T20:27:43.576 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-09-10T20:27:43.576 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9" 2024-09-10T20:27:43.576 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9" 2024-09-10T20:27:43.576 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T20:27:43.577 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-09-10T20:27:43.577 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon" 2024-09-10T20:27:43.577 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T20:27:43.577 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-09-10T20:27:43.577 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T20:27:43.577 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T20:27:43.577 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T20:27:43.577 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-09-10T20:27:43.600 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T20:27:43.609 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T20:27:43.632 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T20:27:43.632 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-09-10T20:27:43.635 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-09-10T20:27:43.652 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T20:27:43.658 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T20:27:43.658 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-09-10T20:27:43.693 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-09-10T20:27:43.730 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T20:27:43.782 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T20:27:43.796 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'} 2024-09-10T20:27:43.796 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-09-10T20:27:43.796 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-09-10T20:27:43.796 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T20:27:43.797 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T20:27:43.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:27:43.814 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-10T20:27:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:27:43.815 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-09-10T20:27:43.873 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-503.el9.x86_64 2024-09-10T20:27:43.873 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-503.el9.x86_64 2024-09-10T20:27:43.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-09-10T20:27:43.874 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-503.el9.x86_64 2024-09-10T20:27:43.874 INFO:teuthology.task.kernel:Running kernel on smithi159: 5.14.0-503.el9.x86_64 2024-09-10T20:27:43.875 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2024-09-10T20:27:47.639 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-09-10T20:27:48.663 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.2 MB 00:03 2024-09-10T20:27:50.939 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-10T20:27:51.365 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 20 MB 00:02 2024-09-10T20:27:56.132 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.5 MB 00:00 2024-09-10T20:27:56.583 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-10T20:27:57.808 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 19 kB 00:00 2024-09-10T20:27:58.023 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 97 kB/s | 19 kB 00:00 2024-09-10T20:27:59.370 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-10T20:27:59.552 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-10T20:28:04.835 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-10T20:28:05.208 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-10T20:28:06.850 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T20:28:06.850 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T20:28:06.946 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-10T20:28:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:Install 4 Packages 2024-09-10T20:28:06.953 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:28:06.953 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 79 M 2024-09-10T20:28:06.953 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 126 M 2024-09-10T20:28:06.953 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-10T20:28:07.303 INFO:teuthology.orchestra.run.smithi159.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 229 kB/s | 23 kB 00:00 2024-09-10T20:28:07.314 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T20:28:07.315 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T20:28:07.409 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T20:28:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T20:28:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-10T20:28:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T20:28:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T20:28:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T20:28:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T20:28:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:28:07.415 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 79 M 2024-09-10T20:28:07.416 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 126 M 2024-09-10T20:28:07.416 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T20:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 163 kB/s | 23 kB 00:00 2024-09-10T20:28:07.687 INFO:teuthology.orchestra.run.smithi159.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-10T20:28:08.128 INFO:teuthology.orchestra.run.smithi159.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-10T20:28:08.395 INFO:teuthology.orchestra.run.smithi159.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-10T20:28:08.395 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:28:08.395 INFO:teuthology.orchestra.run.smithi159.stdout:Total 55 MB/s | 79 MB 00:01 2024-09-10T20:28:08.685 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-10T20:28:08.952 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 21 MB/s | 29 MB 00:01 2024-09-10T20:28:09.084 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-10T20:28:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 9.5 MB/s | 16 MB 00:01 2024-09-10T20:28:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:28:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:Total 47 MB/s | 79 MB 00:01 2024-09-10T20:28:09.450 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-10T20:28:09.450 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-10T20:28:09.810 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T20:28:10.198 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T20:28:10.199 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T20:28:10.400 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-10T20:28:10.401 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-10T20:28:11.206 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T20:28:11.207 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T20:28:12.856 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-10T20:28:13.613 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:28:13.704 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:28:13.804 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T20:28:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:28:14.561 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:28:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:28:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:28:17.352 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:28:17.522 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:28:21.494 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:28:21.625 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:28:22.648 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:28:22.680 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:28:26.493 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:28:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:04.821 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:04.863 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:05.267 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:05.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:29:05.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:29:05.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:29:06.114 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:06.114 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:29:06.114 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-10T20:29:06.115 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T20:29:06.115 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T20:29:06.115 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T20:29:06.115 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T20:29:06.115 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:29:06.115 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-10T20:29:06.460 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:29:06.548 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:29:06.548 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:29:06.548 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T20:29:06.548 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T20:29:06.548 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T20:29:06.549 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-09-10T20:29:06.549 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T20:29:06.549 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:29:06.607 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:29:06.607 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:29:06.608 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2024-09-10T20:29:06.640 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2024-09-10T20:29:06.640 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2024-09-10T20:29:06.640 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.06 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 82.el9 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Group : Unspecified 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 8274670 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : builder@centos.org 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : CentOS 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T20:29:06.641 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:29:06.642 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T20:29:06.643 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T20:29:06.643 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 5.14.0-505.el9.x86_64 2024-09-10T20:29:06.643 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T20:29:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:29:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:29:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:29:07.261 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2024-09-10T20:29:08.111 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:29:08.112 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T20:29:08.444 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:29:08.530 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:29:08.530 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T20:29:08.530 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T20:29:08.530 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T20:29:08.530 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-10T20:29:08.530 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T20:29:08.530 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:29:08.590 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:29:08.590 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:29:08.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-09-10T20:29:08.622 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 82.el9 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8274670 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-09-10T20:29:08.623 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T20:29:08.624 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T20:29:08.625 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-09-10T20:29:08.625 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:29:08.625 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T20:29:08.626 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T20:29:08.626 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-505.el9.x86_64 2024-09-10T20:29:08.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T20:29:08.644 INFO:teuthology.orchestra.run.smithi159.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T20:29:08.667 INFO:teuthology.orchestra.run.smithi159.stderr:done 2024-09-10T20:29:08.669 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-09-10T20:29:08.684 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.dInS0JsbMg 2024-09-10T20:29:08.685 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dInS0JsbMg 2024-09-10T20:29:08.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.dInS0JsbMg 2024-09-10T20:29:08.898 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.dInS0JsbMg is 6KB 2024-09-10T20:29:08.946 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.dInS0JsbMg 2024-09-10T20:29:08.962 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T20:29:09.025 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T20:29:09.025 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T20:29:09.025 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T20:29:09.025 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T20:29:09.026 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T20:29:09.185 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2024-09-10T20:29:09.259 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-09-10T20:29:09.276 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-09-10T20:29:10.523 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T20:29:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-09-10T20:29:10.551 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-10T20:29:10.566 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.vvttTWS5cn 2024-09-10T20:29:10.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vvttTWS5cn 2024-09-10T20:29:10.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.vvttTWS5cn 2024-09-10T20:29:10.779 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.vvttTWS5cn is 6KB 2024-09-10T20:29:10.827 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.vvttTWS5cn 2024-09-10T20:29:10.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T20:29:10.904 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T20:29:10.904 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T20:29:10.904 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T20:29:10.904 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T20:29:10.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T20:29:11.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-09-10T20:29:39.216 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-09-10T20:29:39.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-10T20:29:39.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:29:41.061 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-10T20:29:41.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T20:29:41.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:30:13.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-09-10T20:30:22.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-10T20:30:22.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:30:41.081 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T20:30:41.213 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-09-10T20:30:50.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T20:30:50.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:30:50.467 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-10T20:30:50.815 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T20:30:50.816 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-10T20:30:50.816 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-10T20:30:50.832 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-505.el9.x86_64 2024-09-10T20:30:50.832 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T20:30:50.833 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T20:30:50.833 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T20:30:51.833 DEBUG:teuthology.parallel:result is None 2024-09-10T20:30:53.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-10T20:30:53.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:30:53.606 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-09-10T20:30:53.955 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-10T20:30:53.955 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-505.el9.x86_64"... 2024-09-10T20:30:53.955 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-09-10T20:30:53.970 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-505.el9.x86_64 2024-09-10T20:30:53.970 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T20:30:53.971 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T20:30:53.971 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T20:30:54.972 DEBUG:teuthology.parallel:result is None 2024-09-10T20:30:54.972 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T20:30:54.979 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T20:30:54.980 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T20:30:54.983 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T20:30:54.999 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T20:30:55.024 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T20:30:55.063 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T20:30:55.064 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T20:30:55.067 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T20:30:55.094 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T20:30:55.105 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T20:30:55.105 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T20:30:55.131 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T20:30:55.199 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:55.213 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:55.229 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:55.269 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:55.271 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T20:30:55.277 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T20:30:55.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T20:30:55.280 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T20:30:55.308 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T20:30:55.317 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T20:30:55.317 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T20:30:55.351 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T20:30:55.367 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:30:55.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:30:55.505 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:30:55.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:30:55.636 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:30:55.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T20:30:55.703 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:30:55.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:30:55.800 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:30:55.881 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:30:55.917 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:30:55.918 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T20:30:55.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-09-10T20:30:55.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-09-10T20:30:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:30:56.061 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:30:56.391 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T20:30:56.399 INFO:teuthology.task.internal:Starting timer... 2024-09-10T20:30:56.399 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T20:30:56.429 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T20:30:56.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-09-10T20:30:56.572 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-09-10T20:30:56.573 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-09-10T20:30:56.676 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-09-10T20:30:56.678 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T20:30:56.678 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-09-10T20:30:56.698 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-09-10T20:30:56.698 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-09-10T20:30:56.728 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-09-10T20:30:56.728 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-09-10T20:30:56.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:30:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:30:56.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:30:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:30:56.792 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T20:30:56.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-09-10T20:30:56.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-09-10T20:30:57.143 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T20:30:57.153 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T20:30:57.166 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T20:30:57.167 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T20:34:45.431 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-09-10T20:34:45.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T20:34:45.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:34:45.544 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-10T20:34:45.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T20:34:45.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-10T20:34:45.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:34:45.731 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-09-10T20:34:45.808 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-10T20:34:45.808 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T20:34:45.818 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T20:34:45.818 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T20:34:45.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:34:45.822 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T20:34:45.822 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:34:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T20:34:45.868 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T20:34:45.877 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T20:34:45.892 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T20:34:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-09-10T20:34:45.907 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-09-10T20:34:45.917 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-09-10T20:34:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T20:34:45.928 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-09-10T20:34:45.937 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T20:34:45.942 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T20:34:45.956 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T20:34:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:34:46.003 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:34:46.003 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-10T20:34:46.003 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.003 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.003 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.003 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.009 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:34:46.012 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:34:46.013 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-09-10T20:34:46.013 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.013 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.013 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.013 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:34:46.014 INFO:teuthology.run_tasks:Running task install... 2024-09-10T20:34:46.021 DEBUG:teuthology.task.install:project ceph 2024-09-10T20:34:46.023 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-10T20:34:46.023 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-10T20:34:46.023 INFO:teuthology.task.install:Using flavor: default 2024-09-10T20:34:46.028 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T20:34:46.029 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-10T20:34:46.029 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-10T20:34:46.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-10T20:34:46.032 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-10T20:34:46.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-10T20:34:46.185 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/ 2024-09-10T20:34:46.186 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4854.g8f219c47 2024-09-10T20:34:46.204 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/ 2024-09-10T20:34:46.205 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4854.g8f219c47 2024-09-10T20:34:46.314 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T20:34:46.314 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:34:46.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T20:34:46.327 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T20:34:46.327 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:34:46.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T20:34:46.344 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-10T20:34:46.345 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8f219c476cb4f26fc5719c661c0323719a45cb30/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T20:34:46.354 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-10T20:34:46.354 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8f219c476cb4f26fc5719c661c0323719a45cb30/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T20:34:46.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T20:34:46.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T20:34:46.490 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T20:34:46.495 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T20:34:46.555 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-09-10T20:34:46.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-09-10T20:34:46.558 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-09-10T20:34:46.560 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-09-10T20:34:46.795 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-09-10T20:34:46.796 INFO:teuthology.orchestra.run.smithi159.stdout:45 files removed 2024-09-10T20:34:46.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-10T20:34:46.826 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-10T20:34:47.380 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 329 kB/s | 84 kB 00:00 2024-09-10T20:34:47.386 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 309 kB/s | 84 kB 00:00 2024-09-10T20:34:47.663 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 102 kB/s | 24 kB 00:00 2024-09-10T20:34:47.670 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-10T20:34:47.945 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-10T20:34:47.947 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-10T20:34:48.828 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.2 MB 00:00 2024-09-10T20:34:51.555 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.2 MB 00:03 2024-09-10T20:34:58.863 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 20 MB 00:08 2024-09-10T20:35:03.265 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-10T20:35:05.491 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-10T20:35:07.120 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-10T20:35:12.518 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-10T20:35:13.105 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 20 MB 00:20 2024-09-10T20:35:14.864 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-10T20:35:14.865 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-10T20:35:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-10T20:35:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-10T20:35:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-10T20:35:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-10T20:35:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-10T20:35:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-10T20:35:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-10T20:35:14.953 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-10T20:35:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================= 2024-09-10T20:35:14.973 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-10T20:35:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================= 2024-09-10T20:35:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 6.6 k 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 900 k 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 145 k 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 1.5 M 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 154 k 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 14 M 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 7.4 M 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 49 k 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 11 M 2024-09-10T20:35:14.974 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 50 M 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 285 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 783 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 225 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-shell x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 27 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 18 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 32 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 755 k 2024-09-10T20:35:14.975 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 127 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 161 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 321 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 306 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 99 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 85 k 2024-09-10T20:35:14.976 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 171 k 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.5 M 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-10T20:35:14.977 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 22 M 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 34 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 2.4 M 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 291 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 4.8 M 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 17 M 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 16 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 25 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T20:35:14.978 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 162 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T20:35:14.979 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 507 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-10T20:35:14.980 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 45 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 143 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T20:35:14.981 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T20:35:14.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T20:35:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T20:35:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T20:35:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T20:35:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T20:35:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T20:35:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T20:35:14.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T20:35:14.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T20:35:14.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Installing weak dependencies: 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T20:35:14.986 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:35:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-10T20:35:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================= 2024-09-10T20:35:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Install 149 Packages 2024-09-10T20:35:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 8 Packages 2024-09-10T20:35:14.987 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:35:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 240 M 2024-09-10T20:35:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-10T20:35:15.619 INFO:teuthology.orchestra.run.smithi159.stdout:(1/157): ceph-19.3.0-4854.g8f219c47.el9.x86_64. 50 kB/s | 6.6 kB 00:00 2024-09-10T20:35:15.852 INFO:teuthology.orchestra.run.smithi159.stdout:(2/157): ceph-fuse-19.3.0-4854.g8f219c47.el9.x8 3.8 MB/s | 900 kB 00:00 2024-09-10T20:35:15.953 INFO:teuthology.orchestra.run.smithi159.stdout:(3/157): ceph-base-19.3.0-4854.g8f219c47.el9.x8 12 MB/s | 5.5 MB 00:00 2024-09-10T20:35:15.978 INFO:teuthology.orchestra.run.smithi159.stdout:(4/157): ceph-immutable-object-cache-19.3.0-485 1.1 MB/s | 145 kB 00:00 2024-09-10T20:35:16.112 INFO:teuthology.orchestra.run.smithi159.stdout:(5/157): ceph-mds-19.3.0-4854.g8f219c47.el9.x86 15 MB/s | 2.4 MB 00:00 2024-09-10T20:35:16.170 INFO:teuthology.orchestra.run.smithi159.stdout:(6/157): ceph-mgr-19.3.0-4854.g8f219c47.el9.x86 8.0 MB/s | 1.5 MB 00:00 2024-09-10T20:35:16.379 INFO:teuthology.orchestra.run.smithi159.stdout:(7/157): ceph-mon-19.3.0-4854.g8f219c47.el9.x86 18 MB/s | 4.8 MB 00:00 2024-09-10T20:35:16.829 INFO:teuthology.orchestra.run.smithi159.stdout:(8/157): ceph-radosgw-19.3.0-4854.g8f219c47.el9 24 MB/s | 11 MB 00:00 2024-09-10T20:35:16.879 INFO:teuthology.orchestra.run.smithi159.stdout:(9/157): ceph-selinux-19.3.0-4854.g8f219c47.el9 505 kB/s | 25 kB 00:00 2024-09-10T20:35:17.130 INFO:teuthology.orchestra.run.smithi159.stdout:(10/157): ceph-common-19.3.0-4854.g8f219c47.el9 13 MB/s | 22 MB 00:01 2024-09-10T20:35:17.330 INFO:teuthology.orchestra.run.smithi159.stdout:(11/157): ceph-osd-19.3.0-4854.g8f219c47.el9.x8 15 MB/s | 17 MB 00:01 2024-09-10T20:35:17.355 INFO:teuthology.orchestra.run.smithi159.stdout:(12/157): cephfs-mirror-19.3.0-4854.g8f219c47.e 1.0 MB/s | 225 kB 00:00 2024-09-10T20:35:17.381 INFO:teuthology.orchestra.run.smithi159.stdout:(13/157): cephfs-shell-19.3.0-4854.g8f219c47.el 539 kB/s | 27 kB 00:00 2024-09-10T20:35:17.406 INFO:teuthology.orchestra.run.smithi159.stdout:(14/157): libcephfs-devel-19.3.0-4854.g8f219c47 635 kB/s | 32 kB 00:00 2024-09-10T20:35:17.448 INFO:teuthology.orchestra.run.smithi159.stdout:(15/157): libcephfs2-19.3.0-4854.g8f219c47.el9. 11 MB/s | 755 kB 00:00 2024-09-10T20:35:17.473 INFO:teuthology.orchestra.run.smithi159.stdout:(16/157): libcephsqlite-19.3.0-4854.g8f219c47.e 2.4 MB/s | 162 kB 00:00 2024-09-10T20:35:17.498 INFO:teuthology.orchestra.run.smithi159.stdout:(17/157): librados-devel-19.3.0-4854.g8f219c47. 2.5 MB/s | 127 kB 00:00 2024-09-10T20:35:17.532 INFO:teuthology.orchestra.run.smithi159.stdout:(18/157): libradosstriper1-19.3.0-4854.g8f219c4 8.5 MB/s | 507 kB 00:00 2024-09-10T20:35:17.591 INFO:teuthology.orchestra.run.smithi159.stdout:(19/157): python3-ceph-argparse-19.3.0-4854.g8f 776 kB/s | 45 kB 00:00 2024-09-10T20:35:17.691 INFO:teuthology.orchestra.run.smithi159.stdout:(20/157): python3-ceph-common-19.3.0-4854.g8f21 1.4 MB/s | 143 kB 00:00 2024-09-10T20:35:17.833 INFO:teuthology.orchestra.run.smithi159.stdout:(21/157): librgw2-19.3.0-4854.g8f219c47.el9.x86 17 MB/s | 5.5 MB 00:00 2024-09-10T20:35:17.858 INFO:teuthology.orchestra.run.smithi159.stdout:(22/157): python3-cephfs-19.3.0-4854.g8f219c47. 968 kB/s | 161 kB 00:00 2024-09-10T20:35:17.883 INFO:teuthology.orchestra.run.smithi159.stdout:(23/157): python3-rados-19.3.0-4854.g8f219c47.e 6.3 MB/s | 321 kB 00:00 2024-09-10T20:35:17.917 INFO:teuthology.orchestra.run.smithi159.stdout:(24/157): python3-rbd-19.3.0-4854.g8f219c47.el9 5.1 MB/s | 306 kB 00:00 2024-09-10T20:35:17.942 INFO:teuthology.orchestra.run.smithi159.stdout:(25/157): python3-rgw-19.3.0-4854.g8f219c47.el9 1.7 MB/s | 99 kB 00:00 2024-09-10T20:35:17.968 INFO:teuthology.orchestra.run.smithi159.stdout:(26/157): rbd-fuse-19.3.0-4854.g8f219c47.el9.x8 1.7 MB/s | 85 kB 00:00 2024-09-10T20:35:18.018 INFO:teuthology.orchestra.run.smithi159.stdout:(27/157): rbd-nbd-19.3.0-4854.g8f219c47.el9.x86 3.3 MB/s | 171 kB 00:00 2024-09-10T20:35:18.102 INFO:teuthology.orchestra.run.smithi159.stdout:(28/157): rbd-mirror-19.3.0-4854.g8f219c47.el9. 20 MB/s | 3.2 MB 00:00 2024-09-10T20:35:18.127 INFO:teuthology.orchestra.run.smithi159.stdout:(29/157): ceph-grafana-dashboards-19.3.0-4854.g 315 kB/s | 34 kB 00:00 2024-09-10T20:35:18.152 INFO:teuthology.orchestra.run.smithi159.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4854.g8f219c4 3.0 MB/s | 154 kB 00:00 2024-09-10T20:35:18.619 INFO:teuthology.orchestra.run.smithi159.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 16 MB/s | 7.4 MB 00:00 2024-09-10T20:35:18.678 INFO:teuthology.orchestra.run.smithi159.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4854.g8f 4.9 MB/s | 291 kB 00:00 2024-09-10T20:35:18.753 INFO:teuthology.orchestra.run.smithi159.stdout:(33/157): ceph-mgr-rook-19.3.0-4854.g8f219c47.e 659 kB/s | 49 kB 00:00 2024-09-10T20:35:18.929 INFO:teuthology.orchestra.run.smithi159.stdout:(34/157): ceph-mgr-dashboard-19.3.0-4854.g8f219 18 MB/s | 14 MB 00:00 2024-09-10T20:35:18.945 INFO:teuthology.orchestra.run.smithi159.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4854.g8 86 kB/s | 16 kB 00:00 2024-09-10T20:35:18.979 INFO:teuthology.orchestra.run.smithi159.stdout:(36/157): ceph-volume-19.3.0-4854.g8f219c47.el9 5.6 MB/s | 285 kB 00:00 2024-09-10T20:35:19.021 INFO:teuthology.orchestra.run.smithi159.stdout:(37/157): cephadm-19.3.0-4854.g8f219c47.el9.noa 10 MB/s | 783 kB 00:00 2024-09-10T20:35:19.046 INFO:teuthology.orchestra.run.smithi159.stdout:(38/157): cephfs-top-19.3.0-4854.g8f219c47.el9. 271 kB/s | 18 kB 00:00 2024-09-10T20:35:19.547 INFO:teuthology.orchestra.run.smithi159.stdout:(39/157): ceph-test-19.3.0-4854.g8f219c47.el9.x 19 MB/s | 50 MB 00:02 2024-09-10T20:35:19.572 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-10T20:35:19.572 INFO:teuthology.orchestra.run.smithi159.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 145 kB/s | 80 kB 00:00 2024-09-10T20:35:19.714 INFO:teuthology.orchestra.run.smithi159.stdout:(41/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-10T20:35:19.747 INFO:teuthology.orchestra.run.smithi159.stdout:(42/157): libquadmath-11.5.0-2.el9.x86_64.rpm 956 kB/s | 191 kB 00:00 2024-09-10T20:35:19.773 INFO:teuthology.orchestra.run.smithi159.stdout:(43/157): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-10T20:35:19.806 INFO:teuthology.orchestra.run.smithi159.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-10T20:35:19.832 INFO:teuthology.orchestra.run.smithi159.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.2 MB/s | 135 kB 00:00 2024-09-10T20:35:19.865 INFO:teuthology.orchestra.run.smithi159.stdout:(46/157): python3-cryptography-36.0.1-4.el9.x86 8.1 MB/s | 1.2 MB 00:00 2024-09-10T20:35:19.891 INFO:teuthology.orchestra.run.smithi159.stdout:(47/157): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-10T20:35:19.916 INFO:teuthology.orchestra.run.smithi159.stdout:(48/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-10T20:35:20.376 INFO:teuthology.orchestra.run.smithi159.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 233 kB/s | 107 kB 00:00 2024-09-10T20:35:20.492 INFO:teuthology.orchestra.run.smithi159.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 186 kB/s | 22 kB 00:00 2024-09-10T20:35:20.650 INFO:teuthology.orchestra.run.smithi159.stdout:(51/157): automake-1.16.2-8.el9.noarch.rpm 878 kB/s | 689 kB 00:00 2024-09-10T20:35:20.767 INFO:teuthology.orchestra.run.smithi159.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-09-10T20:35:20.792 INFO:teuthology.orchestra.run.smithi159.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-09-10T20:35:20.817 INFO:teuthology.orchestra.run.smithi159.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 963 kB/s | 313 kB 00:00 2024-09-10T20:35:20.910 INFO:teuthology.orchestra.run.smithi159.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 155 kB/s | 18 kB 00:00 2024-09-10T20:35:20.935 INFO:teuthology.orchestra.run.smithi159.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-09-10T20:35:21.027 INFO:teuthology.orchestra.run.smithi159.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 1.4 MB/s | 171 kB 00:00 2024-09-10T20:35:21.052 INFO:teuthology.orchestra.run.smithi159.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-09-10T20:35:21.144 INFO:teuthology.orchestra.run.smithi159.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-10T20:35:21.261 INFO:teuthology.orchestra.run.smithi159.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-10T20:35:21.353 INFO:teuthology.orchestra.run.smithi159.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 2.7 MB/s | 581 kB 00:00 2024-09-10T20:35:21.378 INFO:teuthology.orchestra.run.smithi159.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 209 kB/s | 24 kB 00:00 2024-09-10T20:35:21.465 INFO:teuthology.orchestra.run.smithi159.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-09-10T20:35:21.525 INFO:teuthology.orchestra.run.smithi159.stdout:(64/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-10T20:35:21.579 INFO:teuthology.orchestra.run.smithi159.stdout:(65/157): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-10T20:35:21.638 INFO:teuthology.orchestra.run.smithi159.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 338 kB/s | 38 kB 00:00 2024-09-10T20:35:21.705 INFO:teuthology.orchestra.run.smithi159.stdout:(67/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.2 MB/s | 3.0 MB 00:00 2024-09-10T20:35:21.839 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-10T20:35:22.180 INFO:teuthology.orchestra.run.smithi159.stdout:(68/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 8.2 MB/s | 4.4 MB 00:00 2024-09-10T20:35:22.314 INFO:teuthology.orchestra.run.smithi159.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 6.7 MB/s | 910 kB 00:00 2024-09-10T20:35:22.464 INFO:teuthology.orchestra.run.smithi159.stdout:(70/157): postgresql-private-libs-13.16-1.el9.x 915 kB/s | 137 kB 00:00 2024-09-10T20:35:22.964 INFO:teuthology.orchestra.run.smithi159.stdout:(71/157): openblas-openmp-0.3.26-2.el9.x86_64.r 3.6 MB/s | 4.9 MB 00:01 2024-09-10T20:35:23.018 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-10T20:35:24.081 INFO:teuthology.orchestra.run.smithi159.stdout:(72/157): postgresql-13.16-1.el9.x86_64.rpm 683 kB/s | 1.6 MB 00:02 2024-09-10T20:35:24.315 INFO:teuthology.orchestra.run.smithi159.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 375 kB/s | 87 kB 00:00 2024-09-10T20:35:24.581 INFO:teuthology.orchestra.run.smithi159.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 639 kB/s | 1.0 MB 00:01 2024-09-10T20:35:24.690 INFO:teuthology.orchestra.run.smithi159.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 565 kB/s | 61 kB 00:00 2024-09-10T20:35:25.107 INFO:teuthology.orchestra.run.smithi159.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 587 kB/s | 245 kB 00:00 2024-09-10T20:35:25.491 INFO:teuthology.orchestra.run.smithi159.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 647 kB/s | 248 kB 00:00 2024-09-10T20:35:25.591 INFO:teuthology.orchestra.run.smithi159.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-09-10T20:35:25.691 INFO:teuthology.orchestra.run.smithi159.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 192 kB/s | 19 kB 00:00 2024-09-10T20:35:25.983 INFO:teuthology.orchestra.run.smithi159.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 590 kB/s | 172 kB 00:00 2024-09-10T20:35:26.092 INFO:teuthology.orchestra.run.smithi159.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 321 kB/s | 35 kB 00:00 2024-09-10T20:35:28.575 INFO:teuthology.orchestra.run.smithi159.stdout:(82/157): python3-babel-2.9.1-2.el9.noarch.rpm 1.4 MB/s | 6.0 MB 00:04 2024-09-10T20:35:28.699 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-10T20:35:28.742 INFO:teuthology.orchestra.run.smithi159.stdout:(83/157): postgresql-server-13.16-1.el9.x86_64. 942 kB/s | 5.8 MB 00:06 2024-09-10T20:35:28.950 INFO:teuthology.orchestra.run.smithi159.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 668 kB/s | 250 kB 00:00 2024-09-10T20:35:29.134 INFO:teuthology.orchestra.run.smithi159.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 567 kB/s | 222 kB 00:00 2024-09-10T20:35:29.159 INFO:teuthology.orchestra.run.smithi159.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 370 kB/s | 77 kB 00:00 2024-09-10T20:35:29.243 INFO:teuthology.orchestra.run.smithi159.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 390 kB/s | 42 kB 00:00 2024-09-10T20:35:29.535 INFO:teuthology.orchestra.run.smithi159.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 544 kB/s | 159 kB 00:00 2024-09-10T20:35:29.652 INFO:teuthology.orchestra.run.smithi159.stdout:(89/157): python3-protobuf-3.14.0-13.el9.noarch 547 kB/s | 269 kB 00:00 2024-09-10T20:35:29.769 INFO:teuthology.orchestra.run.smithi159.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 437 kB/s | 51 kB 00:00 2024-09-10T20:35:29.894 INFO:teuthology.orchestra.run.smithi159.stdout:(91/157): python3-requests-oauthlib-1.3.0-12.el 429 kB/s | 54 kB 00:00 2024-09-10T20:35:30.011 INFO:teuthology.orchestra.run.smithi159.stdout:(92/157): python3-pyasn1-modules-0.4.8-6.el9.no 586 kB/s | 279 kB 00:00 2024-09-10T20:35:30.128 INFO:teuthology.orchestra.run.smithi159.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 42 kB 00:00 2024-09-10T20:35:30.262 INFO:teuthology.orchestra.run.smithi159.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 331 kB/s | 44 kB 00:00 2024-09-10T20:35:30.637 INFO:teuthology.orchestra.run.smithi159.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 589 kB/s | 221 kB 00:00 2024-09-10T20:35:30.754 INFO:teuthology.orchestra.run.smithi159.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 307 kB/s | 36 kB 00:00 2024-09-10T20:35:30.871 INFO:teuthology.orchestra.run.smithi159.stdout:(97/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 406 kB/s | 47 kB 00:00 2024-09-10T20:35:31.137 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-10T20:35:31.138 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-10T20:35:31.153 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-10T20:35:31.153 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-10T20:35:31.154 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-10T20:35:31.155 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-10T20:35:31.157 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-10T20:35:31.160 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-10T20:35:31.161 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-10T20:35:31.228 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 6.6 k 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 900 k 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 145 k 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 1.5 M 2024-09-10T20:35:31.245 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 154 k 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 14 M 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 7.4 M 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 49 k 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 11 M 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 50 M 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 285 k 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 783 k 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 225 k 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 27 k 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 18 k 2024-09-10T20:35:31.246 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 32 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 755 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 127 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-10T20:35:31.247 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-10T20:35:31.248 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-10T20:35:31.248 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 161 k 2024-09-10T20:35:31.248 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 321 k 2024-09-10T20:35:31.248 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 306 k 2024-09-10T20:35:31.248 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 99 k 2024-09-10T20:35:31.248 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 85 k 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 171 k 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-10T20:35:31.249 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.5 M 2024-09-10T20:35:31.250 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-10T20:35:31.250 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-10T20:35:31.250 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-10T20:35:31.250 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T20:35:31.250 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 22 M 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 34 k 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 2.4 M 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 291 k 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 4.8 M 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 17 M 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 16 k 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 25 k 2024-09-10T20:35:31.251 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 162 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T20:35:31.252 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 507 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T20:35:31.253 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-10T20:35:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 45 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 143 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-10T20:35:31.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T20:35:31.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T20:35:31.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-10T20:35:31.258 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-10T20:35:31.259 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-10T20:35:31.260 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout:Install 149 Packages 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 8 Packages 2024-09-10T20:35:31.261 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:35:31.262 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 240 M 2024-09-10T20:35:31.262 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T20:35:31.338 INFO:teuthology.orchestra.run.smithi159.stdout:(98/157): socat-1.7.4.1-6.el9.x86_64.rpm 652 kB/s | 304 kB 00:00 2024-09-10T20:35:31.463 INFO:teuthology.orchestra.run.smithi159.stdout:(99/157): uuid-1.6.2-55.el9.x86_64.rpm 463 kB/s | 58 kB 00:00 2024-09-10T20:35:31.663 INFO:teuthology.orchestra.run.smithi159.stdout:(100/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 479 kB/s | 96 kB 00:00 2024-09-10T20:35:31.864 INFO:teuthology.orchestra.run.smithi159.stdout:(101/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 318 kB/s | 64 kB 00:00 2024-09-10T20:35:31.972 INFO:teuthology.orchestra.run.smithi159.stdout:(102/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 414 kB/s | 45 kB 00:00 2024-09-10T20:35:32.064 INFO:teuthology.orchestra.run.smithi159.stdout:(103/157): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-09-10T20:35:32.189 INFO:teuthology.orchestra.run.smithi159.stdout:(104/157): protobuf-compiler-3.14.0-13.el9.x86_ 6.7 MB/s | 863 kB 00:00 2024-09-10T20:35:32.208 INFO:teuthology.orchestra.run.smithi028.stdout:(1/157): ceph-19.3.0-4854.g8f219c47.el9.x86_64. 50 kB/s | 6.6 kB 00:00 2024-09-10T20:35:32.257 INFO:teuthology.orchestra.run.smithi159.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 8.1 MB/s | 548 kB 00:00 2024-09-10T20:35:32.290 INFO:teuthology.orchestra.run.smithi159.stdout:(106/157): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-09-10T20:35:32.307 INFO:teuthology.orchestra.run.smithi159.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-10T20:35:32.400 INFO:teuthology.orchestra.run.smithi028.stdout:(2/157): ceph-fuse-19.3.0-4854.g8f219c47.el9.x8 4.6 MB/s | 900 kB 00:00 2024-09-10T20:35:32.467 INFO:teuthology.orchestra.run.smithi028.stdout:(3/157): ceph-immutable-object-cache-19.3.0-485 2.1 MB/s | 145 kB 00:00 2024-09-10T20:35:32.549 INFO:teuthology.orchestra.run.smithi159.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-09-10T20:35:32.568 INFO:teuthology.orchestra.run.smithi028.stdout:(4/157): ceph-base-19.3.0-4854.g8f219c47.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-10T20:35:32.574 INFO:teuthology.orchestra.run.smithi159.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-10T20:35:32.599 INFO:teuthology.orchestra.run.smithi159.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-10T20:35:32.701 INFO:teuthology.orchestra.run.smithi028.stdout:(5/157): ceph-mds-19.3.0-4854.g8f219c47.el9.x86 10 MB/s | 2.4 MB 00:00 2024-09-10T20:35:32.708 INFO:teuthology.orchestra.run.smithi159.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-09-10T20:35:32.743 INFO:teuthology.orchestra.run.smithi028.stdout:(6/157): ceph-mgr-19.3.0-4854.g8f219c47.el9.x86 8.7 MB/s | 1.5 MB 00:00 2024-09-10T20:35:32.800 INFO:teuthology.orchestra.run.smithi159.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-09-10T20:35:32.876 INFO:teuthology.orchestra.run.smithi159.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-10T20:35:32.917 INFO:teuthology.orchestra.run.smithi159.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-10T20:35:32.951 INFO:teuthology.orchestra.run.smithi159.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-09-10T20:35:32.976 INFO:teuthology.orchestra.run.smithi159.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-10T20:35:33.002 INFO:teuthology.orchestra.run.smithi159.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 565 kB/s | 14 kB 00:00 2024-09-10T20:35:33.044 INFO:teuthology.orchestra.run.smithi159.stdout:(118/157): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-10T20:35:33.085 INFO:teuthology.orchestra.run.smithi028.stdout:(7/157): ceph-mon-19.3.0-4854.g8f219c47.el9.x86 12 MB/s | 4.8 MB 00:00 2024-09-10T20:35:33.094 INFO:teuthology.orchestra.run.smithi159.stdout:(119/157): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-10T20:35:33.153 INFO:teuthology.orchestra.run.smithi159.stdout:(120/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-10T20:35:33.194 INFO:teuthology.orchestra.run.smithi159.stdout:(121/157): python3-google-auth-2.34.0-1.el9.noa 5.2 MB/s | 220 kB 00:00 2024-09-10T20:35:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:(8/157): ceph-common-19.3.0-4854.g8f219c47.el9. 17 MB/s | 22 MB 00:01 2024-09-10T20:35:33.361 INFO:teuthology.orchestra.run.smithi159.stdout:(122/157): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-09-10T20:35:33.369 INFO:teuthology.orchestra.run.smithi028.stdout:(9/157): ceph-selinux-19.3.0-4854.g8f219c47.el9 607 kB/s | 25 kB 00:00 2024-09-10T20:35:33.404 INFO:teuthology.orchestra.run.smithi159.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-09-10T20:35:33.429 INFO:teuthology.orchestra.run.smithi159.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-10T20:35:33.454 INFO:teuthology.orchestra.run.smithi159.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 708 kB/s | 18 kB 00:00 2024-09-10T20:35:33.479 INFO:teuthology.orchestra.run.smithi159.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 923 kB/s | 23 kB 00:00 2024-09-10T20:35:33.505 INFO:teuthology.orchestra.run.smithi159.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 779 kB/s | 19 kB 00:00 2024-09-10T20:35:33.530 INFO:teuthology.orchestra.run.smithi159.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 788 kB/s | 20 kB 00:00 2024-09-10T20:35:33.605 INFO:teuthology.orchestra.run.smithi159.stdout:(129/157): python3-numpy-1.20.1-5.el9.x86_64.rp 695 kB/s | 5.1 MB 00:07 2024-09-10T20:35:33.656 INFO:teuthology.orchestra.run.smithi159.stdout:(130/157): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-09-10T20:35:33.689 INFO:teuthology.orchestra.run.smithi159.stdout:(131/157): python3-logutils-0.3.5-21.el9.noarch 555 kB/s | 46 kB 00:00 2024-09-10T20:35:33.715 INFO:teuthology.orchestra.run.smithi159.stdout:(132/157): python3-more-itertools-8.12.0-2.el9. 1.4 MB/s | 79 kB 00:00 2024-09-10T20:35:33.727 INFO:teuthology.orchestra.run.smithi028.stdout:(10/157): ceph-osd-19.3.0-4854.g8f219c47.el9.x8 17 MB/s | 17 MB 00:00 2024-09-10T20:35:33.740 INFO:teuthology.orchestra.run.smithi159.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-09-10T20:35:33.765 INFO:teuthology.orchestra.run.smithi159.stdout:(134/157): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-10T20:35:33.790 INFO:teuthology.orchestra.run.smithi159.stdout:(135/157): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-09-10T20:35:33.824 INFO:teuthology.orchestra.run.smithi159.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-09-10T20:35:33.844 INFO:teuthology.orchestra.run.smithi028.stdout:(11/157): ceph-radosgw-19.3.0-4854.g8f219c47.el 14 MB/s | 11 MB 00:00 2024-09-10T20:35:33.849 INFO:teuthology.orchestra.run.smithi159.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-09-10T20:35:33.869 INFO:teuthology.orchestra.run.smithi028.stdout:(12/157): cephfs-mirror-19.3.0-4854.g8f219c47.e 1.6 MB/s | 225 kB 00:00 2024-09-10T20:35:33.874 INFO:teuthology.orchestra.run.smithi159.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-09-10T20:35:33.896 INFO:teuthology.orchestra.run.smithi028.stdout:(13/157): cephfs-shell-19.3.0-4854.g8f219c47.el 542 kB/s | 27 kB 00:00 2024-09-10T20:35:33.900 INFO:teuthology.orchestra.run.smithi159.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-10T20:35:33.920 INFO:teuthology.orchestra.run.smithi028.stdout:(14/157): libcephfs-devel-19.3.0-4854.g8f219c47 632 kB/s | 32 kB 00:00 2024-09-10T20:35:33.925 INFO:teuthology.orchestra.run.smithi159.stdout:(140/157): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-10T20:35:33.950 INFO:teuthology.orchestra.run.smithi159.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-09-10T20:35:33.953 INFO:teuthology.orchestra.run.smithi028.stdout:(15/157): libcephfs2-19.3.0-4854.g8f219c47.el9. 13 MB/s | 755 kB 00:00 2024-09-10T20:35:33.975 INFO:teuthology.orchestra.run.smithi159.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-10T20:35:33.979 INFO:teuthology.orchestra.run.smithi028.stdout:(16/157): libcephsqlite-19.3.0-4854.g8f219c47.e 2.7 MB/s | 162 kB 00:00 2024-09-10T20:35:33.996 INFO:teuthology.orchestra.run.smithi028.stdout:(17/157): librados-devel-19.3.0-4854.g8f219c47. 3.0 MB/s | 127 kB 00:00 2024-09-10T20:35:34.001 INFO:teuthology.orchestra.run.smithi159.stdout:(143/157): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-10T20:35:34.026 INFO:teuthology.orchestra.run.smithi159.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-10T20:35:34.029 INFO:teuthology.orchestra.run.smithi028.stdout:(18/157): libradosstriper1-19.3.0-4854.g8f219c4 9.9 MB/s | 507 kB 00:00 2024-09-10T20:35:34.051 INFO:teuthology.orchestra.run.smithi159.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 802 kB/s | 20 kB 00:00 2024-09-10T20:35:34.085 INFO:teuthology.orchestra.run.smithi159.stdout:(146/157): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-09-10T20:35:34.110 INFO:teuthology.orchestra.run.smithi159.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-10T20:35:34.113 INFO:teuthology.orchestra.run.smithi028.stdout:(19/157): python3-ceph-argparse-19.3.0-4854.g8f 543 kB/s | 45 kB 00:00 2024-09-10T20:35:34.178 INFO:teuthology.orchestra.run.smithi159.stdout:(148/157): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-10T20:35:34.205 INFO:teuthology.orchestra.run.smithi028.stdout:(20/157): python3-ceph-common-19.3.0-4854.g8f21 1.5 MB/s | 143 kB 00:00 2024-09-10T20:35:34.280 INFO:teuthology.orchestra.run.smithi028.stdout:(21/157): librgw2-19.3.0-4854.g8f219c47.el9.x86 19 MB/s | 5.5 MB 00:00 2024-09-10T20:35:34.314 INFO:teuthology.orchestra.run.smithi028.stdout:(22/157): python3-cephfs-19.3.0-4854.g8f219c47. 1.5 MB/s | 161 kB 00:00 2024-09-10T20:35:34.339 INFO:teuthology.orchestra.run.smithi028.stdout:(23/157): python3-rados-19.3.0-4854.g8f219c47.e 5.3 MB/s | 321 kB 00:00 2024-09-10T20:35:34.373 INFO:teuthology.orchestra.run.smithi028.stdout:(24/157): python3-rbd-19.3.0-4854.g8f219c47.el9 5.1 MB/s | 306 kB 00:00 2024-09-10T20:35:34.406 INFO:teuthology.orchestra.run.smithi028.stdout:(25/157): python3-rgw-19.3.0-4854.g8f219c47.el9 1.5 MB/s | 99 kB 00:00 2024-09-10T20:35:34.457 INFO:teuthology.orchestra.run.smithi028.stdout:(26/157): rbd-fuse-19.3.0-4854.g8f219c47.el9.x8 1.0 MB/s | 85 kB 00:00 2024-09-10T20:35:34.486 INFO:teuthology.orchestra.run.smithi159.stdout:(149/157): librbd1-19.3.0-4854.g8f219c47.el9.x8 11 MB/s | 3.2 MB 00:00 2024-09-10T20:35:34.536 INFO:teuthology.orchestra.run.smithi159.stdout:(150/157): librados2-19.3.0-4854.g8f219c47.el9. 8.1 MB/s | 3.5 MB 00:00 2024-09-10T20:35:34.548 INFO:teuthology.orchestra.run.smithi028.stdout:(27/157): rbd-nbd-19.3.0-4854.g8f219c47.el9.x86 1.8 MB/s | 171 kB 00:00 2024-09-10T20:35:34.649 INFO:teuthology.orchestra.run.smithi028.stdout:(28/157): rbd-mirror-19.3.0-4854.g8f219c47.el9. 13 MB/s | 3.2 MB 00:00 2024-09-10T20:35:34.665 INFO:teuthology.orchestra.run.smithi028.stdout:(29/157): ceph-grafana-dashboards-19.3.0-4854.g 293 kB/s | 34 kB 00:00 2024-09-10T20:35:34.699 INFO:teuthology.orchestra.run.smithi028.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4854.g8f219c4 3.0 MB/s | 154 kB 00:00 2024-09-10T20:35:34.814 INFO:teuthology.orchestra.run.smithi159.stdout:(151/157): NetworkManager-1.49.90-1.el9.x86_64. 7.1 MB/s | 2.3 MB 00:00 2024-09-10T20:35:34.862 INFO:teuthology.orchestra.run.smithi159.stdout:(152/157): NetworkManager-libnm-1.49.90-1.el9.x 5.5 MB/s | 1.8 MB 00:00 2024-09-10T20:35:34.887 INFO:teuthology.orchestra.run.smithi159.stdout:(153/157): NetworkManager-team-1.49.90-1.el9.x8 556 kB/s | 41 kB 00:00 2024-09-10T20:35:34.921 INFO:teuthology.orchestra.run.smithi159.stdout:(154/157): NetworkManager-tui-1.49.90-1.el9.x86 4.1 MB/s | 249 kB 00:00 2024-09-10T20:35:34.996 INFO:teuthology.orchestra.run.smithi159.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 12 MB/s | 1.4 MB 00:00 2024-09-10T20:35:35.080 INFO:teuthology.orchestra.run.smithi159.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-09-10T20:35:35.200 INFO:teuthology.orchestra.run.smithi028.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 15 MB/s | 7.4 MB 00:00 2024-09-10T20:35:35.266 INFO:teuthology.orchestra.run.smithi028.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4854.g8f 4.3 MB/s | 291 kB 00:00 2024-09-10T20:35:35.325 INFO:teuthology.orchestra.run.smithi028.stdout:(33/157): ceph-mgr-rook-19.3.0-4854.g8f219c47.e 847 kB/s | 49 kB 00:00 2024-09-10T20:35:35.459 INFO:teuthology.orchestra.run.smithi028.stdout:(34/157): ceph-mgr-dashboard-19.3.0-4854.g8f219 18 MB/s | 14 MB 00:00 2024-09-10T20:35:35.475 INFO:teuthology.orchestra.run.smithi028.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4854.g8 109 kB/s | 16 kB 00:00 2024-09-10T20:35:35.509 INFO:teuthology.orchestra.run.smithi028.stdout:(36/157): ceph-volume-19.3.0-4854.g8f219c47.el9 5.6 MB/s | 285 kB 00:00 2024-09-10T20:35:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:(37/157): cephadm-19.3.0-4854.g8f219c47.el9.noa 10 MB/s | 783 kB 00:00 2024-09-10T20:35:35.576 INFO:teuthology.orchestra.run.smithi028.stdout:(38/157): cephfs-top-19.3.0-4854.g8f219c47.el9. 272 kB/s | 18 kB 00:00 2024-09-10T20:35:36.010 INFO:teuthology.orchestra.run.smithi028.stdout:(39/157): ceph-test-19.3.0-4854.g8f219c47.el9.x 19 MB/s | 50 MB 00:02 2024-09-10T20:35:36.302 INFO:teuthology.orchestra.run.smithi028.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 106 kB/s | 80 kB 00:00 2024-09-10T20:35:36.560 INFO:teuthology.orchestra.run.smithi028.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 348 kB/s | 191 kB 00:00 2024-09-10T20:35:36.586 INFO:teuthology.orchestra.run.smithi028.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 893 kB/s | 253 kB 00:00 2024-09-10T20:35:36.703 INFO:teuthology.orchestra.run.smithi028.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 915 kB/s | 106 kB 00:00 2024-09-10T20:35:36.736 INFO:teuthology.orchestra.run.smithi028.stdout:(44/157): libgfortran-11.5.0-2.el9.x86_64.rpm 691 kB/s | 801 kB 00:01 2024-09-10T20:35:36.820 INFO:teuthology.orchestra.run.smithi028.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-09-10T20:35:36.845 INFO:teuthology.orchestra.run.smithi028.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-09-10T20:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-09-10T20:35:36.970 INFO:teuthology.orchestra.run.smithi028.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 3.0 MB/s | 1.2 MB 00:00 2024-09-10T20:35:37.071 INFO:teuthology.orchestra.run.smithi028.stdout:(49/157): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 689 kB 00:00 2024-09-10T20:35:37.138 INFO:teuthology.orchestra.run.smithi028.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 325 kB/s | 22 kB 00:00 2024-09-10T20:35:37.163 INFO:teuthology.orchestra.run.smithi028.stdout:(51/157): boost-program-options-1.75.0-8.el9.x8 556 kB/s | 107 kB 00:00 2024-09-10T20:35:37.189 INFO:teuthology.orchestra.run.smithi028.stdout:(52/157): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-09-10T20:35:37.222 INFO:teuthology.orchestra.run.smithi028.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-09-10T20:35:37.255 INFO:teuthology.orchestra.run.smithi028.stdout:(54/157): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 943 kB 00:00 2024-09-10T20:35:37.272 INFO:teuthology.orchestra.run.smithi028.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-09-10T20:35:37.289 INFO:teuthology.orchestra.run.smithi028.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 331 kB/s | 11 kB 00:00 2024-09-10T20:35:37.340 INFO:teuthology.orchestra.run.smithi028.stdout:(57/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-10T20:35:37.365 INFO:teuthology.orchestra.run.smithi028.stdout:(58/157): libnbd-1.20.2-2.el9.x86_64.rpm 1.8 MB/s | 171 kB 00:00 2024-09-10T20:35:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-10T20:35:37.441 INFO:teuthology.orchestra.run.smithi028.stdout:(60/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-09-10T20:35:37.474 INFO:teuthology.orchestra.run.smithi028.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 6.2 MB/s | 581 kB 00:00 2024-09-10T20:35:37.491 INFO:teuthology.orchestra.run.smithi028.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 489 kB/s | 24 kB 00:00 2024-09-10T20:35:37.517 INFO:teuthology.orchestra.run.smithi028.stdout:(63/157): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-10T20:35:37.533 INFO:teuthology.orchestra.run.smithi028.stdout:(64/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 47 kB 00:00 2024-09-10T20:35:37.558 INFO:teuthology.orchestra.run.smithi028.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-10T20:35:37.575 INFO:teuthology.orchestra.run.smithi028.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 920 kB/s | 38 kB 00:00 2024-09-10T20:35:37.592 INFO:teuthology.orchestra.run.smithi028.stdout:(67/157): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-09-10T20:35:37.717 INFO:teuthology.orchestra.run.smithi028.stdout:(68/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-09-10T20:35:37.759 INFO:teuthology.orchestra.run.smithi028.stdout:(69/157): postgresql-13.16-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-09-10T20:35:37.793 INFO:teuthology.orchestra.run.smithi028.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 12 MB/s | 910 kB 00:00 2024-09-10T20:35:37.818 INFO:teuthology.orchestra.run.smithi028.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-09-10T20:35:37.885 INFO:teuthology.orchestra.run.smithi028.stdout:(72/157): openblas-openmp-0.3.26-2.el9.x86_64.r 15 MB/s | 4.9 MB 00:00 2024-09-10T20:35:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-09-10T20:35:37.952 INFO:teuthology.orchestra.run.smithi028.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-10T20:35:37.994 INFO:teuthology.orchestra.run.smithi028.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-10T20:35:38.069 INFO:teuthology.orchestra.run.smithi028.stdout:(76/157): postgresql-server-13.16-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-09-10T20:35:38.095 INFO:teuthology.orchestra.run.smithi028.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-09-10T20:35:38.120 INFO:teuthology.orchestra.run.smithi028.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-09-10T20:35:38.145 INFO:teuthology.orchestra.run.smithi028.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 509 kB/s | 26 kB 00:00 2024-09-10T20:35:38.170 INFO:teuthology.orchestra.run.smithi028.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-09-10T20:35:38.196 INFO:teuthology.orchestra.run.smithi028.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-10T20:35:38.221 INFO:teuthology.orchestra.run.smithi028.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 691 kB/s | 35 kB 00:00 2024-09-10T20:35:38.313 INFO:teuthology.orchestra.run.smithi028.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-10T20:35:38.338 INFO:teuthology.orchestra.run.smithi028.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-09-10T20:35:38.363 INFO:teuthology.orchestra.run.smithi028.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-10T20:35:38.391 INFO:teuthology.orchestra.run.smithi028.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 77 kB 00:00 2024-09-10T20:35:38.408 INFO:teuthology.orchestra.run.smithi028.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 965 kB/s | 42 kB 00:00 2024-09-10T20:35:38.441 INFO:teuthology.orchestra.run.smithi028.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-09-10T20:35:38.466 INFO:teuthology.orchestra.run.smithi028.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-09-10T20:35:38.492 INFO:teuthology.orchestra.run.smithi028.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-09-10T20:35:38.509 INFO:teuthology.orchestra.run.smithi028.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-10T20:35:38.525 INFO:teuthology.orchestra.run.smithi028.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-09-10T20:35:38.592 INFO:teuthology.orchestra.run.smithi028.stdout:(93/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-09-10T20:35:38.609 INFO:teuthology.orchestra.run.smithi028.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 498 kB/s | 42 kB 00:00 2024-09-10T20:35:38.627 INFO:teuthology.orchestra.run.smithi028.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.3 MB/s | 44 kB 00:00 2024-09-10T20:35:38.660 INFO:teuthology.orchestra.run.smithi028.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-10T20:35:38.677 INFO:teuthology.orchestra.run.smithi028.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 715 kB/s | 36 kB 00:00 2024-09-10T20:35:38.694 INFO:teuthology.orchestra.run.smithi028.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-10T20:35:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-10T20:35:38.744 INFO:teuthology.orchestra.run.smithi028.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-10T20:35:38.778 INFO:teuthology.orchestra.run.smithi028.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-10T20:35:38.795 INFO:teuthology.orchestra.run.smithi028.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-10T20:35:38.812 INFO:teuthology.orchestra.run.smithi028.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-09-10T20:35:38.862 INFO:teuthology.orchestra.run.smithi028.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 333 kB/s | 22 kB 00:00 2024-09-10T20:35:39.054 INFO:teuthology.orchestra.run.smithi028.stdout:(105/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-09-10T20:35:39.096 INFO:teuthology.orchestra.run.smithi028.stdout:(106/157): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-09-10T20:35:39.113 INFO:teuthology.orchestra.run.smithi028.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.3 MB/s | 19 kB 00:00 2024-09-10T20:35:39.155 INFO:teuthology.orchestra.run.smithi028.stdout:(108/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-09-10T20:35:39.188 INFO:teuthology.orchestra.run.smithi028.stdout:(109/157): gperftools-libs-2.9.1-2.el9.x86_64.r 2.3 MB/s | 309 kB 00:00 2024-09-10T20:35:39.213 INFO:teuthology.orchestra.run.smithi028.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-10T20:35:39.231 INFO:teuthology.orchestra.run.smithi028.stdout:(111/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 344 kB/s | 26 kB 00:00 2024-09-10T20:35:39.255 INFO:teuthology.orchestra.run.smithi028.stdout:(112/157): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-10T20:35:39.290 INFO:teuthology.orchestra.run.smithi028.stdout:(113/157): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-10T20:35:39.373 INFO:teuthology.orchestra.run.smithi028.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-09-10T20:35:39.406 INFO:teuthology.orchestra.run.smithi028.stdout:(115/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-09-10T20:35:39.431 INFO:teuthology.orchestra.run.smithi028.stdout:(116/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-09-10T20:35:39.457 INFO:teuthology.orchestra.run.smithi028.stdout:(117/157): python3-cachetools-4.2.4-1.el9.noarc 641 kB/s | 32 kB 00:00 2024-09-10T20:35:39.482 INFO:teuthology.orchestra.run.smithi028.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 282 kB/s | 14 kB 00:00 2024-09-10T20:35:39.516 INFO:teuthology.orchestra.run.smithi028.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-09-10T20:35:39.566 INFO:teuthology.orchestra.run.smithi028.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-09-10T20:35:39.592 INFO:teuthology.orchestra.run.smithi028.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-09-10T20:35:39.625 INFO:teuthology.orchestra.run.smithi028.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-10T20:35:39.684 INFO:teuthology.orchestra.run.smithi028.stdout:(123/157): libarrow-9.0.0-11.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-09-10T20:35:39.725 INFO:teuthology.orchestra.run.smithi028.stdout:(124/157): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-09-10T20:35:39.751 INFO:teuthology.orchestra.run.smithi028.stdout:(125/157): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-09-10T20:35:39.776 INFO:teuthology.orchestra.run.smithi028.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-09-10T20:35:39.801 INFO:teuthology.orchestra.run.smithi028.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-09-10T20:35:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-10T20:35:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-10T20:35:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-09-10T20:35:39.919 INFO:teuthology.orchestra.run.smithi028.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-10T20:35:39.953 INFO:teuthology.orchestra.run.smithi028.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-09-10T20:35:39.994 INFO:teuthology.orchestra.run.smithi028.stdout:(133/157): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-09-10T20:35:40.036 INFO:teuthology.orchestra.run.smithi028.stdout:(134/157): python3-grpcio-1.46.7-10.el9.x86_64. 4.6 MB/s | 2.0 MB 00:00 2024-09-10T20:35:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:(135/157): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-09-10T20:35:40.078 INFO:teuthology.orchestra.run.smithi028.stdout:(136/157): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-10T20:35:40.095 INFO:teuthology.orchestra.run.smithi028.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 676 kB/s | 28 kB 00:00 2024-09-10T20:35:40.120 INFO:teuthology.orchestra.run.smithi028.stdout:(138/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-09-10T20:35:40.137 INFO:teuthology.orchestra.run.smithi028.stdout:(139/157): python3-repoze-lru-0.7-16.el9.noarch 739 kB/s | 31 kB 00:00 2024-09-10T20:35:40.154 INFO:teuthology.orchestra.run.smithi028.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-10T20:35:40.180 INFO:teuthology.orchestra.run.smithi028.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 859 kB/s | 36 kB 00:00 2024-09-10T20:35:40.205 INFO:teuthology.orchestra.run.smithi028.stdout:(142/157): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-09-10T20:35:40.230 INFO:teuthology.orchestra.run.smithi028.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 989 kB/s | 74 kB 00:00 2024-09-10T20:35:40.255 INFO:teuthology.orchestra.run.smithi028.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-09-10T20:35:40.281 INFO:teuthology.orchestra.run.smithi028.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-10T20:35:40.306 INFO:teuthology.orchestra.run.smithi028.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-09-10T20:35:40.331 INFO:teuthology.orchestra.run.smithi028.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-10T20:35:40.356 INFO:teuthology.orchestra.run.smithi028.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-10T20:35:40.431 INFO:teuthology.orchestra.run.smithi028.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-10T20:35:40.590 INFO:teuthology.orchestra.run.smithi028.stdout:(150/157): librados2-19.3.0-4854.g8f219c47.el9. 13 MB/s | 3.5 MB 00:00 2024-09-10T20:35:40.757 INFO:teuthology.orchestra.run.smithi028.stdout:(151/157): librbd1-19.3.0-4854.g8f219c47.el9.x8 8.1 MB/s | 3.2 MB 00:00 2024-09-10T20:35:41.049 INFO:teuthology.orchestra.run.smithi028.stdout:(152/157): NetworkManager-team-1.49.90-1.el9.x8 140 kB/s | 41 kB 00:00 2024-09-10T20:35:41.458 INFO:teuthology.orchestra.run.smithi028.stdout:(153/157): NetworkManager-1.49.90-1.el9.x86_64. 2.3 MB/s | 2.3 MB 00:01 2024-09-10T20:35:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:(154/157): NetworkManager-libnm-1.49.90-1.el9.x 2.0 MB/s | 1.8 MB 00:00 2024-09-10T20:35:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:(155/157): NetworkManager-tui-1.49.90-1.el9.x86 514 kB/s | 249 kB 00:00 2024-09-10T20:35:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 10 MB/s | 1.4 MB 00:00 2024-09-10T20:35:41.833 INFO:teuthology.orchestra.run.smithi028.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 7.4 MB/s | 2.4 MB 00:00 2024-09-10T20:35:41.840 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:35:41.840 INFO:teuthology.orchestra.run.smithi028.stdout:Total 23 MB/s | 240 MB 00:10 2024-09-10T20:35:43.070 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T20:35:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T20:35:43.240 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T20:35:45.619 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T20:35:45.620 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T20:35:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T20:35:52.504 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-10T20:35:52.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-10T20:35:52.597 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-10T20:35:52.638 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-10T20:35:52.663 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-10T20:35:52.719 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-10T20:35:53.100 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-10T20:35:53.205 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-10T20:35:53.340 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-10T20:35:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-10T20:35:53.466 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-10T20:35:53.500 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-10T20:35:53.533 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-10T20:35:53.718 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-10T20:35:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-10T20:35:53.803 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-10T20:35:53.909 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-10T20:35:53.937 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-10T20:35:54.000 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-10T20:35:54.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-10T20:35:54.052 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-10T20:35:54.204 INFO:teuthology.orchestra.run.smithi159.stdout:(157/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 685 kB/s | 16 MB 00:24 2024-09-10T20:35:54.209 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:35:54.209 INFO:teuthology.orchestra.run.smithi159.stdout:Total 6.1 MB/s | 240 MB 00:39 2024-09-10T20:35:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-10T20:35:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-10T20:35:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 18/165 2024-09-10T20:35:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-10T20:35:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-10T20:35:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-10T20:35:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-10T20:35:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-10T20:35:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-10T20:35:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-10T20:35:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-10T20:35:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-10T20:35:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-10T20:35:55.394 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-10T20:35:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-10T20:35:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-10T20:35:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-10T20:35:55.548 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-10T20:35:55.548 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-10T20:35:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-10T20:35:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-10T20:35:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-10T20:35:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-10T20:35:55.784 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 34/165 2024-09-10T20:35:55.828 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 35/165 2024-09-10T20:35:55.856 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 36/165 2024-09-10T20:35:55.930 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-10T20:35:55.980 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-10T20:35:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-10T20:35:56.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-10T20:35:56.266 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-10T20:35:56.830 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-10T20:35:56.922 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-10T20:35:56.955 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-10T20:35:56.999 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-10T20:35:57.036 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 46/165 2024-09-10T20:35:57.094 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-10T20:35:57.263 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-10T20:35:57.314 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-10T20:35:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-10T20:35:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-10T20:35:57.554 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-10T20:35:57.666 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-10T20:35:57.811 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-10T20:35:57.859 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-10T20:35:57.870 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-10T20:35:57.870 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-10T20:35:57.893 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-10T20:35:57.937 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-10T20:35:57.979 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-10T20:35:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-10T20:35:58.179 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-10T20:35:58.221 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-10T20:35:58.329 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-10T20:35:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-10T20:35:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-10T20:35:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/165 2024-09-10T20:35:58.460 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-10T20:35:58.569 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-10T20:35:58.700 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-10T20:35:58.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-10T20:35:58.778 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-10T20:35:58.821 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-10T20:35:58.860 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-10T20:35:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-10T20:35:58.930 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-10T20:35:58.964 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-10T20:35:58.998 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-10T20:35:59.040 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-10T20:35:59.074 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-10T20:35:59.167 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-10T20:35:59.477 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-10T20:35:59.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-10T20:35:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-10T20:35:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-10T20:36:00.466 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-10T20:36:00.508 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-10T20:36:00.541 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-10T20:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-10T20:36:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-10T20:36:00.751 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-10T20:36:00.842 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-10T20:36:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-10T20:36:01.070 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-10T20:36:01.110 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-10T20:36:01.206 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-10T20:36:01.920 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-10T20:36:02.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-10T20:36:03.392 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-10T20:36:03.455 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-10T20:36:03.481 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-10T20:36:03.514 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-10T20:36:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-10T20:36:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-10T20:36:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-10T20:36:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 103/165 2024-09-10T20:36:04.060 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-10T20:36:04.367 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-10T20:36:04.516 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-10T20:36:04.554 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-10T20:36:04.588 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-10T20:36:04.608 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-10T20:36:04.621 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-10T20:36:04.642 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-10T20:36:04.676 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-10T20:36:04.692 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-10T20:36:04.722 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 107/165 2024-09-10T20:36:04.725 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-10T20:36:05.086 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-10T20:36:05.236 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-10T20:36:05.341 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-10T20:36:05.370 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-10T20:36:05.462 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-10T20:36:05.501 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-10T20:36:05.539 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-10T20:36:05.722 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-10T20:36:05.781 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-10T20:36:05.809 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-10T20:36:05.913 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-10T20:36:05.943 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-10T20:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-10T20:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-10T20:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-10T20:36:06.247 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-10T20:36:06.415 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-10T20:36:06.551 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 18/165 2024-09-10T20:36:06.661 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-10T20:36:06.728 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-10T20:36:06.961 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-10T20:36:07.065 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-10T20:36:07.099 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-10T20:36:07.133 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-10T20:36:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-10T20:36:07.205 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-10T20:36:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-10T20:36:07.250 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-10T20:36:07.283 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-10T20:36:07.336 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-10T20:36:07.376 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-10T20:36:07.403 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-10T20:36:07.472 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-10T20:36:07.525 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-10T20:36:07.557 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-10T20:36:07.600 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-10T20:36:07.677 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-10T20:36:07.711 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 34/165 2024-09-10T20:36:07.753 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 35/165 2024-09-10T20:36:07.791 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 36/165 2024-09-10T20:36:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-10T20:36:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-10T20:36:07.868 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-10T20:36:07.922 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-10T20:36:07.948 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-10T20:36:07.949 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T20:36:07.951 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T20:36:07.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:07.989 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-10T20:36:07.996 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-10T20:36:08.137 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-10T20:36:08.191 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-10T20:36:08.715 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-10T20:36:08.795 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-10T20:36:08.825 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-10T20:36:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-10T20:36:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 46/165 2024-09-10T20:36:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-10T20:36:09.105 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-10T20:36:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-10T20:36:09.250 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-10T20:36:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-10T20:36:09.388 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-10T20:36:09.419 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-10T20:36:09.463 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-10T20:36:09.518 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-10T20:36:09.554 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-10T20:36:09.597 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-10T20:36:09.640 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-10T20:36:09.775 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-10T20:36:09.808 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-10T20:36:09.850 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-10T20:36:09.951 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-10T20:36:09.992 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-10T20:36:10.032 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-10T20:36:10.057 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/165 2024-09-10T20:36:10.082 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-10T20:36:10.187 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-10T20:36:10.317 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-10T20:36:10.348 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-10T20:36:10.376 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-10T20:36:10.413 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-10T20:36:10.453 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-10T20:36:10.492 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-10T20:36:10.522 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-10T20:36:10.557 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-10T20:36:10.590 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-10T20:36:10.631 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-10T20:36:10.675 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-10T20:36:10.761 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-10T20:36:10.804 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-10T20:36:10.836 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-10T20:36:10.938 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-10T20:36:10.995 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-10T20:36:11.709 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-10T20:36:11.751 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-10T20:36:11.784 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-10T20:36:11.882 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-10T20:36:11.939 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-10T20:36:11.984 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-10T20:36:12.071 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-10T20:36:12.104 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-10T20:36:12.286 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-10T20:36:12.318 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-10T20:36:12.395 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-10T20:36:13.045 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-10T20:36:13.168 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-10T20:36:14.411 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-10T20:36:14.508 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-10T20:36:14.535 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-10T20:36:14.568 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-10T20:36:14.885 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-10T20:36:14.939 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-10T20:36:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-10T20:36:15.053 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 103/165 2024-09-10T20:36:15.088 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-10T20:36:15.591 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-10T20:36:15.617 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-10T20:36:15.688 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-10T20:36:15.711 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 107/165 2024-09-10T20:36:18.009 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-10T20:36:18.049 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-10T20:36:18.600 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-10T20:36:18.635 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-10T20:36:18.729 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-10T20:36:18.729 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T20:36:18.729 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T20:36:18.729 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:18.770 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-09-10T20:36:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-09-10T20:36:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-10T20:36:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-10T20:36:20.232 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:20.232 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T20:36:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:21.793 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-10T20:36:21.821 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-10T20:36:21.821 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:21.821 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T20:36:21.821 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:36:21.821 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:36:21.821 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:22.052 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-10T20:36:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-10T20:36:22.077 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:22.077 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T20:36:22.077 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:36:22.077 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:36:22.077 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:22.544 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-10T20:36:22.569 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-10T20:36:22.569 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:22.569 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T20:36:22.569 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:36:22.569 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:36:22.569 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:22.692 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-10T20:36:22.750 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-10T20:36:22.814 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-10T20:36:23.011 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-10T20:36:23.053 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-10T20:36:23.135 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-10T20:36:23.233 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-10T20:36:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-10T20:36:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-10T20:36:23.478 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-10T20:36:23.506 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-10T20:36:23.937 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-10T20:36:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-10T20:36:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 127/165 2024-09-10T20:36:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 128/165 2024-09-10T20:36:24.093 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-10T20:36:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-10T20:36:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-10T20:36:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-10T20:36:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-10T20:36:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-10T20:36:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 132/165 2024-09-10T20:36:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-10T20:36:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-10T20:36:28.951 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:28.951 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T20:36:28.952 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:36:28.952 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:36:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-10T20:36:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-10T20:36:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 135/165 2024-09-10T20:36:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 136/165 2024-09-10T20:36:30.198 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-10T20:36:30.237 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-10T20:36:30.262 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-10T20:36:30.263 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:30.263 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T20:36:30.263 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:36:30.263 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:36:30.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:30.305 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-10T20:36:30.328 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-10T20:36:30.328 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:30.329 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T20:36:30.329 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:30.380 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-10T20:36:30.404 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-10T20:36:30.404 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:30.405 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-10T20:36:30.405 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-10T20:36:30.405 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-10T20:36:30.405 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:30.739 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-10T20:36:30.764 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-10T20:36:30.764 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:30.764 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T20:36:30.764 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:36:30.764 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:36:30.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-10T20:36:30.918 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-09-10T20:36:30.919 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-09-10T20:36:30.919 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-09-10T20:36:30.919 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-09-10T20:36:30.919 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-09-10T20:36:30.919 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /root 2024-09-10T20:36:30.919 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-09-10T20:36:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:31.099 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-10T20:36:31.115 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-10T20:36:31.115 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:31.115 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T20:36:31.115 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:32.653 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-10T20:36:32.680 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-10T20:36:32.680 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:32.680 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T20:36:32.680 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:36:32.680 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:36:32.681 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:32.904 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-10T20:36:32.927 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-10T20:36:32.927 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:32.927 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T20:36:32.927 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:36:32.927 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:36:32.927 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:33.380 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-10T20:36:33.403 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-10T20:36:33.403 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:33.403 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T20:36:33.403 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:36:33.403 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:36:33.403 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:33.510 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-10T20:36:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-10T20:36:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-10T20:36:33.799 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-10T20:36:33.847 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-10T20:36:33.916 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-10T20:36:34.005 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-10T20:36:34.154 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-10T20:36:34.193 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-10T20:36:34.238 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-10T20:36:34.274 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-10T20:36:34.704 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-10T20:36:34.748 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-10T20:36:34.780 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 127/165 2024-09-10T20:36:34.846 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 128/165 2024-09-10T20:36:34.871 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-10T20:36:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 142/165 2024-09-10T20:36:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 143/165 2024-09-10T20:36:35.843 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 144/165 2024-09-10T20:36:35.874 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-10T20:36:35.901 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-10T20:36:36.111 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 147/165 2024-09-10T20:36:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-10T20:36:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 149/165 2024-09-10T20:36:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 150/165 2024-09-10T20:36:36.253 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 151/165 2024-09-10T20:36:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-10T20:36:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-10T20:36:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-10T20:36:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 153/165 2024-09-10T20:36:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-10T20:36:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-10T20:36:38.058 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-10T20:36:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-10T20:36:38.111 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-10T20:36:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-10T20:36:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-10T20:36:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-10T20:36:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-10T20:36:38.293 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-10T20:36:38.293 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-10T20:36:38.368 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-10T20:36:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-10T20:36:38.398 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-10T20:36:38.398 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-10T20:36:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-10T20:36:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-10T20:36:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-10T20:36:38.723 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-10T20:36:39.072 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-10T20:36:39.102 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-10T20:36:39.217 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-10T20:36:39.373 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 132/165 2024-09-10T20:36:39.405 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-10T20:36:39.429 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-10T20:36:39.429 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:39.429 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T20:36:39.429 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:36:39.429 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:36:39.429 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:39.472 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-10T20:36:39.483 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-10T20:36:39.587 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 135/165 2024-09-10T20:36:39.640 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 136/165 2024-09-10T20:36:40.588 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-10T20:36:40.625 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-10T20:36:40.649 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-10T20:36:40.649 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:40.649 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T20:36:40.649 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:36:40.650 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:36:40.650 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:40.692 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-10T20:36:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-10T20:36:40.715 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:40.715 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T20:36:40.715 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:40.759 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-10T20:36:40.782 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-10T20:36:40.782 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:40.782 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-10T20:36:40.782 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-10T20:36:40.782 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-10T20:36:40.783 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:41.093 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-10T20:36:41.116 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-10T20:36:41.116 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:36:41.116 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T20:36:41.116 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:36:41.117 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:36:41.117 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:45.024 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 1/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 3/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 4/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 5/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 6/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 7/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-10T20:36:45.025 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 11/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 12/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 13/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 15/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 16/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 17/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 18/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 19/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 20/165 2024-09-10T20:36:45.026 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 21/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 22/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 23/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 24/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 25/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 26/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 27/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 28/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 29/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 30/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 31/165 2024-09-10T20:36:45.027 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 32/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 33/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 34/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 35/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 36/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 37/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 38/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 39/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-10T20:36:45.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-10T20:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-10T20:36:45.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-10T20:36:45.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-10T20:36:45.032 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-10T20:36:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-10T20:36:45.034 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-10T20:36:45.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 150/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-10T20:36:45.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 152/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 154/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 156/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 158/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 160/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-10T20:36:45.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-10T20:36:46.098 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 142/165 2024-09-10T20:36:46.138 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 143/165 2024-09-10T20:36:46.179 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 144/165 2024-09-10T20:36:46.211 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-10T20:36:46.247 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-10T20:36:46.450 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 147/165 2024-09-10T20:36:46.502 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-10T20:36:46.533 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 149/165 2024-09-10T20:36:46.562 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 150/165 2024-09-10T20:36:46.608 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 151/165 2024-09-10T20:36:47.172 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-10T20:36:47.214 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-10T20:36:47.239 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T20:36:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T20:36:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T20:36:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.16-1.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T20:36:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T20:36:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T20:36:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T20:36:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T20:36:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:36:47.572 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T20:36:47.984 DEBUG:teuthology.parallel:result is None 2024-09-10T20:36:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 153/165 2024-09-10T20:36:48.287 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-10T20:36:48.338 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-10T20:36:48.433 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-10T20:36:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-10T20:36:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-10T20:36:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-10T20:36:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-10T20:36:48.600 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-10T20:36:48.600 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-10T20:36:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-10T20:36:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-10T20:36:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-10T20:36:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-10T20:36:48.894 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-10T20:36:48.895 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-10T20:36:48.979 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-10T20:36:49.043 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-10T20:36:55.894 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 1/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 3/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 4/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 5/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 6/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 7/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 11/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 12/165 2024-09-10T20:36:55.895 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 13/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 15/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 16/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 17/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 18/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 19/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 20/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 21/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 22/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 23/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 24/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 25/165 2024-09-10T20:36:55.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 26/165 2024-09-10T20:36:55.897 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 27/165 2024-09-10T20:36:55.909 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 28/165 2024-09-10T20:36:55.909 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 29/165 2024-09-10T20:36:55.909 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 30/165 2024-09-10T20:36:55.909 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 31/165 2024-09-10T20:36:55.909 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 32/165 2024-09-10T20:36:55.909 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 33/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 34/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 35/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 36/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 37/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 38/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 39/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-10T20:36:55.910 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-10T20:36:55.911 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-10T20:36:55.912 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-10T20:36:55.913 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-10T20:36:55.914 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-10T20:36:55.915 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-10T20:36:55.916 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-10T20:36:55.917 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-10T20:36:55.918 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 150/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 152/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 154/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 156/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 158/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 160/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-10T20:36:55.919 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-10T20:36:55.920 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-10T20:36:55.920 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: automake-1.16.2-8.el9.noarch 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: bison-3.7.4-5.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.261 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: flex-2.6.4-9.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T20:36:58.262 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T20:36:58.263 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T20:36:58.264 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-13.16-1.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T20:36:58.265 INFO:teuthology.orchestra.run.smithi159.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T20:36:58.266 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T20:36:58.267 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T20:36:58.268 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.269 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T20:36:58.270 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.271 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:36:58.272 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-10T20:36:58.734 DEBUG:teuthology.parallel:result is None 2024-09-10T20:36:58.735 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-10T20:36:58.885 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T20:36:58.913 INFO:teuthology.orchestra.run.smithi028.stdout:19.3.0-4854.g8f219c47.el9 2024-09-10T20:36:58.914 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854.g8f219c47.el9 2024-09-10T20:36:58.915 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854.g8f219c47 is installed. 2024-09-10T20:36:58.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-10T20:36:59.065 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T20:36:59.088 INFO:teuthology.orchestra.run.smithi159.stdout:19.3.0-4854.g8f219c47.el9 2024-09-10T20:36:59.088 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854.g8f219c47.el9 2024-09-10T20:36:59.088 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854.g8f219c47 is installed. 2024-09-10T20:36:59.090 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T20:36:59.090 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:36:59.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T20:36:59.125 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:36:59.125 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T20:36:59.158 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T20:36:59.158 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:36:59.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T20:36:59.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T20:36:59.266 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:36:59.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T20:36:59.292 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T20:36:59.358 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T20:36:59.359 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:36:59.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T20:36:59.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T20:36:59.452 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:36:59.453 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T20:36:59.477 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T20:36:59.542 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T20:36:59.543 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:36:59.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T20:36:59.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T20:36:59.635 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:36:59.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T20:36:59.660 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T20:36:59.724 INFO:teuthology.run_tasks:Running task ceph... 2024-09-10T20:36:59.803 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-10T20:36:59.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-09-10T20:36:59.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-09-10T20:36:59.830 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-10T20:36:59.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T20:36:59.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T20:36:59.896 INFO:tasks.ceph:Creating extra log directories... 2024-09-10T20:36:59.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T20:36:59.938 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T20:36:59.969 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-10T20:36:59.969 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-10T20:36:59.969 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-10T20:36:59.969 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:36:59.970 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-10T20:37:00.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-10T20:37:00.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-10T20:37:00.164 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:00.164 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-10T20:37:00.180 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-10T20:37:00.300 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-10T20:37:00.331 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-10T20:37:00.331 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-10T20:37:00.331 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622', 'branch': 'wip-hemanth-testing-2024-09-10-0723', 'description': 'fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}}', 'email': 'hyelloji@redhat.com', 'first_in_suite': False, 'job_id': '7899622', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'owner': 'scheduled_hyelloji@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6190, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'sleep_before_teardown': 0, 'subset': '479/1024', 'suite': 'fs', 'suite_branch': 'wip-hemanth-testing-2024-09-10-0723', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHd80x6m2B/v65Fp3xZZwWlBYopdV0S+7+h+Yw4RPGGn0R3bBxlIHSIrjkKWVKxVNAK2PCp3K51oXAdmXWRbtJg=', 'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJGNyHwlIkvDvh/X2kKAL3uV77D/4ACogTrSKXEu+FxRh+lQteFZPjyPoUK7AluuY8Q8F++HAJPdmb5L7hEawM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, {'ceph': {'create_rbd_pool': False, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestMisc']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_16:59:47', 'tube': 'smithi', 'user': 'hyelloji', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-10T20:37:00.332 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T20:37:00.334 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T20:37:00.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T20:37:00.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T20:37:00.448 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:00.448 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T20:37:00.469 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T20:37:00.470 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 20:36:39.074536042 +0000 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 20:34:24.843969589 +0000 2024-09-10T20:37:00.526 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 20:34:24.843969589 +0000 2024-09-10T20:37:00.527 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 20:34:24.843969589 +0000 2024-09-10T20:37:00.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T20:37:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:37:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:37:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 8.6878e-05 s, 5.9 MB/s 2024-09-10T20:37:00.592 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T20:37:00.649 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-09-10T20:37:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 20:36:39.074536042 +0000 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 20:34:25.185959798 +0000 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 20:34:25.185959798 +0000 2024-09-10T20:37:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 20:34:25.185959798 +0000 2024-09-10T20:37:00.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T20:37:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:37:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:37:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.8546e-05 s, 5.2 MB/s 2024-09-10T20:37:00.770 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T20:37:00.827 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-09-10T20:37:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T20:37:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:00.884 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-10T20:37:00.884 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:00.884 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:00.884 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 20:36:39.075536013 +0000 2024-09-10T20:37:00.884 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 20:34:25.516950323 +0000 2024-09-10T20:37:00.884 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 20:34:25.516950323 +0000 2024-09-10T20:37:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 20:34:25.516950323 +0000 2024-09-10T20:37:00.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T20:37:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:37:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:37:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 8.5994e-05 s, 6.0 MB/s 2024-09-10T20:37:00.949 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T20:37:01.005 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-09-10T20:37:01.061 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T20:37:01.061 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:01.061 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-10T20:37:01.061 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:01.062 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:01.062 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 20:36:39.075536013 +0000 2024-09-10T20:37:01.062 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 20:34:25.840941048 +0000 2024-09-10T20:37:01.062 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 20:34:25.840941048 +0000 2024-09-10T20:37:01.062 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 20:34:25.840941048 +0000 2024-09-10T20:37:01.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T20:37:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:37:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:37:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000101673 s, 5.0 MB/s 2024-09-10T20:37:01.126 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T20:37:01.183 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:01.183 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:01.183 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T20:37:01.200 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T20:37:01.200 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-10 20:36:49.393673272 +0000 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-10 20:34:22.496057847 +0000 2024-09-10T20:37:01.256 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-10 20:34:22.496057847 +0000 2024-09-10T20:37:01.257 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-10 20:34:22.496057847 +0000 2024-09-10T20:37:01.257 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T20:37:01.320 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-10T20:37:01.321 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-10T20:37:01.321 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000189256 s, 2.7 MB/s 2024-09-10T20:37:01.322 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T20:37:01.378 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-09-10T20:37:01.434 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-10 20:36:49.394673254 +0000 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-10 20:34:22.824051872 +0000 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-10 20:34:22.824051872 +0000 2024-09-10T20:37:01.435 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-10 20:34:22.824051872 +0000 2024-09-10T20:37:01.435 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T20:37:01.498 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-10T20:37:01.498 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-10T20:37:01.498 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000192871 s, 2.7 MB/s 2024-09-10T20:37:01.499 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T20:37:01.556 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-09-10T20:37:01.612 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-10 20:36:49.394673254 +0000 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-10 20:34:23.160045750 +0000 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-10 20:34:23.160045750 +0000 2024-09-10T20:37:01.613 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-10 20:34:23.160045750 +0000 2024-09-10T20:37:01.613 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T20:37:01.676 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-10T20:37:01.677 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-10T20:37:01.677 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.00018087 s, 2.8 MB/s 2024-09-10T20:37:01.678 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T20:37:01.734 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-09-10T20:37:01.790 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T20:37:01.790 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:37:01.790 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-10T20:37:01.790 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:37:01.791 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T20:37:01.791 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-10 20:36:49.394673254 +0000 2024-09-10T20:37:01.791 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-10 20:34:23.496039628 +0000 2024-09-10T20:37:01.791 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-10 20:34:23.496039628 +0000 2024-09-10T20:37:01.791 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-10 20:34:23.496039628 +0000 2024-09-10T20:37:01.792 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T20:37:01.853 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-10T20:37:01.853 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-10T20:37:01.854 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000219473 s, 2.3 MB/s 2024-09-10T20:37:01.855 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T20:37:01.912 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:01.912 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-10T20:37:01.912 INFO:tasks.ceph:Generating config... 2024-09-10T20:37:01.914 INFO:tasks.ceph:[client] pid file = 2024-09-10T20:37:01.914 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mgr] client mount timeout = 30 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-10T20:37:01.914 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-10T20:37:01.915 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-10T20:37:01.916 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-10T20:37:01.916 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-10T20:37:01.916 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-10T20:37:01.916 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-10T20:37:01.916 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-10T20:37:01.916 INFO:tasks.ceph:Setting up mon.a... 2024-09-10T20:37:01.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-10T20:37:01.978 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-09-10T20:37:01.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-10T20:37:02.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:37:02.057 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.b', '172.21.15.159'), ('mon.c', '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]')] 2024-09-10T20:37:02.057 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-09-10T20:37:02.058 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:02.058 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T20:37:02.113 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --add b 172.21.15.159 --addv c '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid 943b2354-acb0-43ad-b06a-e7974842506f 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 943b2354-acb0-43ad-b06a-e7974842506f 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-09-10T20:37:02.199733+0000 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-10T20:37:02.199733+0000 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 17 (quincy) 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.b 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] mon.c 2024-09-10T20:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-10T20:37:02.205 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T20:37:02.221 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 943b2354-acb0-43ad-b06a-e7974842506f... 2024-09-10T20:37:02.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T20:37:02.263 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T20:37:02.305 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-10T20:37:02.305 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-10T20:37:02.305 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.306 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-10T20:37:02.307 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-10T20:37:02.308 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 943b2354-acb0-43ad-b06a-e7974842506f 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-10T20:37:02.309 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.310 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: bdev async discard = True 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: bdev enable discard = True 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore allocator = bitmap 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2024-09-10T20:37:02.311 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:37:02.312 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 30 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: debug finisher = 20 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-10T20:37:02.313 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.314 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: mon down mkfs grace = 300 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-10T20:37:02.315 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2024-09-10T20:37:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-09-10T20:37:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T20:37:02.318 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.28,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-09-10T20:37:02.319 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on skipped message = False 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = 943b2354-acb0-43ad-b06a-e7974842506f 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.320 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug trim objects = true 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.321 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: bdev async discard = True 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: bdev enable discard = True 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore allocator = bitmap 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 1/20 2024-09-10T20:37:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 1/20 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 4/10 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-09-10T20:37:02.323 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: client mount timeout = 30 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: debug finisher = 20 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-09-10T20:37:02.324 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.325 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: mon down mkfs grace = 300 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = "" 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2024-09-10T20:37:02.326 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2024-09-10T20:37:02.327 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-09-10T20:37:02.327 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-09-10T20:37:02.327 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-09-10T20:37:02.327 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-09-10T20:37:02.328 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-10T20:37:02.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-10T20:37:02.425 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-10T20:37:02.425 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:02.425 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-10T20:37:02.441 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:02.441 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-10T20:37:02.497 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:37:02.498 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:02.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T20:37:02.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:37:02.573 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:02.573 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:02.630 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:37:02.630 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:02.630 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T20:37:02.630 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:37:02.666 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:02.666 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:02.722 INFO:tasks.ceph:Setting up mon nodes... 2024-09-10T20:37:02.723 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-10T20:37:02.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-10T20:37:02.776 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-10T20:37:02.778 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T20:37:02.834 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T20:37:02.836 INFO:tasks.ceph:Setting up mds nodes... 2024-09-10T20:37:02.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-10T20:37:02.889 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-10T20:37:02.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-10T20:37:02.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-10T20:37:03.010 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-10T20:37:03.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-10T20:37:03.038 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-10T20:37:03.090 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-10T20:37:03.092 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-10T20:37:03.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-10T20:37:03.209 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-10T20:37:03.211 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-10T20:37:03.235 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-10T20:37:03.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T20:37:03.279 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-10T20:37:03.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-10T20:37:03.375 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-09-10T20:37:03.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-10T20:37:03.430 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-10T20:37:03.443 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-10T20:37:03.525 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-09-10T20:37:03.538 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-10T20:37:03.538 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-10T20:37:03.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-10T20:37:03.564 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:03.564 INFO:tasks.ceph:role: osd.0 2024-09-10T20:37:03.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:37:03.564 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:03.661 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-10T20:37:03.663 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-10T20:37:03.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-10T20:37:03.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-10T20:37:03.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-10T20:37:03.869 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:03.869 INFO:tasks.ceph:role: osd.1 2024-09-10T20:37:03.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:37:03.870 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:03.935 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:03.936 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:03.936 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:03.967 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-10T20:37:03.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-10T20:37:03.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-10T20:37:04.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-10T20:37:04.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-10T20:37:04.179 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:04.180 INFO:tasks.ceph:role: osd.2 2024-09-10T20:37:04.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:37:04.180 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T20:37:04.244 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:04.245 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:04.278 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-10T20:37:04.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-10T20:37:04.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-10T20:37:04.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-10T20:37:04.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-10T20:37:04.488 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:04.488 INFO:tasks.ceph:role: osd.3 2024-09-10T20:37:04.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:37:04.489 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-10T20:37:04.553 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:04.554 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:04.586 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-10T20:37:04.588 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-10T20:37:04.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-10T20:37:04.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-10T20:37:04.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:04.816+0000 7f63459ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-10T20:37:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:04.816+0000 7f63459ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-10T20:37:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:04.816+0000 7f63459ae640 -1 bdev(0x55cbac30b800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:04.817+0000 7f63459ae640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-10T20:37:07.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T20:37:07.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:07.857+0000 7fec2ca0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-10T20:37:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:07.858+0000 7fec2ca0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-10T20:37:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:07.858+0000 7fec2ca0f640 -1 bdev(0x55efba3a3800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:07.858+0000 7fec2ca0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-10T20:37:11.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T20:37:11.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:11.153+0000 7f078200e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-10T20:37:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:11.153+0000 7f078200e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-10T20:37:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:11.153+0000 7f078200e640 -1 bdev(0x5629d4ecd800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:11.154+0000 7f078200e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-10T20:37:14.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-10T20:37:14.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:14.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:14.191+0000 7f8fc4213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-10T20:37:14.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:14.191+0000 7f8fc4213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-10T20:37:14.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:14.191+0000 7f8fc4213640 -1 bdev(0x55914f543800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:14.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:14.191+0000 7f8fc4213640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-10T20:37:17.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-10T20:37:17.402 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-10T20:37:17.433 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:17.434 INFO:tasks.ceph:role: osd.4 2024-09-10T20:37:17.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:37:17.434 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T20:37:17.509 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:17.509 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:17.509 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:17.509 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:17.510 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:17.510 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:17.510 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:17.510 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:17.510 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:17.510 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:18.023 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-10T20:37:18.026 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-10T20:37:18.026 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-10T20:37:18.175 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-10T20:37:18.208 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-10T20:37:18.275 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:18.275 INFO:tasks.ceph:role: osd.5 2024-09-10T20:37:18.275 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:37:18.275 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T20:37:18.347 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:18.348 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:18.348 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:18.348 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:18.348 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:18.348 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:18.348 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:18.349 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:18.349 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:18.349 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:18.794 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-10T20:37:18.796 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-10T20:37:18.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-10T20:37:18.873 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-10T20:37:18.943 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-10T20:37:19.009 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:19.009 INFO:tasks.ceph:role: osd.6 2024-09-10T20:37:19.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:37:19.009 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T20:37:19.081 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:19.081 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:19.081 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:19.081 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:19.081 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:19.081 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:19.082 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:19.082 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:19.082 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:19.082 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:19.566 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-10T20:37:19.569 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-10T20:37:19.569 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-10T20:37:19.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-10T20:37:19.726 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-10T20:37:19.790 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-10T20:37:19.791 INFO:tasks.ceph:role: osd.7 2024-09-10T20:37:19.791 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:37:19.791 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:37:19.863 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:37:20.375 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-10T20:37:20.377 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-10T20:37:20.377 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-10T20:37:20.514 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-10T20:37:20.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:20.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:20.642+0000 7fa0489ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-10T20:37:20.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:20.643+0000 7fa0489ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-10T20:37:20.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:20.643+0000 7fa0489ae640 -1 bdev(0x560f8b6ff800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:20.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:20.643+0000 7fa0489ae640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-10T20:37:23.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-10T20:37:23.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:23.920 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:23.916+0000 7f40ba4b4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-10T20:37:23.921 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:23.916+0000 7f40ba4b4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-10T20:37:23.921 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:23.916+0000 7f40ba4b4640 -1 bdev(0x5622f748d800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:23.921 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:23.917+0000 7f40ba4b4640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-10T20:37:27.819 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-10T20:37:27.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:27.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:27.935+0000 7f792cc5e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-10T20:37:27.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:27.935+0000 7f792cc5e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-10T20:37:27.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:27.935+0000 7f792cc5e640 -1 bdev(0x55b4da06f800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:27.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:27.935+0000 7f792cc5e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-10T20:37:30.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:37:30.340 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:37:31.096 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-10T20:37:31.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:31.209 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:31.205+0000 7fee2380c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-10T20:37:31.209 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:31.205+0000 7fee2380c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-10T20:37:31.210 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:31.205+0000 7fee2380c640 -1 bdev(0x55a53a2fb800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:31.210 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:31.205+0000 7fee2380c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-10T20:37:34.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-10T20:37:34.647 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-10T20:37:34.647 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:34.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-10T20:37:34.680 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:34.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-10T20:37:34.749 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:34.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-10T20:37:34.818 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:34.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-10T20:37:34.887 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:34.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-10T20:37:34.954 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:34.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-10T20:37:35.021 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:35.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-10T20:37:35.088 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.088 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-10T20:37:35.118 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.119 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-10T20:37:35.187 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.187 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-10T20:37:35.255 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.255 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-10T20:37:35.321 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.322 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-10T20:37:35.388 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-10T20:37:35.455 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.455 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-10T20:37:35.521 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:35.522 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-10T20:37:35.541 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:35.541 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-09-10T20:37:35.601 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.601 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-10T20:37:35.617 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:35.617 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-09-10T20:37:35.673 INFO:tasks.ceph:Adding keys to all mons... 2024-09-10T20:37:35.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T20:37:35.676 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T20:37:35.717 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.y] 2024-09-10T20:37:35.717 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ== 2024-09-10T20:37:35.717 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A== 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA== 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg== 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw== 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ== 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug== 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-09-10T20:37:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A== 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w== 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg== 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA== 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw== 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.6] 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w== 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.7] 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA== 2024-09-10T20:37:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:37:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:37:35.720 INFO:teuthology.orchestra.run.smithi028.stdout:[client.2] 2024-09-10T20:37:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw== 2024-09-10T20:37:35.720 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2024-09-10T20:37:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw== 2024-09-10T20:37:35.720 INFO:teuthology.orchestra.run.smithi028.stdout:[client.3] 2024-09-10T20:37:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ== 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.y] 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ== 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A== 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA== 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg== 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-09-10T20:37:35.738 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw== 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ== 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.3] 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug== 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A== 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w== 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg== 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.4] 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA== 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.5] 2024-09-10T20:37:35.739 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw== 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.6] 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w== 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.7] 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA== 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout:[client.2] 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw== 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout:[client.1] 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw== 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout:[client.3] 2024-09-10T20:37:35.740 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ== 2024-09-10T20:37:35.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T20:37:35.765 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T20:37:35.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:35.836 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:35.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:35.895 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:35.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:35.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.006 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.061 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.114 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T20:37:36.168 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T20:37:36.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:36.221 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:36.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:36.278 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-10T20:37:36.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.335 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.441 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:37:36.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.548 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.604 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.661 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.718 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:37:36.774 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-10T20:37:36.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-10T20:37:36.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:37:36.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-10T20:37:36.922 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-10T20:37:36.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:37:37.045 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-10T20:37:37.070 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-10T20:37:37.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:37:37.231 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-10T20:37:37.255 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:37.258 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:37:37.311 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-10T20:37:37.312 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T20:37:37.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T20:37:37.314 INFO:tasks.ceph.mon.a:Started 2024-09-10T20:37:37.314 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T20:37:37.314 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T20:37:37.354 INFO:tasks.ceph.mon.b:Started 2024-09-10T20:37:37.355 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T20:37:37.355 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T20:37:37.357 INFO:tasks.ceph.mon.c:Started 2024-09-10T20:37:37.357 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-10T20:37:37.357 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-10T20:37:37.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-10T20:37:37.359 INFO:tasks.ceph.mgr.y:Started 2024-09-10T20:37:37.360 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-10T20:37:37.360 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-10T20:37:37.362 INFO:tasks.ceph.mgr.x:Started 2024-09-10T20:37:37.362 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-10T20:37:37.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-10T20:37:37.364 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-10T20:37:37.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-10T20:37:37.407 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-10T20:37:37.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-10T20:37:37.450 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-10T20:37:37.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-10T20:37:37.494 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-10T20:37:37.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-10T20:37:37.537 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-10T20:37:37.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-10T20:37:37.580 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-10T20:37:37.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-10T20:37:37.623 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-10T20:37:37.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-10T20:37:37.667 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-10T20:37:37.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-10T20:37:37.710 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-10T20:37:37.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-10T20:37:37.753 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-10T20:37:37.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-10T20:37:37.796 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-10T20:37:37.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-10T20:37:37.840 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-10T20:37:37.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-10T20:37:37.883 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-10T20:37:37.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-10T20:37:37.926 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-10T20:37:37.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-10T20:37:37.969 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-10T20:37:37.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-10T20:37:38.012 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-09-10T20:37:38.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-09-10T20:37:38.057 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-10T20:37:38.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-10T20:37:38.103 DEBUG:tasks.ceph:set 18 configs 2024-09-10T20:37:38.103 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi028') 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 -- 172.21.15.28:0/1171878447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70fc004800 msgr2=0x7f71140c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 --2- 172.21.15.28:0/1171878447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70fc004800 0x7f71140c8ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f71080066b0 tx=0x7f7108006ab0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 -- 172.21.15.28:0/1171878447 shutdown_connections 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 --2- 172.21.15.28:0/1171878447 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70fc005290 0x7f71141f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 --2- 172.21.15.28:0/1171878447 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70fc004800 0x7f71140c8ed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 --2- 172.21.15.28:0/1171878447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70fc0039b0 0x7f71140b7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 -- 172.21.15.28:0/1171878447 >> 172.21.15.28:0/1171878447 conn(0x7f71140d1e00 msgr2=0x7f71140d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 -- 172.21.15.28:0/1171878447 shutdown_connections 2024-09-10T20:37:47.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.953+0000 7f70f97fa640 1 -- 172.21.15.28:0/1171878447 wait complete. 2024-09-10T20:37:47.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.984+0000 7f4bfced4640 1 -- 172.21.15.28:0/2503459531 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f4bdc004150 con 0x7f4be0005210 2024-09-10T20:37:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.984+0000 7f761792b640 1 -- 172.21.15.28:0/1913138092 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f76100b1ff0 con 0x7f75fc0039b0 2024-09-10T20:37:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.987+0000 7fdf3ac03640 1 -- 172.21.15.28:0/2420105645 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fdf14000e60 con 0x7fdf1c004800 2024-09-10T20:37:47.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.987+0000 7f1edd2a2640 1 -- 172.21.15.28:0/1273736489 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f1ebc0040f0 con 0x7f1ec40039b0 2024-09-10T20:37:47.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.988+0000 7f4be6ffd640 1 -- 172.21.15.28:0/2503459531 <== mon.2 v2:172.21.15.159:3301/0 8 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v8)=0 v8) ==== 121+0+0 (secure 0 0 0) 0x7f4bf0021060 con 0x7f4be0005210 2024-09-10T20:37:47.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.989+0000 7f4be4ff9640 1 -- 172.21.15.28:0/2503459531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be0005210 msgr2=0x7f4bf8070500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:47.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.989+0000 7f4be4ff9640 1 --2- 172.21.15.28:0/2503459531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be0005210 0x7f4bf8070500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4bf80b6ba0 tx=0x7f4bf00138f0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.990+0000 7f4be4ff9640 1 -- 172.21.15.28:0/2503459531 shutdown_connections 2024-09-10T20:37:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.990+0000 7f4be4ff9640 1 --2- 172.21.15.28:0/2503459531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be0005210 0x7f4bf8070500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.990+0000 7f4be4ff9640 1 --2- 172.21.15.28:0/2503459531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4be0004800 0x7f4bf81f0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.990+0000 7f4be4ff9640 1 --2- 172.21.15.28:0/2503459531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be00039b0 0x7f4bf81efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.990+0000 7f4be4ff9640 1 -- 172.21.15.28:0/2503459531 >> 172.21.15.28:0/2503459531 conn(0x7f4bf80d0c70 msgr2=0x7f4bf81586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.990+0000 7f4be4ff9640 1 -- 172.21.15.28:0/2503459531 shutdown_connections 2024-09-10T20:37:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.991+0000 7f4be4ff9640 1 -- 172.21.15.28:0/2503459531 wait complete. 2024-09-10T20:37:47.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.990+0000 7f3461ffb640 1 -- 172.21.15.28:0/1583485728 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f347c0cc180 con 0x7f347c152210 2024-09-10T20:37:47.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:47.995+0000 7faf08add640 1 -- 172.21.15.28:0/301621299 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7faf040761d0 con 0x7faef0005310 2024-09-10T20:37:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.000+0000 7fdf297fa640 1 -- 172.21.15.28:0/2420105645 <== mon.2 v2:172.21.15.159:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7fdf24020750 con 0x7fdf1c004800 2024-09-10T20:37:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.000+0000 7f3463fff640 1 -- 172.21.15.28:0/1583485728 <== mon.2 v2:172.21.15.159:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f3470020c10 con 0x7f347c152210 2024-09-10T20:37:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.000+0000 7fd295ffb640 1 -- 172.21.15.28:0/2290921529 <== mon.2 v2:172.21.15.159:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7fd28c00a040 con 0x7fd29814aea0 2024-09-10T20:37:48.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.000+0000 7f1a7bfff640 1 -- 172.21.15.28:0/3565713043 <== mon.2 v2:172.21.15.159:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f1a88013770 con 0x7f1a7c004800 2024-09-10T20:37:48.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.000+0000 7f47f0ff9640 1 -- 172.21.15.28:0/2737412202 <== mon.2 v2:172.21.15.159:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f47ec01dbd0 con 0x7f47f414f3e0 2024-09-10T20:37:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.003+0000 7faefaffd640 1 -- 172.21.15.28:0/301621299 <== mon.2 v2:172.21.15.159:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7faefc022b70 con 0x7faef0005310 2024-09-10T20:37:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.002+0000 7fe71b7fe640 1 -- 172.21.15.28:0/318562914 <== mon.1 v2:172.21.15.159:3300/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7fe72401daa0 con 0x7fe71c003fc0 2024-09-10T20:37:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.001+0000 7f1ecf7fe640 1 -- 172.21.15.28:0/1273736489 <== mon.2 v2:172.21.15.159:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f1ec8005020 con 0x7f1ec40039b0 2024-09-10T20:37:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.004+0000 7f7605ffb640 1 -- 172.21.15.28:0/1913138092 <== mon.1 v2:172.21.15.159:3300/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f7600029b90 con 0x7f75fc0039b0 2024-09-10T20:37:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.004+0000 7f7605ffb640 1 -- 172.21.15.28:0/1913138092 <== mon.1 v2:172.21.15.159:3300/0 9 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v9)=0 v9) ==== 123+0+0 (secure 0 0 0) 0x7f760001f650 con 0x7f75fc0039b0 2024-09-10T20:37:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.005+0000 7fe7395c2640 1 -- 172.21.15.28:0/318562914 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fe70c0052f0 con 0x7fe71c003fc0 2024-09-10T20:37:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.004+0000 7f7ceaffd640 1 -- 172.21.15.28:0/3299196949 <== mon.1 v2:172.21.15.159:3300/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f7cec010760 con 0x7f7ce0004800 2024-09-10T20:37:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.005+0000 7f47fa76b640 1 -- 172.21.15.28:0/2737412202 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f47cc005310 con 0x7f47f414f3e0 2024-09-10T20:37:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f7cfc7ec640 1 -- 172.21.15.28:0/3299196949 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f7cd00052f0 con 0x7f7ce0004800 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 -- 172.21.15.28:0/1913138092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75fc0039b0 msgr2=0x7f76100b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 --2- 172.21.15.28:0/1913138092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75fc0039b0 0x7f76100b34e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7600004eb0 tx=0x7f7600013820 comp rx=0 tx=0).stop 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 -- 172.21.15.28:0/1913138092 shutdown_connections 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 --2- 172.21.15.28:0/1913138092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75fc004d40 0x7f76100c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 --2- 172.21.15.28:0/1913138092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f75fc0039b0 0x7f76100b34e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 --2- 172.21.15.28:0/1913138092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f4003b70 0x7f76100b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 -- 172.21.15.28:0/1913138092 >> 172.21.15.28:0/1913138092 conn(0x7f7610074240 msgr2=0x7f76100b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 -- 172.21.15.28:0/1913138092 shutdown_connections 2024-09-10T20:37:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.006+0000 7f761792b640 1 -- 172.21.15.28:0/1913138092 wait complete. 2024-09-10T20:37:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.008+0000 7faefaffd640 1 -- 172.21.15.28:0/301621299 <== mon.2 v2:172.21.15.159:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7faefc022d20 con 0x7faef0005310 2024-09-10T20:37:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.008+0000 7fdf297fa640 1 -- 172.21.15.28:0/2420105645 <== mon.2 v2:172.21.15.159:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fdf240209c0 con 0x7fdf1c004800 2024-09-10T20:37:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.008+0000 7f1a7bfff640 1 -- 172.21.15.28:0/3565713043 <== mon.2 v2:172.21.15.159:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f1a88013990 con 0x7f1a7c004800 2024-09-10T20:37:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.008+0000 7f3463fff640 1 -- 172.21.15.28:0/1583485728 <== mon.2 v2:172.21.15.159:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f34700035f0 con 0x7f347c152210 2024-09-10T20:37:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.009+0000 7fd295ffb640 1 -- 172.21.15.28:0/2290921529 <== mon.2 v2:172.21.15.159:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fd28c03fa90 con 0x7fd29814aea0 2024-09-10T20:37:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.009+0000 7fdf297fa640 1 -- 172.21.15.28:0/2420105645 <== mon.2 v2:172.21.15.159:3301/0 10 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7fdf24020c10 con 0x7fdf1c004800 2024-09-10T20:37:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.009+0000 7fe71b7fe640 1 -- 172.21.15.28:0/318562914 <== mon.1 v2:172.21.15.159:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fe72401dc50 con 0x7fe71c003fc0 2024-09-10T20:37:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.009+0000 7f47f0ff9640 1 -- 172.21.15.28:0/2737412202 <== mon.2 v2:172.21.15.159:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f47ec01dd80 con 0x7f47f414f3e0 2024-09-10T20:37:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.009+0000 7f7ceaffd640 1 -- 172.21.15.28:0/3299196949 <== mon.1 v2:172.21.15.159:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f7cec025e30 con 0x7f7ce0004800 2024-09-10T20:37:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.009+0000 7f1ecf7fe640 1 -- 172.21.15.28:0/1273736489 <== mon.2 v2:172.21.15.159:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f1ec8005250 con 0x7f1ec40039b0 2024-09-10T20:37:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7fdf3ac03640 1 -- 172.21.15.28:0/2420105645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf1c004800 msgr2=0x7fdf340b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7fdf3ac03640 1 --2- 172.21.15.28:0/2420105645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf1c004800 0x7fdf340b3b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdf24004c70 tx=0x7fdf24013820 comp rx=0 tx=0).stop 2024-09-10T20:37:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7fdf3ac03640 1 -- 172.21.15.28:0/2420105645 shutdown_connections 2024-09-10T20:37:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7fdf3ac03640 1 --2- 172.21.15.28:0/2420105645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf1c005290 0x7fdf341e6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7fdf3ac03640 1 --2- 172.21.15.28:0/2420105645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf1c004800 0x7fdf340b3b10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7fdf3ac03640 1 --2- 172.21.15.28:0/2420105645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf1c0039b0 0x7fdf3414b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7fdf3ac03640 1 -- 172.21.15.28:0/2420105645 >> 172.21.15.28:0/2420105645 conn(0x7fdf340d0ce0 msgr2=0x7fdf3415eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.012+0000 7f1ecf7fe640 1 -- 172.21.15.28:0/1273736489 <== mon.2 v2:172.21.15.159:3301/0 10 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v11)=0 v11) ==== 121+0+0 (secure 0 0 0) 0x7f1ec8019040 con 0x7f1ec40039b0 2024-09-10T20:37:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.013+0000 7fdf3ac03640 1 -- 172.21.15.28:0/2420105645 shutdown_connections 2024-09-10T20:37:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.013+0000 7fdf3ac03640 1 -- 172.21.15.28:0/2420105645 wait complete. 2024-09-10T20:37:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.013+0000 7f1a79ffb640 1 -- 172.21.15.28:0/3565713043 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f1a6c0052f0 con 0x7f1a7c004800 2024-09-10T20:37:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.014+0000 7f1edd2a2640 1 -- 172.21.15.28:0/1273736489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec40039b0 msgr2=0x7f1ed80cb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.014+0000 7f1edd2a2640 1 --2- 172.21.15.28:0/1273736489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec40039b0 0x7f1ed80cb480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8005c60 tx=0x7f1ec8012cd0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.014+0000 7f1edd2a2640 1 -- 172.21.15.28:0/1273736489 shutdown_connections 2024-09-10T20:37:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.014+0000 7f1edd2a2640 1 --2- 172.21.15.28:0/1273736489 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ec4005210 0x7f1ed81fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.014+0000 7f1edd2a2640 1 --2- 172.21.15.28:0/1273736489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ec4004800 0x7f1ed80c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.014+0000 7f1edd2a2640 1 --2- 172.21.15.28:0/1273736489 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ec40039b0 0x7f1ed80cb480 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.014+0000 7f1edd2a2640 1 -- 172.21.15.28:0/1273736489 >> 172.21.15.28:0/1273736489 conn(0x7f1ed80d0ce0 msgr2=0x7f1ed80cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.015+0000 7f1edd2a2640 1 -- 172.21.15.28:0/1273736489 shutdown_connections 2024-09-10T20:37:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.015+0000 7f1edd2a2640 1 -- 172.21.15.28:0/1273736489 wait complete. 2024-09-10T20:37:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.017+0000 7f3463fff640 1 -- 172.21.15.28:0/1583485728 <== mon.2 v2:172.21.15.159:3301/0 10 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v12)=0 v12) ==== 120+0+0 (secure 0 0 0) 0x7f34700037a0 con 0x7f347c152210 2024-09-10T20:37:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.019+0000 7f34818e2640 1 -- 172.21.15.28:0/1583485728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c152210 msgr2=0x7f347c0b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.019+0000 7f34818e2640 1 --2- 172.21.15.28:0/1583485728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c152210 0x7f347c0b3b10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3470012870 tx=0x7f3470012d30 comp rx=0 tx=0).stop 2024-09-10T20:37:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.019+0000 7f34818e2640 1 -- 172.21.15.28:0/1583485728 shutdown_connections 2024-09-10T20:37:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.019+0000 7f34818e2640 1 --2- 172.21.15.28:0/1583485728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347c1f1c00 0x7f347c0b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.019+0000 7f34818e2640 1 --2- 172.21.15.28:0/1583485728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f347c152210 0x7f347c0b3b10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.019+0000 7f34818e2640 1 --2- 172.21.15.28:0/1583485728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3468005c10 0x7f347c1480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.019+0000 7f34818e2640 1 -- 172.21.15.28:0/1583485728 >> 172.21.15.28:0/1583485728 conn(0x7f347c0d0ce0 msgr2=0x7f347c15dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.020+0000 7f34818e2640 1 -- 172.21.15.28:0/1583485728 shutdown_connections 2024-09-10T20:37:48.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.020+0000 7f34818e2640 1 -- 172.21.15.28:0/1583485728 wait complete. 2024-09-10T20:37:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.022+0000 7fe71b7fe640 1 -- 172.21.15.28:0/318562914 <== mon.1 v2:172.21.15.159:3300/0 10 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v13)=0 v13) ==== 108+0+0 (secure 0 0 0) 0x7fe724027440 con 0x7fe71c003fc0 2024-09-10T20:37:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.025+0000 7fe7395c2640 1 -- 172.21.15.28:0/318562914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71c003fc0 msgr2=0x7fe7340cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.025+0000 7fe7395c2640 1 --2- 172.21.15.28:0/318562914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71c003fc0 0x7fe7340cbb80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe724000c00 tx=0x7fe724013e90 comp rx=0 tx=0).stop 2024-09-10T20:37:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.025+0000 7fe7395c2640 1 -- 172.21.15.28:0/318562914 shutdown_connections 2024-09-10T20:37:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.025+0000 7fe7395c2640 1 --2- 172.21.15.28:0/318562914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe72000a720 0x7fe7341f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.025+0000 7fe7395c2640 1 --2- 172.21.15.28:0/318562914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c004d40 0x7fe7341f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.025+0000 7fe7395c2640 1 --2- 172.21.15.28:0/318562914 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71c003fc0 0x7fe7340cbb80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.025+0000 7fe7395c2640 1 -- 172.21.15.28:0/318562914 >> 172.21.15.28:0/318562914 conn(0x7fe7340d1e00 msgr2=0x7fe734071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.026+0000 7fe7395c2640 1 -- 172.21.15.28:0/318562914 shutdown_connections 2024-09-10T20:37:48.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.026+0000 7fe7395c2640 1 -- 172.21.15.28:0/318562914 wait complete. 2024-09-10T20:37:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.029+0000 7f1a7bfff640 1 -- 172.21.15.28:0/3565713043 <== mon.2 v2:172.21.15.159:3301/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f1a88013c80 con 0x7f1a7c004800 2024-09-10T20:37:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.031+0000 7faefaffd640 1 -- 172.21.15.28:0/301621299 <== mon.2 v2:172.21.15.159:3301/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7faefc0197b0 con 0x7faef0005310 2024-09-10T20:37:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.031+0000 7faefaffd640 1 -- 172.21.15.28:0/301621299 <== mon.2 v2:172.21.15.159:3301/0 11 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v14)=0 v14) ==== 114+0+0 (secure 0 0 0) 0x7faefc016070 con 0x7faef0005310 2024-09-10T20:37:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.030+0000 7f7ceaffd640 1 -- 172.21.15.28:0/3299196949 <== mon.1 v2:172.21.15.159:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f7cec0255a0 con 0x7f7ce0004800 2024-09-10T20:37:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.032+0000 7fd295ffb640 1 -- 172.21.15.28:0/2290921529 <== mon.2 v2:172.21.15.159:3301/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fd28c03fcc0 con 0x7fd29814aea0 2024-09-10T20:37:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.032+0000 7f47f0ff9640 1 -- 172.21.15.28:0/2737412202 <== mon.2 v2:172.21.15.159:3301/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f47ec01c960 con 0x7f47f414f3e0 2024-09-10T20:37:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 -- 172.21.15.28:0/301621299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faef0005310 msgr2=0x7faf0415f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 --2- 172.21.15.28:0/301621299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faef0005310 0x7faf0415f390 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faf040b36b0 tx=0x7faefc00f780 comp rx=0 tx=0).stop 2024-09-10T20:37:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 -- 172.21.15.28:0/301621299 shutdown_connections 2024-09-10T20:37:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 --2- 172.21.15.28:0/301621299 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faef0005310 0x7faf0415f390 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 --2- 172.21.15.28:0/301621299 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faef0004800 0x7faf041f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 --2- 172.21.15.28:0/301621299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00039e0 0x7faf040b9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 -- 172.21.15.28:0/301621299 >> 172.21.15.28:0/301621299 conn(0x7faf040d4070 msgr2=0x7faf040b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 -- 172.21.15.28:0/301621299 shutdown_connections 2024-09-10T20:37:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.034+0000 7faf08add640 1 -- 172.21.15.28:0/301621299 wait complete. 2024-09-10T20:37:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.035+0000 7f47f0ff9640 1 -- 172.21.15.28:0/2737412202 <== mon.2 v2:172.21.15.159:3301/0 11 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v15)=0 v15) ==== 125+0+0 (secure 0 0 0) 0x7f47ec01cb10 con 0x7f47f414f3e0 2024-09-10T20:37:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.036+0000 7f47fa76b640 1 -- 172.21.15.28:0/2737412202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47f414f3e0 msgr2=0x7f47f406d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.036+0000 7f47fa76b640 1 --2- 172.21.15.28:0/2737412202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47f414f3e0 0x7f47f406d310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009e70 tx=0x7f47ec001b10 comp rx=0 tx=0).stop 2024-09-10T20:37:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.036+0000 7f47fa76b640 1 -- 172.21.15.28:0/2737412202 shutdown_connections 2024-09-10T20:37:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.036+0000 7f47fa76b640 1 --2- 172.21.15.28:0/2737412202 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47f414f3e0 0x7f47f406d310 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.036+0000 7f47fa76b640 1 --2- 172.21.15.28:0/2737412202 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47dc004d40 0x7f47f406cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.036+0000 7f47fa76b640 1 --2- 172.21.15.28:0/2737412202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47dc004020 0x7f47f41f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.036+0000 7f47fa76b640 1 -- 172.21.15.28:0/2737412202 >> 172.21.15.28:0/2737412202 conn(0x7f47f40d0c70 msgr2=0x7f47f40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.037+0000 7f47fa76b640 1 -- 172.21.15.28:0/2737412202 shutdown_connections 2024-09-10T20:37:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.037+0000 7f47fa76b640 1 -- 172.21.15.28:0/2737412202 wait complete. 2024-09-10T20:37:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.039+0000 7f7ceaffd640 1 -- 172.21.15.28:0/3299196949 <== mon.1 v2:172.21.15.159:3300/0 11 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v16)=0 v16) ==== 121+0+0 (secure 0 0 0) 0x7f7cec01cad0 con 0x7f7ce0004800 2024-09-10T20:37:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.041+0000 7f7cfc7ec640 1 -- 172.21.15.28:0/3299196949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ce0004800 msgr2=0x7f7cf415cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.041+0000 7f7cfc7ec640 1 --2- 172.21.15.28:0/3299196949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ce0004800 0x7f7cf415cde0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40b7eb0 tx=0x7f7cec013b20 comp rx=0 tx=0).stop 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.041+0000 7f7cfc7ec640 1 -- 172.21.15.28:0/3299196949 shutdown_connections 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.041+0000 7f7cfc7ec640 1 --2- 172.21.15.28:0/3299196949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ce0004800 0x7f7cf415cde0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.041+0000 7f7cfc7ec640 1 --2- 172.21.15.28:0/3299196949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ce00039b0 0x7f7cf40c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.041+0000 7f7cfc7ec640 1 --2- 172.21.15.28:0/3299196949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd8003b60 0x7f7cf4160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.041+0000 7f7cfc7ec640 1 -- 172.21.15.28:0/3299196949 >> 172.21.15.28:0/3299196949 conn(0x7f7cf40d1e00 msgr2=0x7f7cf4070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.042+0000 7f7cfc7ec640 1 -- 172.21.15.28:0/3299196949 shutdown_connections 2024-09-10T20:37:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.043+0000 7f7cfc7ec640 1 -- 172.21.15.28:0/3299196949 wait complete. 2024-09-10T20:37:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.046+0000 7f1a7bfff640 1 -- 172.21.15.28:0/3565713043 <== mon.2 v2:172.21.15.159:3301/0 11 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v17)=0 v17) ==== 106+0+0 (secure 0 0 0) 0x7f1a88032450 con 0x7f1a7c004800 2024-09-10T20:37:48.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 -- 172.21.15.28:0/3565713043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a7c004800 msgr2=0x7f1a940c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 --2- 172.21.15.28:0/3565713043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a7c004800 0x7f1a940c97f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a88010b50 tx=0x7f1a88004600 comp rx=0 tx=0).stop 2024-09-10T20:37:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 -- 172.21.15.28:0/3565713043 shutdown_connections 2024-09-10T20:37:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 --2- 172.21.15.28:0/3565713043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a80006ba0 0x7f1a940c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 --2- 172.21.15.28:0/3565713043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a7c004800 0x7f1a940c97f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 --2- 172.21.15.28:0/3565713043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a7c0039e0 0x7f1a940b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 -- 172.21.15.28:0/3565713043 >> 172.21.15.28:0/3565713043 conn(0x7f1a940d0ce0 msgr2=0x7f1a9415d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 -- 172.21.15.28:0/3565713043 shutdown_connections 2024-09-10T20:37:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.047+0000 7f1a79ffb640 1 -- 172.21.15.28:0/3565713043 wait complete. 2024-09-10T20:37:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.075+0000 7fd27f7fe640 1 -- 172.21.15.28:0/2290921529 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7fd270005310 con 0x7fd29814aea0 2024-09-10T20:37:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.079+0000 7fd295ffb640 1 -- 172.21.15.28:0/2290921529 <== mon.2 v2:172.21.15.159:3301/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd28c052430 con 0x7fd29814aea0 2024-09-10T20:37:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.079+0000 7fd295ffb640 1 -- 172.21.15.28:0/2290921529 <== mon.2 v2:172.21.15.159:3301/0 12 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v18)=0 v18) ==== 123+0+0 (secure 0 0 0) 0x7fd28c0527a0 con 0x7fd29814aea0 2024-09-10T20:37:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 -- 172.21.15.28:0/2290921529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29814aea0 msgr2=0x7fd29814c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 --2- 172.21.15.28:0/2290921529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29814aea0 0x7fd29814c080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd2840143f0 tx=0x7fd28c004730 comp rx=0 tx=0).stop 2024-09-10T20:37:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 -- 172.21.15.28:0/2290921529 shutdown_connections 2024-09-10T20:37:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 --2- 172.21.15.28:0/2290921529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2980b5b80 0x7fd29814c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 --2- 172.21.15.28:0/2290921529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29814aea0 0x7fd29814c080 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 --2- 172.21.15.28:0/2290921529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840118e0 0x7fd29814a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 -- 172.21.15.28:0/2290921529 >> 172.21.15.28:0/2290921529 conn(0x7fd2980d0ce0 msgr2=0x7fd2980cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 -- 172.21.15.28:0/2290921529 shutdown_connections 2024-09-10T20:37:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.080+0000 7fd27f7fe640 1 -- 172.21.15.28:0/2290921529 wait complete. 2024-09-10T20:37:48.096 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.096 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.096 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.096 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.097 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.097 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.097 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.097 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.219+0000 7f434e9a1640 1 -- 172.21.15.28:0/3305594040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f432c0087d0 msgr2=0x7f432c011360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.219+0000 7f434e9a1640 1 --2- 172.21.15.28:0/3305594040 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f432c0087d0 0x7f432c011360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4344005fd0 tx=0x7f434402d900 comp rx=0 tx=0).stop 2024-09-10T20:37:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.219+0000 7f434e1a0640 1 -- 172.21.15.28:0/3305594040 reap_dead start 2024-09-10T20:37:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.219+0000 7f434e9a1640 1 -- 172.21.15.28:0/3305594040 shutdown_connections 2024-09-10T20:37:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.219+0000 7f434e9a1640 1 -- 172.21.15.28:0/3305594040 >> 172.21.15.28:0/3305594040 conn(0x7f43480b26a0 msgr2=0x7f43480b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.219+0000 7f434e9a1640 1 -- 172.21.15.28:0/3305594040 shutdown_connections 2024-09-10T20:37:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.219+0000 7f434e9a1640 1 -- 172.21.15.28:0/3305594040 wait complete. 2024-09-10T20:37:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.220+0000 7f434e9a1640 1 Processor -- start 2024-09-10T20:37:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.220+0000 7f434e9a1640 1 -- start start 2024-09-10T20:37:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e9a1640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43480a9710 0x7f43480c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e9a1640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43480a9af0 0x7f43480c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e9a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480b15b0 0x7f43480b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e9a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4348070eb0 con 0x7f43480b15b0 2024-09-10T20:37:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e9a1640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4348070d30 con 0x7f43480a9710 2024-09-10T20:37:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e9a1640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f434815bf40 con 0x7f43480a9af0 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e1a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480b15b0 0x7f43480b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e1a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480b15b0 0x7f43480b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43982/0 (socket says 172.21.15.28:43982) 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434e1a0640 1 -- 172.21.15.28:0/801796506 learned_addr learned my addr 172.21.15.28:0/801796506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434d99f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43480a9710 0x7f43480c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.221+0000 7f434d19e640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43480a9af0 0x7f43480c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e1a0640 1 -- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43480a9af0 msgr2=0x7f43480c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e1a0640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43480a9af0 0x7f43480c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e1a0640 1 -- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43480a9710 msgr2=0x7f43480c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e1a0640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43480a9710 0x7f43480c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e1a0640 1 -- 172.21.15.28:0/801796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43481f3090 con 0x7f43480b15b0 2024-09-10T20:37:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434d99f640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43480a9710 0x7f43480c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e1a0640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480b15b0 0x7f43480b19f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f434806db90 tx=0x7f4340013a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f433e7fc640 1 -- 172.21.15.28:0/801796506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43400107f0 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f433e7fc640 1 -- 172.21.15.28:0/801796506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4340010aa0 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43480c97e0 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f433e7fc640 1 -- 172.21.15.28:0/801796506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434001c440 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43480c9da0 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f433e7fc640 1 -- 172.21.15.28:0/801796506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f434001a070 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.222+0000 7f433e7fc640 1 -- 172.21.15.28:0/801796506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f434001bab0 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.223+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4348073a30 con 0x7f43480b15b0 2024-09-10T20:37:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.226+0000 7f433e7fc640 1 -- 172.21.15.28:0/801796506 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f4340022060 con 0x7f43480b15b0 2024-09-10T20:37:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.288+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f43480cd790 con 0x7f43480b15b0 2024-09-10T20:37:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.290+0000 7f433e7fc640 1 -- 172.21.15.28:0/801796506 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v19)=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7f4340018070 con 0x7f43480b15b0 2024-09-10T20:37:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480b15b0 msgr2=0x7f43480b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480b15b0 0x7f43480b19f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f434806db90 tx=0x7f4340013a60 comp rx=0 tx=0).stop 2024-09-10T20:37:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 shutdown_connections 2024-09-10T20:37:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480b15b0 0x7f43480b19f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43480a9af0 0x7f43480c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 --2- 172.21.15.28:0/801796506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43480a9710 0x7f43480c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 >> 172.21.15.28:0/801796506 conn(0x7f43480b26a0 msgr2=0x7f43480b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 shutdown_connections 2024-09-10T20:37:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.291+0000 7f434e9a1640 1 -- 172.21.15.28:0/801796506 wait complete. 2024-09-10T20:37:48.306 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.306 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.306 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.306 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.306 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.306 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.306 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.307 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.307 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi028') 2024-09-10T20:37:48.307 INFO:tasks.ceph:Setting crush tunables to default 2024-09-10T20:37:48.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-09-10T20:37:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 -- 172.21.15.28:0/1482603493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a00cb880 msgr2=0x7fa0a00736c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/1482603493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a00cb880 0x7fa0a00736c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0075280 tx=0x7fa09802aa30 comp rx=0 tx=0).stop 2024-09-10T20:37:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 -- 172.21.15.28:0/1482603493 shutdown_connections 2024-09-10T20:37:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/1482603493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a00cb880 0x7fa0a00736c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/1482603493 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0a015e900 0x7fa0a00cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/1482603493 >> v2:172.21.15.159:3300/0 conn(0x7fa0a0151d50 0x7fa0a00d0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 -- 172.21.15.28:0/1482603493 >> 172.21.15.28:0/1482603493 conn(0x7fa0a00d0cf0 msgr2=0x7fa0a00d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 -- 172.21.15.28:0/1482603493 shutdown_connections 2024-09-10T20:37:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.408+0000 7fa0a4c40640 1 -- 172.21.15.28:0/1482603493 wait complete. 2024-09-10T20:37:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.409+0000 7fa0a4c40640 1 Processor -- start 2024-09-10T20:37:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa0a4c40640 1 -- start start 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa0a4c40640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0a00cb880 0x7fa0a0147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa0a4c40640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0a0151d50 0x7fa0a00ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa0a4c40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a015e900 0x7fa0a00c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa0a4c40640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a0073f90 con 0x7fa0a015e900 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa0a4c40640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a0073e10 con 0x7fa0a0151d50 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa0a4c40640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a015b900 con 0x7fa0a00cb880 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa09ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a015e900 0x7fa0a00c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa09ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a015e900 0x7fa0a00c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44006/0 (socket says 172.21.15.28:44006) 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.410+0000 7fa09ed9d640 1 -- 172.21.15.28:0/987184218 learned_addr learned my addr 172.21.15.28:0/987184218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:37:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa095d9b640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0a0151d50 0x7fa0a00ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa09e59c640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0a00cb880 0x7fa0a0147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa09ed9d640 1 -- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0a00cb880 msgr2=0x7fa0a0147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa09ed9d640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0a00cb880 0x7fa0a0147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa09ed9d640 1 -- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0a0151d50 msgr2=0x7fa0a00ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa09ed9d640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0a0151d50 0x7fa0a00ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa09ed9d640 1 -- 172.21.15.28:0/987184218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a00cb020 con 0x7fa0a015e900 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa09ed9d640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a015e900 0x7fa0a00c8620 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa09802b020 tx=0x7fa098003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa0977fe640 1 -- 172.21.15.28:0/987184218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098046070 con 0x7fa0a015e900 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa0977fe640 1 -- 172.21.15.28:0/987184218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa098004250 con 0x7fa0a015e900 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a0153580 con 0x7fa0a015e900 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa0977fe640 1 -- 172.21.15.28:0/987184218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098033050 con 0x7fa0a015e900 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a0153b40 con 0x7fa0a015e900 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa0977fe640 1 -- 172.21.15.28:0/987184218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fa098004680 con 0x7fa0a015e900 2024-09-10T20:37:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.411+0000 7fa0977fe640 1 -- 172.21.15.28:0/987184218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa09804b440 con 0x7fa0a015e900 2024-09-10T20:37:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.412+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a00d0360 con 0x7fa0a015e900 2024-09-10T20:37:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.414+0000 7fa0977fe640 1 -- 172.21.15.28:0/987184218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fa098050080 con 0x7fa0a015e900 2024-09-10T20:37:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.480+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa0a0074860 con 0x7fa0a015e900 2024-09-10T20:37:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.481+0000 7fa0977fe640 1 -- 172.21.15.28:0/987184218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fa09802f780 con 0x7fa0a015e900 2024-09-10T20:37:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-09-10T20:37:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.482+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a015e900 msgr2=0x7fa0a00c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.482+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a015e900 0x7fa0a00c8620 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa09802b020 tx=0x7fa098003f10 comp rx=0 tx=0).stop 2024-09-10T20:37:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.482+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 shutdown_connections 2024-09-10T20:37:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.482+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0a015e900 0x7fa0a00c8620 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.482+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0a0151d50 0x7fa0a00ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.482+0000 7fa0a4c40640 1 --2- 172.21.15.28:0/987184218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0a00cb880 0x7fa0a0147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.482+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 >> 172.21.15.28:0/987184218 conn(0x7fa0a00d0cf0 msgr2=0x7fa0a015bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.483+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 shutdown_connections 2024-09-10T20:37:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:48.483+0000 7fa0a4c40640 1 -- 172.21.15.28:0/987184218 wait complete. 2024-09-10T20:37:48.497 INFO:tasks.ceph:check_enable_crimson: False 2024-09-10T20:37:48.497 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-10T20:37:48.497 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:48.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-10T20:37:48.523 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:48.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-10T20:37:48.588 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:48.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-10T20:37:48.654 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:37:48.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-10T20:37:48.720 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:48.720 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-10T20:37:48.746 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:48.746 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-10T20:37:48.812 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:48.813 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-10T20:37:48.877 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:37:48.878 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-10T20:37:48.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 547ada29-9f8c-465c-a0ce-218d9a56bff4 0 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 -- 172.21.15.159:0/1877242122 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 msgr2=0x7f13880b0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 --2- 172.21.15.159:0/1877242122 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 0x7f13880b0340 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f137c005fd0 tx=0x7f137c02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 -- 172.21.15.159:0/1877242122 shutdown_connections 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 --2- 172.21.15.159:0/1877242122 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 0x7f13880b0340 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 --2- 172.21.15.159:0/1877242122 >> v2:172.21.15.159:3300/0 conn(0x7f13880aff60 0x7f13880ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 -- 172.21.15.159:0/1877242122 >> 172.21.15.159:0/1877242122 conn(0x7f1388073390 msgr2=0x7f1388074240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 -- 172.21.15.159:0/1877242122 shutdown_connections 2024-09-10T20:37:49.089 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.084+0000 7f138ffad640 1 -- 172.21.15.159:0/1877242122 wait complete. 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.085+0000 7f138ffad640 1 Processor -- start 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138ffad640 1 -- start start 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138ffad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13880aff60 0x7f138815e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138ffad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880b11c0 0x7f13880d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138ffad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 0x7f13881f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138ffad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13880ce420 con 0x7f13880b11c0 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138ffad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f13880ce2a0 con 0x7f13880aff60 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138ffad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f138815f200 con 0x7f13880cefe0 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138dd49640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13880aff60 0x7f138815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138e54a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 0x7f13881f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.092 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138dd49640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13880aff60 0x7f138815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:60294/0 (socket says 172.21.15.159:60294) 2024-09-10T20:37:49.093 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138dd49640 1 -- 172.21.15.159:0/1207589491 learned_addr learned my addr 172.21.15.159:0/1207589491 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:49.093 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.086+0000 7f138d548640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880b11c0 0x7f13880d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.093 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138e54a640 1 -- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13880aff60 msgr2=0x7f138815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.093 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138e54a640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13880aff60 0x7f138815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.093 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138e54a640 1 -- 172.21.15.159:0/1207589491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880b11c0 msgr2=0x7f13880d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138e54a640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880b11c0 0x7f13880d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138e54a640 1 -- 172.21.15.159:0/1207589491 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13880d1360 con 0x7f13880cefe0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138dd49640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13880aff60 0x7f138815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138e54a640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 0x7f13881f45a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f137800c3c0 tx=0x7f137800c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f13867fc640 1 -- 172.21.15.159:0/1207589491 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378017070 con 0x7f13880cefe0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f13867fc640 1 -- 172.21.15.159:0/1207589491 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13780053c0 con 0x7f13880cefe0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f13867fc640 1 -- 172.21.15.159:0/1207589491 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378003850 con 0x7f13880cefe0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13880c4070 con 0x7f13880cefe0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13880c4630 con 0x7f13880cefe0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f13867fc640 1 -- 172.21.15.159:0/1207589491 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f13780039f0 con 0x7f13880cefe0 2024-09-10T20:37:49.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.087+0000 7f13867fc640 1 -- 172.21.15.159:0/1207589491 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f137801c8e0 con 0x7f13880cefe0 2024-09-10T20:37:49.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.088+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13880cb320 con 0x7f13880cefe0 2024-09-10T20:37:49.095 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.090+0000 7f13867fc640 1 -- 172.21.15.159:0/1207589491 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f1378021020 con 0x7f13880cefe0 2024-09-10T20:37:49.164 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.160+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "547ada29-9f8c-465c-a0ce-218d9a56bff4", "id": 0} v 0) -- 0x7f1388070bc0 con 0x7f13880cefe0 2024-09-10T20:37:49.170 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.166+0000 7f13867fc640 1 -- 172.21.15.159:0/1207589491 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "547ada29-9f8c-465c-a0ce-218d9a56bff4", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f137801cb30 con 0x7f13880cefe0 2024-09-10T20:37:49.170 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-09-10T20:37:49.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 msgr2=0x7f13881f45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 0x7f13881f45a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f137800c3c0 tx=0x7f137800c7c0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.171 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 shutdown_connections 2024-09-10T20:37:49.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13880cefe0 0x7f13881f45a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880b11c0 0x7f13880d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 --2- 172.21.15.159:0/1207589491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13880aff60 0x7f138815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 >> 172.21.15.159:0/1207589491 conn(0x7f1388073390 msgr2=0x7f13880cbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 shutdown_connections 2024-09-10T20:37:49.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.167+0000 7f138ffad640 1 -- 172.21.15.159:0/1207589491 wait complete. 2024-09-10T20:37:49.181 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new c8bd37ec-f775-41d8-a6bd-517095803a9e 1 2024-09-10T20:37:49.284 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 -- 172.21.15.159:0/2923923816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc415e900 msgr2=0x7fabc40cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.285 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 --2- 172.21.15.159:0/2923923816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc415e900 0x7fabc40cb340 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fabac005fd0 tx=0x7fabac02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.285 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 -- 172.21.15.159:0/2923923816 shutdown_connections 2024-09-10T20:37:49.285 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 --2- 172.21.15.159:0/2923923816 >> v2:172.21.15.28:3300/0 conn(0x7fabc40cb880 0x7fabc40736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.285 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 --2- 172.21.15.159:0/2923923816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc415e900 0x7fabc40cb340 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.285 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 -- 172.21.15.159:0/2923923816 >> 172.21.15.159:0/2923923816 conn(0x7fabc40d0cf0 msgr2=0x7fabc40d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.285 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 -- 172.21.15.159:0/2923923816 shutdown_connections 2024-09-10T20:37:49.285 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.280+0000 7fabc8bca640 1 -- 172.21.15.159:0/2923923816 wait complete. 2024-09-10T20:37:49.286 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.281+0000 7fabc8bca640 1 Processor -- start 2024-09-10T20:37:49.286 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc8bca640 1 -- start start 2024-09-10T20:37:49.288 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc8bca640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc40cb880 0x7fabc4151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.288 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc8bca640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabc4151d50 0x7fabc40cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.288 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc8bca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc415e900 0x7fabc40cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.288 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc8bca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabc415b900 con 0x7fabc415e900 2024-09-10T20:37:49.288 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc8bca640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fabc4074050 con 0x7fabc4151d50 2024-09-10T20:37:49.288 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc8bca640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fabc415ba80 con 0x7fabc40cb880 2024-09-10T20:37:49.288 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc259c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc40cb880 0x7fabc4151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc1d9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabc4151d50 0x7fabc40cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc259c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc40cb880 0x7fabc4151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54048/0 (socket says 172.21.15.159:54048) 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc259c640 1 -- 172.21.15.159:0/1852117106 learned_addr learned my addr 172.21.15.159:0/1852117106 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.282+0000 7fabc2d9d640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc415e900 0x7fabc40cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc259c640 1 -- 172.21.15.159:0/1852117106 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabc4151d50 msgr2=0x7fabc40cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc259c640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabc4151d50 0x7fabc40cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc259c640 1 -- 172.21.15.159:0/1852117106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc415e900 msgr2=0x7fabc40cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc259c640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc415e900 0x7fabc40cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc259c640 1 -- 172.21.15.159:0/1852117106 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc40cad00 con 0x7fabc40cb880 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc2d9d640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc415e900 0x7fabc40cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc259c640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc40cb880 0x7fabc4151190 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c450 tx=0x7fabb400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabbaffd640 1 -- 172.21.15.159:0/1852117106 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb40052d0 con 0x7fabc40cb880 2024-09-10T20:37:49.289 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabbaffd640 1 -- 172.21.15.159:0/1852117106 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabb4005470 con 0x7fabc40cb880 2024-09-10T20:37:49.290 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc41516d0 con 0x7fabc40cb880 2024-09-10T20:37:49.290 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabbaffd640 1 -- 172.21.15.159:0/1852117106 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4012870 con 0x7fabc40cb880 2024-09-10T20:37:49.290 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc41e6fd0 con 0x7fabc40cb880 2024-09-10T20:37:49.290 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabbaffd640 1 -- 172.21.15.159:0/1852117106 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fabb4012a10 con 0x7fabc40cb880 2024-09-10T20:37:49.290 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.283+0000 7fabbaffd640 1 -- 172.21.15.159:0/1852117106 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fabb401b520 con 0x7fabc40cb880 2024-09-10T20:37:49.290 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.284+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc415bfa0 con 0x7fabc40cb880 2024-09-10T20:37:49.290 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.286+0000 7fabbaffd640 1 -- 172.21.15.159:0/1852117106 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fabb40035f0 con 0x7fabc40cb880 2024-09-10T20:37:49.361 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.357+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c8bd37ec-f775-41d8-a6bd-517095803a9e", "id": 1} v 0) -- 0x7fabc40b68e0 con 0x7fabc40cb880 2024-09-10T20:37:49.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.365+0000 7fabbaffd640 1 -- 172.21.15.159:0/1852117106 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c8bd37ec-f775-41d8-a6bd-517095803a9e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fabb40037d0 con 0x7fabc40cb880 2024-09-10T20:37:49.370 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:37:49.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc40cb880 msgr2=0x7fabc4151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc40cb880 0x7fabc4151190 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c450 tx=0x7fabb400c910 comp rx=0 tx=0).stop 2024-09-10T20:37:49.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 shutdown_connections 2024-09-10T20:37:49.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc415e900 0x7fabc40cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabc4151d50 0x7fabc40cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 --2- 172.21.15.159:0/1852117106 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabc40cb880 0x7fabc4151190 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 >> 172.21.15.159:0/1852117106 conn(0x7fabc40d0cf0 msgr2=0x7fabc40c7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 shutdown_connections 2024-09-10T20:37:49.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.366+0000 7fabc8bca640 1 -- 172.21.15.159:0/1852117106 wait complete. 2024-09-10T20:37:49.381 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 082d2815-a747-4bf3-bdb2-02af942ae051 2 2024-09-10T20:37:49.482 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 -- 172.21.15.159:0/3909126900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc0cc4d0 msgr2=0x7fc6fc15b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 --2- 172.21.15.159:0/3909126900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc0cc4d0 0x7fc6fc15b900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4005fd0 tx=0x7fc6e402a540 comp rx=0 tx=0).stop 2024-09-10T20:37:49.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 -- 172.21.15.159:0/3909126900 shutdown_connections 2024-09-10T20:37:49.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 --2- 172.21.15.159:0/3909126900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc0cc4d0 0x7fc6fc15b900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 --2- 172.21.15.159:0/3909126900 >> v2:172.21.15.159:3300/0 conn(0x7fc6fc073dc0 0x7fc6fc15e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 -- 172.21.15.159:0/3909126900 >> 172.21.15.159:0/3909126900 conn(0x7fc6fc0d0d60 msgr2=0x7fc6fc0d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 -- 172.21.15.159:0/3909126900 shutdown_connections 2024-09-10T20:37:49.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.478+0000 7fc700f32640 1 -- 172.21.15.159:0/3909126900 wait complete. 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.479+0000 7fc700f32640 1 Processor -- start 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.479+0000 7fc700f32640 1 -- start start 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc700f32640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc073dc0 0x7fc6fc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc700f32640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6fc0cc4d0 0x7fc6fc0c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc700f32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6fc15be40 0x7fc6fc0b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc700f32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6fc0d00f0 con 0x7fc6fc15be40 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc700f32640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc6fc0743f0 con 0x7fc6fc0cc4d0 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc700f32640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc6fc0d0270 con 0x7fc6fc073dc0 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc6fa59c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc073dc0 0x7fc6fc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc6f3fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6fc0cc4d0 0x7fc6fc0c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc6fa59c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc073dc0 0x7fc6fc06d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54066/0 (socket says 172.21.15.159:54066) 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc6fa59c640 1 -- 172.21.15.159:0/471841270 learned_addr learned my addr 172.21.15.159:0/471841270 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.480+0000 7fc6fad9d640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6fc15be40 0x7fc6fc0b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6fa59c640 1 -- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6fc0cc4d0 msgr2=0x7fc6fc0c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6fa59c640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6fc0cc4d0 0x7fc6fc0c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6fa59c640 1 -- 172.21.15.159:0/471841270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6fc15be40 msgr2=0x7fc6fc0b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6fa59c640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6fc15be40 0x7fc6fc0b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6fa59c640 1 -- 172.21.15.159:0/471841270 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6fc1f5fe0 con 0x7fc6fc073dc0 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6f3fff640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6fc0cc4d0 0x7fc6fc0c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6fa59c640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc073dc0 0x7fc6fc06d540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00e9f0 tx=0x7fc6ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6f2ffd640 1 -- 172.21.15.159:0/471841270 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec00ccc0 con 0x7fc6fc073dc0 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6f2ffd640 1 -- 172.21.15.159:0/471841270 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6ec010430 con 0x7fc6fc073dc0 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6f2ffd640 1 -- 172.21.15.159:0/471841270 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec010840 con 0x7fc6fc073dc0 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6fc1f62c0 con 0x7fc6fc073dc0 2024-09-10T20:37:49.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6fc1f6880 con 0x7fc6fc073dc0 2024-09-10T20:37:49.488 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6f2ffd640 1 -- 172.21.15.159:0/471841270 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fc6ec0109e0 con 0x7fc6fc073dc0 2024-09-10T20:37:49.488 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.481+0000 7fc6f2ffd640 1 -- 172.21.15.159:0/471841270 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fc6ec018d30 con 0x7fc6fc073dc0 2024-09-10T20:37:49.488 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.482+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6fc0d0720 con 0x7fc6fc073dc0 2024-09-10T20:37:49.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.485+0000 7fc6f2ffd640 1 -- 172.21.15.159:0/471841270 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fc6ec0213f0 con 0x7fc6fc073dc0 2024-09-10T20:37:49.559 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.555+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "082d2815-a747-4bf3-bdb2-02af942ae051", "id": 2} v 0) -- 0x7fc6fc0c66d0 con 0x7fc6fc073dc0 2024-09-10T20:37:49.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.558+0000 7fc6f2ffd640 1 -- 172.21.15.159:0/471841270 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "082d2815-a747-4bf3-bdb2-02af942ae051", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fc6ec018420 con 0x7fc6fc073dc0 2024-09-10T20:37:49.563 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-09-10T20:37:49.564 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc073dc0 msgr2=0x7fc6fc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.564 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc073dc0 0x7fc6fc06d540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00e9f0 tx=0x7fc6ec00eeb0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.564 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 shutdown_connections 2024-09-10T20:37:49.564 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6fc15be40 0x7fc6fc0b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6fc0cc4d0 0x7fc6fc0c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 --2- 172.21.15.159:0/471841270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6fc073dc0 0x7fc6fc06d540 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 >> 172.21.15.159:0/471841270 conn(0x7fc6fc0d0d60 msgr2=0x7fc6fc0d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 shutdown_connections 2024-09-10T20:37:49.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.559+0000 7fc700f32640 1 -- 172.21.15.159:0/471841270 wait complete. 2024-09-10T20:37:49.574 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new d222baa7-bb15-4bc0-9aca-a3a95301665d 3 2024-09-10T20:37:49.677 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.673+0000 7f381add3640 1 -- 172.21.15.159:0/2106180462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 msgr2=0x7f38140afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.677 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.673+0000 7f381add3640 1 --2- 172.21.15.159:0/2106180462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 0x7f38140afde0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f381415fb90 tx=0x7f380802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.678 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.673+0000 7f381add3640 1 -- 172.21.15.159:0/2106180462 shutdown_connections 2024-09-10T20:37:49.678 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.673+0000 7f381add3640 1 --2- 172.21.15.159:0/2106180462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 0x7f38140afde0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.678 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.673+0000 7f381add3640 1 -- 172.21.15.159:0/2106180462 >> 172.21.15.159:0/2106180462 conn(0x7f3814073390 msgr2=0x7f3814074240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.678 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.673+0000 7f381add3640 1 -- 172.21.15.159:0/2106180462 shutdown_connections 2024-09-10T20:37:49.678 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.673+0000 7f381add3640 1 -- 172.21.15.159:0/2106180462 wait complete. 2024-09-10T20:37:49.680 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.674+0000 7f381add3640 1 Processor -- start 2024-09-10T20:37:49.680 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.674+0000 7f381add3640 1 -- start start 2024-09-10T20:37:49.680 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f381add3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3814074950 0x7f38140c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.680 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f381add3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 0x7f38140c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f381add3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38140d0b20 0x7f38141576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f381add3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38140ccb30 con 0x7f38140d0b20 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f381add3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f381415ce60 con 0x7f38140b1960 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f381add3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3814157c20 con 0x7f3814074950 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3813fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 0x7f38140c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3814074950 0x7f38140c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3814074950 0x7f38140c3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54076/0 (socket says 172.21.15.159:54076) 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 -- 172.21.15.159:0/2173080422 learned_addr learned my addr 172.21.15.159:0/2173080422 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 -- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 msgr2=0x7f38140c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 0x7f38140c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 -- 172.21.15.159:0/2173080422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38140d0b20 msgr2=0x7f38141576e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38140d0b20 0x7f38141576e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 -- 172.21.15.159:0/2173080422 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38140c7c50 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3813fff640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 0x7f38140c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.675+0000 7f3818b6f640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3814074950 0x7f38140c3fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3808009850 tx=0x7f38080038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.676+0000 7f38117fa640 1 -- 172.21.15.159:0/2173080422 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380803f070 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.676+0000 7f38117fa640 1 -- 172.21.15.159:0/2173080422 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38080047a0 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.676+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38140b3da0 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.676+0000 7f38117fa640 1 -- 172.21.15.159:0/2173080422 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808035030 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.676+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f38140b4360 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.676+0000 7f38117fa640 1 -- 172.21.15.159:0/2173080422 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f3808004bb0 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.676+0000 7f38117fa640 1 -- 172.21.15.159:0/2173080422 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f380803bd00 con 0x7f3814074950 2024-09-10T20:37:49.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.677+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381415d320 con 0x7f3814074950 2024-09-10T20:37:49.683 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.678+0000 7f38117fa640 1 -- 172.21.15.159:0/2173080422 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f3808005060 con 0x7f3814074950 2024-09-10T20:37:49.753 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.749+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d222baa7-bb15-4bc0-9aca-a3a95301665d", "id": 3} v 0) -- 0x7f38140d1210 con 0x7f3814074950 2024-09-10T20:37:49.756 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.752+0000 7f38117fa640 1 -- 172.21.15.159:0/2173080422 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d222baa7-bb15-4bc0-9aca-a3a95301665d", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f38080052e0 con 0x7f3814074950 2024-09-10T20:37:49.756 INFO:teuthology.orchestra.run.smithi159.stdout:3 2024-09-10T20:37:49.757 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3814074950 msgr2=0x7f38140c3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3814074950 0x7f38140c3fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3808009850 tx=0x7f38080038a0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 shutdown_connections 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38140d0b20 0x7f38141576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38140b1960 0x7f38140c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 --2- 172.21.15.159:0/2173080422 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3814074950 0x7f38140c3fd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 >> 172.21.15.159:0/2173080422 conn(0x7f3814073390 msgr2=0x7f381415f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 shutdown_connections 2024-09-10T20:37:49.758 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.753+0000 7f381add3640 1 -- 172.21.15.159:0/2173080422 wait complete. 2024-09-10T20:37:49.768 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new c143c33b-86b2-466c-9cff-cd0494dd6675 4 2024-09-10T20:37:49.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 -- 172.21.15.159:0/33146708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7100ccc70 msgr2=0x7fe7100cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 --2- 172.21.15.159:0/33146708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7100ccc70 0x7fe7100cb650 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe708005fd0 tx=0x7fe70802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 -- 172.21.15.159:0/33146708 shutdown_connections 2024-09-10T20:37:49.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 --2- 172.21.15.159:0/33146708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7100ccc70 0x7fe7100cb650 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 --2- 172.21.15.159:0/33146708 >> v2:172.21.15.159:3300/0 conn(0x7fe71006fe30 0x7fe710071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 -- 172.21.15.159:0/33146708 >> 172.21.15.159:0/33146708 conn(0x7fe7100d0d60 msgr2=0x7fe7100d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 -- 172.21.15.159:0/33146708 shutdown_connections 2024-09-10T20:37:49.871 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.865+0000 7fe714f16640 1 -- 172.21.15.159:0/33146708 wait complete. 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 Processor -- start 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 -- start start 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe71006fe30 0x7fe710159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7100ccc70 0x7fe7100c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71015f140 0x7fe7100c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7100cbd10 con 0x7fe7100ccc70 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe7100cbb90 con 0x7fe71015f140 2024-09-10T20:37:49.873 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe714f16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe7100cbe90 con 0x7fe71006fe30 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe70e59c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe71006fe30 0x7fe710159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe70ed9d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71015f140 0x7fe7100c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe70e59c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe71006fe30 0x7fe710159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54080/0 (socket says 172.21.15.159:54080) 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe70e59c640 1 -- 172.21.15.159:0/1002518478 learned_addr learned my addr 172.21.15.159:0/1002518478 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.867+0000 7fe70ed9d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71015f140 0x7fe7100c6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:60342/0 (socket says 172.21.15.159:60342) 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe707fff640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7100ccc70 0x7fe7100c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe70e59c640 1 -- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71015f140 msgr2=0x7fe7100c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe70e59c640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71015f140 0x7fe7100c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe70e59c640 1 -- 172.21.15.159:0/1002518478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7100ccc70 msgr2=0x7fe7100c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe70e59c640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7100ccc70 0x7fe7100c6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe70e59c640 1 -- 172.21.15.159:0/1002518478 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7100c7300 con 0x7fe71006fe30 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe70ed9d640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71015f140 0x7fe7100c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:49.874 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe70e59c640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe71006fe30 0x7fe710159ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c3c0 tx=0x7fe70000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe706ffd640 1 -- 172.21.15.159:0/1002518478 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700017070 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe706ffd640 1 -- 172.21.15.159:0/1002518478 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7000035f0 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe706ffd640 1 -- 172.21.15.159:0/1002518478 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700003a20 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7100c75e0 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7100c49e0 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe706ffd640 1 -- 172.21.15.159:0/1002518478 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fe700003bc0 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.868+0000 7fe706ffd640 1 -- 172.21.15.159:0/1002518478 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fe700012910 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.869+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7100cc340 con 0x7fe71006fe30 2024-09-10T20:37:49.875 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.871+0000 7fe706ffd640 1 -- 172.21.15.159:0/1002518478 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fe700021070 con 0x7fe71006fe30 2024-09-10T20:37:49.946 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.942+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c143c33b-86b2-466c-9cff-cd0494dd6675", "id": 4} v 0) -- 0x7fe71015e8d0 con 0x7fe71006fe30 2024-09-10T20:37:49.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.945+0000 7fe706ffd640 1 -- 172.21.15.159:0/1002518478 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c143c33b-86b2-466c-9cff-cd0494dd6675", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fe700012d60 con 0x7fe71006fe30 2024-09-10T20:37:49.949 INFO:teuthology.orchestra.run.smithi159.stdout:4 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe71006fe30 msgr2=0x7fe710159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe71006fe30 0x7fe710159ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c3c0 tx=0x7fe70000c880 comp rx=0 tx=0).stop 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 shutdown_connections 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe71015f140 0x7fe7100c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7100ccc70 0x7fe7100c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 --2- 172.21.15.159:0/1002518478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe71006fe30 0x7fe710159ed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 >> 172.21.15.159:0/1002518478 conn(0x7fe7100d0d60 msgr2=0x7fe7100afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 shutdown_connections 2024-09-10T20:37:49.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:49.946+0000 7fe714f16640 1 -- 172.21.15.159:0/1002518478 wait complete. 2024-09-10T20:37:49.960 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 4ec51bc6-6e6d-4aaf-b889-9f47e66698b0 5 2024-09-10T20:37:50.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.057+0000 7f3cc26a3640 1 -- 172.21.15.159:0/3577497165 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0d01c0 msgr2=0x7f3cbc0ced60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.057+0000 7f3cc26a3640 1 --2- 172.21.15.159:0/3577497165 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0d01c0 0x7f3cbc0ced60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0d14c0 tx=0x7f3cb402aac0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.058+0000 7f3cc26a3640 1 -- 172.21.15.159:0/3577497165 shutdown_connections 2024-09-10T20:37:50.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.058+0000 7f3cc26a3640 1 --2- 172.21.15.159:0/3577497165 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0d01c0 0x7f3cbc0ced60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.063 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.058+0000 7f3cc26a3640 1 --2- 172.21.15.159:0/3577497165 >> v2:172.21.15.28:3300/0 conn(0x7f3cbc15bf60 0x7f3cbc0ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.063 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.058+0000 7f3cc26a3640 1 -- 172.21.15.159:0/3577497165 >> 172.21.15.159:0/3577497165 conn(0x7f3cbc09b1b0 msgr2=0x7f3cbc0d1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.063 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.058+0000 7f3cc26a3640 1 -- 172.21.15.159:0/3577497165 shutdown_connections 2024-09-10T20:37:50.063 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.058+0000 7f3cc26a3640 1 -- 172.21.15.159:0/3577497165 wait complete. 2024-09-10T20:37:50.067 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.059+0000 7f3cc26a3640 1 Processor -- start 2024-09-10T20:37:50.067 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.059+0000 7f3cc26a3640 1 -- start start 2024-09-10T20:37:50.067 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc26a3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0cda50 0x7f3cbc1f8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.067 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc26a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cbc0d01c0 0x7f3cbc1eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.067 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc26a3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cbc15bf60 0x7f3cbc0b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc26a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc0ccfc0 con 0x7f3cbc0d01c0 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc26a3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc0cfbf0 con 0x7f3cbc0cda50 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc26a3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3cbc0cd140 con 0x7f3cbc15bf60 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0cda50 0x7f3cbc1f8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0cda50 0x7f3cbc1f8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:60352/0 (socket says 172.21.15.159:60352) 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 -- 172.21.15.159:0/2544294448 learned_addr learned my addr 172.21.15.159:0/2544294448 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc0c40640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cbc15bf60 0x7f3cbc0b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbb7fe640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cbc0d01c0 0x7f3cbc1eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 -- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cbc15bf60 msgr2=0x7f3cbc0b6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.068 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cbc15bf60 0x7f3cbc0b6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 -- 172.21.15.159:0/2544294448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cbc0d01c0 msgr2=0x7f3cbc1eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cbc0d01c0 0x7f3cbc1eb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 -- 172.21.15.159:0/2544294448 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc1f9240 con 0x7f3cbc0cda50 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc0c40640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cbc15bf60 0x7f3cbc0b6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cbbfff640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0cda50 0x7f3cbc1f8d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00e790 tx=0x7f3cac00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cb8ff9640 1 -- 172.21.15.159:0/2544294448 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac00eea0 con 0x7f3cbc0cda50 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cb8ff9640 1 -- 172.21.15.159:0/2544294448 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cac00cdb0 con 0x7f3cbc0cda50 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cbc1f52c0 con 0x7f3cbc0cda50 2024-09-10T20:37:50.069 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.060+0000 7f3cb8ff9640 1 -- 172.21.15.159:0/2544294448 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac010690 con 0x7f3cbc0cda50 2024-09-10T20:37:50.070 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.061+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cbc1f5880 con 0x7f3cbc0cda50 2024-09-10T20:37:50.070 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.061+0000 7f3cb8ff9640 1 -- 172.21.15.159:0/2544294448 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f3cac010830 con 0x7f3cbc0cda50 2024-09-10T20:37:50.070 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.061+0000 7f3cb8ff9640 1 -- 172.21.15.159:0/2544294448 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f3cac018d50 con 0x7f3cbc0cda50 2024-09-10T20:37:50.070 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.061+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cbc0cd660 con 0x7f3cbc0cda50 2024-09-10T20:37:50.070 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.063+0000 7f3cb8ff9640 1 -- 172.21.15.159:0/2544294448 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f3cac010d30 con 0x7f3cbc0cda50 2024-09-10T20:37:50.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.133+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4ec51bc6-6e6d-4aaf-b889-9f47e66698b0", "id": 5} v 0) -- 0x7f3cbc0c55e0 con 0x7f3cbc0cda50 2024-09-10T20:37:50.140 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.136+0000 7f3cb8ff9640 1 -- 172.21.15.159:0/2544294448 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4ec51bc6-6e6d-4aaf-b889-9f47e66698b0", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f3cac018420 con 0x7f3cbc0cda50 2024-09-10T20:37:50.141 INFO:teuthology.orchestra.run.smithi159.stdout:5 2024-09-10T20:37:50.142 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.137+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0cda50 msgr2=0x7f3cbc1f8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.137+0000 7f3cc26a3640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0cda50 0x7f3cbc1f8d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00e790 tx=0x7f3cac00ec50 comp rx=0 tx=0).stop 2024-09-10T20:37:50.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.137+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 shutdown_connections 2024-09-10T20:37:50.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.137+0000 7f3cc26a3640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cbc15bf60 0x7f3cbc0b6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.137+0000 7f3cc26a3640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cbc0d01c0 0x7f3cbc1eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.137+0000 7f3cc26a3640 1 --2- 172.21.15.159:0/2544294448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cbc0cda50 0x7f3cbc1f8d00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.137+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 >> 172.21.15.159:0/2544294448 conn(0x7f3cbc09b1b0 msgr2=0x7f3cbc070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.144 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.138+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 shutdown_connections 2024-09-10T20:37:50.144 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.138+0000 7f3cc26a3640 1 -- 172.21.15.159:0/2544294448 wait complete. 2024-09-10T20:37:50.152 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 26884790-c3f6-4a07-8df5-9ddb7f0cbfd9 6 2024-09-10T20:37:50.256 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.251+0000 7f01279ad640 1 -- 172.21.15.159:0/1835161782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01200cc4d0 msgr2=0x7f012015b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.256 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.251+0000 7f01279ad640 1 --2- 172.21.15.159:0/1835161782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01200cc4d0 0x7f012015b900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0114005fd0 tx=0x7f011402a540 comp rx=0 tx=0).stop 2024-09-10T20:37:50.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.251+0000 7f01279ad640 1 -- 172.21.15.159:0/1835161782 shutdown_connections 2024-09-10T20:37:50.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.251+0000 7f01279ad640 1 --2- 172.21.15.159:0/1835161782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01200cc4d0 0x7f012015b900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.251+0000 7f01279ad640 1 --2- 172.21.15.159:0/1835161782 >> v2:172.21.15.28:3300/0 conn(0x7f0120073dc0 0x7f012015e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.251+0000 7f01279ad640 1 -- 172.21.15.159:0/1835161782 >> 172.21.15.159:0/1835161782 conn(0x7f01200d0d60 msgr2=0x7f01200d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.252+0000 7f01279ad640 1 -- 172.21.15.159:0/1835161782 shutdown_connections 2024-09-10T20:37:50.257 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.252+0000 7f01279ad640 1 -- 172.21.15.159:0/1835161782 wait complete. 2024-09-10T20:37:50.260 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 Processor -- start 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 -- start start 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0120073dc0 0x7f012014f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cc4d0 0x7f01200c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012015be40 0x7f01200c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01200d00f0 con 0x7f01200cc4d0 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f01200743f0 con 0x7f0120073dc0 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.253+0000 7f01279ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f01200d0270 con 0x7f012015be40 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125749640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0120073dc0 0x7f012014f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125f4a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012015be40 0x7f01200c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125749640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0120073dc0 0x7f012014f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:60356/0 (socket says 172.21.15.159:60356) 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125749640 1 -- 172.21.15.159:0/598898997 learned_addr learned my addr 172.21.15.159:0/598898997 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0124f48640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cc4d0 0x7f01200c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125f4a640 1 -- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0120073dc0 msgr2=0x7f012014f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125f4a640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0120073dc0 0x7f012014f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125f4a640 1 -- 172.21.15.159:0/598898997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cc4d0 msgr2=0x7f01200c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125f4a640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cc4d0 0x7f01200c56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125f4a640 1 -- 172.21.15.159:0/598898997 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01201f7230 con 0x7f012015be40 2024-09-10T20:37:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125749640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0120073dc0 0x7f012014f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0124f48640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cc4d0 0x7f01200c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0125f4a640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012015be40 0x7f01200c5c30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f011800c730 tx=0x7f011800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0111ffb640 1 -- 172.21.15.159:0/598898997 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118003b70 con 0x7f012015be40 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0111ffb640 1 -- 172.21.15.159:0/598898997 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0118003d10 con 0x7f012015be40 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0111ffb640 1 -- 172.21.15.159:0/598898997 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118005330 con 0x7f012015be40 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01200c4f60 con 0x7f012015be40 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.254+0000 7f0111ffb640 1 -- 172.21.15.159:0/598898997 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f01180054d0 con 0x7f012015be40 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.255+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f01200c49a0 con 0x7f012015be40 2024-09-10T20:37:50.263 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.255+0000 7f0111ffb640 1 -- 172.21.15.159:0/598898997 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f0118012df0 con 0x7f012015be40 2024-09-10T20:37:50.264 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.255+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01200d0720 con 0x7f012015be40 2024-09-10T20:37:50.264 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.257+0000 7f0111ffb640 1 -- 172.21.15.159:0/598898997 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f01200cd1a0 con 0x7f012015be40 2024-09-10T20:37:50.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.328+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "26884790-c3f6-4a07-8df5-9ddb7f0cbfd9", "id": 6} v 0) -- 0x7f0120070de0 con 0x7f012015be40 2024-09-10T20:37:50.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.331+0000 7f0111ffb640 1 -- 172.21.15.159:0/598898997 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26884790-c3f6-4a07-8df5-9ddb7f0cbfd9", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f0118012800 con 0x7f012015be40 2024-09-10T20:37:50.335 INFO:teuthology.orchestra.run.smithi159.stdout:6 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012015be40 msgr2=0x7f01200c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012015be40 0x7f01200c5c30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f011800c730 tx=0x7f011800cbf0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 shutdown_connections 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012015be40 0x7f01200c5c30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cc4d0 0x7f01200c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 --2- 172.21.15.159:0/598898997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0120073dc0 0x7f012014f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 >> 172.21.15.159:0/598898997 conn(0x7f01200d0d60 msgr2=0x7f01200b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 shutdown_connections 2024-09-10T20:37:50.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.332+0000 7f01279ad640 1 -- 172.21.15.159:0/598898997 wait complete. 2024-09-10T20:37:50.347 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 2318f19e-1d8f-4383-97c5-e10e7331c8b2 7 2024-09-10T20:37:50.449 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 -- 172.21.15.159:0/4175490078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0ccc70 msgr2=0x7fafcc0cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.449 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 --2- 172.21.15.159:0/4175490078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0ccc70 0x7fafcc0cb650 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005fd0 tx=0x7fafc402a8b0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.449 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 -- 172.21.15.159:0/4175490078 shutdown_connections 2024-09-10T20:37:50.449 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 --2- 172.21.15.159:0/4175490078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc0ccc70 0x7fafcc0cb650 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.450 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 --2- 172.21.15.159:0/4175490078 >> v2:172.21.15.28:3300/0 conn(0x7fafcc06fe30 0x7fafcc071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.450 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 -- 172.21.15.159:0/4175490078 >> 172.21.15.159:0/4175490078 conn(0x7fafcc0d0d60 msgr2=0x7fafcc0d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.450 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 -- 172.21.15.159:0/4175490078 shutdown_connections 2024-09-10T20:37:50.450 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.445+0000 7fafd105a640 1 -- 172.21.15.159:0/4175490078 wait complete. 2024-09-10T20:37:50.450 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.446+0000 7fafd105a640 1 Processor -- start 2024-09-10T20:37:50.450 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.446+0000 7fafd105a640 1 -- start start 2024-09-10T20:37:50.453 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafd105a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc06fe30 0x7fafcc15b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.453 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafd105a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafcc0ccc70 0x7fafcc0a9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.453 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafd105a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc15f140 0x7fafcc0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.453 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafd105a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc0cbd10 con 0x7fafcc0ccc70 2024-09-10T20:37:50.453 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafd105a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc0cbb90 con 0x7fafcc15f140 2024-09-10T20:37:50.453 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafd105a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fafcc0cbe90 con 0x7fafcc06fe30 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafcad9d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc06fe30 0x7fafcc15b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafcad9d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc06fe30 0x7fafcc15b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:54132/0 (socket says 172.21.15.159:54132) 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafcad9d640 1 -- 172.21.15.159:0/1427028326 learned_addr learned my addr 172.21.15.159:0/1427028326 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafcb59e640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc15f140 0x7fafcc0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.447+0000 7fafca59c640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafcc0ccc70 0x7fafcc0a9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafca59c640 1 -- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc06fe30 msgr2=0x7fafcc15b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafca59c640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc06fe30 0x7fafcc15b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafca59c640 1 -- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc15f140 msgr2=0x7fafcc0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafca59c640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc15f140 0x7fafcc0ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafca59c640 1 -- 172.21.15.159:0/1427028326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc0c8060 con 0x7fafcc0ccc70 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafcad9d640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc06fe30 0x7fafcc15b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafca59c640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafcc0ccc70 0x7fafcc0a9c00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fafc000e9c0 tx=0x7fafc000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:50.454 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafb37fe640 1 -- 172.21.15.159:0/1427028326 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc000ccc0 con 0x7fafcc0ccc70 2024-09-10T20:37:50.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafb37fe640 1 -- 172.21.15.159:0/1427028326 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafc0010430 con 0x7fafcc0ccc70 2024-09-10T20:37:50.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafcc0c8340 con 0x7fafcc0ccc70 2024-09-10T20:37:50.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafb37fe640 1 -- 172.21.15.159:0/1427028326 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0010860 con 0x7fafcc0ccc70 2024-09-10T20:37:50.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafcc0c8900 con 0x7fafcc0ccc70 2024-09-10T20:37:50.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafb37fe640 1 -- 172.21.15.159:0/1427028326 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fafc0010a00 con 0x7fafcc0ccc70 2024-09-10T20:37:50.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.448+0000 7fafb37fe640 1 -- 172.21.15.159:0/1427028326 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fafc0018d30 con 0x7fafcc0ccc70 2024-09-10T20:37:50.455 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.449+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafcc0cc340 con 0x7fafcc0ccc70 2024-09-10T20:37:50.457 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.453+0000 7fafb37fe640 1 -- 172.21.15.159:0/1427028326 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fafc0018420 con 0x7fafcc0ccc70 2024-09-10T20:37:50.528 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.524+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2318f19e-1d8f-4383-97c5-e10e7331c8b2", "id": 7} v 0) -- 0x7fafcc074e70 con 0x7fafcc0ccc70 2024-09-10T20:37:50.530 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.526+0000 7fafb37fe640 1 -- 172.21.15.159:0/1427028326 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2318f19e-1d8f-4383-97c5-e10e7331c8b2", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fafc0018600 con 0x7fafcc0ccc70 2024-09-10T20:37:50.530 INFO:teuthology.orchestra.run.smithi159.stdout:7 2024-09-10T20:37:50.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafcc0ccc70 msgr2=0x7fafcc0a9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafcc0ccc70 0x7fafcc0a9c00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fafc000e9c0 tx=0x7fafc000ee80 comp rx=0 tx=0).stop 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 shutdown_connections 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafcc15f140 0x7fafcc0ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafcc0ccc70 0x7fafcc0a9c00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 --2- 172.21.15.159:0/1427028326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafcc06fe30 0x7fafcc15b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 >> 172.21.15.159:0/1427028326 conn(0x7fafcc0d0d60 msgr2=0x7fafcc0c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 shutdown_connections 2024-09-10T20:37:50.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:37:50.527+0000 7fafd105a640 1 -- 172.21.15.159:0/1427028326 wait complete. 2024-09-10T20:37:50.542 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-10T20:37:50.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-10T20:37:50.544 INFO:tasks.ceph.osd.0:Started 2024-09-10T20:37:50.544 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-10T20:37:50.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-10T20:37:50.547 INFO:tasks.ceph.osd.1:Started 2024-09-10T20:37:50.547 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-10T20:37:50.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-10T20:37:50.549 INFO:tasks.ceph.osd.2:Started 2024-09-10T20:37:50.549 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-10T20:37:50.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-10T20:37:50.551 INFO:tasks.ceph.osd.3:Started 2024-09-10T20:37:50.552 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-10T20:37:50.552 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-10T20:37:50.554 INFO:tasks.ceph.osd.4:Started 2024-09-10T20:37:50.554 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-10T20:37:50.554 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-10T20:37:50.556 INFO:tasks.ceph.osd.5:Started 2024-09-10T20:37:50.556 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-10T20:37:50.556 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-10T20:37:50.558 INFO:tasks.ceph.osd.6:Started 2024-09-10T20:37:50.558 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-10T20:37:50.558 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-10T20:37:50.561 INFO:tasks.ceph.osd.7:Started 2024-09-10T20:37:50.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:37:50.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 -- 172.21.15.28:0/1308418561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015f660 msgr2=0x7f00a00ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 --2- 172.21.15.28:0/1308418561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015f660 0x7f00a00ce710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0094009830 tx=0x7f009402a510 comp rx=0 tx=0).stop 2024-09-10T20:37:50.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 -- 172.21.15.28:0/1308418561 shutdown_connections 2024-09-10T20:37:50.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 --2- 172.21.15.28:0/1308418561 >> v2:172.21.15.159:3300/0 conn(0x7f00a00cdaf0 0x7f00a00cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 --2- 172.21.15.28:0/1308418561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a015f660 0x7f00a00ce710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 -- 172.21.15.28:0/1308418561 >> 172.21.15.28:0/1308418561 conn(0x7f00a00d1ed0 msgr2=0x7f00a00d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 -- 172.21.15.28:0/1308418561 shutdown_connections 2024-09-10T20:37:50.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.695+0000 7f00a6d17640 1 -- 172.21.15.28:0/1308418561 wait complete. 2024-09-10T20:37:50.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.697+0000 7f00a6d17640 1 Processor -- start 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.697+0000 7f00a6d17640 1 -- start start 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.697+0000 7f00a6d17640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00cdaf0 0x7f00a009ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.697+0000 7f00a6d17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00d29f0 0x7f00a01503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a6d17640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a015f660 0x7f00a01537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a6d17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00a00b5e50 con 0x7f00a00d29f0 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a6d17640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f00a00b5cd0 con 0x7f00a015f660 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a6d17640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f00a00b5fd0 con 0x7f00a00cdaf0 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a4ab3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00cdaf0 0x7f00a009ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00d29f0 0x7f00a01503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a4ab3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00cdaf0 0x7f00a009ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:58858/0 (socket says 172.21.15.28:58858) 2024-09-10T20:37:50.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a4ab3640 1 -- 172.21.15.28:0/1705775290 learned_addr learned my addr 172.21.15.28:0/1705775290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a52b4640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a015f660 0x7f00a01537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009ffff640 1 -- 172.21.15.28:0/1705775290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00cdaf0 msgr2=0x7f00a009ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009ffff640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00cdaf0 0x7f00a009ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009ffff640 1 -- 172.21.15.28:0/1705775290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a015f660 msgr2=0x7f00a01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009ffff640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a015f660 0x7f00a01537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009ffff640 1 -- 172.21.15.28:0/1705775290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a009bd60 con 0x7f00a00d29f0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009ffff640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00d29f0 0x7f00a01503e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c400 tx=0x7f008c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009d7fa640 1 -- 172.21.15.28:0/1705775290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c011790 con 0x7f00a00d29f0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009d7fa640 1 -- 172.21.15.28:0/1705775290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f008c017070 con 0x7f00a00d29f0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a009c040 con 0x7f00a00d29f0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.698+0000 7f009d7fa640 1 -- 172.21.15.28:0/1705775290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c011d00 con 0x7f00a00d29f0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.699+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a009c600 con 0x7f00a00d29f0 2024-09-10T20:37:50.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.699+0000 7f009d7fa640 1 -- 172.21.15.28:0/1705775290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f008c01c460 con 0x7f00a00d29f0 2024-09-10T20:37:50.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.699+0000 7f009d7fa640 1 -- 172.21.15.28:0/1705775290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f008c012420 con 0x7f00a00d29f0 2024-09-10T20:37:50.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.700+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a0069580 con 0x7f00a00d29f0 2024-09-10T20:37:50.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.702+0000 7f009d7fa640 1 -- 172.21.15.28:0/1705775290 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f00a0069580 con 0x7f00a00d29f0 2024-09-10T20:37:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.764+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00a00b64f0 con 0x7f00a00d29f0 2024-09-10T20:37:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.764+0000 7f009d7fa640 1 -- 172.21.15.28:0/1705775290 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f008c012a00 con 0x7f00a00d29f0 2024-09-10T20:37:50.768 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:37:50.769 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:37:50.526381+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:37:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00d29f0 msgr2=0x7f00a01503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00d29f0 0x7f00a01503e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c400 tx=0x7f008c00c8c0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 shutdown_connections 2024-09-10T20:37:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00a015f660 0x7f00a01537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00d29f0 0x7f00a01503e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 --2- 172.21.15.28:0/1705775290 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00a00cdaf0 0x7f00a009ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 >> 172.21.15.28:0/1705775290 conn(0x7f00a00d1ed0 msgr2=0x7f00a00708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 shutdown_connections 2024-09-10T20:37:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:37:50.765+0000 7f00a6d17640 1 -- 172.21.15.28:0/1705775290 wait complete. 2024-09-10T20:37:50.780 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-10T20:37:50.780 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-10T20:37:51.586 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:37:51.682 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 -- 172.21.15.28:0/2906428303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c0d4c40 msgr2=0x7f1f5c0d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 --2- 172.21.15.28:0/2906428303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c0d4c40 0x7f1f5c0d0fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1f48005fd0 tx=0x7f1f4802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 -- 172.21.15.28:0/2906428303 shutdown_connections 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 --2- 172.21.15.28:0/2906428303 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f5c0c8da0 0x7f1f5c0ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 --2- 172.21.15.28:0/2906428303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c0d4c40 0x7f1f5c0d0fc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 -- 172.21.15.28:0/2906428303 >> 172.21.15.28:0/2906428303 conn(0x7f1f5c0d4090 msgr2=0x7f1f5c0d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 -- 172.21.15.28:0/2906428303 shutdown_connections 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.678+0000 7f1f631c2640 1 -- 172.21.15.28:0/2906428303 wait complete. 2024-09-10T20:37:51.683 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.680+0000 7f1f631c2640 1 Processor -- start 2024-09-10T20:37:51.684 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.680+0000 7f1f631c2640 1 -- start start 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.680+0000 7f1f631c2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c0c8da0 0x7f1f5c14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.680+0000 7f1f631c2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f5c0d4c40 0x7f1f5c0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.680+0000 7f1f631c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c161700 0x7f1f5c0d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.680+0000 7f1f631c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c0cf450 con 0x7f1f5c161700 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f631c2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c0cf2d0 con 0x7f1f5c0c8da0 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f631c2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5c0cf5d0 con 0x7f1f5c0d4c40 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c161700 0x7f1f5c0d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f5bfff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f5c0d4c40 0x7f1f5c0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f60f5e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c0c8da0 0x7f1f5c14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:51.686 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c161700 0x7f1f5c0d0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48288/0 (socket says 172.21.15.28:48288) 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 -- 172.21.15.28:0/2272601825 learned_addr learned my addr 172.21.15.28:0/2272601825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f5bfff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f5c0d4c40 0x7f1f5c0d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:58874/0 (socket says 172.21.15.28:58874) 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 -- 172.21.15.28:0/2272601825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f5c0d4c40 msgr2=0x7f1f5c0d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 --2- 172.21.15.28:0/2272601825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f5c0d4c40 0x7f1f5c0d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 -- 172.21.15.28:0/2272601825 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c0c8da0 msgr2=0x7f1f5c14fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 --2- 172.21.15.28:0/2272601825 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c0c8da0 0x7f1f5c14fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 -- 172.21.15.28:0/2272601825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c1f5760 con 0x7f1f5c161700 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f6175f640 1 --2- 172.21.15.28:0/2272601825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c161700 0x7f1f5c0d0c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400eb10 tx=0x7f1f5400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f597fa640 1 -- 172.21.15.28:0/2272601825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54004b20 con 0x7f1f5c161700 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f597fa640 1 -- 172.21.15.28:0/2272601825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f54004ce0 con 0x7f1f5c161700 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5c1f49a0 con 0x7f1f5c161700 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f597fa640 1 -- 172.21.15.28:0/2272601825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54005280 con 0x7f1f5c161700 2024-09-10T20:37:51.687 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.681+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5c1f4f60 con 0x7f1f5c161700 2024-09-10T20:37:51.688 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.682+0000 7f1f597fa640 1 -- 172.21.15.28:0/2272601825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f1f540057b0 con 0x7f1f5c161700 2024-09-10T20:37:51.688 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.682+0000 7f1f597fa640 1 -- 172.21.15.28:0/2272601825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f1f54018b00 con 0x7f1f5c161700 2024-09-10T20:37:51.688 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.682+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f5c0ceb80 con 0x7f1f5c161700 2024-09-10T20:37:51.688 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.684+0000 7f1f597fa640 1 -- 172.21.15.28:0/2272601825 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f1f54038950 con 0x7f1f5c161700 2024-09-10T20:37:51.750 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.747+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f5c160b80 con 0x7f1f5c161700 2024-09-10T20:37:51.751 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.747+0000 7f1f597fa640 1 -- 172.21.15.28:0/2272601825 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f1f54026020 con 0x7f1f5c161700 2024-09-10T20:37:51.751 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-10T20:37:51.751 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:37:50.526381+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:37:51.752 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c161700 msgr2=0x7f1f5c0d0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:51.752 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 --2- 172.21.15.28:0/2272601825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c161700 0x7f1f5c0d0c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400eb10 tx=0x7f1f5400c2d0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.753 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 shutdown_connections 2024-09-10T20:37:51.753 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 --2- 172.21.15.28:0/2272601825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c161700 0x7f1f5c0d0c30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.753 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 --2- 172.21.15.28:0/2272601825 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f5c0d4c40 0x7f1f5c0d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.753 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 --2- 172.21.15.28:0/2272601825 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c0c8da0 0x7f1f5c14fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:51.753 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 >> 172.21.15.28:0/2272601825 conn(0x7f1f5c0d4090 msgr2=0x7f1f5c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:51.753 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 shutdown_connections 2024-09-10T20:37:51.753 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:51.748+0000 7f1f631c2640 1 -- 172.21.15.28:0/2272601825 wait complete. 2024-09-10T20:37:51.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-10T20:37:52.007 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-10T20:37:52.003+0000 7f26e0a0d640 -1 Falling back to public interface 2024-09-10T20:37:52.010 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-09-10T20:37:52.006+0000 7f397b41d640 -1 Falling back to public interface 2024-09-10T20:37:52.018 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-09-10T20:37:52.014+0000 7fcdaf42e640 -1 Falling back to public interface 2024-09-10T20:37:52.035 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-10T20:37:52.032+0000 7fa382817640 -1 Falling back to public interface 2024-09-10T20:37:52.272 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-10T20:37:52.268+0000 7f2a04618640 -1 Falling back to public interface 2024-09-10T20:37:52.289 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-09-10T20:37:52.285+0000 7f9abcbd4640 -1 Falling back to public interface 2024-09-10T20:37:52.530 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-10T20:37:52.526+0000 7f3f16af1640 -1 Falling back to public interface 2024-09-10T20:37:52.534 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-10T20:37:52.530+0000 7f65bb8e0640 -1 Falling back to public interface 2024-09-10T20:37:52.983 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:37:52.979+0000 7f97f4c8f640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-10T20:37:52.984 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:37:52.980+0000 7f97f4c8f640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-10T20:37:55.583 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-09-10T20:37:55.579+0000 7f397b41d640 -1 osd.4 0 log_to_monitors true 2024-09-10T20:37:55.830 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-10T20:37:55.826+0000 7f2a04618640 -1 osd.7 0 log_to_monitors true 2024-09-10T20:37:56.102 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-10T20:37:56.098+0000 7fa382817640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:37:56.335 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-10T20:37:56.331+0000 7f3f16af1640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:37:56.339 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-10T20:37:56.336+0000 7f26e0a0d640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:37:56.389 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-09-10T20:37:56.385+0000 7fcdaf42e640 -1 osd.6 0 log_to_monitors true 2024-09-10T20:37:56.622 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-09-10T20:37:56.618+0000 7f9abcbd4640 -1 osd.5 0 log_to_monitors true 2024-09-10T20:37:56.829 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-10T20:37:56.825+0000 7f65bb8e0640 -1 osd.3 0 log_to_monitors true 2024-09-10T20:37:58.326 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-09-10T20:37:58.322+0000 7f397761c640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T20:37:58.349 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-09-10T20:37:58.345+0000 7f396ec27640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:37:58.568 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:37:58.611 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-10T20:37:58.607+0000 7f29fff7a640 -1 osd.7 0 waiting for initial osdmap 2024-09-10T20:37:58.695 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.690+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3825125794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf600b35b0 msgr2=0x7fbf6015da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:58.695 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.690+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3825125794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf600b35b0 0x7fbf6015da20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400dbc0 tx=0x7fbf5402e490 comp rx=0 tx=0).stop 2024-09-10T20:37:58.695 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.691+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3825125794 shutdown_connections 2024-09-10T20:37:58.695 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.691+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3825125794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf600b35b0 0x7fbf6015da20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:58.695 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.691+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3825125794 >> 172.21.15.28:0/3825125794 conn(0x7fbf6006cec0 msgr2=0x7fbf6006d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:58.695 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.691+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3825125794 shutdown_connections 2024-09-10T20:37:58.695 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.691+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3825125794 wait complete. 2024-09-10T20:37:58.696 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.693+0000 7fbf67d0e640 1 Processor -- start 2024-09-10T20:37:58.696 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.693+0000 7fbf67d0e640 1 -- start start 2024-09-10T20:37:58.699 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf67d0e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf600b35b0 0x7fbf601f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf67d0e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf600cf270 0x7fbf600aea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf67d0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6015aa60 0x7fbf600aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf67d0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf600d00d0 con 0x7fbf6015aa60 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf67d0e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbf60070a60 con 0x7fbf600b35b0 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf65aaa640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf600b35b0 0x7fbf601f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf652a9640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf600cf270 0x7fbf600aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf652a9640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf600cf270 0x7fbf600aea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:58914/0 (socket says 172.21.15.28:58914) 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf65aaa640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf600b35b0 0x7fbf601f4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:37060/0 (socket says 172.21.15.28:37060) 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf652a9640 1 -- 172.21.15.28:0/3651245655 learned_addr learned my addr 172.21.15.28:0/3651245655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.694+0000 7fbf67d0e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbf600d0250 con 0x7fbf600cf270 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf662ab640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6015aa60 0x7fbf600aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf652a9640 1 -- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf600b35b0 msgr2=0x7fbf601f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:58.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf652a9640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf600b35b0 0x7fbf601f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf652a9640 1 -- 172.21.15.28:0/3651245655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6015aa60 msgr2=0x7fbf600aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf652a9640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6015aa60 0x7fbf600aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf652a9640 1 -- 172.21.15.28:0/3651245655 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf601425e0 con 0x7fbf600cf270 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf65aaa640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf600b35b0 0x7fbf601f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf662ab640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6015aa60 0x7fbf600aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf652a9640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf600cf270 0x7fbf600aea90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf54000c00 tx=0x7fbf540038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.695+0000 7fbf5e7fc640 1 -- 172.21.15.28:0/3651245655 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5403a050 con 0x7fbf600cf270 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.696+0000 7fbf5e7fc640 1 -- 172.21.15.28:0/3651245655 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf5403b040 con 0x7fbf600cf270 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.696+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf60141a60 con 0x7fbf600cf270 2024-09-10T20:37:58.701 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.696+0000 7fbf5e7fc640 1 -- 172.21.15.28:0/3651245655 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf54003ca0 con 0x7fbf600cf270 2024-09-10T20:37:58.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.696+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf60142020 con 0x7fbf600cf270 2024-09-10T20:37:58.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.698+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6006e2d0 con 0x7fbf600cf270 2024-09-10T20:37:58.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.699+0000 7fbf5e7fc640 1 -- 172.21.15.28:0/3651245655 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 4) ==== 100545+0+0 (secure 0 0 0) 0x7fbf54041450 con 0x7fbf600cf270 2024-09-10T20:37:58.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.699+0000 7fbf5e7fc640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbf4c077ee0 0x7fbf4c07a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:37:58.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.700+0000 7fbf5e7fc640 1 -- 172.21.15.28:0/3651245655 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 2231+0+0 (secure 0 0 0) 0x7fbf5400ce70 con 0x7fbf600cf270 2024-09-10T20:37:58.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.700+0000 7fbf65aaa640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbf4c077ee0 0x7fbf4c07a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:37:58.704 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.700+0000 7fbf65aaa640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbf4c077ee0 0x7fbf4c07a3a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbf50000f60 tx=0x7fbf50009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:37:58.707 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:58.704+0000 7fbf5e7fc640 1 -- 172.21.15.28:0/3651245655 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbf540874b0 con 0x7fbf600cf270 2024-09-10T20:37:58.741 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-10T20:37:58.737+0000 7f29f7d99640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:37:59.104 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.100+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf60150960 con 0x7fbf600cf270 2024-09-10T20:37:59.104 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.101+0000 7fbf5e7fc640 1 -- 172.21.15.28:0/3651245655 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7058 (secure 0 0 0) 0x7fbf54086db0 con 0x7fbf600cf270 2024-09-10T20:37:59.105 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-10T20:37:59.105 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":13,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:37:57.984685+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:37:59.109 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbf4c077ee0 msgr2=0x7fbf4c07a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:59.109 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbf4c077ee0 0x7fbf4c07a3a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbf50000f60 tx=0x7fbf50009040 comp rx=0 tx=0).stop 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf600cf270 msgr2=0x7fbf600aea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf600cf270 0x7fbf600aea90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf54000c00 tx=0x7fbf540038a0 comp rx=0 tx=0).stop 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 shutdown_connections 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbf4c077ee0 0x7fbf4c07a3a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6015aa60 0x7fbf600aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf600cf270 0x7fbf600aea90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.105+0000 7fbf67d0e640 1 --2- 172.21.15.28:0/3651245655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf600b35b0 0x7fbf601f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.106+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 >> 172.21.15.28:0/3651245655 conn(0x7fbf6006cec0 msgr2=0x7fbf6015ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.106+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 shutdown_connections 2024-09-10T20:37:59.110 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:37:59.106+0000 7fbf67d0e640 1 -- 172.21.15.28:0/3651245655 wait complete. 2024-09-10T20:37:59.124 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-10T20:37:59.308 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-09-10T20:37:59.304+0000 7f9ab9377640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T20:37:59.308 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-09-10T20:37:59.305+0000 7fcdaaa3a640 -1 osd.6 0 waiting for initial osdmap 2024-09-10T20:37:59.318 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-09-10T20:37:59.314+0000 7f9ab0982640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:37:59.321 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-09-10T20:37:59.318+0000 7fcda2859640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:37:59.505 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-10T20:37:59.501+0000 7f26dc44b640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T20:37:59.517 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-10T20:37:59.514+0000 7f26d426a640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:37:59.528 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-10T20:37:59.525+0000 7fa37e91a640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T20:37:59.539 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-10T20:37:59.536+0000 7fa375f25640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:37:59.558 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-10T20:37:59.555+0000 7f3f12a80640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T20:37:59.573 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-10T20:37:59.570+0000 7f3f0a89f640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:37:59.594 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-10T20:37:59.590+0000 7f65b8083640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T20:37:59.609 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-10T20:37:59.606+0000 7f65af68e640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:38:00.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:38:00.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:38:05.931 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.036+0000 7f0bd353d640 1 -- 172.21.15.28:0/1470714458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 msgr2=0x7f0bcc0d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.036+0000 7f0bd353d640 1 --2- 172.21.15.28:0/1470714458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc0d0fc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009830 tx=0x7f0bc002a510 comp rx=0 tx=0).stop 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.036+0000 7f0bd353d640 1 -- 172.21.15.28:0/1470714458 shutdown_connections 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.036+0000 7f0bd353d640 1 --2- 172.21.15.28:0/1470714458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bcc0c8da0 0x7f0bcc0ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.037+0000 7f0bd353d640 1 --2- 172.21.15.28:0/1470714458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc0d0fc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.037+0000 7f0bd353d640 1 -- 172.21.15.28:0/1470714458 >> 172.21.15.28:0/1470714458 conn(0x7f0bcc0d4090 msgr2=0x7f0bcc0d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.037+0000 7f0bd353d640 1 -- 172.21.15.28:0/1470714458 shutdown_connections 2024-09-10T20:38:06.041 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.037+0000 7f0bd353d640 1 -- 172.21.15.28:0/1470714458 wait complete. 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.038+0000 7f0bd353d640 1 Processor -- start 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.038+0000 7f0bd353d640 1 -- start start 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd353d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bcc0c8da0 0x7f0bcc15de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd353d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc15af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd353d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bcc161700 0x7f0bcc0cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd353d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc0cf450 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd353d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc0cf2d0 con 0x7f0bcc161700 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd353d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0bcc0cf5d0 con 0x7f0bcc0c8da0 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc15af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.044 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc15af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47958/0 (socket says 172.21.15.28:47958) 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 -- 172.21.15.28:0/821998800 learned_addr learned my addr 172.21.15.28:0/821998800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd12d9640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bcc0c8da0 0x7f0bcc15de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd1ada640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bcc161700 0x7f0bcc0cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 -- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bcc0c8da0 msgr2=0x7f0bcc15de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bcc0c8da0 0x7f0bcc15de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 -- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bcc161700 msgr2=0x7f0bcc0cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bcc161700 0x7f0bcc0cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 -- 172.21.15.28:0/821998800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bcc0cd300 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd1ada640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bcc161700 0x7f0bcc0cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd12d9640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bcc0c8da0 0x7f0bcc15de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.039+0000 7f0bd0ad8640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc15af30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000c00 tx=0x7f0bc0032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.040+0000 7f0bbdffb640 1 -- 172.21.15.28:0/821998800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0040070 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.040+0000 7f0bbdffb640 1 -- 172.21.15.28:0/821998800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0bc0032e70 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.045 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.040+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bcc0cd5e0 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.046 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.040+0000 7f0bbdffb640 1 -- 172.21.15.28:0/821998800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0003d90 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.046 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.040+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bcc0cd8c0 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.046 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.041+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bcc0ceb80 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.047 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.041+0000 7f0bbdffb640 1 -- 172.21.15.28:0/821998800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0bc004d030 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.047 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.042+0000 7f0bbdffb640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0bb0077f70 0x7f0bb007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.047 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.042+0000 7f0bbdffb640 1 -- 172.21.15.28:0/821998800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0bc00bafd0 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.047 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.042+0000 7f0bd12d9640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0bb0077f70 0x7f0bb007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.047 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.043+0000 7f0bd12d9640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0bb0077f70 0x7f0bb007a430 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8004a30 tx=0x7f0bc8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:06.050 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.047+0000 7f0bbdffb640 1 -- 172.21.15.28:0/821998800 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0bc0083e20 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.237 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.234+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bcc15e370 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.238 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.234+0000 7f0bbdffb640 1 -- 172.21.15.28:0/821998800 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12896 (secure 0 0 0) 0x7f0bc0083720 con 0x7f0bcc0d4c40 2024-09-10T20:38:06.238 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-10T20:38:06.238 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":17,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:02.346556+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:06.240 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0bb0077f70 msgr2=0x7f0bb007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0bb0077f70 0x7f0bb007a430 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8004a30 tx=0x7f0bc8009040 comp rx=0 tx=0).stop 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 msgr2=0x7f0bcc15af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc15af30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000c00 tx=0x7f0bc0032c80 comp rx=0 tx=0).stop 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 shutdown_connections 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0bb0077f70 0x7f0bb007a430 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bcc161700 0x7f0bcc0cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bcc0d4c40 0x7f0bcc15af30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 --2- 172.21.15.28:0/821998800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bcc0c8da0 0x7f0bcc15de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.236+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 >> 172.21.15.28:0/821998800 conn(0x7f0bcc0d4090 msgr2=0x7f0bcc0c7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.237+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 shutdown_connections 2024-09-10T20:38:06.241 INFO:teuthology.misc.health.smithi028.stderr:2024-09-10T20:38:06.237+0000 7f0bd353d640 1 -- 172.21.15.28:0/821998800 wait complete. 2024-09-10T20:38:06.249 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-10T20:38:06.250 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-10T20:38:06.250 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:38:06.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:38:06.253 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:38:06.253 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:38:06.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:38:06.255 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:38:06.255 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:38:06.255 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:38:06.258 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:38:06.258 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:38:06.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:38:06.260 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:38:06.260 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-10T20:38:06.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:06.330 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:38:06.331 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:38:06.343 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:38:06.345 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:38:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 -- 172.21.15.28:0/134817842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080d29f0 msgr2=0x7f1c080cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 --2- 172.21.15.28:0/134817842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080d29f0 0x7f1c080cec50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009830 tx=0x7f1bfc02a510 comp rx=0 tx=0).stop 2024-09-10T20:38:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 -- 172.21.15.28:0/134817842 shutdown_connections 2024-09-10T20:38:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 --2- 172.21.15.28:0/134817842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c080d29f0 0x7f1c080cec50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 --2- 172.21.15.28:0/134817842 >> v2:172.21.15.159:3300/0 conn(0x7f1c0815f660 0x7f1c080ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 -- 172.21.15.28:0/134817842 >> 172.21.15.28:0/134817842 conn(0x7f1c080d1ed0 msgr2=0x7f1c080d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 -- 172.21.15.28:0/134817842 shutdown_connections 2024-09-10T20:38:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.366+0000 7f1c0e324640 1 -- 172.21.15.28:0/134817842 wait complete. 2024-09-10T20:38:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.367+0000 7f1c0e324640 1 Processor -- start 2024-09-10T20:38:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.368+0000 7f1c0e324640 1 -- start start 2024-09-10T20:38:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.368+0000 7f1c0e324640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c080b55c0 0x7f1c08157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.368+0000 7f1c0e324640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c080d29f0 0x7f1c08155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.368+0000 7f1c0e324640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c0815f660 0x7f1c08154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.368+0000 7f1c0e324640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c080cdc70 con 0x7f1c080d29f0 2024-09-10T20:38:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.368+0000 7f1c0e324640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1c080cdaf0 con 0x7f1c080b55c0 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.368+0000 7f1c0e324640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c080cddf0 con 0x7f1c0815f660 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c080d29f0 0x7f1c08155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c080d29f0 0x7f1c08155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47976/0 (socket says 172.21.15.28:47976) 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 -- 172.21.15.28:0/2547868577 learned_addr learned my addr 172.21.15.28:0/2547868577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c07fff640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c080b55c0 0x7f1c08157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c0c8c1640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c0815f660 0x7f1c08154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 -- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c0815f660 msgr2=0x7f1c08154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c0815f660 0x7f1c08154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 -- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c080b55c0 msgr2=0x7f1c08157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c080b55c0 0x7f1c08157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 -- 172.21.15.28:0/2547868577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c081f60d0 con 0x7f1c080d29f0 2024-09-10T20:38:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c07fff640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c080b55c0 0x7f1c08157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c077fe640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c080d29f0 0x7f1c08155f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c04ff9640 1 -- 172.21.15.28:0/2547868577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc040070 con 0x7f1c080d29f0 2024-09-10T20:38:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c04ff9640 1 -- 172.21.15.28:0/2547868577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bfc0038e0 con 0x7f1c080d29f0 2024-09-10T20:38:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c081f63b0 con 0x7f1c080d29f0 2024-09-10T20:38:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c04ff9640 1 -- 172.21.15.28:0/2547868577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc005c90 con 0x7f1c080d29f0 2024-09-10T20:38:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.369+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c081f6970 con 0x7f1c080d29f0 2024-09-10T20:38:06.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.370+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c080b63e0 con 0x7f1c080d29f0 2024-09-10T20:38:06.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.371+0000 7f1c04ff9640 1 -- 172.21.15.28:0/2547868577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1bfc04d030 con 0x7f1c080d29f0 2024-09-10T20:38:06.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.371+0000 7f1c04ff9640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf4077f70 0x7f1bf407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:06.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.371+0000 7f1c04ff9640 1 -- 172.21.15.28:0/2547868577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f1bfc0bac30 con 0x7f1c080d29f0 2024-09-10T20:38:06.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.372+0000 7f1c07fff640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf4077f70 0x7f1bf407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:06.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.372+0000 7f1c07fff640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf4077f70 0x7f1bf407a430 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1c080724e0 tx=0x7f1bf8005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:06.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.376+0000 7f1c04ff9640 1 -- 172.21.15.28:0/2547868577 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1bfc0834c0 con 0x7f1c080d29f0 2024-09-10T20:38:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.598+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c0806cf40 con 0x7f1c080d29f0 2024-09-10T20:38:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.598+0000 7f1c04ff9640 1 -- 172.21.15.28:0/2547868577 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7f1bfc087ef0 con 0x7f1c080d29f0 2024-09-10T20:38:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"btime":"2024-09-10T20:38:06:332153+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-10T20:38:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf4077f70 msgr2=0x7f1bf407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf4077f70 0x7f1bf407a430 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1c080724e0 tx=0x7f1bf8005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c080d29f0 msgr2=0x7f1c08155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c080d29f0 0x7f1c08155f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc032c10 comp rx=0 tx=0).stop 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 shutdown_connections 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf4077f70 0x7f1bf407a430 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c0815f660 0x7f1c08154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c080d29f0 0x7f1c08155f40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 --2- 172.21.15.28:0/2547868577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c080b55c0 0x7f1c08157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 >> 172.21.15.28:0/2547868577 conn(0x7f1c080d1ed0 msgr2=0x7f1c080d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 shutdown_connections 2024-09-10T20:38:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:06.601+0000 7f1c0e324640 1 -- 172.21.15.28:0/2547868577 wait complete. 2024-09-10T20:38:08.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:08.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 -- 172.21.15.28:0/1336580389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79840d0c80 msgr2=0x7f798415bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:08.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 --2- 172.21.15.28:0/1336580389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79840d0c80 0x7f798415bc40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f79840b5e20 tx=0x7f797402a4d0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 -- 172.21.15.28:0/1336580389 shutdown_connections 2024-09-10T20:38:08.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 --2- 172.21.15.28:0/1336580389 >> v2:172.21.15.159:3300/0 conn(0x7f7984070280 0x7f79840706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 --2- 172.21.15.28:0/1336580389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79840d0c80 0x7f798415bc40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 -- 172.21.15.28:0/1336580389 >> 172.21.15.28:0/1336580389 conn(0x7f798406dbf0 msgr2=0x7f798406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:08.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 -- 172.21.15.28:0/1336580389 shutdown_connections 2024-09-10T20:38:08.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.729+0000 7f7988a6a640 1 -- 172.21.15.28:0/1336580389 wait complete. 2024-09-10T20:38:08.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.730+0000 7f7988a6a640 1 Processor -- start 2024-09-10T20:38:08.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.730+0000 7f7988a6a640 1 -- start start 2024-09-10T20:38:08.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f7988a6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7984070280 0x7f798409e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:08.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f7988a6a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79840d0c80 0x7f798414ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:08.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f7988a6a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79840d1060 0x7f79840cf810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:08.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f7988a6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f798415edf0 con 0x7f7984070280 2024-09-10T20:38:08.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f7988a6a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f79840b54a0 con 0x7f79840d1060 2024-09-10T20:38:08.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f7988a6a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f798415ef70 con 0x7f79840d0c80 2024-09-10T20:38:08.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f798259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7984070280 0x7f798409e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:08.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f798259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7984070280 0x7f798409e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47984/0 (socket says 172.21.15.28:47984) 2024-09-10T20:38:08.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f798259c640 1 -- 172.21.15.28:0/4158120864 learned_addr learned my addr 172.21.15.28:0/4158120864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f797bfff640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79840d0c80 0x7f798414ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.731+0000 7f7982d9d640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79840d1060 0x7f79840cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f798259c640 1 -- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79840d0c80 msgr2=0x7f798414ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f798259c640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79840d0c80 0x7f798414ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f798259c640 1 -- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79840d1060 msgr2=0x7f79840cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f798259c640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79840d1060 0x7f79840cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f798259c640 1 -- 172.21.15.28:0/4158120864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79841509d0 con 0x7f7984070280 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f798259c640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7984070280 0x7f798409e4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7974000c00 tx=0x7f7974003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:08.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f797affd640 1 -- 172.21.15.28:0/4158120864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797403f070 con 0x7f7984070280 2024-09-10T20:38:08.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f797affd640 1 -- 172.21.15.28:0/4158120864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79740047a0 con 0x7f7984070280 2024-09-10T20:38:08.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79840cd0d0 con 0x7f7984070280 2024-09-10T20:38:08.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f797affd640 1 -- 172.21.15.28:0/4158120864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974035030 con 0x7f7984070280 2024-09-10T20:38:08.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.732+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79840cd690 con 0x7f7984070280 2024-09-10T20:38:08.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.733+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7984070eb0 con 0x7f7984070280 2024-09-10T20:38:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.734+0000 7f797affd640 1 -- 172.21.15.28:0/4158120864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7974004bb0 con 0x7f7984070280 2024-09-10T20:38:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.735+0000 7f797affd640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7964077f70 0x7f796407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.735+0000 7f797affd640 1 -- 172.21.15.28:0/4158120864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f79740babe0 con 0x7f7984070280 2024-09-10T20:38:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.735+0000 7f797bfff640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7964077f70 0x7f796407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.735+0000 7f797bfff640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7964077f70 0x7f796407a430 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79840d0490 tx=0x7f796c009060 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.739+0000 7f797affd640 1 -- 172.21.15.28:0/4158120864 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7974083360 con 0x7f7984070280 2024-09-10T20:38:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.961+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79840d0a60 con 0x7f7984070280 2024-09-10T20:38:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.962+0000 7f797affd640 1 -- 172.21.15.28:0/4158120864 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3909 (secure 0 0 0) 0x7f797408c0d0 con 0x7f7984070280 2024-09-10T20:38:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:08.967 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"btime":"2024-09-10T20:38:07:333107+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-10T20:38:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7964077f70 msgr2=0x7f796407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7964077f70 0x7f796407a430 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79840d0490 tx=0x7f796c009060 comp rx=0 tx=0).stop 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7984070280 msgr2=0x7f798409e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7984070280 0x7f798409e4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7974000c00 tx=0x7f7974003910 comp rx=0 tx=0).stop 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 shutdown_connections 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7964077f70 0x7f796407a430 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79840d1060 0x7f79840cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79840d0c80 0x7f798414ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 --2- 172.21.15.28:0/4158120864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7984070280 0x7f798409e4c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 >> 172.21.15.28:0/4158120864 conn(0x7f798406dbf0 msgr2=0x7f798406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 shutdown_connections 2024-09-10T20:38:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:08.964+0000 7f7988a6a640 1 -- 172.21.15.28:0/4158120864 wait complete. 2024-09-10T20:38:08.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:38:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 -- 172.21.15.28:0/1390648416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f395015e0f0 msgr2=0x7f39500d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 --2- 172.21.15.28:0/1390648416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f395015e0f0 0x7f39500d2030 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f394c005fd0 tx=0x7f394c02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 -- 172.21.15.28:0/1390648416 shutdown_connections 2024-09-10T20:38:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 --2- 172.21.15.28:0/1390648416 >> v2:172.21.15.159:3300/0 conn(0x7f39500b77f0 0x7f39500b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 --2- 172.21.15.28:0/1390648416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f395015e0f0 0x7f39500d2030 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 -- 172.21.15.28:0/1390648416 >> 172.21.15.28:0/1390648416 conn(0x7f39500d4040 msgr2=0x7f39500d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 -- 172.21.15.28:0/1390648416 shutdown_connections 2024-09-10T20:38:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.079+0000 7f3957e47640 1 -- 172.21.15.28:0/1390648416 wait complete. 2024-09-10T20:38:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.080+0000 7f3957e47640 1 Processor -- start 2024-09-10T20:38:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.080+0000 7f3957e47640 1 -- start start 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f3957e47640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39500b77f0 0x7f39501f55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f3957e47640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39500d4c40 0x7f39500cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f3957e47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395015e0f0 0x7f3950147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f3957e47640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39500b8300 con 0x7f395015e0f0 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f3957e47640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f39500b8180 con 0x7f39500b77f0 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f3957e47640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f39500b8480 con 0x7f39500d4c40 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f39563e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395015e0f0 0x7f3950147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f39563e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395015e0f0 0x7f3950147270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47990/0 (socket says 172.21.15.28:47990) 2024-09-10T20:38:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f39563e4640 1 -- 172.21.15.28:0/3979755545 learned_addr learned my addr 172.21.15.28:0/3979755545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f3955be3640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39500b77f0 0x7f39501f55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f39553e2640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39500d4c40 0x7f39500cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f39563e4640 1 -- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39500d4c40 msgr2=0x7f39500cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.081+0000 7f39563e4640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39500d4c40 0x7f39500cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f39563e4640 1 -- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39500b77f0 msgr2=0x7f39501f55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f39563e4640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39500b77f0 0x7f39501f55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f39563e4640 1 -- 172.21.15.28:0/3979755545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39501477b0 con 0x7f395015e0f0 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f39563e4640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395015e0f0 0x7f3950147270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f394800efd0 tx=0x7f394800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f39467fc640 1 -- 172.21.15.28:0/3979755545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948003b70 con 0x7f395015e0f0 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f39467fc640 1 -- 172.21.15.28:0/3979755545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3948003e00 con 0x7f395015e0f0 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f39467fc640 1 -- 172.21.15.28:0/3979755545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948004280 con 0x7f395015e0f0 2024-09-10T20:38:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3950147a30 con 0x7f395015e0f0 2024-09-10T20:38:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.082+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3950147ff0 con 0x7f395015e0f0 2024-09-10T20:38:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.083+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395015c760 con 0x7f395015e0f0 2024-09-10T20:38:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.083+0000 7f39467fc640 1 -- 172.21.15.28:0/3979755545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3948004420 con 0x7f395015e0f0 2024-09-10T20:38:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.084+0000 7f39467fc640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3934077f70 0x7f393407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.084+0000 7f39467fc640 1 -- 172.21.15.28:0/3979755545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f394809a860 con 0x7f395015e0f0 2024-09-10T20:38:09.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.084+0000 7f3955be3640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3934077f70 0x7f393407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.085+0000 7f3955be3640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3934077f70 0x7f393407a430 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f394c033000 tx=0x7f394c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.088+0000 7f39467fc640 1 -- 172.21.15.28:0/3979755545 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f39501477b0 con 0x7f395015e0f0 2024-09-10T20:38:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.275+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f39501481c0 con 0x7f395015e0f0 2024-09-10T20:38:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.275+0000 7f39467fc640 1 -- 172.21.15.28:0/3979755545 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f39501481c0 con 0x7f395015e0f0 2024-09-10T20:38:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:09.281 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:38:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:38:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:38:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:09.282 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.277+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3934077f70 msgr2=0x7f393407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.277+0000 7f3957e47640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3934077f70 0x7f393407a430 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f394c033000 tx=0x7f394c004730 comp rx=0 tx=0).stop 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.277+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395015e0f0 msgr2=0x7f3950147270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.277+0000 7f3957e47640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395015e0f0 0x7f3950147270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f394800efd0 tx=0x7f394800c6a0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 shutdown_connections 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3934077f70 0x7f393407a430 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395015e0f0 0x7f3950147270 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39500d4c40 0x7f39500cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 --2- 172.21.15.28:0/3979755545 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39500b77f0 0x7f39501f55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 >> 172.21.15.28:0/3979755545 conn(0x7f39500d4040 msgr2=0x7f3950161390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 shutdown_connections 2024-09-10T20:38:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.278+0000 7f3957e47640 1 -- 172.21.15.28:0/3979755545 wait complete. 2024-09-10T20:38:09.293 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:38:09.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:38:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.398+0000 7ff14a471640 1 -- 172.21.15.28:0/108673409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 msgr2=0x7ff12c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.398+0000 7ff14a471640 1 --2- 172.21.15.28:0/108673409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 0x7ff12c004800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff138009b20 tx=0x7ff12c005830 comp rx=0 tx=0).stop 2024-09-10T20:38:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.398+0000 7ff148a0e640 1 -- 172.21.15.28:0/108673409 reap_dead start 2024-09-10T20:38:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.398+0000 7ff14a471640 1 -- 172.21.15.28:0/108673409 shutdown_connections 2024-09-10T20:38:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.398+0000 7ff14a471640 1 -- 172.21.15.28:0/108673409 >> 172.21.15.28:0/108673409 conn(0x7ff1440d1eb0 msgr2=0x7ff1440d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.398+0000 7ff14a471640 1 -- 172.21.15.28:0/108673409 shutdown_connections 2024-09-10T20:38:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.398+0000 7ff14a471640 1 -- 172.21.15.28:0/108673409 wait complete. 2024-09-10T20:38:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.399+0000 7ff14a471640 1 Processor -- start 2024-09-10T20:38:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.399+0000 7ff14a471640 1 -- start start 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff14a471640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 0x7ff144152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff14a471640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff14414eba0 0x7ff14414f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff14a471640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff14409c770 0x7ff14409cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff14a471640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff144160120 con 0x7ff12c003fd0 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff14a471640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff14415ffa0 con 0x7ff14409c770 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff14a471640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff1440b4fa0 con 0x7ff14414eba0 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 0x7ff144152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 0x7ff144152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48000/0 (socket says 172.21.15.28:48000) 2024-09-10T20:38:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 -- 172.21.15.28:0/3051638108 learned_addr learned my addr 172.21.15.28:0/3051638108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff1437fe640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff14414eba0 0x7ff14414f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff148a0e640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff14409c770 0x7ff14409cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 -- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff14414eba0 msgr2=0x7ff14414f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff14414eba0 0x7ff14414f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 -- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff14409c770 msgr2=0x7ff14409cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff14409c770 0x7ff14409cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 -- 172.21.15.28:0/3051638108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14414cbb0 con 0x7ff12c003fd0 2024-09-10T20:38:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.400+0000 7ff143fff640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 0x7ff144152a80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff134006660 tx=0x7ff134003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.401+0000 7ff140ff9640 1 -- 172.21.15.28:0/3051638108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134015070 con 0x7ff12c003fd0 2024-09-10T20:38:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.401+0000 7ff140ff9640 1 -- 172.21.15.28:0/3051638108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff13400f960 con 0x7ff12c003fd0 2024-09-10T20:38:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.401+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14414ce20 con 0x7ff12c003fd0 2024-09-10T20:38:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.401+0000 7ff140ff9640 1 -- 172.21.15.28:0/3051638108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13400fd70 con 0x7ff12c003fd0 2024-09-10T20:38:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.401+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff144152fc0 con 0x7ff12c003fd0 2024-09-10T20:38:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.402+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1440b6cf0 con 0x7ff12c003fd0 2024-09-10T20:38:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.402+0000 7ff140ff9640 1 -- 172.21.15.28:0/3051638108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff134011430 con 0x7ff12c003fd0 2024-09-10T20:38:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.403+0000 7ff140ff9640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff130077f70 0x7ff13007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.403+0000 7ff140ff9640 1 -- 172.21.15.28:0/3051638108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff13409a6c0 con 0x7ff12c003fd0 2024-09-10T20:38:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.403+0000 7ff1437fe640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff130077f70 0x7ff13007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.404+0000 7ff1437fe640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff130077f70 0x7ff13007a430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff138035000 tx=0x7ff1380046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.407+0000 7ff140ff9640 1 -- 172.21.15.28:0/3051638108 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff1440b6cf0 con 0x7ff12c003fd0 2024-09-10T20:38:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:09.592+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff144160760 con 0x7ff12c003fd0 2024-09-10T20:38:10.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.371+0000 7ff140ff9640 1 -- 172.21.15.28:0/3051638108 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7ff134062ed0 con 0x7ff12c003fd0 2024-09-10T20:38:10.376 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:38:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff130077f70 msgr2=0x7ff13007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff130077f70 0x7ff13007a430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff138035000 tx=0x7ff1380046c0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 msgr2=0x7ff144152a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 0x7ff144152a80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff134006660 tx=0x7ff134003a80 comp rx=0 tx=0).stop 2024-09-10T20:38:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 shutdown_connections 2024-09-10T20:38:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff130077f70 0x7ff13007a430 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff14409c770 0x7ff14409cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff14414eba0 0x7ff14414f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 --2- 172.21.15.28:0/3051638108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c003fd0 0x7ff144152a80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.374+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 >> 172.21.15.28:0/3051638108 conn(0x7ff1440d1eb0 msgr2=0x7ff1440b6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.375+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 shutdown_connections 2024-09-10T20:38:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.375+0000 7ff14a471640 1 -- 172.21.15.28:0/3051638108 wait complete. 2024-09-10T20:38:10.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:38:10.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 -- 172.21.15.28:0/1379399683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f3d0 msgr2=0x7f01240b14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 --2- 172.21.15.28:0/1379399683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f3d0 0x7f01240b14e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0120009900 tx=0x7f012002a610 comp rx=0 tx=0).stop 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 -- 172.21.15.28:0/1379399683 shutdown_connections 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 --2- 172.21.15.28:0/1379399683 >> v2:172.21.15.159:3300/0 conn(0x7f01240b2dd0 0x7f01240af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 --2- 172.21.15.28:0/1379399683 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f3d0 0x7f01240b14e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 -- 172.21.15.28:0/1379399683 >> 172.21.15.28:0/1379399683 conn(0x7f01240d1e10 msgr2=0x7f01240d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 -- 172.21.15.28:0/1379399683 shutdown_connections 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.511+0000 7f012c647640 1 -- 172.21.15.28:0/1379399683 wait complete. 2024-09-10T20:38:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 Processor -- start 2024-09-10T20:38:10.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 -- start start 2024-09-10T20:38:10.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01240b2dd0 0x7f01240cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:10.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f012415f3d0 0x7f01241f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f7b0 0x7f01241f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01240afd90 con 0x7f01240b2dd0 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f01240afc10 con 0x7f012415f3d0 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.513+0000 7f012c647640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f01240aff10 con 0x7f012415f7b0 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01240b2dd0 0x7f01240cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01240b2dd0 0x7f01240cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34974/0 (socket says 172.21.15.28:34974) 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 -- 172.21.15.28:0/757822232 learned_addr learned my addr 172.21.15.28:0/757822232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012abe4640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f7b0 0x7f01241f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f0129be2640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f012415f3d0 0x7f01241f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 -- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f7b0 msgr2=0x7f01241f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f7b0 0x7f01241f68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 -- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f012415f3d0 msgr2=0x7f01241f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f012415f3d0 0x7f01241f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 -- 172.21.15.28:0/757822232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01240cf770 con 0x7f01240b2dd0 2024-09-10T20:38:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012a3e3640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01240b2dd0 0x7f01240cf230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0120006010 tx=0x7f01200035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f011affd640 1 -- 172.21.15.28:0/757822232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012003f070 con 0x7f01240b2dd0 2024-09-10T20:38:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f011affd640 1 -- 172.21.15.28:0/757822232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0120003ce0 con 0x7f01240b2dd0 2024-09-10T20:38:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01241f71f0 con 0x7f01240b2dd0 2024-09-10T20:38:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f011affd640 1 -- 172.21.15.28:0/757822232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120035030 con 0x7f01240b2dd0 2024-09-10T20:38:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.514+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0124154520 con 0x7f01240b2dd0 2024-09-10T20:38:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.516+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01240b04a0 con 0x7f01240b2dd0 2024-09-10T20:38:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.516+0000 7f011affd640 1 -- 172.21.15.28:0/757822232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0120004a00 con 0x7f01240b2dd0 2024-09-10T20:38:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.517+0000 7f011affd640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0108077f70 0x7f010807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.517+0000 7f011affd640 1 -- 172.21.15.28:0/757822232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f01200badd0 con 0x7f01240b2dd0 2024-09-10T20:38:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.517+0000 7f0129be2640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0108077f70 0x7f010807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.517+0000 7f0129be2640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0108077f70 0x7f010807a430 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01240c88e0 tx=0x7f0114009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:10.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.521+0000 7f011affd640 1 -- 172.21.15.28:0/757822232 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0120083460 con 0x7f01240b2dd0 2024-09-10T20:38:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:10.706+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0124072520 con 0x7f01240b2dd0 2024-09-10T20:38:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.373+0000 7f011affd640 1 -- 172.21.15.28:0/757822232 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f0120087e90 con 0x7f01240b2dd0 2024-09-10T20:38:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:38:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0108077f70 msgr2=0x7f010807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0108077f70 0x7f010807a430 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01240c88e0 tx=0x7f0114009040 comp rx=0 tx=0).stop 2024-09-10T20:38:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01240b2dd0 msgr2=0x7f01240cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01240b2dd0 0x7f01240cf230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0120006010 tx=0x7f01200035f0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 shutdown_connections 2024-09-10T20:38:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0108077f70 0x7f010807a430 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f012415f7b0 0x7f01241f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f012415f3d0 0x7f01241f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 --2- 172.21.15.28:0/757822232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01240b2dd0 0x7f01240cf230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 >> 172.21.15.28:0/757822232 conn(0x7f01240d1e10 msgr2=0x7f01240b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 shutdown_connections 2024-09-10T20:38:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.376+0000 7f012c647640 1 -- 172.21.15.28:0/757822232 wait complete. 2024-09-10T20:38:11.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:38:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 -- 172.21.15.28:0/1760952855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d2ba0 msgr2=0x7f96180b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 --2- 172.21.15.28:0/1760952855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d2ba0 0x7f96180b1870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9610005fd0 tx=0x7f961002a670 comp rx=0 tx=0).stop 2024-09-10T20:38:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 -- 172.21.15.28:0/1760952855 shutdown_connections 2024-09-10T20:38:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 --2- 172.21.15.28:0/1760952855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d2ba0 0x7f96180b1870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 --2- 172.21.15.28:0/1760952855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96180d1f00 0x7f96180d2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 -- 172.21.15.28:0/1760952855 >> 172.21.15.28:0/1760952855 conn(0x7f96180d4130 msgr2=0x7f96180d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 -- 172.21.15.28:0/1760952855 shutdown_connections 2024-09-10T20:38:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.518+0000 7f961db05640 1 -- 172.21.15.28:0/1760952855 wait complete. 2024-09-10T20:38:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.519+0000 7f961db05640 1 Processor -- start 2024-09-10T20:38:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.519+0000 7f961db05640 1 -- start start 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f961db05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d1f00 0x7f96180b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f961db05640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96180d2ba0 0x7f961814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f961db05640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f961815e220 0x7f96180c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f961db05640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96180b2140 con 0x7f96180d1f00 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f961db05640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f96180b1fc0 con 0x7f961815e220 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f961db05640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f96180b22c0 con 0x7f96180d2ba0 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f96177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d1f00 0x7f96180b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f96177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d1f00 0x7f96180b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34990/0 (socket says 172.21.15.28:34990) 2024-09-10T20:38:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f96177fe640 1 -- 172.21.15.28:0/1849687762 learned_addr learned my addr 172.21.15.28:0/1849687762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f9616ffd640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96180d2ba0 0x7f961814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.520+0000 7f9617fff640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f961815e220 0x7f96180c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f96177fe640 1 -- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96180d2ba0 msgr2=0x7f961814d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f96177fe640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96180d2ba0 0x7f961814d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f96177fe640 1 -- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f961815e220 msgr2=0x7f96180c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f96177fe640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f961815e220 0x7f96180c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f96177fe640 1 -- 172.21.15.28:0/1849687762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96180c9c70 con 0x7f96180d1f00 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f96177fe640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d1f00 0x7f96180b4820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f960800c760 tx=0x7f960800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f95fffff640 1 -- 172.21.15.28:0/1849687762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608003b70 con 0x7f96180d1f00 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f95fffff640 1 -- 172.21.15.28:0/1849687762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9608003d30 con 0x7f96180d1f00 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96181f8cd0 con 0x7f96180d1f00 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f95fffff640 1 -- 172.21.15.28:0/1849687762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960800aab0 con 0x7f96180d1f00 2024-09-10T20:38:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.521+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96181f9290 con 0x7f96180d1f00 2024-09-10T20:38:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.522+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96180d2660 con 0x7f96180d1f00 2024-09-10T20:38:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.523+0000 7f95fffff640 1 -- 172.21.15.28:0/1849687762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f960800ac50 con 0x7f96180d1f00 2024-09-10T20:38:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.523+0000 7f95fffff640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9604077fa0 0x7f960407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.524+0000 7f95fffff640 1 -- 172.21.15.28:0/1849687762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f960809a330 con 0x7f96180d1f00 2024-09-10T20:38:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.524+0000 7f9616ffd640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9604077fa0 0x7f960407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.524+0000 7f9616ffd640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9604077fa0 0x7f960407a460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96180ceaa0 tx=0x7f960c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.527+0000 7f95fffff640 1 -- 172.21.15.28:0/1849687762 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f96080629b0 con 0x7f96180d1f00 2024-09-10T20:38:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:11.749+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f96180c9e70 con 0x7f96180d1f00 2024-09-10T20:38:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.394+0000 7f95fffff640 1 -- 172.21.15.28:0/1849687762 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f96180c9e70 con 0x7f96180d1f00 2024-09-10T20:38:12.404 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-09-10T20:38:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.406+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9604077fa0 msgr2=0x7f960407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.407+0000 7f961db05640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9604077fa0 0x7f960407a460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96180ceaa0 tx=0x7f960c005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.407+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d1f00 msgr2=0x7f96180b4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.407+0000 7f961db05640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d1f00 0x7f96180b4820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f960800c760 tx=0x7f960800cc20 comp rx=0 tx=0).stop 2024-09-10T20:38:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.408+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 shutdown_connections 2024-09-10T20:38:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.408+0000 7f961db05640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9604077fa0 0x7f960407a460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.409+0000 7f961db05640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f961815e220 0x7f96180c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.409+0000 7f961db05640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96180d2ba0 0x7f961814d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.409+0000 7f961db05640 1 --2- 172.21.15.28:0/1849687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96180d1f00 0x7f96180b4820 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.409+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 >> 172.21.15.28:0/1849687762 conn(0x7f96180d4130 msgr2=0x7f96180b98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.412+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 shutdown_connections 2024-09-10T20:38:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.414+0000 7f961db05640 1 -- 172.21.15.28:0/1849687762 wait complete. 2024-09-10T20:38:12.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 -- 172.21.15.28:0/3169639626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 msgr2=0x7f761406eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 --2- 172.21.15.28:0/3169639626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 0x7f761406eb10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f760c005fd0 tx=0x7f760c02aa90 comp rx=0 tx=0).stop 2024-09-10T20:38:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 -- 172.21.15.28:0/3169639626 shutdown_connections 2024-09-10T20:38:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 --2- 172.21.15.28:0/3169639626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 0x7f761406eb10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 --2- 172.21.15.28:0/3169639626 >> v2:172.21.15.159:3300/0 conn(0x7f76141564e0 0x7f761415c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 -- 172.21.15.28:0/3169639626 >> 172.21.15.28:0/3169639626 conn(0x7f76140b1c40 msgr2=0x7f76140b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 -- 172.21.15.28:0/3169639626 shutdown_connections 2024-09-10T20:38:12.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.542+0000 7f76196ef640 1 -- 172.21.15.28:0/3169639626 wait complete. 2024-09-10T20:38:12.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.543+0000 7f76196ef640 1 Processor -- start 2024-09-10T20:38:12.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.543+0000 7f76196ef640 1 -- start start 2024-09-10T20:38:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f76196ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614072b50 0x7f761415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f76196ef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 0x7f7614158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f76196ef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76141564e0 0x7f76140b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f76196ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76140af130 con 0x7f7614072b50 2024-09-10T20:38:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f76196ef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f76140aefb0 con 0x7f76141564e0 2024-09-10T20:38:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f76196ef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f76140af2b0 con 0x7f76140acf20 2024-09-10T20:38:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614072b50 0x7f761415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f760bfff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 0x7f7614158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614072b50 0x7f761415bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35004/0 (socket says 172.21.15.28:35004) 2024-09-10T20:38:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 -- 172.21.15.28:0/3867686860 learned_addr learned my addr 172.21.15.28:0/3867686860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f76137fe640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76141564e0 0x7f76140b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 -- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 msgr2=0x7f7614158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 0x7f7614158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 -- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76141564e0 msgr2=0x7f76140b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76141564e0 0x7f76140b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f7612ffd640 1 -- 172.21.15.28:0/3867686860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76141f5a50 con 0x7f7614072b50 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.544+0000 7f760bfff640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 0x7f7614158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.545+0000 7f7612ffd640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614072b50 0x7f761415bb70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f760400c450 tx=0x7f760400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.545+0000 7f760b7fe640 1 -- 172.21.15.28:0/3867686860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76040052d0 con 0x7f7614072b50 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.545+0000 7f760b7fe640 1 -- 172.21.15.28:0/3867686860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7604005470 con 0x7f7614072b50 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.545+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761415b0d0 con 0x7f7614072b50 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.545+0000 7f760b7fe640 1 -- 172.21.15.28:0/3867686860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7604012870 con 0x7f7614072b50 2024-09-10T20:38:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.545+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f761415b370 con 0x7f7614072b50 2024-09-10T20:38:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.546+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76140af760 con 0x7f7614072b50 2024-09-10T20:38:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.546+0000 7f760b7fe640 1 -- 172.21.15.28:0/3867686860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f76040035f0 con 0x7f7614072b50 2024-09-10T20:38:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.547+0000 7f760b7fe640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f75f4077f70 0x7f75f407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.547+0000 7f760b7fe640 1 -- 172.21.15.28:0/3867686860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f76040997c0 con 0x7f7614072b50 2024-09-10T20:38:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.547+0000 7f760bfff640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f75f4077f70 0x7f75f407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.548+0000 7f760bfff640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f75f4077f70 0x7f75f407a430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f76141f9150 tx=0x7f75fc005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:12.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.551+0000 7f760b7fe640 1 -- 172.21.15.28:0/3867686860 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7604061c60 con 0x7f7614072b50 2024-09-10T20:38:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.738+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7614070960 con 0x7f7614072b50 2024-09-10T20:38:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.739+0000 7f760b7fe640 1 -- 172.21.15.28:0/3867686860 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16682 (secure 0 0 0) 0x7f7604066690 con 0x7f7614072b50 2024-09-10T20:38:12.744 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:12.744 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:11.751785+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f75f4077f70 msgr2=0x7f75f407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f75f4077f70 0x7f75f407a430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f76141f9150 tx=0x7f75fc005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614072b50 msgr2=0x7f761415bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614072b50 0x7f761415bb70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f760400c450 tx=0x7f760400c910 comp rx=0 tx=0).stop 2024-09-10T20:38:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 shutdown_connections 2024-09-10T20:38:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f75f4077f70 0x7f75f407a430 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76141564e0 0x7f76140b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76140acf20 0x7f7614158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 --2- 172.21.15.28:0/3867686860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614072b50 0x7f761415bb70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 >> 172.21.15.28:0/3867686860 conn(0x7f76140b1c40 msgr2=0x7f7614151ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 shutdown_connections 2024-09-10T20:38:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.741+0000 7f76196ef640 1 -- 172.21.15.28:0/3867686860 wait complete. 2024-09-10T20:38:12.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.858+0000 7f01df5c6640 1 -- 172.21.15.28:0/2653259430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01c00039e0 msgr2=0x7f01c0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.858+0000 7f01df5c6640 1 --2- 172.21.15.28:0/2653259430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01c00039e0 0x7f01c0003dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f01c8001e90 tx=0x7f01c0005830 comp rx=0 tx=0).stop 2024-09-10T20:38:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.858+0000 7f01ddb63640 1 -- 172.21.15.28:0/2653259430 reap_dead start 2024-09-10T20:38:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.858+0000 7f01df5c6640 1 -- 172.21.15.28:0/2653259430 shutdown_connections 2024-09-10T20:38:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.858+0000 7f01df5c6640 1 -- 172.21.15.28:0/2653259430 >> 172.21.15.28:0/2653259430 conn(0x7f01d806cf00 msgr2=0x7f01d806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.858+0000 7f01df5c6640 1 -- 172.21.15.28:0/2653259430 shutdown_connections 2024-09-10T20:38:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.858+0000 7f01df5c6640 1 -- 172.21.15.28:0/2653259430 wait complete. 2024-09-10T20:38:12.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.859+0000 7f01df5c6640 1 Processor -- start 2024-09-10T20:38:12.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.859+0000 7f01df5c6640 1 -- start start 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01df5c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01c00039e0 0x7f01d81f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01df5c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01d815a630 0x7f01d80af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01df5c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d815aab0 0x7f01d81510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01df5c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01d8072bd0 con 0x7f01d815aab0 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01df5c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f01d80d08e0 con 0x7f01c00039e0 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01df5c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f01d8072d50 con 0x7f01d815a630 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d815aab0 0x7f01d81510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d815aab0 0x7f01d81510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35022/0 (socket says 172.21.15.28:35022) 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 -- 172.21.15.28:0/392824053 learned_addr learned my addr 172.21.15.28:0/392824053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01dd362640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01c00039e0 0x7f01d81f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01dcb61640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01d815a630 0x7f01d80af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 -- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01d815a630 msgr2=0x7f01d80af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01d815a630 0x7f01d80af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 -- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01c00039e0 msgr2=0x7f01d81f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01c00039e0 0x7f01d81f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 -- 172.21.15.28:0/392824053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d809d3e0 con 0x7f01d815aab0 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01ddb63640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d815aab0 0x7f01d81510c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f01d80ced30 tx=0x7f01d0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.860+0000 7f01d5ffb640 1 -- 172.21.15.28:0/392824053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0004440 con 0x7f01d815aab0 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.861+0000 7f01d5ffb640 1 -- 172.21.15.28:0/392824053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01d000f8f0 con 0x7f01d815aab0 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.861+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d809d6c0 con 0x7f01d815aab0 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.861+0000 7f01d5ffb640 1 -- 172.21.15.28:0/392824053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d000fd40 con 0x7f01d815aab0 2024-09-10T20:38:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.861+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d81f0660 con 0x7f01d815aab0 2024-09-10T20:38:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.862+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d815bd50 con 0x7f01d815aab0 2024-09-10T20:38:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.862+0000 7f01d5ffb640 1 -- 172.21.15.28:0/392824053 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f01d0022030 con 0x7f01d815aab0 2024-09-10T20:38:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.863+0000 7f01d5ffb640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01c4077f70 0x7f01c407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.863+0000 7f01d5ffb640 1 -- 172.21.15.28:0/392824053 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f01d009a620 con 0x7f01d815aab0 2024-09-10T20:38:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.863+0000 7f01dd362640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01c4077f70 0x7f01c407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.864+0000 7f01dd362640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01c4077f70 0x7f01c407a430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01c8004fd0 tx=0x7f01c802d000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:12.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:12.867+0000 7f01d5ffb640 1 -- 172.21.15.28:0/392824053 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f01d0005a70 con 0x7f01d815aab0 2024-09-10T20:38:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.053+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01d81f0830 con 0x7f01d815aab0 2024-09-10T20:38:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.054+0000 7f01d5ffb640 1 -- 172.21.15.28:0/392824053 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16682 (secure 0 0 0) 0x7f01d0062a30 con 0x7f01d815aab0 2024-09-10T20:38:13.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:13.059 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:11.751785+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.056+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01c4077f70 msgr2=0x7f01c407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.056+0000 7f01df5c6640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01c4077f70 0x7f01c407a430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01c8004fd0 tx=0x7f01c802d000 comp rx=0 tx=0).stop 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.056+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d815aab0 msgr2=0x7f01d81510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.056+0000 7f01df5c6640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d815aab0 0x7f01d81510c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f01d80ced30 tx=0x7f01d0011b50 comp rx=0 tx=0).stop 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 shutdown_connections 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01c4077f70 0x7f01c407a430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d815aab0 0x7f01d81510c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01d815a630 0x7f01d80af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 --2- 172.21.15.28:0/392824053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01c00039e0 0x7f01d81f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 >> 172.21.15.28:0/392824053 conn(0x7f01d806cf00 msgr2=0x7f01d80cc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 shutdown_connections 2024-09-10T20:38:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.057+0000 7f01df5c6640 1 -- 172.21.15.28:0/392824053 wait complete. 2024-09-10T20:38:13.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.175+0000 7f5fa782a640 1 -- 172.21.15.28:0/1983637960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa015f580 msgr2=0x7f5fa00ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.175+0000 7f5fa782a640 1 --2- 172.21.15.28:0/1983637960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa015f580 0x7f5fa00ce750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009830 tx=0x7f5f9c02a510 comp rx=0 tx=0).stop 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.176+0000 7f5fa782a640 1 -- 172.21.15.28:0/1983637960 shutdown_connections 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.176+0000 7f5fa782a640 1 --2- 172.21.15.28:0/1983637960 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fa00cdb30 0x7f5fa00cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.176+0000 7f5fa782a640 1 --2- 172.21.15.28:0/1983637960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa015f580 0x7f5fa00ce750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.176+0000 7f5fa782a640 1 -- 172.21.15.28:0/1983637960 >> 172.21.15.28:0/1983637960 conn(0x7f5fa00d1f30 msgr2=0x7f5fa00d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.176+0000 7f5fa782a640 1 -- 172.21.15.28:0/1983637960 shutdown_connections 2024-09-10T20:38:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.176+0000 7f5fa782a640 1 -- 172.21.15.28:0/1983637960 wait complete. 2024-09-10T20:38:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.177+0000 7f5fa782a640 1 Processor -- start 2024-09-10T20:38:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.177+0000 7f5fa782a640 1 -- start start 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa782a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fa00b3110 0x7f5fa01f84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa782a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa00cdb30 0x7f5fa006f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa782a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fa015f580 0x7f5fa006e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa782a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa00d1030 con 0x7f5fa00cdb30 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa782a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa00d0eb0 con 0x7f5fa00b3110 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa782a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa00b5470 con 0x7f5fa015f580 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa00cdb30 0x7f5fa006f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa00cdb30 0x7f5fa006f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35050/0 (socket says 172.21.15.28:35050) 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 -- 172.21.15.28:0/2112270228 learned_addr learned my addr 172.21.15.28:0/2112270228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa55c6640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fa00b3110 0x7f5fa01f84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa5dc7640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fa015f580 0x7f5fa006e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 -- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fa015f580 msgr2=0x7f5fa006e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fa015f580 0x7f5fa006e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 -- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fa00b3110 msgr2=0x7f5fa01f84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fa00b3110 0x7f5fa01f84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 -- 172.21.15.28:0/2112270228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa006fc30 con 0x7f5fa00cdb30 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5fa4dc5640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa00cdb30 0x7f5fa006f6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c430 tx=0x7f5f9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.178+0000 7f5f95ffb640 1 -- 172.21.15.28:0/2112270228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90006890 con 0x7f5fa00cdb30 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.179+0000 7f5f95ffb640 1 -- 172.21.15.28:0/2112270228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f90006b40 con 0x7f5fa00cdb30 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.179+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa01f9350 con 0x7f5fa00cdb30 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.179+0000 7f5f95ffb640 1 -- 172.21.15.28:0/2112270228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90007020 con 0x7f5fa00cdb30 2024-09-10T20:38:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.179+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa01f8ab0 con 0x7f5fa00cdb30 2024-09-10T20:38:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.180+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa00b5a00 con 0x7f5fa00cdb30 2024-09-10T20:38:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.180+0000 7f5f95ffb640 1 -- 172.21.15.28:0/2112270228 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5f900071c0 con 0x7f5fa00cdb30 2024-09-10T20:38:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.181+0000 7f5f95ffb640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5f84077fa0 0x7f5f8407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.181+0000 7f5f95ffb640 1 -- 172.21.15.28:0/2112270228 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5f90099750 con 0x7f5fa00cdb30 2024-09-10T20:38:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.181+0000 7f5fa55c6640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5f84077fa0 0x7f5f8407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.181+0000 7f5fa55c6640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5f84077fa0 0x7f5f8407a460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c034000 tx=0x7f5f9c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:13.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.185+0000 7f5f95ffb640 1 -- 172.21.15.28:0/2112270228 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5f90061b60 con 0x7f5fa00cdb30 2024-09-10T20:38:13.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:13.414+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5fa00ce290 con 0x7f5fa00cdb30 2024-09-10T20:38:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.408+0000 7f5f95ffb640 1 -- 172.21.15.28:0/2112270228 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f5f90066590 con 0x7f5fa00cdb30 2024-09-10T20:38:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5f84077fa0 msgr2=0x7f5f8407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5f84077fa0 0x7f5f8407a460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c034000 tx=0x7f5f9c004730 comp rx=0 tx=0).stop 2024-09-10T20:38:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa00cdb30 msgr2=0x7f5fa006f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa00cdb30 0x7f5fa006f6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c430 tx=0x7f5f9000c8f0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 shutdown_connections 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5f84077fa0 0x7f5f8407a460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fa015f580 0x7f5fa006e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa00cdb30 0x7f5fa006f6f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 --2- 172.21.15.28:0/2112270228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fa00b3110 0x7f5fa01f84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 >> 172.21.15.28:0/2112270228 conn(0x7f5fa00d1f30 msgr2=0x7f5fa00ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 shutdown_connections 2024-09-10T20:38:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.411+0000 7f5fa782a640 1 -- 172.21.15.28:0/2112270228 wait complete. 2024-09-10T20:38:14.429 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:38:14.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:38:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.541+0000 7f63877e2640 1 -- 172.21.15.28:0/37645618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6368003c30 msgr2=0x7f6368004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.541+0000 7f63877e2640 1 --2- 172.21.15.28:0/37645618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6368003c30 0x7f6368004030 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6378001830 tx=0x7f6368005830 comp rx=0 tx=0).stop 2024-09-10T20:38:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.541+0000 7f63877e2640 1 -- 172.21.15.28:0/37645618 shutdown_connections 2024-09-10T20:38:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.541+0000 7f63877e2640 1 --2- 172.21.15.28:0/37645618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6368003c30 0x7f6368004030 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.541+0000 7f63877e2640 1 -- 172.21.15.28:0/37645618 >> 172.21.15.28:0/37645618 conn(0x7f63800d0090 msgr2=0x7f63800d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.541+0000 7f63877e2640 1 -- 172.21.15.28:0/37645618 shutdown_connections 2024-09-10T20:38:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.541+0000 7f63877e2640 1 -- 172.21.15.28:0/37645618 wait complete. 2024-09-10T20:38:14.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.542+0000 7f63877e2640 1 Processor -- start 2024-09-10T20:38:14.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.542+0000 7f63877e2640 1 -- start start 2024-09-10T20:38:14.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f63877e2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6368003c30 0x7f63800746b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:14.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f63877e2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6380159fe0 0x7f63801f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:14.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f63877e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63800b0780 0x7f63800b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:14.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f63877e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63800ce1d0 con 0x7f63800b0780 2024-09-10T20:38:14.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f63877e2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f63800cc8f0 con 0x7f6368003c30 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f63877e2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f63800ce350 con 0x7f6380159fe0 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63800b0780 0x7f63800b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63800b0780 0x7f63800b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35068/0 (socket says 172.21.15.28:35068) 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 -- 172.21.15.28:0/3381989232 learned_addr learned my addr 172.21.15.28:0/3381989232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6384d7d640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6380159fe0 0x7f63801f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f638557e640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6368003c30 0x7f63800746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 -- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6380159fe0 msgr2=0x7f63801f86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6380159fe0 0x7f63801f86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 -- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6368003c30 msgr2=0x7f63800746b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6368003c30 0x7f63800746b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 -- 172.21.15.28:0/3381989232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638015a510 con 0x7f63800b0780 2024-09-10T20:38:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f6385d7f640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63800b0780 0x7f63800b0be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f63680057e0 tx=0x7f6370006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f637dffb640 1 -- 172.21.15.28:0/3381989232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637000fa20 con 0x7f63800b0780 2024-09-10T20:38:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f637dffb640 1 -- 172.21.15.28:0/3381989232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f637000fbc0 con 0x7f63800b0780 2024-09-10T20:38:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f637dffb640 1 -- 172.21.15.28:0/3381989232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370011430 con 0x7f63800b0780 2024-09-10T20:38:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.543+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6380073bd0 con 0x7f63800b0780 2024-09-10T20:38:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.544+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6380074190 con 0x7f63800b0780 2024-09-10T20:38:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.545+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f638006e990 con 0x7f63800b0780 2024-09-10T20:38:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.545+0000 7f637dffb640 1 -- 172.21.15.28:0/3381989232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f63700115d0 con 0x7f63800b0780 2024-09-10T20:38:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.546+0000 7f637dffb640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f636c077f70 0x7f636c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.546+0000 7f637dffb640 1 -- 172.21.15.28:0/3381989232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f637009a500 con 0x7f63800b0780 2024-09-10T20:38:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.546+0000 7f638557e640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f636c077f70 0x7f636c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:14.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.546+0000 7f638557e640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f636c077f70 0x7f636c07a430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6378032000 tx=0x7f6378001360 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:14.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.550+0000 7f637dffb640 1 -- 172.21.15.28:0/3381989232 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6370063090 con 0x7f63800b0780 2024-09-10T20:38:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:14.772+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6380074360 con 0x7f63800b0780 2024-09-10T20:38:15.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.410+0000 7f637dffb640 1 -- 172.21.15.28:0/3381989232 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f6370062990 con 0x7f63800b0780 2024-09-10T20:38:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f636c077f70 msgr2=0x7f636c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f636c077f70 0x7f636c07a430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6378032000 tx=0x7f6378001360 comp rx=0 tx=0).stop 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63800b0780 msgr2=0x7f63800b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63800b0780 0x7f63800b0be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f63680057e0 tx=0x7f6370006660 comp rx=0 tx=0).stop 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 shutdown_connections 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f636c077f70 0x7f636c07a430 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63800b0780 0x7f63800b0be0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6380159fe0 0x7f63801f86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 --2- 172.21.15.28:0/3381989232 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6368003c30 0x7f63800746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 >> 172.21.15.28:0/3381989232 conn(0x7f63800d0090 msgr2=0x7f6380160b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 shutdown_connections 2024-09-10T20:38:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.413+0000 7f63877e2640 1 -- 172.21.15.28:0/3381989232 wait complete. 2024-09-10T20:38:15.431 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:38:15.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.539+0000 7f27ff46b640 1 -- 172.21.15.28:0/220365479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e00052e0 msgr2=0x7f27e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.539+0000 7f27ff46b640 1 --2- 172.21.15.28:0/220365479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e00052e0 0x7f27e000b9c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27e8001dc0 tx=0x7f27e0005830 comp rx=0 tx=0).stop 2024-09-10T20:38:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.539+0000 7f27ff46b640 1 -- 172.21.15.28:0/220365479 shutdown_connections 2024-09-10T20:38:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.539+0000 7f27ff46b640 1 --2- 172.21.15.28:0/220365479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e00052e0 0x7f27e000b9c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.539+0000 7f27ff46b640 1 -- 172.21.15.28:0/220365479 >> 172.21.15.28:0/220365479 conn(0x7f27f80d1e40 msgr2=0x7f27f80d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.539+0000 7f27ff46b640 1 -- 172.21.15.28:0/220365479 shutdown_connections 2024-09-10T20:38:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.539+0000 7f27ff46b640 1 -- 172.21.15.28:0/220365479 wait complete. 2024-09-10T20:38:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.540+0000 7f27ff46b640 1 Processor -- start 2024-09-10T20:38:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.540+0000 7f27ff46b640 1 -- start start 2024-09-10T20:38:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27ff46b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27e00052e0 0x7f27f81f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27ff46b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f80c8380 0x7f27f81f80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27ff46b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27f81f4ae0 0x7f27f81f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27ff46b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27f80ce850 con 0x7f27f80c8380 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27ff46b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f27f80cfef0 con 0x7f27f81f4ae0 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27ff46b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f27f80ce9d0 con 0x7f27e00052e0 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f80c8380 0x7f27f81f80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f80c8380 0x7f27f81f80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35094/0 (socket says 172.21.15.28:35094) 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 -- 172.21.15.28:0/438955564 learned_addr learned my addr 172.21.15.28:0/438955564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fd207640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27e00052e0 0x7f27f81f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fda08640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27f81f4ae0 0x7f27f81f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 -- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27e00052e0 msgr2=0x7f27f81f5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27e00052e0 0x7f27f81f5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 -- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27f81f4ae0 msgr2=0x7f27f81f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27f81f4ae0 0x7f27f81f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fca06640 1 -- 172.21.15.28:0/438955564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f81f8600 con 0x7f27f80c8380 2024-09-10T20:38:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.541+0000 7f27fd207640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27e00052e0 0x7f27f81f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.542+0000 7f27fca06640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f80c8380 0x7f27f81f80c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f27ec003fb0 tx=0x7f27ec00ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.542+0000 7f27f5ffb640 1 -- 172.21.15.28:0/438955564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec010a00 con 0x7f27f80c8380 2024-09-10T20:38:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.542+0000 7f27f5ffb640 1 -- 172.21.15.28:0/438955564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27ec015070 con 0x7f27f80c8380 2024-09-10T20:38:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.542+0000 7f27f5ffb640 1 -- 172.21.15.28:0/438955564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec010e20 con 0x7f27f80c8380 2024-09-10T20:38:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.542+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f81f54a0 con 0x7f27f80c8380 2024-09-10T20:38:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.542+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f81f88a0 con 0x7f27f80c8380 2024-09-10T20:38:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.543+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f80cef30 con 0x7f27f80c8380 2024-09-10T20:38:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.543+0000 7f27f5ffb640 1 -- 172.21.15.28:0/438955564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f27ec022030 con 0x7f27f80c8380 2024-09-10T20:38:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.544+0000 7f27f5ffb640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27e4077f70 0x7f27e407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.544+0000 7f27f5ffb640 1 -- 172.21.15.28:0/438955564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f27ec09a5d0 con 0x7f27f80c8380 2024-09-10T20:38:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.544+0000 7f27fd207640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27e4077f70 0x7f27e407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.545+0000 7f27fd207640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27e4077f70 0x7f27e407a430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f27f0006c10 tx=0x7f27f0006720 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.548+0000 7f27f5ffb640 1 -- 172.21.15.28:0/438955564 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f27ec062a90 con 0x7f27f80c8380 2024-09-10T20:38:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.771+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27f80724d0 con 0x7f27f80c8380 2024-09-10T20:38:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.771+0000 7f27f5ffb640 1 -- 172.21.15.28:0/438955564 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f27ec0674c0 con 0x7f27f80c8380 2024-09-10T20:38:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:15.777 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:15.777 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.773+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27e4077f70 msgr2=0x7f27e407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.773+0000 7f27ff46b640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27e4077f70 0x7f27e407a430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f27f0006c10 tx=0x7f27f0006720 comp rx=0 tx=0).stop 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.773+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f80c8380 msgr2=0x7f27f81f80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.773+0000 7f27ff46b640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f80c8380 0x7f27f81f80c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f27ec003fb0 tx=0x7f27ec00ba40 comp rx=0 tx=0).stop 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 shutdown_connections 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27e4077f70 0x7f27e407a430 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27f81f4ae0 0x7f27f81f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f80c8380 0x7f27f81f80c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 --2- 172.21.15.28:0/438955564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27e00052e0 0x7f27f81f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 >> 172.21.15.28:0/438955564 conn(0x7f27f80d1e40 msgr2=0x7f27f80b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 shutdown_connections 2024-09-10T20:38:15.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.774+0000 7f27ff46b640 1 -- 172.21.15.28:0/438955564 wait complete. 2024-09-10T20:38:15.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 -- 172.21.15.28:0/3870096485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 msgr2=0x7fe4880ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/3870096485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ce850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe478009830 tx=0x7fe47802a480 comp rx=0 tx=0).stop 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 -- 172.21.15.28:0/3870096485 shutdown_connections 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/3870096485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4880cd9b0 0x7fe4880ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/3870096485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ce850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 -- 172.21.15.28:0/3870096485 >> 172.21.15.28:0/3870096485 conn(0x7fe4880d1e40 msgr2=0x7fe4880d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 -- 172.21.15.28:0/3870096485 shutdown_connections 2024-09-10T20:38:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.890+0000 7fe48f7f3640 1 -- 172.21.15.28:0/3870096485 wait complete. 2024-09-10T20:38:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.891+0000 7fe48f7f3640 1 Processor -- start 2024-09-10T20:38:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.891+0000 7fe48f7f3640 1 -- start start 2024-09-10T20:38:15.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48f7f3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4880cd9b0 0x7fe48815aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48f7f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48f7f3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe48815c050 0x7fe4881533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48f7f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4880b1960 con 0x7fe4880d2960 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48f7f3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe4880b17e0 con 0x7fe48815c050 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48f7f3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe4880b1ae0 con 0x7fe4880cd9b0 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35116/0 (socket says 172.21.15.28:35116) 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 -- 172.21.15.28:0/301100293 learned_addr learned my addr 172.21.15.28:0/301100293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48d58f640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4880cd9b0 0x7fe48815aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48dd90640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe48815c050 0x7fe4881533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 -- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4880cd9b0 msgr2=0x7fe48815aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4880cd9b0 0x7fe48815aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 -- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe48815c050 msgr2=0x7fe4881533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe48815c050 0x7fe4881533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 -- 172.21.15.28:0/301100293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4880ca9e0 con 0x7fe4880d2960 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48d58f640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4880cd9b0 0x7fe48815aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48cd8e640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ca2e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe478009950 tx=0x7fe478032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe47dffb640 1 -- 172.21.15.28:0/301100293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe478040070 con 0x7fe4880d2960 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe47dffb640 1 -- 172.21.15.28:0/301100293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe478005310 con 0x7fe4880d2960 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.892+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4880cacc0 con 0x7fe4880d2960 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.893+0000 7fe47dffb640 1 -- 172.21.15.28:0/301100293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe478005c90 con 0x7fe4880d2960 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.893+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4880cb280 con 0x7fe4880d2960 2024-09-10T20:38:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.894+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4880b2070 con 0x7fe4880d2960 2024-09-10T20:38:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.894+0000 7fe47dffb640 1 -- 172.21.15.28:0/301100293 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe47804c030 con 0x7fe4880d2960 2024-09-10T20:38:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.895+0000 7fe47dffb640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe46c077f70 0x7fe46c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.895+0000 7fe47dffb640 1 -- 172.21.15.28:0/301100293 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe4780badc0 con 0x7fe4880d2960 2024-09-10T20:38:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.895+0000 7fe48d58f640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe46c077f70 0x7fe46c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.895+0000 7fe48d58f640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe46c077f70 0x7fe46c07a430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe484007a50 tx=0x7fe484009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:15.899+0000 7fe47dffb640 1 -- 172.21.15.28:0/301100293 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe4780838c0 con 0x7fe4880d2960 2024-09-10T20:38:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.085+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe488076360 con 0x7fe4880d2960 2024-09-10T20:38:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.086+0000 7fe47dffb640 1 -- 172.21.15.28:0/301100293 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16669 (secure 0 0 0) 0x7fe4780831c0 con 0x7fe4880d2960 2024-09-10T20:38:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:16.091 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:13.395359+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:16.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe46c077f70 msgr2=0x7fe46c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe46c077f70 0x7fe46c07a430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe484007a50 tx=0x7fe484009040 comp rx=0 tx=0).stop 2024-09-10T20:38:16.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 msgr2=0x7fe4880ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ca2e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe478009950 tx=0x7fe478032c10 comp rx=0 tx=0).stop 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 shutdown_connections 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe46c077f70 0x7fe46c07a430 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe48815c050 0x7fe4881533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4880d2960 0x7fe4880ca2e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 --2- 172.21.15.28:0/301100293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4880cd9b0 0x7fe48815aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 >> 172.21.15.28:0/301100293 conn(0x7fe4880d1e40 msgr2=0x7fe488070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 shutdown_connections 2024-09-10T20:38:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.088+0000 7fe48f7f3640 1 -- 172.21.15.28:0/301100293 wait complete. 2024-09-10T20:38:16.104 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:38:16.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.203+0000 7f2ce446e640 1 -- 172.21.15.28:0/3854543051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0d2960 msgr2=0x7f2cdc0ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.203+0000 7f2ce446e640 1 --2- 172.21.15.28:0/3854543051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0d2960 0x7f2cdc0ce850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc073cb0 tx=0x7f2ccc02a5f0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.204+0000 7f2ce446e640 1 -- 172.21.15.28:0/3854543051 shutdown_connections 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.204+0000 7f2ce446e640 1 --2- 172.21.15.28:0/3854543051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0d2960 0x7f2cdc0ce850 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.204+0000 7f2ce446e640 1 --2- 172.21.15.28:0/3854543051 >> v2:172.21.15.159:3300/0 conn(0x7f2cdc15c050 0x7f2cdc0cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.204+0000 7f2ce446e640 1 -- 172.21.15.28:0/3854543051 >> 172.21.15.28:0/3854543051 conn(0x7f2cdc0d1e40 msgr2=0x7f2cdc0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.204+0000 7f2ce446e640 1 -- 172.21.15.28:0/3854543051 shutdown_connections 2024-09-10T20:38:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.204+0000 7f2ce446e640 1 -- 172.21.15.28:0/3854543051 wait complete. 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.205+0000 7f2ce446e640 1 Processor -- start 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.205+0000 7f2ce446e640 1 -- start start 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0cd9b0 0x7f2cdc0cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cdc0d2960 0x7f2cdc06f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cdc15c050 0x7f2cdc15aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc0cf1d0 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc0cf050 con 0x7f2cdc0d2960 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc0b1af0 con 0x7f2cdc15c050 2024-09-10T20:38:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0cd9b0 0x7f2cdc0cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0cd9b0 0x7f2cdc0cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35136/0 (socket says 172.21.15.28:35136) 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 -- 172.21.15.28:0/81976830 learned_addr learned my addr 172.21.15.28:0/81976830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce2a0b640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cdc15c050 0x7f2cdc15aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 -- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cdc15c050 msgr2=0x7f2cdc15aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cdc15c050 0x7f2cdc15aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce1a09640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cdc0d2960 0x7f2cdc06f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 -- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cdc0d2960 msgr2=0x7f2cdc06f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cdc0d2960 0x7f2cdc06f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 -- 172.21.15.28:0/81976830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc06dd50 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce2a0b640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cdc15c050 0x7f2cdc15aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce220a640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0cd9b0 0x7f2cdc0cb350 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800b410 tx=0x7f2cd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2cd2ffd640 1 -- 172.21.15.28:0/81976830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8015020 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2cd2ffd640 1 -- 172.21.15.28:0/81976830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cd8011940 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc06e2e0 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2cd2ffd640 1 -- 172.21.15.28:0/81976830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8011d90 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.206+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc06e8a0 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.207+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cdc1e6ac0 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.208+0000 7f2cd2ffd640 1 -- 172.21.15.28:0/81976830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2cd801a400 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.208+0000 7f2cd2ffd640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2cc0077f70 0x7f2cc007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.208+0000 7f2cd2ffd640 1 -- 172.21.15.28:0/81976830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2cd8099ed0 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.209+0000 7f2ce1a09640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2cc0077f70 0x7f2cc007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.209+0000 7f2ce1a09640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2cc0077f70 0x7f2cc007a430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc033000 tx=0x7f2ccc004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.213+0000 7f2cd2ffd640 1 -- 172.21.15.28:0/81976830 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2cdc1e6ac0 con 0x7f2cdc0cd9b0 2024-09-10T20:38:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.396+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2cdc0cfd70 con 0x7f2cc0077f70 2024-09-10T20:38:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.401+0000 7f2cd2ffd640 1 -- 172.21.15.28:0/81976830 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+245574 (secure 0 0 0) 0x7f2cdc0cfd70 con 0x7f2cc0077f70 2024-09-10T20:38:16.409 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2cc0077f70 msgr2=0x7f2cc007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2cc0077f70 0x7f2cc007a430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc033000 tx=0x7f2ccc004730 comp rx=0 tx=0).stop 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0cd9b0 msgr2=0x7f2cdc0cb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0cd9b0 0x7f2cdc0cb350 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800b410 tx=0x7f2cd800b8d0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 shutdown_connections 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2cc0077f70 0x7f2cc007a430 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2cdc15c050 0x7f2cdc15aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2cdc0d2960 0x7f2cdc06f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 --2- 172.21.15.28:0/81976830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cdc0cd9b0 0x7f2cdc0cb350 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 >> 172.21.15.28:0/81976830 conn(0x7f2cdc0d1e40 msgr2=0x7f2cdc0d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 shutdown_connections 2024-09-10T20:38:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.404+0000 7f2ce446e640 1 -- 172.21.15.28:0/81976830 wait complete. 2024-09-10T20:38:16.419 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:38:15.976782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591144,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":129,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":124,"acting":124,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235700,"kb_used_data":2636,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739020,"statfs":{"total":773094113280,"available":772852756480,"internally_reserved":0,"allocated":2699264,"data_stored":1662301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.603193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400675+0000","last_change":"2024-09-10T20:38:12.413418+0000","last_active":"2024-09-10T20:38:13.400675+0000","last_peered":"2024-09-10T20:38:13.400675+0000","last_clean":"2024-09-10T20:38:13.400675+0000","last_became_active":"2024-09-10T20:38:12.413063+0000","last_became_peered":"2024-09-10T20:38:12.413063+0000","last_unstale":"2024-09-10T20:38:13.400675+0000","last_undegraded":"2024-09-10T20:38:13.400675+0000","last_fullsized":"2024-09-10T20:38:13.400675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:12.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401706+0000","last_change":"2024-09-10T20:38:12.404482+0000","last_active":"2024-09-10T20:38:13.401706+0000","last_peered":"2024-09-10T20:38:13.401706+0000","last_clean":"2024-09-10T20:38:13.401706+0000","last_became_active":"2024-09-10T20:38:12.404181+0000","last_became_peered":"2024-09-10T20:38:12.404181+0000","last_unstale":"2024-09-10T20:38:13.401706+0000","last_undegraded":"2024-09-10T20:38:13.401706+0000","last_fullsized":"2024-09-10T20:38:13.401706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:02.847206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401300+0000","last_change":"2024-09-10T20:38:12.403203+0000","last_active":"2024-09-10T20:38:13.401300+0000","last_peered":"2024-09-10T20:38:13.401300+0000","last_clean":"2024-09-10T20:38:13.401300+0000","last_became_active":"2024-09-10T20:38:12.402272+0000","last_became_peered":"2024-09-10T20:38:12.402272+0000","last_unstale":"2024-09-10T20:38:13.401300+0000","last_undegraded":"2024-09-10T20:38:13.401300+0000","last_fullsized":"2024-09-10T20:38:13.401300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343562+0000","last_change":"2024-09-10T20:38:12.415360+0000","last_active":"2024-09-10T20:38:14.343562+0000","last_peered":"2024-09-10T20:38:14.343562+0000","last_clean":"2024-09-10T20:38:14.343562+0000","last_became_active":"2024-09-10T20:38:12.415094+0000","last_became_peered":"2024-09-10T20:38:12.415094+0000","last_unstale":"2024-09-10T20:38:14.343562+0000","last_undegraded":"2024-09-10T20:38:14.343562+0000","last_fullsized":"2024-09-10T20:38:14.343562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:36.623506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343324+0000","last_change":"2024-09-10T20:38:12.411600+0000","last_active":"2024-09-10T20:38:14.343324+0000","last_peered":"2024-09-10T20:38:14.343324+0000","last_clean":"2024-09-10T20:38:14.343324+0000","last_became_active":"2024-09-10T20:38:12.411328+0000","last_became_peered":"2024-09-10T20:38:12.411328+0000","last_unstale":"2024-09-10T20:38:14.343324+0000","last_undegraded":"2024-09-10T20:38:14.343324+0000","last_fullsized":"2024-09-10T20:38:14.343324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:30.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400233+0000","last_change":"2024-09-10T20:38:12.404816+0000","last_active":"2024-09-10T20:38:13.400233+0000","last_peered":"2024-09-10T20:38:13.400233+0000","last_clean":"2024-09-10T20:38:13.400233+0000","last_became_active":"2024-09-10T20:38:12.404568+0000","last_became_peered":"2024-09-10T20:38:12.404568+0000","last_unstale":"2024-09-10T20:38:13.400233+0000","last_undegraded":"2024-09-10T20:38:13.400233+0000","last_fullsized":"2024-09-10T20:38:13.400233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:05:49.714066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633628+0000","last_change":"2024-09-10T20:38:12.407265+0000","last_active":"2024-09-10T20:38:13.633628+0000","last_peered":"2024-09-10T20:38:13.633628+0000","last_clean":"2024-09-10T20:38:13.633628+0000","last_became_active":"2024-09-10T20:38:12.406927+0000","last_became_peered":"2024-09-10T20:38:12.406927+0000","last_unstale":"2024-09-10T20:38:13.633628+0000","last_undegraded":"2024-09-10T20:38:13.633628+0000","last_fullsized":"2024-09-10T20:38:13.633628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:14.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400878+0000","last_change":"2024-09-10T20:38:12.407874+0000","last_active":"2024-09-10T20:38:13.400878+0000","last_peered":"2024-09-10T20:38:13.400878+0000","last_clean":"2024-09-10T20:38:13.400878+0000","last_became_active":"2024-09-10T20:38:12.407623+0000","last_became_peered":"2024-09-10T20:38:12.407623+0000","last_unstale":"2024-09-10T20:38:13.400878+0000","last_undegraded":"2024-09-10T20:38:13.400878+0000","last_fullsized":"2024-09-10T20:38:13.400878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:56.990230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400518+0000","last_change":"2024-09-10T20:38:12.408020+0000","last_active":"2024-09-10T20:38:13.400518+0000","last_peered":"2024-09-10T20:38:13.400518+0000","last_clean":"2024-09-10T20:38:13.400518+0000","last_became_active":"2024-09-10T20:38:12.407757+0000","last_became_peered":"2024-09-10T20:38:12.407757+0000","last_unstale":"2024-09-10T20:38:13.400518+0000","last_undegraded":"2024-09-10T20:38:13.400518+0000","last_fullsized":"2024-09-10T20:38:13.400518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:32.513086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401654+0000","last_change":"2024-09-10T20:38:12.416773+0000","last_active":"2024-09-10T20:38:13.401654+0000","last_peered":"2024-09-10T20:38:13.401654+0000","last_clean":"2024-09-10T20:38:13.401654+0000","last_became_active":"2024-09-10T20:38:12.416498+0000","last_became_peered":"2024-09-10T20:38:12.416498+0000","last_unstale":"2024-09-10T20:38:13.401654+0000","last_undegraded":"2024-09-10T20:38:13.401654+0000","last_fullsized":"2024-09-10T20:38:13.401654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:42.492324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343671+0000","last_change":"2024-09-10T20:38:12.406673+0000","last_active":"2024-09-10T20:38:14.343671+0000","last_peered":"2024-09-10T20:38:14.343671+0000","last_clean":"2024-09-10T20:38:14.343671+0000","last_became_active":"2024-09-10T20:38:12.406401+0000","last_became_peered":"2024-09-10T20:38:12.406401+0000","last_unstale":"2024-09-10T20:38:14.343671+0000","last_undegraded":"2024-09-10T20:38:14.343671+0000","last_fullsized":"2024-09-10T20:38:14.343671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:31.845414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343502+0000","last_change":"2024-09-10T20:38:12.414290+0000","last_active":"2024-09-10T20:38:14.343502+0000","last_peered":"2024-09-10T20:38:14.343502+0000","last_clean":"2024-09-10T20:38:14.343502+0000","last_became_active":"2024-09-10T20:38:12.413712+0000","last_became_peered":"2024-09-10T20:38:12.413712+0000","last_unstale":"2024-09-10T20:38:14.343502+0000","last_undegraded":"2024-09-10T20:38:14.343502+0000","last_fullsized":"2024-09-10T20:38:14.343502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:48.033101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400477+0000","last_change":"2024-09-10T20:38:12.406921+0000","last_active":"2024-09-10T20:38:13.400477+0000","last_peered":"2024-09-10T20:38:13.400477+0000","last_clean":"2024-09-10T20:38:13.400477+0000","last_became_active":"2024-09-10T20:38:12.406642+0000","last_became_peered":"2024-09-10T20:38:12.406642+0000","last_unstale":"2024-09-10T20:38:13.400477+0000","last_undegraded":"2024-09-10T20:38:13.400477+0000","last_fullsized":"2024-09-10T20:38:13.400477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:02.877370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633368+0000","last_change":"2024-09-10T20:38:12.415776+0000","last_active":"2024-09-10T20:38:13.633368+0000","last_peered":"2024-09-10T20:38:13.633368+0000","last_clean":"2024-09-10T20:38:13.633368+0000","last_became_active":"2024-09-10T20:38:12.415480+0000","last_became_peered":"2024-09-10T20:38:12.415480+0000","last_unstale":"2024-09-10T20:38:13.633368+0000","last_undegraded":"2024-09-10T20:38:13.633368+0000","last_fullsized":"2024-09-10T20:38:13.633368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:13.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400454+0000","last_change":"2024-09-10T20:38:11.394869+0000","last_active":"2024-09-10T20:38:13.400454+0000","last_peered":"2024-09-10T20:38:13.400454+0000","last_clean":"2024-09-10T20:38:13.400454+0000","last_became_active":"2024-09-10T20:38:11.394564+0000","last_became_peered":"2024-09-10T20:38:11.394564+0000","last_unstale":"2024-09-10T20:38:13.400454+0000","last_undegraded":"2024-09-10T20:38:13.400454+0000","last_fullsized":"2024-09-10T20:38:13.400454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:19.365550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401125+0000","last_change":"2024-09-10T20:38:12.401521+0000","last_active":"2024-09-10T20:38:13.401125+0000","last_peered":"2024-09-10T20:38:13.401125+0000","last_clean":"2024-09-10T20:38:13.401125+0000","last_became_active":"2024-09-10T20:38:12.401218+0000","last_became_peered":"2024-09-10T20:38:12.401218+0000","last_unstale":"2024-09-10T20:38:13.401125+0000","last_undegraded":"2024-09-10T20:38:13.401125+0000","last_fullsized":"2024-09-10T20:38:13.401125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:33.954998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378820+0000","last_change":"2024-09-10T20:38:10.378820+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378820+0000","last_undegraded":"2024-09-10T20:38:10.378820+0000","last_fullsized":"2024-09-10T20:38:10.378820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343167+0000","last_change":"2024-09-10T20:38:11.398400+0000","last_active":"2024-09-10T20:38:14.343167+0000","last_peered":"2024-09-10T20:38:14.343167+0000","last_clean":"2024-09-10T20:38:14.343167+0000","last_became_active":"2024-09-10T20:38:11.398097+0000","last_became_peered":"2024-09-10T20:38:11.398097+0000","last_unstale":"2024-09-10T20:38:14.343167+0000","last_undegraded":"2024-09-10T20:38:14.343167+0000","last_fullsized":"2024-09-10T20:38:14.343167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.599838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401478+0000","last_change":"2024-09-10T20:38:12.413177+0000","last_active":"2024-09-10T20:38:13.401478+0000","last_peered":"2024-09-10T20:38:13.401478+0000","last_clean":"2024-09-10T20:38:13.401478+0000","last_became_active":"2024-09-10T20:38:12.412925+0000","last_became_peered":"2024-09-10T20:38:12.412925+0000","last_unstale":"2024-09-10T20:38:13.401478+0000","last_undegraded":"2024-09-10T20:38:13.401478+0000","last_fullsized":"2024-09-10T20:38:13.401478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:18.693832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378650+0000","last_change":"2024-09-10T20:38:10.378650+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378650+0000","last_undegraded":"2024-09-10T20:38:10.378650+0000","last_fullsized":"2024-09-10T20:38:10.378650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401415+0000","last_change":"2024-09-10T20:38:12.415606+0000","last_active":"2024-09-10T20:38:13.401415+0000","last_peered":"2024-09-10T20:38:13.401415+0000","last_clean":"2024-09-10T20:38:13.401415+0000","last_became_active":"2024-09-10T20:38:12.415330+0000","last_became_peered":"2024-09-10T20:38:12.415330+0000","last_unstale":"2024-09-10T20:38:13.401415+0000","last_undegraded":"2024-09-10T20:38:13.401415+0000","last_fullsized":"2024-09-10T20:38:13.401415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:53.050014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380287+0000","last_change":"2024-09-10T20:38:10.380287+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380287+0000","last_undegraded":"2024-09-10T20:38:10.380287+0000","last_fullsized":"2024-09-10T20:38:10.380287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380912+0000","last_change":"2024-09-10T20:38:10.380912+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380912+0000","last_undegraded":"2024-09-10T20:38:10.380912+0000","last_fullsized":"2024-09-10T20:38:10.380912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401902+0000","last_change":"2024-09-10T20:38:12.410504+0000","last_active":"2024-09-10T20:38:13.401902+0000","last_peered":"2024-09-10T20:38:13.401902+0000","last_clean":"2024-09-10T20:38:13.401902+0000","last_became_active":"2024-09-10T20:38:12.410008+0000","last_became_peered":"2024-09-10T20:38:12.410008+0000","last_unstale":"2024-09-10T20:38:13.401902+0000","last_undegraded":"2024-09-10T20:38:13.401902+0000","last_fullsized":"2024-09-10T20:38:13.401902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:57.544501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633468+0000","last_change":"2024-09-10T20:38:11.395962+0000","last_active":"2024-09-10T20:38:13.633468+0000","last_peered":"2024-09-10T20:38:13.633468+0000","last_clean":"2024-09-10T20:38:13.633468+0000","last_became_active":"2024-09-10T20:38:11.395687+0000","last_became_peered":"2024-09-10T20:38:11.395687+0000","last_unstale":"2024-09-10T20:38:13.633468+0000","last_undegraded":"2024-09-10T20:38:13.633468+0000","last_fullsized":"2024-09-10T20:38:13.633468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:15.421827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380034+0000","last_change":"2024-09-10T20:38:10.380034+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380034+0000","last_undegraded":"2024-09-10T20:38:10.380034+0000","last_fullsized":"2024-09-10T20:38:10.380034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633539+0000","last_change":"2024-09-10T20:38:11.398330+0000","last_active":"2024-09-10T20:38:13.633539+0000","last_peered":"2024-09-10T20:38:13.633539+0000","last_clean":"2024-09-10T20:38:13.633539+0000","last_became_active":"2024-09-10T20:38:11.398059+0000","last_became_peered":"2024-09-10T20:38:11.398059+0000","last_unstale":"2024-09-10T20:38:13.633539+0000","last_undegraded":"2024-09-10T20:38:13.633539+0000","last_fullsized":"2024-09-10T20:38:13.633539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343446+0000","last_change":"2024-09-10T20:38:12.412484+0000","last_active":"2024-09-10T20:38:14.343446+0000","last_peered":"2024-09-10T20:38:14.343446+0000","last_clean":"2024-09-10T20:38:14.343446+0000","last_became_active":"2024-09-10T20:38:12.412165+0000","last_became_peered":"2024-09-10T20:38:12.412165+0000","last_unstale":"2024-09-10T20:38:14.343446+0000","last_undegraded":"2024-09-10T20:38:14.343446+0000","last_fullsized":"2024-09-10T20:38:14.343446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:09.972659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378166+0000","last_change":"2024-09-10T20:38:10.378166+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378166+0000","last_undegraded":"2024-09-10T20:38:10.378166+0000","last_fullsized":"2024-09-10T20:38:10.378166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380952+0000","last_change":"2024-09-10T20:38:10.380952+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380952+0000","last_undegraded":"2024-09-10T20:38:10.380952+0000","last_fullsized":"2024-09-10T20:38:10.380952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400564+0000","last_change":"2024-09-10T20:38:11.393414+0000","last_active":"2024-09-10T20:38:13.400564+0000","last_peered":"2024-09-10T20:38:13.400564+0000","last_clean":"2024-09-10T20:38:13.400564+0000","last_became_active":"2024-09-10T20:38:11.393078+0000","last_became_peered":"2024-09-10T20:38:11.393078+0000","last_unstale":"2024-09-10T20:38:13.400564+0000","last_undegraded":"2024-09-10T20:38:13.400564+0000","last_fullsized":"2024-09-10T20:38:13.400564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:58.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381915+0000","last_change":"2024-09-10T20:38:10.381915+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381915+0000","last_undegraded":"2024-09-10T20:38:10.381915+0000","last_fullsized":"2024-09-10T20:38:10.381915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401433+0000","last_change":"2024-09-10T20:38:11.388247+0000","last_active":"2024-09-10T20:38:13.401433+0000","last_peered":"2024-09-10T20:38:13.401433+0000","last_clean":"2024-09-10T20:38:13.401433+0000","last_became_active":"2024-09-10T20:38:11.387924+0000","last_became_peered":"2024-09-10T20:38:11.387924+0000","last_unstale":"2024-09-10T20:38:13.401433+0000","last_undegraded":"2024-09-10T20:38:13.401433+0000","last_fullsized":"2024-09-10T20:38:13.401433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:29.584118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400616+0000","last_change":"2024-09-10T20:38:11.399015+0000","last_active":"2024-09-10T20:38:13.400616+0000","last_peered":"2024-09-10T20:38:13.400616+0000","last_clean":"2024-09-10T20:38:13.400616+0000","last_became_active":"2024-09-10T20:38:11.398826+0000","last_became_peered":"2024-09-10T20:38:11.398826+0000","last_unstale":"2024-09-10T20:38:13.400616+0000","last_undegraded":"2024-09-10T20:38:13.400616+0000","last_fullsized":"2024-09-10T20:38:13.400616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:03.958637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343682+0000","last_change":"2024-09-10T20:38:11.398337+0000","last_active":"2024-09-10T20:38:14.343682+0000","last_peered":"2024-09-10T20:38:14.343682+0000","last_clean":"2024-09-10T20:38:14.343682+0000","last_became_active":"2024-09-10T20:38:11.398068+0000","last_became_peered":"2024-09-10T20:38:11.398068+0000","last_unstale":"2024-09-10T20:38:14.343682+0000","last_undegraded":"2024-09-10T20:38:14.343682+0000","last_fullsized":"2024-09-10T20:38:14.343682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:00:28.236504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":105,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-10T20:38:10.380805+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:10.380805+0000","last_peered":"2024-09-10T20:38:10.380805+0000","last_clean":"2024-09-10T20:38:10.380805+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:10.380805+0000","last_undegraded":"2024-09-10T20:38:10.380805+0000","last_fullsized":"2024-09-10T20:38:10.380805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377085+0000","last_change":"2024-09-10T20:38:10.377085+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377085+0000","last_undegraded":"2024-09-10T20:38:10.377085+0000","last_fullsized":"2024-09-10T20:38:10.377085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400902+0000","last_change":"2024-09-10T20:38:12.407263+0000","last_active":"2024-09-10T20:38:13.400902+0000","last_peered":"2024-09-10T20:38:13.400902+0000","last_clean":"2024-09-10T20:38:13.400902+0000","last_became_active":"2024-09-10T20:38:12.406952+0000","last_became_peered":"2024-09-10T20:38:12.406952+0000","last_unstale":"2024-09-10T20:38:13.400902+0000","last_undegraded":"2024-09-10T20:38:13.400902+0000","last_fullsized":"2024-09-10T20:38:13.400902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.171100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.376990+0000","last_change":"2024-09-10T20:38:10.376990+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.376990+0000","last_undegraded":"2024-09-10T20:38:10.376990+0000","last_fullsized":"2024-09-10T20:38:10.376990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343358+0000","last_change":"2024-09-10T20:38:12.407025+0000","last_active":"2024-09-10T20:38:14.343358+0000","last_peered":"2024-09-10T20:38:14.343358+0000","last_clean":"2024-09-10T20:38:14.343358+0000","last_became_active":"2024-09-10T20:38:12.406124+0000","last_became_peered":"2024-09-10T20:38:12.406124+0000","last_unstale":"2024-09-10T20:38:14.343358+0000","last_undegraded":"2024-09-10T20:38:14.343358+0000","last_fullsized":"2024-09-10T20:38:14.343358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633299+0000","last_change":"2024-09-10T20:38:11.397217+0000","last_active":"2024-09-10T20:38:13.633299+0000","last_peered":"2024-09-10T20:38:13.633299+0000","last_clean":"2024-09-10T20:38:13.633299+0000","last_became_active":"2024-09-10T20:38:11.396928+0000","last_became_peered":"2024-09-10T20:38:11.396928+0000","last_unstale":"2024-09-10T20:38:13.633299+0000","last_undegraded":"2024-09-10T20:38:13.633299+0000","last_fullsized":"2024-09-10T20:38:13.633299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:38.900087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401469+0000","last_change":"2024-09-10T20:38:12.407595+0000","last_active":"2024-09-10T20:38:13.401469+0000","last_peered":"2024-09-10T20:38:13.401469+0000","last_clean":"2024-09-10T20:38:13.401469+0000","last_became_active":"2024-09-10T20:38:12.407355+0000","last_became_peered":"2024-09-10T20:38:12.407355+0000","last_unstale":"2024-09-10T20:38:13.401469+0000","last_undegraded":"2024-09-10T20:38:13.401469+0000","last_fullsized":"2024-09-10T20:38:13.401469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:05.419998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377007+0000","last_change":"2024-09-10T20:38:10.377007+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377007+0000","last_undegraded":"2024-09-10T20:38:10.377007+0000","last_fullsized":"2024-09-10T20:38:10.377007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400255+0000","last_change":"2024-09-10T20:38:11.397463+0000","last_active":"2024-09-10T20:38:13.400255+0000","last_peered":"2024-09-10T20:38:13.400255+0000","last_clean":"2024-09-10T20:38:13.400255+0000","last_became_active":"2024-09-10T20:38:11.397205+0000","last_became_peered":"2024-09-10T20:38:11.397205+0000","last_unstale":"2024-09-10T20:38:13.400255+0000","last_undegraded":"2024-09-10T20:38:13.400255+0000","last_fullsized":"2024-09-10T20:38:13.400255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:53.003074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377042+0000","last_change":"2024-09-10T20:38:10.377042+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377042+0000","last_undegraded":"2024-09-10T20:38:10.377042+0000","last_fullsized":"2024-09-10T20:38:10.377042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343576+0000","last_change":"2024-09-10T20:38:12.408779+0000","last_active":"2024-09-10T20:38:14.343576+0000","last_peered":"2024-09-10T20:38:14.343576+0000","last_clean":"2024-09-10T20:38:14.343576+0000","last_became_active":"2024-09-10T20:38:12.408490+0000","last_became_peered":"2024-09-10T20:38:12.408490+0000","last_unstale":"2024-09-10T20:38:14.343576+0000","last_undegraded":"2024-09-10T20:38:14.343576+0000","last_fullsized":"2024-09-10T20:38:14.343576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:25.888623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407174+0000","last_change":"2024-09-10T20:38:11.394170+0000","last_active":"2024-09-10T20:38:13.407174+0000","last_peered":"2024-09-10T20:38:13.407174+0000","last_clean":"2024-09-10T20:38:13.407174+0000","last_became_active":"2024-09-10T20:38:11.393850+0000","last_became_peered":"2024-09-10T20:38:11.393850+0000","last_unstale":"2024-09-10T20:38:13.407174+0000","last_undegraded":"2024-09-10T20:38:13.407174+0000","last_fullsized":"2024-09-10T20:38:13.407174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:15.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401623+0000","last_change":"2024-09-10T20:38:12.404386+0000","last_active":"2024-09-10T20:38:13.401623+0000","last_peered":"2024-09-10T20:38:13.401623+0000","last_clean":"2024-09-10T20:38:13.401623+0000","last_became_active":"2024-09-10T20:38:12.404045+0000","last_became_peered":"2024-09-10T20:38:12.404045+0000","last_unstale":"2024-09-10T20:38:13.401623+0000","last_undegraded":"2024-09-10T20:38:13.401623+0000","last_fullsized":"2024-09-10T20:38:13.401623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:49.410849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377015+0000","last_change":"2024-09-10T20:38:10.377015+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377015+0000","last_undegraded":"2024-09-10T20:38:10.377015+0000","last_fullsized":"2024-09-10T20:38:10.377015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400970+0000","last_change":"2024-09-10T20:38:12.409091+0000","last_active":"2024-09-10T20:38:13.400970+0000","last_peered":"2024-09-10T20:38:13.400970+0000","last_clean":"2024-09-10T20:38:13.400970+0000","last_became_active":"2024-09-10T20:38:12.408815+0000","last_became_peered":"2024-09-10T20:38:12.408815+0000","last_unstale":"2024-09-10T20:38:13.400970+0000","last_undegraded":"2024-09-10T20:38:13.400970+0000","last_fullsized":"2024-09-10T20:38:13.400970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.870149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.382686+0000","last_change":"2024-09-10T20:38:10.382686+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.382686+0000","last_undegraded":"2024-09-10T20:38:10.382686+0000","last_fullsized":"2024-09-10T20:38:10.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401287+0000","last_change":"2024-09-10T20:38:11.397172+0000","last_active":"2024-09-10T20:38:13.401287+0000","last_peered":"2024-09-10T20:38:13.401287+0000","last_clean":"2024-09-10T20:38:13.401287+0000","last_became_active":"2024-09-10T20:38:11.396670+0000","last_became_peered":"2024-09-10T20:38:11.396670+0000","last_unstale":"2024-09-10T20:38:13.401287+0000","last_undegraded":"2024-09-10T20:38:13.401287+0000","last_fullsized":"2024-09-10T20:38:13.401287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400195+0000","last_change":"2024-09-10T20:38:12.405875+0000","last_active":"2024-09-10T20:38:13.400195+0000","last_peered":"2024-09-10T20:38:13.400195+0000","last_clean":"2024-09-10T20:38:13.400195+0000","last_became_active":"2024-09-10T20:38:12.405583+0000","last_became_peered":"2024-09-10T20:38:12.405583+0000","last_unstale":"2024-09-10T20:38:13.400195+0000","last_undegraded":"2024-09-10T20:38:13.400195+0000","last_fullsized":"2024-09-10T20:38:13.400195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:30.185620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343471+0000","last_change":"2024-09-10T20:38:11.395992+0000","last_active":"2024-09-10T20:38:14.343471+0000","last_peered":"2024-09-10T20:38:14.343471+0000","last_clean":"2024-09-10T20:38:14.343471+0000","last_became_active":"2024-09-10T20:38:11.394935+0000","last_became_peered":"2024-09-10T20:38:11.394935+0000","last_unstale":"2024-09-10T20:38:14.343471+0000","last_undegraded":"2024-09-10T20:38:14.343471+0000","last_fullsized":"2024-09-10T20:38:14.343471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:49.040222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633574+0000","last_change":"2024-09-10T20:38:12.412620+0000","last_active":"2024-09-10T20:38:13.633574+0000","last_peered":"2024-09-10T20:38:13.633574+0000","last_clean":"2024-09-10T20:38:13.633574+0000","last_became_active":"2024-09-10T20:38:12.412329+0000","last_became_peered":"2024-09-10T20:38:12.412329+0000","last_unstale":"2024-09-10T20:38:13.633574+0000","last_undegraded":"2024-09-10T20:38:13.633574+0000","last_fullsized":"2024-09-10T20:38:13.633574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:40.609805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377791+0000","last_change":"2024-09-10T20:38:10.377791+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377791+0000","last_undegraded":"2024-09-10T20:38:10.377791+0000","last_fullsized":"2024-09-10T20:38:10.377791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381267+0000","last_change":"2024-09-10T20:38:10.381267+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381267+0000","last_undegraded":"2024-09-10T20:38:10.381267+0000","last_fullsized":"2024-09-10T20:38:10.381267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381405+0000","last_change":"2024-09-10T20:38:10.381405+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381405+0000","last_undegraded":"2024-09-10T20:38:10.381405+0000","last_fullsized":"2024-09-10T20:38:10.381405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401905+0000","last_change":"2024-09-10T20:38:12.416588+0000","last_active":"2024-09-10T20:38:13.401905+0000","last_peered":"2024-09-10T20:38:13.401905+0000","last_clean":"2024-09-10T20:38:13.401905+0000","last_became_active":"2024-09-10T20:38:12.416297+0000","last_became_peered":"2024-09-10T20:38:12.416297+0000","last_unstale":"2024-09-10T20:38:13.401905+0000","last_undegraded":"2024-09-10T20:38:13.401905+0000","last_fullsized":"2024-09-10T20:38:13.401905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401853+0000","last_change":"2024-09-10T20:38:11.396382+0000","last_active":"2024-09-10T20:38:13.401853+0000","last_peered":"2024-09-10T20:38:13.401853+0000","last_clean":"2024-09-10T20:38:13.401853+0000","last_became_active":"2024-09-10T20:38:11.395934+0000","last_became_peered":"2024-09-10T20:38:11.395934+0000","last_unstale":"2024-09-10T20:38:13.401853+0000","last_undegraded":"2024-09-10T20:38:13.401853+0000","last_fullsized":"2024-09-10T20:38:13.401853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:20:39.553068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401634+0000","last_change":"2024-09-10T20:38:12.402624+0000","last_active":"2024-09-10T20:38:13.401634+0000","last_peered":"2024-09-10T20:38:13.401634+0000","last_clean":"2024-09-10T20:38:13.401634+0000","last_became_active":"2024-09-10T20:38:12.401924+0000","last_became_peered":"2024-09-10T20:38:12.401924+0000","last_unstale":"2024-09-10T20:38:13.401634+0000","last_undegraded":"2024-09-10T20:38:13.401634+0000","last_fullsized":"2024-09-10T20:38:13.401634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:33.683744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401708+0000","last_change":"2024-09-10T20:38:11.401264+0000","last_active":"2024-09-10T20:38:13.401708+0000","last_peered":"2024-09-10T20:38:13.401708+0000","last_clean":"2024-09-10T20:38:13.401708+0000","last_became_active":"2024-09-10T20:38:11.400986+0000","last_became_peered":"2024-09-10T20:38:11.400986+0000","last_unstale":"2024-09-10T20:38:13.401708+0000","last_undegraded":"2024-09-10T20:38:13.401708+0000","last_fullsized":"2024-09-10T20:38:13.401708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:50.668458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":172032,"data_stored":53177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":753664,"data_stored":637728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":66227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":662987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":74811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:16.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.559+0000 7fb238662640 1 -- 172.21.15.28:0/1726155272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300cd3d0 msgr2=0x7fb2300cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.559+0000 7fb238662640 1 --2- 172.21.15.28:0/1726155272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300cd3d0 0x7fb2300cc570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009830 tx=0x7fb22c02a510 comp rx=0 tx=0).stop 2024-09-10T20:38:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.560+0000 7fb238662640 1 -- 172.21.15.28:0/1726155272 shutdown_connections 2024-09-10T20:38:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.560+0000 7fb238662640 1 --2- 172.21.15.28:0/1726155272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2300d07a0 0x7fb2300b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.560+0000 7fb238662640 1 --2- 172.21.15.28:0/1726155272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300cd3d0 0x7fb2300cc570 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.560+0000 7fb238662640 1 -- 172.21.15.28:0/1726155272 >> 172.21.15.28:0/1726155272 conn(0x7fb23006cf00 msgr2=0x7fb23006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.560+0000 7fb238662640 1 -- 172.21.15.28:0/1726155272 shutdown_connections 2024-09-10T20:38:16.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.560+0000 7fb238662640 1 -- 172.21.15.28:0/1726155272 wait complete. 2024-09-10T20:38:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.561+0000 7fb238662640 1 Processor -- start 2024-09-10T20:38:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.561+0000 7fb238662640 1 -- start start 2024-09-10T20:38:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb238662640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300b4340 0x7fb2300b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb238662640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2300cd3d0 0x7fb23015dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb238662640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2300d07a0 0x7fb2300cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb238662640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2300b33c0 con 0x7fb2300b4340 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb238662640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb2300b4af0 con 0x7fb2300cd3d0 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb238662640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb2300b3540 con 0x7fb2300d07a0 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300b4340 0x7fb2300b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300b4340 0x7fb2300b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35140/0 (socket says 172.21.15.28:35140) 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 -- 172.21.15.28:0/1937378142 learned_addr learned my addr 172.21.15.28:0/1937378142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb236bff640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2300d07a0 0x7fb2300cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb235bfd640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2300cd3d0 0x7fb23015dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 -- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2300d07a0 msgr2=0x7fb2300cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2300d07a0 0x7fb2300cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 -- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2300cd3d0 msgr2=0x7fb23015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2300cd3d0 0x7fb23015dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 -- 172.21.15.28:0/1937378142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2301f5490 con 0x7fb2300b4340 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb2363fe640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300b4340 0x7fb2300b0f40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009950 tx=0x7fb22c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb226ffd640 1 -- 172.21.15.28:0/1937378142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c040070 con 0x7fb2300b4340 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb226ffd640 1 -- 172.21.15.28:0/1937378142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb22c005310 con 0x7fb2300b4340 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb226ffd640 1 -- 172.21.15.28:0/1937378142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0047a0 con 0x7fb2300b4340 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.562+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2301f4910 con 0x7fb2300b4340 2024-09-10T20:38:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.563+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2301f4ed0 con 0x7fb2300b4340 2024-09-10T20:38:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.564+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2300b3a10 con 0x7fb2300b4340 2024-09-10T20:38:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.564+0000 7fb226ffd640 1 -- 172.21.15.28:0/1937378142 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb22c004b50 con 0x7fb2300b4340 2024-09-10T20:38:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.565+0000 7fb226ffd640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb214077fa0 0x7fb21407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.565+0000 7fb226ffd640 1 -- 172.21.15.28:0/1937378142 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb22c0bb450 con 0x7fb2300b4340 2024-09-10T20:38:16.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.565+0000 7fb235bfd640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb214077fa0 0x7fb21407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.565+0000 7fb235bfd640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb214077fa0 0x7fb21407a460 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb220000f60 tx=0x7fb220009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:16.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.570+0000 7fb226ffd640 1 -- 172.21.15.28:0/1937378142 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb22c083f40 con 0x7fb2300b4340 2024-09-10T20:38:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.749+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2300ced50 con 0x7fb214077fa0 2024-09-10T20:38:16.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.754+0000 7fb226ffd640 1 -- 172.21.15.28:0/1937378142 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+245574 (secure 0 0 0) 0x7fb2300ced50 con 0x7fb214077fa0 2024-09-10T20:38:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:16.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb214077fa0 msgr2=0x7fb21407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb214077fa0 0x7fb21407a460 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb220000f60 tx=0x7fb220009040 comp rx=0 tx=0).stop 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300b4340 msgr2=0x7fb2300b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300b4340 0x7fb2300b0f40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009950 tx=0x7fb22c032c10 comp rx=0 tx=0).stop 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 shutdown_connections 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb214077fa0 0x7fb21407a460 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2300d07a0 0x7fb2300cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2300cd3d0 0x7fb23015dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 --2- 172.21.15.28:0/1937378142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2300b4340 0x7fb2300b0f40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 >> 172.21.15.28:0/1937378142 conn(0x7fb23006cf00 msgr2=0x7fb23015d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 shutdown_connections 2024-09-10T20:38:16.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.757+0000 7fb238662640 1 -- 172.21.15.28:0/1937378142 wait complete. 2024-09-10T20:38:16.772 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:38:15.976782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591144,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":129,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":124,"acting":124,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235700,"kb_used_data":2636,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739020,"statfs":{"total":773094113280,"available":772852756480,"internally_reserved":0,"allocated":2699264,"data_stored":1662301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.603193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400675+0000","last_change":"2024-09-10T20:38:12.413418+0000","last_active":"2024-09-10T20:38:13.400675+0000","last_peered":"2024-09-10T20:38:13.400675+0000","last_clean":"2024-09-10T20:38:13.400675+0000","last_became_active":"2024-09-10T20:38:12.413063+0000","last_became_peered":"2024-09-10T20:38:12.413063+0000","last_unstale":"2024-09-10T20:38:13.400675+0000","last_undegraded":"2024-09-10T20:38:13.400675+0000","last_fullsized":"2024-09-10T20:38:13.400675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:12.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401706+0000","last_change":"2024-09-10T20:38:12.404482+0000","last_active":"2024-09-10T20:38:13.401706+0000","last_peered":"2024-09-10T20:38:13.401706+0000","last_clean":"2024-09-10T20:38:13.401706+0000","last_became_active":"2024-09-10T20:38:12.404181+0000","last_became_peered":"2024-09-10T20:38:12.404181+0000","last_unstale":"2024-09-10T20:38:13.401706+0000","last_undegraded":"2024-09-10T20:38:13.401706+0000","last_fullsized":"2024-09-10T20:38:13.401706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:02.847206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401300+0000","last_change":"2024-09-10T20:38:12.403203+0000","last_active":"2024-09-10T20:38:13.401300+0000","last_peered":"2024-09-10T20:38:13.401300+0000","last_clean":"2024-09-10T20:38:13.401300+0000","last_became_active":"2024-09-10T20:38:12.402272+0000","last_became_peered":"2024-09-10T20:38:12.402272+0000","last_unstale":"2024-09-10T20:38:13.401300+0000","last_undegraded":"2024-09-10T20:38:13.401300+0000","last_fullsized":"2024-09-10T20:38:13.401300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343562+0000","last_change":"2024-09-10T20:38:12.415360+0000","last_active":"2024-09-10T20:38:14.343562+0000","last_peered":"2024-09-10T20:38:14.343562+0000","last_clean":"2024-09-10T20:38:14.343562+0000","last_became_active":"2024-09-10T20:38:12.415094+0000","last_became_peered":"2024-09-10T20:38:12.415094+0000","last_unstale":"2024-09-10T20:38:14.343562+0000","last_undegraded":"2024-09-10T20:38:14.343562+0000","last_fullsized":"2024-09-10T20:38:14.343562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:36.623506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343324+0000","last_change":"2024-09-10T20:38:12.411600+0000","last_active":"2024-09-10T20:38:14.343324+0000","last_peered":"2024-09-10T20:38:14.343324+0000","last_clean":"2024-09-10T20:38:14.343324+0000","last_became_active":"2024-09-10T20:38:12.411328+0000","last_became_peered":"2024-09-10T20:38:12.411328+0000","last_unstale":"2024-09-10T20:38:14.343324+0000","last_undegraded":"2024-09-10T20:38:14.343324+0000","last_fullsized":"2024-09-10T20:38:14.343324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:30.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400233+0000","last_change":"2024-09-10T20:38:12.404816+0000","last_active":"2024-09-10T20:38:13.400233+0000","last_peered":"2024-09-10T20:38:13.400233+0000","last_clean":"2024-09-10T20:38:13.400233+0000","last_became_active":"2024-09-10T20:38:12.404568+0000","last_became_peered":"2024-09-10T20:38:12.404568+0000","last_unstale":"2024-09-10T20:38:13.400233+0000","last_undegraded":"2024-09-10T20:38:13.400233+0000","last_fullsized":"2024-09-10T20:38:13.400233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:05:49.714066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633628+0000","last_change":"2024-09-10T20:38:12.407265+0000","last_active":"2024-09-10T20:38:13.633628+0000","last_peered":"2024-09-10T20:38:13.633628+0000","last_clean":"2024-09-10T20:38:13.633628+0000","last_became_active":"2024-09-10T20:38:12.406927+0000","last_became_peered":"2024-09-10T20:38:12.406927+0000","last_unstale":"2024-09-10T20:38:13.633628+0000","last_undegraded":"2024-09-10T20:38:13.633628+0000","last_fullsized":"2024-09-10T20:38:13.633628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:14.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400878+0000","last_change":"2024-09-10T20:38:12.407874+0000","last_active":"2024-09-10T20:38:13.400878+0000","last_peered":"2024-09-10T20:38:13.400878+0000","last_clean":"2024-09-10T20:38:13.400878+0000","last_became_active":"2024-09-10T20:38:12.407623+0000","last_became_peered":"2024-09-10T20:38:12.407623+0000","last_unstale":"2024-09-10T20:38:13.400878+0000","last_undegraded":"2024-09-10T20:38:13.400878+0000","last_fullsized":"2024-09-10T20:38:13.400878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:56.990230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400518+0000","last_change":"2024-09-10T20:38:12.408020+0000","last_active":"2024-09-10T20:38:13.400518+0000","last_peered":"2024-09-10T20:38:13.400518+0000","last_clean":"2024-09-10T20:38:13.400518+0000","last_became_active":"2024-09-10T20:38:12.407757+0000","last_became_peered":"2024-09-10T20:38:12.407757+0000","last_unstale":"2024-09-10T20:38:13.400518+0000","last_undegraded":"2024-09-10T20:38:13.400518+0000","last_fullsized":"2024-09-10T20:38:13.400518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:32.513086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401654+0000","last_change":"2024-09-10T20:38:12.416773+0000","last_active":"2024-09-10T20:38:13.401654+0000","last_peered":"2024-09-10T20:38:13.401654+0000","last_clean":"2024-09-10T20:38:13.401654+0000","last_became_active":"2024-09-10T20:38:12.416498+0000","last_became_peered":"2024-09-10T20:38:12.416498+0000","last_unstale":"2024-09-10T20:38:13.401654+0000","last_undegraded":"2024-09-10T20:38:13.401654+0000","last_fullsized":"2024-09-10T20:38:13.401654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:42.492324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343671+0000","last_change":"2024-09-10T20:38:12.406673+0000","last_active":"2024-09-10T20:38:14.343671+0000","last_peered":"2024-09-10T20:38:14.343671+0000","last_clean":"2024-09-10T20:38:14.343671+0000","last_became_active":"2024-09-10T20:38:12.406401+0000","last_became_peered":"2024-09-10T20:38:12.406401+0000","last_unstale":"2024-09-10T20:38:14.343671+0000","last_undegraded":"2024-09-10T20:38:14.343671+0000","last_fullsized":"2024-09-10T20:38:14.343671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:31.845414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343502+0000","last_change":"2024-09-10T20:38:12.414290+0000","last_active":"2024-09-10T20:38:14.343502+0000","last_peered":"2024-09-10T20:38:14.343502+0000","last_clean":"2024-09-10T20:38:14.343502+0000","last_became_active":"2024-09-10T20:38:12.413712+0000","last_became_peered":"2024-09-10T20:38:12.413712+0000","last_unstale":"2024-09-10T20:38:14.343502+0000","last_undegraded":"2024-09-10T20:38:14.343502+0000","last_fullsized":"2024-09-10T20:38:14.343502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:48.033101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400477+0000","last_change":"2024-09-10T20:38:12.406921+0000","last_active":"2024-09-10T20:38:13.400477+0000","last_peered":"2024-09-10T20:38:13.400477+0000","last_clean":"2024-09-10T20:38:13.400477+0000","last_became_active":"2024-09-10T20:38:12.406642+0000","last_became_peered":"2024-09-10T20:38:12.406642+0000","last_unstale":"2024-09-10T20:38:13.400477+0000","last_undegraded":"2024-09-10T20:38:13.400477+0000","last_fullsized":"2024-09-10T20:38:13.400477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:02.877370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633368+0000","last_change":"2024-09-10T20:38:12.415776+0000","last_active":"2024-09-10T20:38:13.633368+0000","last_peered":"2024-09-10T20:38:13.633368+0000","last_clean":"2024-09-10T20:38:13.633368+0000","last_became_active":"2024-09-10T20:38:12.415480+0000","last_became_peered":"2024-09-10T20:38:12.415480+0000","last_unstale":"2024-09-10T20:38:13.633368+0000","last_undegraded":"2024-09-10T20:38:13.633368+0000","last_fullsized":"2024-09-10T20:38:13.633368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:13.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400454+0000","last_change":"2024-09-10T20:38:11.394869+0000","last_active":"2024-09-10T20:38:13.400454+0000","last_peered":"2024-09-10T20:38:13.400454+0000","last_clean":"2024-09-10T20:38:13.400454+0000","last_became_active":"2024-09-10T20:38:11.394564+0000","last_became_peered":"2024-09-10T20:38:11.394564+0000","last_unstale":"2024-09-10T20:38:13.400454+0000","last_undegraded":"2024-09-10T20:38:13.400454+0000","last_fullsized":"2024-09-10T20:38:13.400454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:19.365550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401125+0000","last_change":"2024-09-10T20:38:12.401521+0000","last_active":"2024-09-10T20:38:13.401125+0000","last_peered":"2024-09-10T20:38:13.401125+0000","last_clean":"2024-09-10T20:38:13.401125+0000","last_became_active":"2024-09-10T20:38:12.401218+0000","last_became_peered":"2024-09-10T20:38:12.401218+0000","last_unstale":"2024-09-10T20:38:13.401125+0000","last_undegraded":"2024-09-10T20:38:13.401125+0000","last_fullsized":"2024-09-10T20:38:13.401125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:33.954998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378820+0000","last_change":"2024-09-10T20:38:10.378820+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378820+0000","last_undegraded":"2024-09-10T20:38:10.378820+0000","last_fullsized":"2024-09-10T20:38:10.378820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343167+0000","last_change":"2024-09-10T20:38:11.398400+0000","last_active":"2024-09-10T20:38:14.343167+0000","last_peered":"2024-09-10T20:38:14.343167+0000","last_clean":"2024-09-10T20:38:14.343167+0000","last_became_active":"2024-09-10T20:38:11.398097+0000","last_became_peered":"2024-09-10T20:38:11.398097+0000","last_unstale":"2024-09-10T20:38:14.343167+0000","last_undegraded":"2024-09-10T20:38:14.343167+0000","last_fullsized":"2024-09-10T20:38:14.343167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.599838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401478+0000","last_change":"2024-09-10T20:38:12.413177+0000","last_active":"2024-09-10T20:38:13.401478+0000","last_peered":"2024-09-10T20:38:13.401478+0000","last_clean":"2024-09-10T20:38:13.401478+0000","last_became_active":"2024-09-10T20:38:12.412925+0000","last_became_peered":"2024-09-10T20:38:12.412925+0000","last_unstale":"2024-09-10T20:38:13.401478+0000","last_undegraded":"2024-09-10T20:38:13.401478+0000","last_fullsized":"2024-09-10T20:38:13.401478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:18.693832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378650+0000","last_change":"2024-09-10T20:38:10.378650+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378650+0000","last_undegraded":"2024-09-10T20:38:10.378650+0000","last_fullsized":"2024-09-10T20:38:10.378650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401415+0000","last_change":"2024-09-10T20:38:12.415606+0000","last_active":"2024-09-10T20:38:13.401415+0000","last_peered":"2024-09-10T20:38:13.401415+0000","last_clean":"2024-09-10T20:38:13.401415+0000","last_became_active":"2024-09-10T20:38:12.415330+0000","last_became_peered":"2024-09-10T20:38:12.415330+0000","last_unstale":"2024-09-10T20:38:13.401415+0000","last_undegraded":"2024-09-10T20:38:13.401415+0000","last_fullsized":"2024-09-10T20:38:13.401415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:53.050014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380287+0000","last_change":"2024-09-10T20:38:10.380287+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380287+0000","last_undegraded":"2024-09-10T20:38:10.380287+0000","last_fullsized":"2024-09-10T20:38:10.380287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380912+0000","last_change":"2024-09-10T20:38:10.380912+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380912+0000","last_undegraded":"2024-09-10T20:38:10.380912+0000","last_fullsized":"2024-09-10T20:38:10.380912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401902+0000","last_change":"2024-09-10T20:38:12.410504+0000","last_active":"2024-09-10T20:38:13.401902+0000","last_peered":"2024-09-10T20:38:13.401902+0000","last_clean":"2024-09-10T20:38:13.401902+0000","last_became_active":"2024-09-10T20:38:12.410008+0000","last_became_peered":"2024-09-10T20:38:12.410008+0000","last_unstale":"2024-09-10T20:38:13.401902+0000","last_undegraded":"2024-09-10T20:38:13.401902+0000","last_fullsized":"2024-09-10T20:38:13.401902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:57.544501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633468+0000","last_change":"2024-09-10T20:38:11.395962+0000","last_active":"2024-09-10T20:38:13.633468+0000","last_peered":"2024-09-10T20:38:13.633468+0000","last_clean":"2024-09-10T20:38:13.633468+0000","last_became_active":"2024-09-10T20:38:11.395687+0000","last_became_peered":"2024-09-10T20:38:11.395687+0000","last_unstale":"2024-09-10T20:38:13.633468+0000","last_undegraded":"2024-09-10T20:38:13.633468+0000","last_fullsized":"2024-09-10T20:38:13.633468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:15.421827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380034+0000","last_change":"2024-09-10T20:38:10.380034+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380034+0000","last_undegraded":"2024-09-10T20:38:10.380034+0000","last_fullsized":"2024-09-10T20:38:10.380034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633539+0000","last_change":"2024-09-10T20:38:11.398330+0000","last_active":"2024-09-10T20:38:13.633539+0000","last_peered":"2024-09-10T20:38:13.633539+0000","last_clean":"2024-09-10T20:38:13.633539+0000","last_became_active":"2024-09-10T20:38:11.398059+0000","last_became_peered":"2024-09-10T20:38:11.398059+0000","last_unstale":"2024-09-10T20:38:13.633539+0000","last_undegraded":"2024-09-10T20:38:13.633539+0000","last_fullsized":"2024-09-10T20:38:13.633539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343446+0000","last_change":"2024-09-10T20:38:12.412484+0000","last_active":"2024-09-10T20:38:14.343446+0000","last_peered":"2024-09-10T20:38:14.343446+0000","last_clean":"2024-09-10T20:38:14.343446+0000","last_became_active":"2024-09-10T20:38:12.412165+0000","last_became_peered":"2024-09-10T20:38:12.412165+0000","last_unstale":"2024-09-10T20:38:14.343446+0000","last_undegraded":"2024-09-10T20:38:14.343446+0000","last_fullsized":"2024-09-10T20:38:14.343446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:09.972659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378166+0000","last_change":"2024-09-10T20:38:10.378166+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378166+0000","last_undegraded":"2024-09-10T20:38:10.378166+0000","last_fullsized":"2024-09-10T20:38:10.378166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380952+0000","last_change":"2024-09-10T20:38:10.380952+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380952+0000","last_undegraded":"2024-09-10T20:38:10.380952+0000","last_fullsized":"2024-09-10T20:38:10.380952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400564+0000","last_change":"2024-09-10T20:38:11.393414+0000","last_active":"2024-09-10T20:38:13.400564+0000","last_peered":"2024-09-10T20:38:13.400564+0000","last_clean":"2024-09-10T20:38:13.400564+0000","last_became_active":"2024-09-10T20:38:11.393078+0000","last_became_peered":"2024-09-10T20:38:11.393078+0000","last_unstale":"2024-09-10T20:38:13.400564+0000","last_undegraded":"2024-09-10T20:38:13.400564+0000","last_fullsized":"2024-09-10T20:38:13.400564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:58.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381915+0000","last_change":"2024-09-10T20:38:10.381915+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381915+0000","last_undegraded":"2024-09-10T20:38:10.381915+0000","last_fullsized":"2024-09-10T20:38:10.381915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401433+0000","last_change":"2024-09-10T20:38:11.388247+0000","last_active":"2024-09-10T20:38:13.401433+0000","last_peered":"2024-09-10T20:38:13.401433+0000","last_clean":"2024-09-10T20:38:13.401433+0000","last_became_active":"2024-09-10T20:38:11.387924+0000","last_became_peered":"2024-09-10T20:38:11.387924+0000","last_unstale":"2024-09-10T20:38:13.401433+0000","last_undegraded":"2024-09-10T20:38:13.401433+0000","last_fullsized":"2024-09-10T20:38:13.401433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:29.584118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400616+0000","last_change":"2024-09-10T20:38:11.399015+0000","last_active":"2024-09-10T20:38:13.400616+0000","last_peered":"2024-09-10T20:38:13.400616+0000","last_clean":"2024-09-10T20:38:13.400616+0000","last_became_active":"2024-09-10T20:38:11.398826+0000","last_became_peered":"2024-09-10T20:38:11.398826+0000","last_unstale":"2024-09-10T20:38:13.400616+0000","last_undegraded":"2024-09-10T20:38:13.400616+0000","last_fullsized":"2024-09-10T20:38:13.400616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:03.958637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343682+0000","last_change":"2024-09-10T20:38:11.398337+0000","last_active":"2024-09-10T20:38:14.343682+0000","last_peered":"2024-09-10T20:38:14.343682+0000","last_clean":"2024-09-10T20:38:14.343682+0000","last_became_active":"2024-09-10T20:38:11.398068+0000","last_became_peered":"2024-09-10T20:38:11.398068+0000","last_unstale":"2024-09-10T20:38:14.343682+0000","last_undegraded":"2024-09-10T20:38:14.343682+0000","last_fullsized":"2024-09-10T20:38:14.343682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:00:28.236504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":105,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-10T20:38:10.380805+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:10.380805+0000","last_peered":"2024-09-10T20:38:10.380805+0000","last_clean":"2024-09-10T20:38:10.380805+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:10.380805+0000","last_undegraded":"2024-09-10T20:38:10.380805+0000","last_fullsized":"2024-09-10T20:38:10.380805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377085+0000","last_change":"2024-09-10T20:38:10.377085+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377085+0000","last_undegraded":"2024-09-10T20:38:10.377085+0000","last_fullsized":"2024-09-10T20:38:10.377085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400902+0000","last_change":"2024-09-10T20:38:12.407263+0000","last_active":"2024-09-10T20:38:13.400902+0000","last_peered":"2024-09-10T20:38:13.400902+0000","last_clean":"2024-09-10T20:38:13.400902+0000","last_became_active":"2024-09-10T20:38:12.406952+0000","last_became_peered":"2024-09-10T20:38:12.406952+0000","last_unstale":"2024-09-10T20:38:13.400902+0000","last_undegraded":"2024-09-10T20:38:13.400902+0000","last_fullsized":"2024-09-10T20:38:13.400902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.171100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.376990+0000","last_change":"2024-09-10T20:38:10.376990+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.376990+0000","last_undegraded":"2024-09-10T20:38:10.376990+0000","last_fullsized":"2024-09-10T20:38:10.376990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343358+0000","last_change":"2024-09-10T20:38:12.407025+0000","last_active":"2024-09-10T20:38:14.343358+0000","last_peered":"2024-09-10T20:38:14.343358+0000","last_clean":"2024-09-10T20:38:14.343358+0000","last_became_active":"2024-09-10T20:38:12.406124+0000","last_became_peered":"2024-09-10T20:38:12.406124+0000","last_unstale":"2024-09-10T20:38:14.343358+0000","last_undegraded":"2024-09-10T20:38:14.343358+0000","last_fullsized":"2024-09-10T20:38:14.343358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633299+0000","last_change":"2024-09-10T20:38:11.397217+0000","last_active":"2024-09-10T20:38:13.633299+0000","last_peered":"2024-09-10T20:38:13.633299+0000","last_clean":"2024-09-10T20:38:13.633299+0000","last_became_active":"2024-09-10T20:38:11.396928+0000","last_became_peered":"2024-09-10T20:38:11.396928+0000","last_unstale":"2024-09-10T20:38:13.633299+0000","last_undegraded":"2024-09-10T20:38:13.633299+0000","last_fullsized":"2024-09-10T20:38:13.633299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:38.900087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401469+0000","last_change":"2024-09-10T20:38:12.407595+0000","last_active":"2024-09-10T20:38:13.401469+0000","last_peered":"2024-09-10T20:38:13.401469+0000","last_clean":"2024-09-10T20:38:13.401469+0000","last_became_active":"2024-09-10T20:38:12.407355+0000","last_became_peered":"2024-09-10T20:38:12.407355+0000","last_unstale":"2024-09-10T20:38:13.401469+0000","last_undegraded":"2024-09-10T20:38:13.401469+0000","last_fullsized":"2024-09-10T20:38:13.401469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:05.419998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377007+0000","last_change":"2024-09-10T20:38:10.377007+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377007+0000","last_undegraded":"2024-09-10T20:38:10.377007+0000","last_fullsized":"2024-09-10T20:38:10.377007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400255+0000","last_change":"2024-09-10T20:38:11.397463+0000","last_active":"2024-09-10T20:38:13.400255+0000","last_peered":"2024-09-10T20:38:13.400255+0000","last_clean":"2024-09-10T20:38:13.400255+0000","last_became_active":"2024-09-10T20:38:11.397205+0000","last_became_peered":"2024-09-10T20:38:11.397205+0000","last_unstale":"2024-09-10T20:38:13.400255+0000","last_undegraded":"2024-09-10T20:38:13.400255+0000","last_fullsized":"2024-09-10T20:38:13.400255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:53.003074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377042+0000","last_change":"2024-09-10T20:38:10.377042+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377042+0000","last_undegraded":"2024-09-10T20:38:10.377042+0000","last_fullsized":"2024-09-10T20:38:10.377042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343576+0000","last_change":"2024-09-10T20:38:12.408779+0000","last_active":"2024-09-10T20:38:14.343576+0000","last_peered":"2024-09-10T20:38:14.343576+0000","last_clean":"2024-09-10T20:38:14.343576+0000","last_became_active":"2024-09-10T20:38:12.408490+0000","last_became_peered":"2024-09-10T20:38:12.408490+0000","last_unstale":"2024-09-10T20:38:14.343576+0000","last_undegraded":"2024-09-10T20:38:14.343576+0000","last_fullsized":"2024-09-10T20:38:14.343576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:25.888623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407174+0000","last_change":"2024-09-10T20:38:11.394170+0000","last_active":"2024-09-10T20:38:13.407174+0000","last_peered":"2024-09-10T20:38:13.407174+0000","last_clean":"2024-09-10T20:38:13.407174+0000","last_became_active":"2024-09-10T20:38:11.393850+0000","last_became_peered":"2024-09-10T20:38:11.393850+0000","last_unstale":"2024-09-10T20:38:13.407174+0000","last_undegraded":"2024-09-10T20:38:13.407174+0000","last_fullsized":"2024-09-10T20:38:13.407174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:15.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401623+0000","last_change":"2024-09-10T20:38:12.404386+0000","last_active":"2024-09-10T20:38:13.401623+0000","last_peered":"2024-09-10T20:38:13.401623+0000","last_clean":"2024-09-10T20:38:13.401623+0000","last_became_active":"2024-09-10T20:38:12.404045+0000","last_became_peered":"2024-09-10T20:38:12.404045+0000","last_unstale":"2024-09-10T20:38:13.401623+0000","last_undegraded":"2024-09-10T20:38:13.401623+0000","last_fullsized":"2024-09-10T20:38:13.401623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:49.410849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377015+0000","last_change":"2024-09-10T20:38:10.377015+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377015+0000","last_undegraded":"2024-09-10T20:38:10.377015+0000","last_fullsized":"2024-09-10T20:38:10.377015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400970+0000","last_change":"2024-09-10T20:38:12.409091+0000","last_active":"2024-09-10T20:38:13.400970+0000","last_peered":"2024-09-10T20:38:13.400970+0000","last_clean":"2024-09-10T20:38:13.400970+0000","last_became_active":"2024-09-10T20:38:12.408815+0000","last_became_peered":"2024-09-10T20:38:12.408815+0000","last_unstale":"2024-09-10T20:38:13.400970+0000","last_undegraded":"2024-09-10T20:38:13.400970+0000","last_fullsized":"2024-09-10T20:38:13.400970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.870149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.382686+0000","last_change":"2024-09-10T20:38:10.382686+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.382686+0000","last_undegraded":"2024-09-10T20:38:10.382686+0000","last_fullsized":"2024-09-10T20:38:10.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401287+0000","last_change":"2024-09-10T20:38:11.397172+0000","last_active":"2024-09-10T20:38:13.401287+0000","last_peered":"2024-09-10T20:38:13.401287+0000","last_clean":"2024-09-10T20:38:13.401287+0000","last_became_active":"2024-09-10T20:38:11.396670+0000","last_became_peered":"2024-09-10T20:38:11.396670+0000","last_unstale":"2024-09-10T20:38:13.401287+0000","last_undegraded":"2024-09-10T20:38:13.401287+0000","last_fullsized":"2024-09-10T20:38:13.401287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400195+0000","last_change":"2024-09-10T20:38:12.405875+0000","last_active":"2024-09-10T20:38:13.400195+0000","last_peered":"2024-09-10T20:38:13.400195+0000","last_clean":"2024-09-10T20:38:13.400195+0000","last_became_active":"2024-09-10T20:38:12.405583+0000","last_became_peered":"2024-09-10T20:38:12.405583+0000","last_unstale":"2024-09-10T20:38:13.400195+0000","last_undegraded":"2024-09-10T20:38:13.400195+0000","last_fullsized":"2024-09-10T20:38:13.400195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:30.185620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343471+0000","last_change":"2024-09-10T20:38:11.395992+0000","last_active":"2024-09-10T20:38:14.343471+0000","last_peered":"2024-09-10T20:38:14.343471+0000","last_clean":"2024-09-10T20:38:14.343471+0000","last_became_active":"2024-09-10T20:38:11.394935+0000","last_became_peered":"2024-09-10T20:38:11.394935+0000","last_unstale":"2024-09-10T20:38:14.343471+0000","last_undegraded":"2024-09-10T20:38:14.343471+0000","last_fullsized":"2024-09-10T20:38:14.343471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:49.040222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633574+0000","last_change":"2024-09-10T20:38:12.412620+0000","last_active":"2024-09-10T20:38:13.633574+0000","last_peered":"2024-09-10T20:38:13.633574+0000","last_clean":"2024-09-10T20:38:13.633574+0000","last_became_active":"2024-09-10T20:38:12.412329+0000","last_became_peered":"2024-09-10T20:38:12.412329+0000","last_unstale":"2024-09-10T20:38:13.633574+0000","last_undegraded":"2024-09-10T20:38:13.633574+0000","last_fullsized":"2024-09-10T20:38:13.633574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:40.609805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377791+0000","last_change":"2024-09-10T20:38:10.377791+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377791+0000","last_undegraded":"2024-09-10T20:38:10.377791+0000","last_fullsized":"2024-09-10T20:38:10.377791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381267+0000","last_change":"2024-09-10T20:38:10.381267+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381267+0000","last_undegraded":"2024-09-10T20:38:10.381267+0000","last_fullsized":"2024-09-10T20:38:10.381267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381405+0000","last_change":"2024-09-10T20:38:10.381405+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381405+0000","last_undegraded":"2024-09-10T20:38:10.381405+0000","last_fullsized":"2024-09-10T20:38:10.381405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401905+0000","last_change":"2024-09-10T20:38:12.416588+0000","last_active":"2024-09-10T20:38:13.401905+0000","last_peered":"2024-09-10T20:38:13.401905+0000","last_clean":"2024-09-10T20:38:13.401905+0000","last_became_active":"2024-09-10T20:38:12.416297+0000","last_became_peered":"2024-09-10T20:38:12.416297+0000","last_unstale":"2024-09-10T20:38:13.401905+0000","last_undegraded":"2024-09-10T20:38:13.401905+0000","last_fullsized":"2024-09-10T20:38:13.401905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401853+0000","last_change":"2024-09-10T20:38:11.396382+0000","last_active":"2024-09-10T20:38:13.401853+0000","last_peered":"2024-09-10T20:38:13.401853+0000","last_clean":"2024-09-10T20:38:13.401853+0000","last_became_active":"2024-09-10T20:38:11.395934+0000","last_became_peered":"2024-09-10T20:38:11.395934+0000","last_unstale":"2024-09-10T20:38:13.401853+0000","last_undegraded":"2024-09-10T20:38:13.401853+0000","last_fullsized":"2024-09-10T20:38:13.401853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:20:39.553068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401634+0000","last_change":"2024-09-10T20:38:12.402624+0000","last_active":"2024-09-10T20:38:13.401634+0000","last_peered":"2024-09-10T20:38:13.401634+0000","last_clean":"2024-09-10T20:38:13.401634+0000","last_became_active":"2024-09-10T20:38:12.401924+0000","last_became_peered":"2024-09-10T20:38:12.401924+0000","last_unstale":"2024-09-10T20:38:13.401634+0000","last_undegraded":"2024-09-10T20:38:13.401634+0000","last_fullsized":"2024-09-10T20:38:13.401634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:33.683744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401708+0000","last_change":"2024-09-10T20:38:11.401264+0000","last_active":"2024-09-10T20:38:13.401708+0000","last_peered":"2024-09-10T20:38:13.401708+0000","last_clean":"2024-09-10T20:38:13.401708+0000","last_became_active":"2024-09-10T20:38:11.400986+0000","last_became_peered":"2024-09-10T20:38:11.400986+0000","last_unstale":"2024-09-10T20:38:13.401708+0000","last_undegraded":"2024-09-10T20:38:13.401708+0000","last_fullsized":"2024-09-10T20:38:13.401708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:50.668458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":172032,"data_stored":53177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":753664,"data_stored":637728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":66227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":662987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":74811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:16.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:16.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 -- 172.21.15.28:0/710835267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 msgr2=0x7f0bac0cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/710835267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac0cfbf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0bac08c020 tx=0x7f0ba002a560 comp rx=0 tx=0).stop 2024-09-10T20:38:16.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 -- 172.21.15.28:0/710835267 shutdown_connections 2024-09-10T20:38:16.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/710835267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac0cfbf0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/710835267 >> v2:172.21.15.159:3300/0 conn(0x7f0bac15de70 0x7f0bac0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 -- 172.21.15.28:0/710835267 >> 172.21.15.28:0/710835267 conn(0x7f0bac0d3fb0 msgr2=0x7f0bac0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:16.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 -- 172.21.15.28:0/710835267 shutdown_connections 2024-09-10T20:38:16.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.912+0000 7f0bb2b90640 1 -- 172.21.15.28:0/710835267 wait complete. 2024-09-10T20:38:16.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.913+0000 7f0bb2b90640 1 Processor -- start 2024-09-10T20:38:16.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.914+0000 7f0bb2b90640 1 -- start start 2024-09-10T20:38:16.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.914+0000 7f0bb2b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac1f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.914+0000 7f0bb2b90640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bac15de70 0x7f0bac0b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.914+0000 7f0bb2b90640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bac161b50 0x7f0bac0b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.914+0000 7f0bb2b90640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac0b7690 con 0x7f0bac0cf780 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.914+0000 7f0bb2b90640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac0b7510 con 0x7f0bac15de70 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.914+0000 7f0bb2b90640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0bac0b7810 con 0x7f0bac161b50 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac1f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac1f53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35164/0 (socket says 172.21.15.28:35164) 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 -- 172.21.15.28:0/2983278319 learned_addr learned my addr 172.21.15.28:0/2983278319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb112d640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bac161b50 0x7f0bac0b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0babfff640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bac15de70 0x7f0bac0b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 -- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bac161b50 msgr2=0x7f0bac0b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bac161b50 0x7f0bac0b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 -- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bac15de70 msgr2=0x7f0bac0b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bac15de70 0x7f0bac0b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 -- 172.21.15.28:0/2983278319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac1607c0 con 0x7f0bac0cf780 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb112d640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bac161b50 0x7f0bac0b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0babfff640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bac15de70 0x7f0bac0b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb092c640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac1f53c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00d770 tx=0x7f0b9c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0ba97fa640 1 -- 172.21.15.28:0/2983278319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c017070 con 0x7f0bac0cf780 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0ba97fa640 1 -- 172.21.15.28:0/2983278319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b9c009d20 con 0x7f0bac0cf780 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bac0b17f0 con 0x7f0bac0cf780 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0ba97fa640 1 -- 172.21.15.28:0/2983278319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c005280 con 0x7f0bac0cf780 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.915+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bac0b1510 con 0x7f0bac0cf780 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.917+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bac0cfbf0 con 0x7f0bac0cf780 2024-09-10T20:38:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.917+0000 7f0ba97fa640 1 -- 172.21.15.28:0/2983278319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0b9c005420 con 0x7f0bac0cf780 2024-09-10T20:38:16.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.917+0000 7f0ba97fa640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b98077f70 0x7f0b9807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:16.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.917+0000 7f0ba97fa640 1 -- 172.21.15.28:0/2983278319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0b9c099fc0 con 0x7f0bac0cf780 2024-09-10T20:38:16.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.918+0000 7f0babfff640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b98077f70 0x7f0b9807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:16.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.918+0000 7f0babfff640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b98077f70 0x7f0b9807a430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0033000 tx=0x7f0ba0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:16.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:16.921+0000 7f0ba97fa640 1 -- 172.21.15.28:0/2983278319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0b9c062ad0 con 0x7f0bac0cf780 2024-09-10T20:38:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.106+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0bac0b9f30 con 0x7f0b98077f70 2024-09-10T20:38:17.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.111+0000 7f0ba97fa640 1 -- 172.21.15.28:0/2983278319 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+245574 (secure 0 0 0) 0x7f0bac0b9f30 con 0x7f0b98077f70 2024-09-10T20:38:17.119 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:17.120 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:17.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b98077f70 msgr2=0x7f0b9807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b98077f70 0x7f0b9807a430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0033000 tx=0x7f0ba0004730 comp rx=0 tx=0).stop 2024-09-10T20:38:17.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 msgr2=0x7f0bac1f53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac1f53c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00d770 tx=0x7f0b9c00dc30 comp rx=0 tx=0).stop 2024-09-10T20:38:17.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 shutdown_connections 2024-09-10T20:38:17.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b98077f70 0x7f0b9807a430 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0bac161b50 0x7f0bac0b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0bac15de70 0x7f0bac0b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.113+0000 7f0bb2b90640 1 --2- 172.21.15.28:0/2983278319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bac0cf780 0x7f0bac1f53c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.114+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 >> 172.21.15.28:0/2983278319 conn(0x7f0bac0d3fb0 msgr2=0x7f0bac075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:17.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.114+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 shutdown_connections 2024-09-10T20:38:17.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.114+0000 7f0bb2b90640 1 -- 172.21.15.28:0/2983278319 wait complete. 2024-09-10T20:38:17.129 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:38:15.976782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591144,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":129,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":124,"acting":124,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235700,"kb_used_data":2636,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739020,"statfs":{"total":773094113280,"available":772852756480,"internally_reserved":0,"allocated":2699264,"data_stored":1662301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.603193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400675+0000","last_change":"2024-09-10T20:38:12.413418+0000","last_active":"2024-09-10T20:38:13.400675+0000","last_peered":"2024-09-10T20:38:13.400675+0000","last_clean":"2024-09-10T20:38:13.400675+0000","last_became_active":"2024-09-10T20:38:12.413063+0000","last_became_peered":"2024-09-10T20:38:12.413063+0000","last_unstale":"2024-09-10T20:38:13.400675+0000","last_undegraded":"2024-09-10T20:38:13.400675+0000","last_fullsized":"2024-09-10T20:38:13.400675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:12.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401706+0000","last_change":"2024-09-10T20:38:12.404482+0000","last_active":"2024-09-10T20:38:13.401706+0000","last_peered":"2024-09-10T20:38:13.401706+0000","last_clean":"2024-09-10T20:38:13.401706+0000","last_became_active":"2024-09-10T20:38:12.404181+0000","last_became_peered":"2024-09-10T20:38:12.404181+0000","last_unstale":"2024-09-10T20:38:13.401706+0000","last_undegraded":"2024-09-10T20:38:13.401706+0000","last_fullsized":"2024-09-10T20:38:13.401706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:02.847206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401300+0000","last_change":"2024-09-10T20:38:12.403203+0000","last_active":"2024-09-10T20:38:13.401300+0000","last_peered":"2024-09-10T20:38:13.401300+0000","last_clean":"2024-09-10T20:38:13.401300+0000","last_became_active":"2024-09-10T20:38:12.402272+0000","last_became_peered":"2024-09-10T20:38:12.402272+0000","last_unstale":"2024-09-10T20:38:13.401300+0000","last_undegraded":"2024-09-10T20:38:13.401300+0000","last_fullsized":"2024-09-10T20:38:13.401300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343562+0000","last_change":"2024-09-10T20:38:12.415360+0000","last_active":"2024-09-10T20:38:14.343562+0000","last_peered":"2024-09-10T20:38:14.343562+0000","last_clean":"2024-09-10T20:38:14.343562+0000","last_became_active":"2024-09-10T20:38:12.415094+0000","last_became_peered":"2024-09-10T20:38:12.415094+0000","last_unstale":"2024-09-10T20:38:14.343562+0000","last_undegraded":"2024-09-10T20:38:14.343562+0000","last_fullsized":"2024-09-10T20:38:14.343562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:36.623506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343324+0000","last_change":"2024-09-10T20:38:12.411600+0000","last_active":"2024-09-10T20:38:14.343324+0000","last_peered":"2024-09-10T20:38:14.343324+0000","last_clean":"2024-09-10T20:38:14.343324+0000","last_became_active":"2024-09-10T20:38:12.411328+0000","last_became_peered":"2024-09-10T20:38:12.411328+0000","last_unstale":"2024-09-10T20:38:14.343324+0000","last_undegraded":"2024-09-10T20:38:14.343324+0000","last_fullsized":"2024-09-10T20:38:14.343324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:30.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400233+0000","last_change":"2024-09-10T20:38:12.404816+0000","last_active":"2024-09-10T20:38:13.400233+0000","last_peered":"2024-09-10T20:38:13.400233+0000","last_clean":"2024-09-10T20:38:13.400233+0000","last_became_active":"2024-09-10T20:38:12.404568+0000","last_became_peered":"2024-09-10T20:38:12.404568+0000","last_unstale":"2024-09-10T20:38:13.400233+0000","last_undegraded":"2024-09-10T20:38:13.400233+0000","last_fullsized":"2024-09-10T20:38:13.400233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:05:49.714066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633628+0000","last_change":"2024-09-10T20:38:12.407265+0000","last_active":"2024-09-10T20:38:13.633628+0000","last_peered":"2024-09-10T20:38:13.633628+0000","last_clean":"2024-09-10T20:38:13.633628+0000","last_became_active":"2024-09-10T20:38:12.406927+0000","last_became_peered":"2024-09-10T20:38:12.406927+0000","last_unstale":"2024-09-10T20:38:13.633628+0000","last_undegraded":"2024-09-10T20:38:13.633628+0000","last_fullsized":"2024-09-10T20:38:13.633628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:14.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400878+0000","last_change":"2024-09-10T20:38:12.407874+0000","last_active":"2024-09-10T20:38:13.400878+0000","last_peered":"2024-09-10T20:38:13.400878+0000","last_clean":"2024-09-10T20:38:13.400878+0000","last_became_active":"2024-09-10T20:38:12.407623+0000","last_became_peered":"2024-09-10T20:38:12.407623+0000","last_unstale":"2024-09-10T20:38:13.400878+0000","last_undegraded":"2024-09-10T20:38:13.400878+0000","last_fullsized":"2024-09-10T20:38:13.400878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:56.990230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400518+0000","last_change":"2024-09-10T20:38:12.408020+0000","last_active":"2024-09-10T20:38:13.400518+0000","last_peered":"2024-09-10T20:38:13.400518+0000","last_clean":"2024-09-10T20:38:13.400518+0000","last_became_active":"2024-09-10T20:38:12.407757+0000","last_became_peered":"2024-09-10T20:38:12.407757+0000","last_unstale":"2024-09-10T20:38:13.400518+0000","last_undegraded":"2024-09-10T20:38:13.400518+0000","last_fullsized":"2024-09-10T20:38:13.400518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:32.513086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401654+0000","last_change":"2024-09-10T20:38:12.416773+0000","last_active":"2024-09-10T20:38:13.401654+0000","last_peered":"2024-09-10T20:38:13.401654+0000","last_clean":"2024-09-10T20:38:13.401654+0000","last_became_active":"2024-09-10T20:38:12.416498+0000","last_became_peered":"2024-09-10T20:38:12.416498+0000","last_unstale":"2024-09-10T20:38:13.401654+0000","last_undegraded":"2024-09-10T20:38:13.401654+0000","last_fullsized":"2024-09-10T20:38:13.401654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:42.492324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343671+0000","last_change":"2024-09-10T20:38:12.406673+0000","last_active":"2024-09-10T20:38:14.343671+0000","last_peered":"2024-09-10T20:38:14.343671+0000","last_clean":"2024-09-10T20:38:14.343671+0000","last_became_active":"2024-09-10T20:38:12.406401+0000","last_became_peered":"2024-09-10T20:38:12.406401+0000","last_unstale":"2024-09-10T20:38:14.343671+0000","last_undegraded":"2024-09-10T20:38:14.343671+0000","last_fullsized":"2024-09-10T20:38:14.343671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:31.845414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343502+0000","last_change":"2024-09-10T20:38:12.414290+0000","last_active":"2024-09-10T20:38:14.343502+0000","last_peered":"2024-09-10T20:38:14.343502+0000","last_clean":"2024-09-10T20:38:14.343502+0000","last_became_active":"2024-09-10T20:38:12.413712+0000","last_became_peered":"2024-09-10T20:38:12.413712+0000","last_unstale":"2024-09-10T20:38:14.343502+0000","last_undegraded":"2024-09-10T20:38:14.343502+0000","last_fullsized":"2024-09-10T20:38:14.343502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:48.033101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400477+0000","last_change":"2024-09-10T20:38:12.406921+0000","last_active":"2024-09-10T20:38:13.400477+0000","last_peered":"2024-09-10T20:38:13.400477+0000","last_clean":"2024-09-10T20:38:13.400477+0000","last_became_active":"2024-09-10T20:38:12.406642+0000","last_became_peered":"2024-09-10T20:38:12.406642+0000","last_unstale":"2024-09-10T20:38:13.400477+0000","last_undegraded":"2024-09-10T20:38:13.400477+0000","last_fullsized":"2024-09-10T20:38:13.400477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:02.877370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633368+0000","last_change":"2024-09-10T20:38:12.415776+0000","last_active":"2024-09-10T20:38:13.633368+0000","last_peered":"2024-09-10T20:38:13.633368+0000","last_clean":"2024-09-10T20:38:13.633368+0000","last_became_active":"2024-09-10T20:38:12.415480+0000","last_became_peered":"2024-09-10T20:38:12.415480+0000","last_unstale":"2024-09-10T20:38:13.633368+0000","last_undegraded":"2024-09-10T20:38:13.633368+0000","last_fullsized":"2024-09-10T20:38:13.633368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:13.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400454+0000","last_change":"2024-09-10T20:38:11.394869+0000","last_active":"2024-09-10T20:38:13.400454+0000","last_peered":"2024-09-10T20:38:13.400454+0000","last_clean":"2024-09-10T20:38:13.400454+0000","last_became_active":"2024-09-10T20:38:11.394564+0000","last_became_peered":"2024-09-10T20:38:11.394564+0000","last_unstale":"2024-09-10T20:38:13.400454+0000","last_undegraded":"2024-09-10T20:38:13.400454+0000","last_fullsized":"2024-09-10T20:38:13.400454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:19.365550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401125+0000","last_change":"2024-09-10T20:38:12.401521+0000","last_active":"2024-09-10T20:38:13.401125+0000","last_peered":"2024-09-10T20:38:13.401125+0000","last_clean":"2024-09-10T20:38:13.401125+0000","last_became_active":"2024-09-10T20:38:12.401218+0000","last_became_peered":"2024-09-10T20:38:12.401218+0000","last_unstale":"2024-09-10T20:38:13.401125+0000","last_undegraded":"2024-09-10T20:38:13.401125+0000","last_fullsized":"2024-09-10T20:38:13.401125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:33.954998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378820+0000","last_change":"2024-09-10T20:38:10.378820+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378820+0000","last_undegraded":"2024-09-10T20:38:10.378820+0000","last_fullsized":"2024-09-10T20:38:10.378820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343167+0000","last_change":"2024-09-10T20:38:11.398400+0000","last_active":"2024-09-10T20:38:14.343167+0000","last_peered":"2024-09-10T20:38:14.343167+0000","last_clean":"2024-09-10T20:38:14.343167+0000","last_became_active":"2024-09-10T20:38:11.398097+0000","last_became_peered":"2024-09-10T20:38:11.398097+0000","last_unstale":"2024-09-10T20:38:14.343167+0000","last_undegraded":"2024-09-10T20:38:14.343167+0000","last_fullsized":"2024-09-10T20:38:14.343167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.599838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401478+0000","last_change":"2024-09-10T20:38:12.413177+0000","last_active":"2024-09-10T20:38:13.401478+0000","last_peered":"2024-09-10T20:38:13.401478+0000","last_clean":"2024-09-10T20:38:13.401478+0000","last_became_active":"2024-09-10T20:38:12.412925+0000","last_became_peered":"2024-09-10T20:38:12.412925+0000","last_unstale":"2024-09-10T20:38:13.401478+0000","last_undegraded":"2024-09-10T20:38:13.401478+0000","last_fullsized":"2024-09-10T20:38:13.401478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:18.693832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378650+0000","last_change":"2024-09-10T20:38:10.378650+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378650+0000","last_undegraded":"2024-09-10T20:38:10.378650+0000","last_fullsized":"2024-09-10T20:38:10.378650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401415+0000","last_change":"2024-09-10T20:38:12.415606+0000","last_active":"2024-09-10T20:38:13.401415+0000","last_peered":"2024-09-10T20:38:13.401415+0000","last_clean":"2024-09-10T20:38:13.401415+0000","last_became_active":"2024-09-10T20:38:12.415330+0000","last_became_peered":"2024-09-10T20:38:12.415330+0000","last_unstale":"2024-09-10T20:38:13.401415+0000","last_undegraded":"2024-09-10T20:38:13.401415+0000","last_fullsized":"2024-09-10T20:38:13.401415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:53.050014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380287+0000","last_change":"2024-09-10T20:38:10.380287+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380287+0000","last_undegraded":"2024-09-10T20:38:10.380287+0000","last_fullsized":"2024-09-10T20:38:10.380287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380912+0000","last_change":"2024-09-10T20:38:10.380912+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380912+0000","last_undegraded":"2024-09-10T20:38:10.380912+0000","last_fullsized":"2024-09-10T20:38:10.380912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401902+0000","last_change":"2024-09-10T20:38:12.410504+0000","last_active":"2024-09-10T20:38:13.401902+0000","last_peered":"2024-09-10T20:38:13.401902+0000","last_clean":"2024-09-10T20:38:13.401902+0000","last_became_active":"2024-09-10T20:38:12.410008+0000","last_became_peered":"2024-09-10T20:38:12.410008+0000","last_unstale":"2024-09-10T20:38:13.401902+0000","last_undegraded":"2024-09-10T20:38:13.401902+0000","last_fullsized":"2024-09-10T20:38:13.401902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:57.544501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633468+0000","last_change":"2024-09-10T20:38:11.395962+0000","last_active":"2024-09-10T20:38:13.633468+0000","last_peered":"2024-09-10T20:38:13.633468+0000","last_clean":"2024-09-10T20:38:13.633468+0000","last_became_active":"2024-09-10T20:38:11.395687+0000","last_became_peered":"2024-09-10T20:38:11.395687+0000","last_unstale":"2024-09-10T20:38:13.633468+0000","last_undegraded":"2024-09-10T20:38:13.633468+0000","last_fullsized":"2024-09-10T20:38:13.633468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:15.421827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380034+0000","last_change":"2024-09-10T20:38:10.380034+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380034+0000","last_undegraded":"2024-09-10T20:38:10.380034+0000","last_fullsized":"2024-09-10T20:38:10.380034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633539+0000","last_change":"2024-09-10T20:38:11.398330+0000","last_active":"2024-09-10T20:38:13.633539+0000","last_peered":"2024-09-10T20:38:13.633539+0000","last_clean":"2024-09-10T20:38:13.633539+0000","last_became_active":"2024-09-10T20:38:11.398059+0000","last_became_peered":"2024-09-10T20:38:11.398059+0000","last_unstale":"2024-09-10T20:38:13.633539+0000","last_undegraded":"2024-09-10T20:38:13.633539+0000","last_fullsized":"2024-09-10T20:38:13.633539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343446+0000","last_change":"2024-09-10T20:38:12.412484+0000","last_active":"2024-09-10T20:38:14.343446+0000","last_peered":"2024-09-10T20:38:14.343446+0000","last_clean":"2024-09-10T20:38:14.343446+0000","last_became_active":"2024-09-10T20:38:12.412165+0000","last_became_peered":"2024-09-10T20:38:12.412165+0000","last_unstale":"2024-09-10T20:38:14.343446+0000","last_undegraded":"2024-09-10T20:38:14.343446+0000","last_fullsized":"2024-09-10T20:38:14.343446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:09.972659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378166+0000","last_change":"2024-09-10T20:38:10.378166+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378166+0000","last_undegraded":"2024-09-10T20:38:10.378166+0000","last_fullsized":"2024-09-10T20:38:10.378166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380952+0000","last_change":"2024-09-10T20:38:10.380952+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380952+0000","last_undegraded":"2024-09-10T20:38:10.380952+0000","last_fullsized":"2024-09-10T20:38:10.380952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400564+0000","last_change":"2024-09-10T20:38:11.393414+0000","last_active":"2024-09-10T20:38:13.400564+0000","last_peered":"2024-09-10T20:38:13.400564+0000","last_clean":"2024-09-10T20:38:13.400564+0000","last_became_active":"2024-09-10T20:38:11.393078+0000","last_became_peered":"2024-09-10T20:38:11.393078+0000","last_unstale":"2024-09-10T20:38:13.400564+0000","last_undegraded":"2024-09-10T20:38:13.400564+0000","last_fullsized":"2024-09-10T20:38:13.400564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:58.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381915+0000","last_change":"2024-09-10T20:38:10.381915+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381915+0000","last_undegraded":"2024-09-10T20:38:10.381915+0000","last_fullsized":"2024-09-10T20:38:10.381915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401433+0000","last_change":"2024-09-10T20:38:11.388247+0000","last_active":"2024-09-10T20:38:13.401433+0000","last_peered":"2024-09-10T20:38:13.401433+0000","last_clean":"2024-09-10T20:38:13.401433+0000","last_became_active":"2024-09-10T20:38:11.387924+0000","last_became_peered":"2024-09-10T20:38:11.387924+0000","last_unstale":"2024-09-10T20:38:13.401433+0000","last_undegraded":"2024-09-10T20:38:13.401433+0000","last_fullsized":"2024-09-10T20:38:13.401433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:29.584118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400616+0000","last_change":"2024-09-10T20:38:11.399015+0000","last_active":"2024-09-10T20:38:13.400616+0000","last_peered":"2024-09-10T20:38:13.400616+0000","last_clean":"2024-09-10T20:38:13.400616+0000","last_became_active":"2024-09-10T20:38:11.398826+0000","last_became_peered":"2024-09-10T20:38:11.398826+0000","last_unstale":"2024-09-10T20:38:13.400616+0000","last_undegraded":"2024-09-10T20:38:13.400616+0000","last_fullsized":"2024-09-10T20:38:13.400616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:03.958637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343682+0000","last_change":"2024-09-10T20:38:11.398337+0000","last_active":"2024-09-10T20:38:14.343682+0000","last_peered":"2024-09-10T20:38:14.343682+0000","last_clean":"2024-09-10T20:38:14.343682+0000","last_became_active":"2024-09-10T20:38:11.398068+0000","last_became_peered":"2024-09-10T20:38:11.398068+0000","last_unstale":"2024-09-10T20:38:14.343682+0000","last_undegraded":"2024-09-10T20:38:14.343682+0000","last_fullsized":"2024-09-10T20:38:14.343682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:00:28.236504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":105,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-10T20:38:10.380805+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:10.380805+0000","last_peered":"2024-09-10T20:38:10.380805+0000","last_clean":"2024-09-10T20:38:10.380805+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:10.380805+0000","last_undegraded":"2024-09-10T20:38:10.380805+0000","last_fullsized":"2024-09-10T20:38:10.380805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377085+0000","last_change":"2024-09-10T20:38:10.377085+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377085+0000","last_undegraded":"2024-09-10T20:38:10.377085+0000","last_fullsized":"2024-09-10T20:38:10.377085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400902+0000","last_change":"2024-09-10T20:38:12.407263+0000","last_active":"2024-09-10T20:38:13.400902+0000","last_peered":"2024-09-10T20:38:13.400902+0000","last_clean":"2024-09-10T20:38:13.400902+0000","last_became_active":"2024-09-10T20:38:12.406952+0000","last_became_peered":"2024-09-10T20:38:12.406952+0000","last_unstale":"2024-09-10T20:38:13.400902+0000","last_undegraded":"2024-09-10T20:38:13.400902+0000","last_fullsized":"2024-09-10T20:38:13.400902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.171100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.376990+0000","last_change":"2024-09-10T20:38:10.376990+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.376990+0000","last_undegraded":"2024-09-10T20:38:10.376990+0000","last_fullsized":"2024-09-10T20:38:10.376990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343358+0000","last_change":"2024-09-10T20:38:12.407025+0000","last_active":"2024-09-10T20:38:14.343358+0000","last_peered":"2024-09-10T20:38:14.343358+0000","last_clean":"2024-09-10T20:38:14.343358+0000","last_became_active":"2024-09-10T20:38:12.406124+0000","last_became_peered":"2024-09-10T20:38:12.406124+0000","last_unstale":"2024-09-10T20:38:14.343358+0000","last_undegraded":"2024-09-10T20:38:14.343358+0000","last_fullsized":"2024-09-10T20:38:14.343358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633299+0000","last_change":"2024-09-10T20:38:11.397217+0000","last_active":"2024-09-10T20:38:13.633299+0000","last_peered":"2024-09-10T20:38:13.633299+0000","last_clean":"2024-09-10T20:38:13.633299+0000","last_became_active":"2024-09-10T20:38:11.396928+0000","last_became_peered":"2024-09-10T20:38:11.396928+0000","last_unstale":"2024-09-10T20:38:13.633299+0000","last_undegraded":"2024-09-10T20:38:13.633299+0000","last_fullsized":"2024-09-10T20:38:13.633299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:38.900087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401469+0000","last_change":"2024-09-10T20:38:12.407595+0000","last_active":"2024-09-10T20:38:13.401469+0000","last_peered":"2024-09-10T20:38:13.401469+0000","last_clean":"2024-09-10T20:38:13.401469+0000","last_became_active":"2024-09-10T20:38:12.407355+0000","last_became_peered":"2024-09-10T20:38:12.407355+0000","last_unstale":"2024-09-10T20:38:13.401469+0000","last_undegraded":"2024-09-10T20:38:13.401469+0000","last_fullsized":"2024-09-10T20:38:13.401469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:05.419998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377007+0000","last_change":"2024-09-10T20:38:10.377007+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377007+0000","last_undegraded":"2024-09-10T20:38:10.377007+0000","last_fullsized":"2024-09-10T20:38:10.377007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400255+0000","last_change":"2024-09-10T20:38:11.397463+0000","last_active":"2024-09-10T20:38:13.400255+0000","last_peered":"2024-09-10T20:38:13.400255+0000","last_clean":"2024-09-10T20:38:13.400255+0000","last_became_active":"2024-09-10T20:38:11.397205+0000","last_became_peered":"2024-09-10T20:38:11.397205+0000","last_unstale":"2024-09-10T20:38:13.400255+0000","last_undegraded":"2024-09-10T20:38:13.400255+0000","last_fullsized":"2024-09-10T20:38:13.400255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:53.003074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377042+0000","last_change":"2024-09-10T20:38:10.377042+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377042+0000","last_undegraded":"2024-09-10T20:38:10.377042+0000","last_fullsized":"2024-09-10T20:38:10.377042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343576+0000","last_change":"2024-09-10T20:38:12.408779+0000","last_active":"2024-09-10T20:38:14.343576+0000","last_peered":"2024-09-10T20:38:14.343576+0000","last_clean":"2024-09-10T20:38:14.343576+0000","last_became_active":"2024-09-10T20:38:12.408490+0000","last_became_peered":"2024-09-10T20:38:12.408490+0000","last_unstale":"2024-09-10T20:38:14.343576+0000","last_undegraded":"2024-09-10T20:38:14.343576+0000","last_fullsized":"2024-09-10T20:38:14.343576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:25.888623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407174+0000","last_change":"2024-09-10T20:38:11.394170+0000","last_active":"2024-09-10T20:38:13.407174+0000","last_peered":"2024-09-10T20:38:13.407174+0000","last_clean":"2024-09-10T20:38:13.407174+0000","last_became_active":"2024-09-10T20:38:11.393850+0000","last_became_peered":"2024-09-10T20:38:11.393850+0000","last_unstale":"2024-09-10T20:38:13.407174+0000","last_undegraded":"2024-09-10T20:38:13.407174+0000","last_fullsized":"2024-09-10T20:38:13.407174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:15.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401623+0000","last_change":"2024-09-10T20:38:12.404386+0000","last_active":"2024-09-10T20:38:13.401623+0000","last_peered":"2024-09-10T20:38:13.401623+0000","last_clean":"2024-09-10T20:38:13.401623+0000","last_became_active":"2024-09-10T20:38:12.404045+0000","last_became_peered":"2024-09-10T20:38:12.404045+0000","last_unstale":"2024-09-10T20:38:13.401623+0000","last_undegraded":"2024-09-10T20:38:13.401623+0000","last_fullsized":"2024-09-10T20:38:13.401623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:49.410849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377015+0000","last_change":"2024-09-10T20:38:10.377015+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377015+0000","last_undegraded":"2024-09-10T20:38:10.377015+0000","last_fullsized":"2024-09-10T20:38:10.377015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400970+0000","last_change":"2024-09-10T20:38:12.409091+0000","last_active":"2024-09-10T20:38:13.400970+0000","last_peered":"2024-09-10T20:38:13.400970+0000","last_clean":"2024-09-10T20:38:13.400970+0000","last_became_active":"2024-09-10T20:38:12.408815+0000","last_became_peered":"2024-09-10T20:38:12.408815+0000","last_unstale":"2024-09-10T20:38:13.400970+0000","last_undegraded":"2024-09-10T20:38:13.400970+0000","last_fullsized":"2024-09-10T20:38:13.400970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.870149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.382686+0000","last_change":"2024-09-10T20:38:10.382686+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.382686+0000","last_undegraded":"2024-09-10T20:38:10.382686+0000","last_fullsized":"2024-09-10T20:38:10.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401287+0000","last_change":"2024-09-10T20:38:11.397172+0000","last_active":"2024-09-10T20:38:13.401287+0000","last_peered":"2024-09-10T20:38:13.401287+0000","last_clean":"2024-09-10T20:38:13.401287+0000","last_became_active":"2024-09-10T20:38:11.396670+0000","last_became_peered":"2024-09-10T20:38:11.396670+0000","last_unstale":"2024-09-10T20:38:13.401287+0000","last_undegraded":"2024-09-10T20:38:13.401287+0000","last_fullsized":"2024-09-10T20:38:13.401287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400195+0000","last_change":"2024-09-10T20:38:12.405875+0000","last_active":"2024-09-10T20:38:13.400195+0000","last_peered":"2024-09-10T20:38:13.400195+0000","last_clean":"2024-09-10T20:38:13.400195+0000","last_became_active":"2024-09-10T20:38:12.405583+0000","last_became_peered":"2024-09-10T20:38:12.405583+0000","last_unstale":"2024-09-10T20:38:13.400195+0000","last_undegraded":"2024-09-10T20:38:13.400195+0000","last_fullsized":"2024-09-10T20:38:13.400195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:30.185620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343471+0000","last_change":"2024-09-10T20:38:11.395992+0000","last_active":"2024-09-10T20:38:14.343471+0000","last_peered":"2024-09-10T20:38:14.343471+0000","last_clean":"2024-09-10T20:38:14.343471+0000","last_became_active":"2024-09-10T20:38:11.394935+0000","last_became_peered":"2024-09-10T20:38:11.394935+0000","last_unstale":"2024-09-10T20:38:14.343471+0000","last_undegraded":"2024-09-10T20:38:14.343471+0000","last_fullsized":"2024-09-10T20:38:14.343471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:49.040222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633574+0000","last_change":"2024-09-10T20:38:12.412620+0000","last_active":"2024-09-10T20:38:13.633574+0000","last_peered":"2024-09-10T20:38:13.633574+0000","last_clean":"2024-09-10T20:38:13.633574+0000","last_became_active":"2024-09-10T20:38:12.412329+0000","last_became_peered":"2024-09-10T20:38:12.412329+0000","last_unstale":"2024-09-10T20:38:13.633574+0000","last_undegraded":"2024-09-10T20:38:13.633574+0000","last_fullsized":"2024-09-10T20:38:13.633574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:40.609805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377791+0000","last_change":"2024-09-10T20:38:10.377791+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377791+0000","last_undegraded":"2024-09-10T20:38:10.377791+0000","last_fullsized":"2024-09-10T20:38:10.377791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381267+0000","last_change":"2024-09-10T20:38:10.381267+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381267+0000","last_undegraded":"2024-09-10T20:38:10.381267+0000","last_fullsized":"2024-09-10T20:38:10.381267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381405+0000","last_change":"2024-09-10T20:38:10.381405+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381405+0000","last_undegraded":"2024-09-10T20:38:10.381405+0000","last_fullsized":"2024-09-10T20:38:10.381405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401905+0000","last_change":"2024-09-10T20:38:12.416588+0000","last_active":"2024-09-10T20:38:13.401905+0000","last_peered":"2024-09-10T20:38:13.401905+0000","last_clean":"2024-09-10T20:38:13.401905+0000","last_became_active":"2024-09-10T20:38:12.416297+0000","last_became_peered":"2024-09-10T20:38:12.416297+0000","last_unstale":"2024-09-10T20:38:13.401905+0000","last_undegraded":"2024-09-10T20:38:13.401905+0000","last_fullsized":"2024-09-10T20:38:13.401905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401853+0000","last_change":"2024-09-10T20:38:11.396382+0000","last_active":"2024-09-10T20:38:13.401853+0000","last_peered":"2024-09-10T20:38:13.401853+0000","last_clean":"2024-09-10T20:38:13.401853+0000","last_became_active":"2024-09-10T20:38:11.395934+0000","last_became_peered":"2024-09-10T20:38:11.395934+0000","last_unstale":"2024-09-10T20:38:13.401853+0000","last_undegraded":"2024-09-10T20:38:13.401853+0000","last_fullsized":"2024-09-10T20:38:13.401853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:20:39.553068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401634+0000","last_change":"2024-09-10T20:38:12.402624+0000","last_active":"2024-09-10T20:38:13.401634+0000","last_peered":"2024-09-10T20:38:13.401634+0000","last_clean":"2024-09-10T20:38:13.401634+0000","last_became_active":"2024-09-10T20:38:12.401924+0000","last_became_peered":"2024-09-10T20:38:12.401924+0000","last_unstale":"2024-09-10T20:38:13.401634+0000","last_undegraded":"2024-09-10T20:38:13.401634+0000","last_fullsized":"2024-09-10T20:38:13.401634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:33.683744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401708+0000","last_change":"2024-09-10T20:38:11.401264+0000","last_active":"2024-09-10T20:38:13.401708+0000","last_peered":"2024-09-10T20:38:13.401708+0000","last_clean":"2024-09-10T20:38:13.401708+0000","last_became_active":"2024-09-10T20:38:11.400986+0000","last_became_peered":"2024-09-10T20:38:11.400986+0000","last_unstale":"2024-09-10T20:38:13.401708+0000","last_undegraded":"2024-09-10T20:38:13.401708+0000","last_fullsized":"2024-09-10T20:38:13.401708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:50.668458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":172032,"data_stored":53177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":753664,"data_stored":637728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":66227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":662987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":74811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3f is not active+clean 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3c is not active+clean 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3b is not active+clean 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3a is not active+clean 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:PG 3.39 is not active+clean 2024-09-10T20:38:17.133 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:PG 3.37 is not active+clean 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:PG 3.36 is not active+clean 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:PG 3.35 is not active+clean 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:PG 3.34 is not active+clean 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:PG 3.30 is not active+clean 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.134 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2f is not active+clean 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2d is not active+clean 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2c is not active+clean 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2a is not active+clean 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:PG 3.28 is not active+clean 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:PG 3.26 is not active+clean 2024-09-10T20:38:17.135 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:PG 3.22 is not active+clean 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:PG 3.20 is not active+clean 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1a is not active+clean 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1a is not active+clean 2024-09-10T20:38:17.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.378820+0000', 'last_change': '2024-09-10T20:38:10.378820+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.378820+0000', 'last_undegraded': '2024-09-10T20:38:10.378820+0000', 'last_fullsized': '2024-09-10T20:38:10.378820+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:PG 2.18 is not active+clean 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.378650+0000', 'last_change': '2024-09-10T20:38:10.378650+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.378650+0000', 'last_undegraded': '2024-09-10T20:38:10.378650+0000', 'last_fullsized': '2024-09-10T20:38:10.378650+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:PG 2.17 is not active+clean 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.380287+0000', 'last_change': '2024-09-10T20:38:10.380287+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.380287+0000', 'last_undegraded': '2024-09-10T20:38:10.380287+0000', 'last_fullsized': '2024-09-10T20:38:10.380287+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:PG 3.16 is not active+clean 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:PG 2.16 is not active+clean 2024-09-10T20:38:17.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.380912+0000', 'last_change': '2024-09-10T20:38:10.380912+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.380912+0000', 'last_undegraded': '2024-09-10T20:38:10.380912+0000', 'last_fullsized': '2024-09-10T20:38:10.380912+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:PG 3.14 is not active+clean 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:PG 2.14 is not active+clean 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.380034+0000', 'last_change': '2024-09-10T20:38:10.380034+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.380034+0000', 'last_undegraded': '2024-09-10T20:38:10.380034+0000', 'last_fullsized': '2024-09-10T20:38:10.380034+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:PG 3.15 is not active+clean 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:PG 2.12 is not active+clean 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.378166+0000', 'last_change': '2024-09-10T20:38:10.378166+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.378166+0000', 'last_undegraded': '2024-09-10T20:38:10.378166+0000', 'last_fullsized': '2024-09-10T20:38:10.378166+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:PG 3.13 is not active+clean 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.138 INFO:tasks.ceph.ceph_manager.ceph:PG 2.11 is not active+clean 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.380952+0000', 'last_change': '2024-09-10T20:38:10.380952+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.380952+0000', 'last_undegraded': '2024-09-10T20:38:10.380952+0000', 'last_fullsized': '2024-09-10T20:38:10.380952+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:PG 3.10 is not active+clean 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:PG 3.11 is not active+clean 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:PG 2.f is not active+clean 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.381915+0000', 'last_change': '2024-09-10T20:38:10.381915+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.381915+0000', 'last_undegraded': '2024-09-10T20:38:10.381915+0000', 'last_fullsized': '2024-09-10T20:38:10.381915+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:PG 3.e is not active+clean 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:PG 3.f is not active+clean 2024-09-10T20:38:17.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:PG 3.c is not active+clean 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:PG 3.d is not active+clean 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.377085+0000', 'last_change': '2024-09-10T20:38:10.377085+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.377085+0000', 'last_undegraded': '2024-09-10T20:38:10.377085+0000', 'last_fullsized': '2024-09-10T20:38:10.377085+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.376990+0000', 'last_change': '2024-09-10T20:38:10.376990+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.376990+0000', 'last_undegraded': '2024-09-10T20:38:10.376990+0000', 'last_fullsized': '2024-09-10T20:38:10.376990+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.377007+0000', 'last_change': '2024-09-10T20:38:10.377007+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.377007+0000', 'last_undegraded': '2024-09-10T20:38:10.377007+0000', 'last_fullsized': '2024-09-10T20:38:10.377007+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.140 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3 is not active+clean 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:PG 3.5 is not active+clean 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.377042+0000', 'last_change': '2024-09-10T20:38:10.377042+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.377042+0000', 'last_undegraded': '2024-09-10T20:38:10.377042+0000', 'last_fullsized': '2024-09-10T20:38:10.377042+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.377015+0000', 'last_change': '2024-09-10T20:38:10.377015+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.377015+0000', 'last_undegraded': '2024-09-10T20:38:10.377015+0000', 'last_fullsized': '2024-09-10T20:38:10.377015+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:PG 2.8 is not active+clean 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.382686+0000', 'last_change': '2024-09-10T20:38:10.382686+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.382686+0000', 'last_undegraded': '2024-09-10T20:38:10.382686+0000', 'last_fullsized': '2024-09-10T20:38:10.382686+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.141 INFO:tasks.ceph.ceph_manager.ceph:PG 2.b is not active+clean 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.377791+0000', 'last_change': '2024-09-10T20:38:10.377791+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.377791+0000', 'last_undegraded': '2024-09-10T20:38:10.377791+0000', 'last_fullsized': '2024-09-10T20:38:10.377791+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:PG 3.a is not active+clean 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1d is not active+clean 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1c is not active+clean 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.381267+0000', 'last_change': '2024-09-10T20:38:10.381267+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.381267+0000', 'last_undegraded': '2024-09-10T20:38:10.381267+0000', 'last_fullsized': '2024-09-10T20:38:10.381267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1c is not active+clean 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-10T20:38:11.372745+0000', 'last_change': '2024-09-10T20:38:11.372745+0000', 'last_active': '2024-09-10T20:38:11.372745+0000', 'last_peered': '2024-09-10T20:38:11.372745+0000', 'last_clean': '2024-09-10T20:38:11.372745+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:11.372745+0000', 'last_undegraded': '2024-09-10T20:38:11.372745+0000', 'last_fullsized': '2024-09-10T20:38:11.372745+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:11.372745+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1d is not active+clean 2024-09-10T20:38:17.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:38:10.381405+0000', 'last_change': '2024-09-10T20:38:10.381405+0000', 'last_active': '2024-09-10T20:38:10.367851+0000', 'last_peered': '2024-09-10T20:38:10.367851+0000', 'last_clean': '2024-09-10T20:38:10.367851+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:38:10.381405+0000', 'last_undegraded': '2024-09-10T20:38:10.381405+0000', 'last_fullsized': '2024-09-10T20:38:10.381405+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'last_clean_scrub_stamp': '2024-09-10T20:38:10.367851+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-10T20:38:17.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:38:17.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.269+0000 7fd744b84640 1 -- 172.21.15.28:0/880436283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 msgr2=0x7fd7400736c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.269+0000 7fd744b84640 1 --2- 172.21.15.28:0/880436283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 0x7fd7400736c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd738005fd0 tx=0x7fd73802aa90 comp rx=0 tx=0).stop 2024-09-10T20:38:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.270+0000 7fd744b84640 1 -- 172.21.15.28:0/880436283 shutdown_connections 2024-09-10T20:38:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.270+0000 7fd744b84640 1 --2- 172.21.15.28:0/880436283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 0x7fd7400736c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.270+0000 7fd744b84640 1 --2- 172.21.15.28:0/880436283 >> v2:172.21.15.159:3300/0 conn(0x7fd7400d0120 0x7fd740141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.270+0000 7fd744b84640 1 -- 172.21.15.28:0/880436283 >> 172.21.15.28:0/880436283 conn(0x7fd74006ccc0 msgr2=0x7fd74006d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.270+0000 7fd744b84640 1 -- 172.21.15.28:0/880436283 shutdown_connections 2024-09-10T20:38:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.270+0000 7fd744b84640 1 -- 172.21.15.28:0/880436283 wait complete. 2024-09-10T20:38:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.271+0000 7fd744b84640 1 Processor -- start 2024-09-10T20:38:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.271+0000 7fd744b84640 1 -- start start 2024-09-10T20:38:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd744b84640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7400d0120 0x7fd7400b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd744b84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7400d0500 0x7fd74012c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd744b84640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 0x7fd7400b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd744b84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7400ce4a0 con 0x7fd7400d0500 2024-09-10T20:38:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd744b84640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd7400ce320 con 0x7fd7400d0120 2024-09-10T20:38:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd744b84640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd740073c00 con 0x7fd74015cd40 2024-09-10T20:38:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7400d0500 0x7fd74012c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7400d0500 0x7fd74012c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35166/0 (socket says 172.21.15.28:35166) 2024-09-10T20:38:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 -- 172.21.15.28:0/4245046714 learned_addr learned my addr 172.21.15.28:0/4245046714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd73e59c640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7400d0120 0x7fd7400b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd73ed9d640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 0x7fd7400b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 -- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 msgr2=0x7fd7400b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 0x7fd7400b16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 -- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7400d0120 msgr2=0x7fd7400b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7400d0120 0x7fd7400b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 -- 172.21.15.28:0/4245046714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7401f29e0 con 0x7fd7400d0500 2024-09-10T20:38:17.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd737fff640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7400d0500 0x7fd74012c3f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c450 tx=0x7fd72800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd736ffd640 1 -- 172.21.15.28:0/4245046714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280052d0 con 0x7fd7400d0500 2024-09-10T20:38:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd736ffd640 1 -- 172.21.15.28:0/4245046714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd728005470 con 0x7fd7400d0500 2024-09-10T20:38:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.272+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd74013bca0 con 0x7fd7400d0500 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.273+0000 7fd736ffd640 1 -- 172.21.15.28:0/4245046714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728012870 con 0x7fd7400d0500 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.273+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd740155f80 con 0x7fd7400d0500 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.273+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd740074120 con 0x7fd7400d0500 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.274+0000 7fd736ffd640 1 -- 172.21.15.28:0/4245046714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd7280035f0 con 0x7fd7400d0500 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.275+0000 7fd736ffd640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd720077f70 0x7fd72007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.275+0000 7fd736ffd640 1 -- 172.21.15.28:0/4245046714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd7280997a0 con 0x7fd7400d0500 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.275+0000 7fd73e59c640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd720077f70 0x7fd72007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.275+0000 7fd73e59c640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd720077f70 0x7fd72007a430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd730009bd0 tx=0x7fd730005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:17.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.279+0000 7fd736ffd640 1 -- 172.21.15.28:0/4245046714 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd728061c50 con 0x7fd7400d0500 2024-09-10T20:38:17.346 INFO:tasks.ceph.mds.d.smithi159.stderr:2024-09-10T20:38:17.342+0000 7f57378c9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:38:17.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.535+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fd74015e150 con 0x7fd7400d0500 2024-09-10T20:38:17.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.536+0000 7fd736ffd640 1 -- 172.21.15.28:0/4245046714 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1215 (secure 0 0 0) 0x7fd728066680 con 0x7fd7400d0500 2024-09-10T20:38:17.540 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:17.540 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":29,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":21,"num_osds":8,"num_up_osds":8,"osd_up_since":1726000680,"num_in_osds":8,"osd_in_since":1726000670,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":45},{"state_name":"unknown","count":35},{"state_name":"creating+peering","count":17}],"num_pgs":97,"num_pools":3,"num_objects":15,"data_bytes":591144,"bytes_used":241356800,"bytes_avail":772852756480,"bytes_total":773094113280,"unknown_pgs_ratio":0.36082473397254944,"inactive_pgs_ratio":0.17525772750377655,"write_bytes_sec":1827,"read_op_per_sec":0,"write_op_per_sec":4},"fsmap":{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"d","status":"up:active","gid":4357}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-10T20:37:47.617910+0000","services":{}},"progress_events":{}} 2024-09-10T20:38:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd720077f70 msgr2=0x7fd72007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd720077f70 0x7fd72007a430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd730009bd0 tx=0x7fd730005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7400d0500 msgr2=0x7fd74012c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7400d0500 0x7fd74012c3f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c450 tx=0x7fd72800c910 comp rx=0 tx=0).stop 2024-09-10T20:38:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 shutdown_connections 2024-09-10T20:38:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd720077f70 0x7fd72007a430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd74015cd40 0x7fd7400b16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7400d0500 0x7fd74012c3f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 --2- 172.21.15.28:0/4245046714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7400d0120 0x7fd7400b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 >> 172.21.15.28:0/4245046714 conn(0x7fd74006ccc0 msgr2=0x7fd74015d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 shutdown_connections 2024-09-10T20:38:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.538+0000 7fd744b84640 1 -- 172.21.15.28:0/4245046714 wait complete. 2024-09-10T20:38:17.553 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T20:38:17.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 -- 172.21.15.28:0/71396704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40cee20 msgr2=0x7fafc40ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/71396704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40cee20 0x7fafc40ccae0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fafc40747e0 tx=0x7fafb402a560 comp rx=0 tx=0).stop 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 -- 172.21.15.28:0/71396704 shutdown_connections 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/71396704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40cee20 0x7fafc40ccae0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/71396704 >> v2:172.21.15.159:3300/0 conn(0x7fafc40b65d0 0x7fafc40cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 -- 172.21.15.28:0/71396704 >> 172.21.15.28:0/71396704 conn(0x7fafc40d1ed0 msgr2=0x7fafc40d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 -- 172.21.15.28:0/71396704 shutdown_connections 2024-09-10T20:38:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.654+0000 7fafcbb6b640 1 -- 172.21.15.28:0/71396704 wait complete. 2024-09-10T20:38:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.655+0000 7fafcbb6b640 1 Processor -- start 2024-09-10T20:38:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.655+0000 7fafcbb6b640 1 -- start start 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafcbb6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40b65d0 0x7fafc40b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafcbb6b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafc40cee20 0x7fafc406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafcbb6b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafc415f660 0x7fafc40c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafcbb6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40d2b70 con 0x7fafc40b65d0 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafcbb6b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40d29f0 con 0x7fafc40cee20 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafcbb6b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fafc40d2cf0 con 0x7fafc415f660 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40b65d0 0x7fafc40b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40b65d0 0x7fafc40b2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35198/0 (socket says 172.21.15.28:35198) 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 -- 172.21.15.28:0/4238123838 learned_addr learned my addr 172.21.15.28:0/4238123838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9106640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafc40cee20 0x7fafc406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafca108640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafc415f660 0x7fafc40c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 -- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafc415f660 msgr2=0x7fafc40c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafc415f660 0x7fafc40c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 -- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafc40cee20 msgr2=0x7fafc406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafc40cee20 0x7fafc406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 -- 172.21.15.28:0/4238123838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc406e4f0 con 0x7fafc40b65d0 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafc9907640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40b65d0 0x7fafc40b2490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafc000b3f0 tx=0x7fafc000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafba7fc640 1 -- 172.21.15.28:0/4238123838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0015020 con 0x7fafc40b65d0 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafba7fc640 1 -- 172.21.15.28:0/4238123838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafc0011920 con 0x7fafc40b65d0 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc406e7d0 con 0x7fafc40b65d0 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.656+0000 7fafba7fc640 1 -- 172.21.15.28:0/4238123838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0011d70 con 0x7fafc40b65d0 2024-09-10T20:38:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.657+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc406ed90 con 0x7fafc40b65d0 2024-09-10T20:38:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.658+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc40ce710 con 0x7fafc40b65d0 2024-09-10T20:38:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.658+0000 7fafba7fc640 1 -- 172.21.15.28:0/4238123838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fafc001a400 con 0x7fafc40b65d0 2024-09-10T20:38:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.659+0000 7fafba7fc640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fafa8077f70 0x7fafa807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.659+0000 7fafba7fc640 1 -- 172.21.15.28:0/4238123838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fafc0099c30 con 0x7fafc40b65d0 2024-09-10T20:38:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.659+0000 7fafc9106640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fafa8077f70 0x7fafa807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.660+0000 7fafc9106640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fafa8077f70 0x7fafa807a430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fafb4033000 tx=0x7fafb4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.663+0000 7fafba7fc640 1 -- 172.21.15.28:0/4238123838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fafc001f030 con 0x7fafc40b65d0 2024-09-10T20:38:17.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.844+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fafc40d16a0 con 0x7fafa8077f70 2024-09-10T20:38:17.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.848+0000 7fafba7fc640 1 -- 172.21.15.28:0/4238123838 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+245574 (secure 0 0 0) 0x7fafc40d16a0 con 0x7fafa8077f70 2024-09-10T20:38:17.854 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:17.855 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fafa8077f70 msgr2=0x7fafa807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fafa8077f70 0x7fafa807a430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fafb4033000 tx=0x7fafb4004730 comp rx=0 tx=0).stop 2024-09-10T20:38:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40b65d0 msgr2=0x7fafc40b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40b65d0 0x7fafc40b2490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafc000b3f0 tx=0x7fafc000b8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 shutdown_connections 2024-09-10T20:38:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fafa8077f70 0x7fafa807a430 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fafc415f660 0x7fafc40c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fafc40cee20 0x7fafc406dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 --2- 172.21.15.28:0/4238123838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc40b65d0 0x7fafc40b2490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 >> 172.21.15.28:0/4238123838 conn(0x7fafc40d1ed0 msgr2=0x7fafc40cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 shutdown_connections 2024-09-10T20:38:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:17.851+0000 7fafcbb6b640 1 -- 172.21.15.28:0/4238123838 wait complete. 2024-09-10T20:38:17.866 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:38:15.976782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591144,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":129,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":124,"acting":124,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235700,"kb_used_data":2636,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739020,"statfs":{"total":773094113280,"available":772852756480,"internally_reserved":0,"allocated":2699264,"data_stored":1662301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.603193"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400675+0000","last_change":"2024-09-10T20:38:12.413418+0000","last_active":"2024-09-10T20:38:13.400675+0000","last_peered":"2024-09-10T20:38:13.400675+0000","last_clean":"2024-09-10T20:38:13.400675+0000","last_became_active":"2024-09-10T20:38:12.413063+0000","last_became_peered":"2024-09-10T20:38:12.413063+0000","last_unstale":"2024-09-10T20:38:13.400675+0000","last_undegraded":"2024-09-10T20:38:13.400675+0000","last_fullsized":"2024-09-10T20:38:13.400675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:12.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401706+0000","last_change":"2024-09-10T20:38:12.404482+0000","last_active":"2024-09-10T20:38:13.401706+0000","last_peered":"2024-09-10T20:38:13.401706+0000","last_clean":"2024-09-10T20:38:13.401706+0000","last_became_active":"2024-09-10T20:38:12.404181+0000","last_became_peered":"2024-09-10T20:38:12.404181+0000","last_unstale":"2024-09-10T20:38:13.401706+0000","last_undegraded":"2024-09-10T20:38:13.401706+0000","last_fullsized":"2024-09-10T20:38:13.401706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:02.847206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401300+0000","last_change":"2024-09-10T20:38:12.403203+0000","last_active":"2024-09-10T20:38:13.401300+0000","last_peered":"2024-09-10T20:38:13.401300+0000","last_clean":"2024-09-10T20:38:13.401300+0000","last_became_active":"2024-09-10T20:38:12.402272+0000","last_became_peered":"2024-09-10T20:38:12.402272+0000","last_unstale":"2024-09-10T20:38:13.401300+0000","last_undegraded":"2024-09-10T20:38:13.401300+0000","last_fullsized":"2024-09-10T20:38:13.401300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343562+0000","last_change":"2024-09-10T20:38:12.415360+0000","last_active":"2024-09-10T20:38:14.343562+0000","last_peered":"2024-09-10T20:38:14.343562+0000","last_clean":"2024-09-10T20:38:14.343562+0000","last_became_active":"2024-09-10T20:38:12.415094+0000","last_became_peered":"2024-09-10T20:38:12.415094+0000","last_unstale":"2024-09-10T20:38:14.343562+0000","last_undegraded":"2024-09-10T20:38:14.343562+0000","last_fullsized":"2024-09-10T20:38:14.343562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:36.623506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343324+0000","last_change":"2024-09-10T20:38:12.411600+0000","last_active":"2024-09-10T20:38:14.343324+0000","last_peered":"2024-09-10T20:38:14.343324+0000","last_clean":"2024-09-10T20:38:14.343324+0000","last_became_active":"2024-09-10T20:38:12.411328+0000","last_became_peered":"2024-09-10T20:38:12.411328+0000","last_unstale":"2024-09-10T20:38:14.343324+0000","last_undegraded":"2024-09-10T20:38:14.343324+0000","last_fullsized":"2024-09-10T20:38:14.343324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:30.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400233+0000","last_change":"2024-09-10T20:38:12.404816+0000","last_active":"2024-09-10T20:38:13.400233+0000","last_peered":"2024-09-10T20:38:13.400233+0000","last_clean":"2024-09-10T20:38:13.400233+0000","last_became_active":"2024-09-10T20:38:12.404568+0000","last_became_peered":"2024-09-10T20:38:12.404568+0000","last_unstale":"2024-09-10T20:38:13.400233+0000","last_undegraded":"2024-09-10T20:38:13.400233+0000","last_fullsized":"2024-09-10T20:38:13.400233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:05:49.714066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633628+0000","last_change":"2024-09-10T20:38:12.407265+0000","last_active":"2024-09-10T20:38:13.633628+0000","last_peered":"2024-09-10T20:38:13.633628+0000","last_clean":"2024-09-10T20:38:13.633628+0000","last_became_active":"2024-09-10T20:38:12.406927+0000","last_became_peered":"2024-09-10T20:38:12.406927+0000","last_unstale":"2024-09-10T20:38:13.633628+0000","last_undegraded":"2024-09-10T20:38:13.633628+0000","last_fullsized":"2024-09-10T20:38:13.633628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:14.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400878+0000","last_change":"2024-09-10T20:38:12.407874+0000","last_active":"2024-09-10T20:38:13.400878+0000","last_peered":"2024-09-10T20:38:13.400878+0000","last_clean":"2024-09-10T20:38:13.400878+0000","last_became_active":"2024-09-10T20:38:12.407623+0000","last_became_peered":"2024-09-10T20:38:12.407623+0000","last_unstale":"2024-09-10T20:38:13.400878+0000","last_undegraded":"2024-09-10T20:38:13.400878+0000","last_fullsized":"2024-09-10T20:38:13.400878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:56.990230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400518+0000","last_change":"2024-09-10T20:38:12.408020+0000","last_active":"2024-09-10T20:38:13.400518+0000","last_peered":"2024-09-10T20:38:13.400518+0000","last_clean":"2024-09-10T20:38:13.400518+0000","last_became_active":"2024-09-10T20:38:12.407757+0000","last_became_peered":"2024-09-10T20:38:12.407757+0000","last_unstale":"2024-09-10T20:38:13.400518+0000","last_undegraded":"2024-09-10T20:38:13.400518+0000","last_fullsized":"2024-09-10T20:38:13.400518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:32.513086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401654+0000","last_change":"2024-09-10T20:38:12.416773+0000","last_active":"2024-09-10T20:38:13.401654+0000","last_peered":"2024-09-10T20:38:13.401654+0000","last_clean":"2024-09-10T20:38:13.401654+0000","last_became_active":"2024-09-10T20:38:12.416498+0000","last_became_peered":"2024-09-10T20:38:12.416498+0000","last_unstale":"2024-09-10T20:38:13.401654+0000","last_undegraded":"2024-09-10T20:38:13.401654+0000","last_fullsized":"2024-09-10T20:38:13.401654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:42.492324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343671+0000","last_change":"2024-09-10T20:38:12.406673+0000","last_active":"2024-09-10T20:38:14.343671+0000","last_peered":"2024-09-10T20:38:14.343671+0000","last_clean":"2024-09-10T20:38:14.343671+0000","last_became_active":"2024-09-10T20:38:12.406401+0000","last_became_peered":"2024-09-10T20:38:12.406401+0000","last_unstale":"2024-09-10T20:38:14.343671+0000","last_undegraded":"2024-09-10T20:38:14.343671+0000","last_fullsized":"2024-09-10T20:38:14.343671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:31.845414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343502+0000","last_change":"2024-09-10T20:38:12.414290+0000","last_active":"2024-09-10T20:38:14.343502+0000","last_peered":"2024-09-10T20:38:14.343502+0000","last_clean":"2024-09-10T20:38:14.343502+0000","last_became_active":"2024-09-10T20:38:12.413712+0000","last_became_peered":"2024-09-10T20:38:12.413712+0000","last_unstale":"2024-09-10T20:38:14.343502+0000","last_undegraded":"2024-09-10T20:38:14.343502+0000","last_fullsized":"2024-09-10T20:38:14.343502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:48.033101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400477+0000","last_change":"2024-09-10T20:38:12.406921+0000","last_active":"2024-09-10T20:38:13.400477+0000","last_peered":"2024-09-10T20:38:13.400477+0000","last_clean":"2024-09-10T20:38:13.400477+0000","last_became_active":"2024-09-10T20:38:12.406642+0000","last_became_peered":"2024-09-10T20:38:12.406642+0000","last_unstale":"2024-09-10T20:38:13.400477+0000","last_undegraded":"2024-09-10T20:38:13.400477+0000","last_fullsized":"2024-09-10T20:38:13.400477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:02.877370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633368+0000","last_change":"2024-09-10T20:38:12.415776+0000","last_active":"2024-09-10T20:38:13.633368+0000","last_peered":"2024-09-10T20:38:13.633368+0000","last_clean":"2024-09-10T20:38:13.633368+0000","last_became_active":"2024-09-10T20:38:12.415480+0000","last_became_peered":"2024-09-10T20:38:12.415480+0000","last_unstale":"2024-09-10T20:38:13.633368+0000","last_undegraded":"2024-09-10T20:38:13.633368+0000","last_fullsized":"2024-09-10T20:38:13.633368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:13.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400454+0000","last_change":"2024-09-10T20:38:11.394869+0000","last_active":"2024-09-10T20:38:13.400454+0000","last_peered":"2024-09-10T20:38:13.400454+0000","last_clean":"2024-09-10T20:38:13.400454+0000","last_became_active":"2024-09-10T20:38:11.394564+0000","last_became_peered":"2024-09-10T20:38:11.394564+0000","last_unstale":"2024-09-10T20:38:13.400454+0000","last_undegraded":"2024-09-10T20:38:13.400454+0000","last_fullsized":"2024-09-10T20:38:13.400454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:19.365550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401125+0000","last_change":"2024-09-10T20:38:12.401521+0000","last_active":"2024-09-10T20:38:13.401125+0000","last_peered":"2024-09-10T20:38:13.401125+0000","last_clean":"2024-09-10T20:38:13.401125+0000","last_became_active":"2024-09-10T20:38:12.401218+0000","last_became_peered":"2024-09-10T20:38:12.401218+0000","last_unstale":"2024-09-10T20:38:13.401125+0000","last_undegraded":"2024-09-10T20:38:13.401125+0000","last_fullsized":"2024-09-10T20:38:13.401125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:33.954998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378820+0000","last_change":"2024-09-10T20:38:10.378820+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378820+0000","last_undegraded":"2024-09-10T20:38:10.378820+0000","last_fullsized":"2024-09-10T20:38:10.378820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343167+0000","last_change":"2024-09-10T20:38:11.398400+0000","last_active":"2024-09-10T20:38:14.343167+0000","last_peered":"2024-09-10T20:38:14.343167+0000","last_clean":"2024-09-10T20:38:14.343167+0000","last_became_active":"2024-09-10T20:38:11.398097+0000","last_became_peered":"2024-09-10T20:38:11.398097+0000","last_unstale":"2024-09-10T20:38:14.343167+0000","last_undegraded":"2024-09-10T20:38:14.343167+0000","last_fullsized":"2024-09-10T20:38:14.343167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.599838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401478+0000","last_change":"2024-09-10T20:38:12.413177+0000","last_active":"2024-09-10T20:38:13.401478+0000","last_peered":"2024-09-10T20:38:13.401478+0000","last_clean":"2024-09-10T20:38:13.401478+0000","last_became_active":"2024-09-10T20:38:12.412925+0000","last_became_peered":"2024-09-10T20:38:12.412925+0000","last_unstale":"2024-09-10T20:38:13.401478+0000","last_undegraded":"2024-09-10T20:38:13.401478+0000","last_fullsized":"2024-09-10T20:38:13.401478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:18.693832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378650+0000","last_change":"2024-09-10T20:38:10.378650+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378650+0000","last_undegraded":"2024-09-10T20:38:10.378650+0000","last_fullsized":"2024-09-10T20:38:10.378650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401415+0000","last_change":"2024-09-10T20:38:12.415606+0000","last_active":"2024-09-10T20:38:13.401415+0000","last_peered":"2024-09-10T20:38:13.401415+0000","last_clean":"2024-09-10T20:38:13.401415+0000","last_became_active":"2024-09-10T20:38:12.415330+0000","last_became_peered":"2024-09-10T20:38:12.415330+0000","last_unstale":"2024-09-10T20:38:13.401415+0000","last_undegraded":"2024-09-10T20:38:13.401415+0000","last_fullsized":"2024-09-10T20:38:13.401415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:53.050014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380287+0000","last_change":"2024-09-10T20:38:10.380287+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380287+0000","last_undegraded":"2024-09-10T20:38:10.380287+0000","last_fullsized":"2024-09-10T20:38:10.380287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380912+0000","last_change":"2024-09-10T20:38:10.380912+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380912+0000","last_undegraded":"2024-09-10T20:38:10.380912+0000","last_fullsized":"2024-09-10T20:38:10.380912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401902+0000","last_change":"2024-09-10T20:38:12.410504+0000","last_active":"2024-09-10T20:38:13.401902+0000","last_peered":"2024-09-10T20:38:13.401902+0000","last_clean":"2024-09-10T20:38:13.401902+0000","last_became_active":"2024-09-10T20:38:12.410008+0000","last_became_peered":"2024-09-10T20:38:12.410008+0000","last_unstale":"2024-09-10T20:38:13.401902+0000","last_undegraded":"2024-09-10T20:38:13.401902+0000","last_fullsized":"2024-09-10T20:38:13.401902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:57.544501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633468+0000","last_change":"2024-09-10T20:38:11.395962+0000","last_active":"2024-09-10T20:38:13.633468+0000","last_peered":"2024-09-10T20:38:13.633468+0000","last_clean":"2024-09-10T20:38:13.633468+0000","last_became_active":"2024-09-10T20:38:11.395687+0000","last_became_peered":"2024-09-10T20:38:11.395687+0000","last_unstale":"2024-09-10T20:38:13.633468+0000","last_undegraded":"2024-09-10T20:38:13.633468+0000","last_fullsized":"2024-09-10T20:38:13.633468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:15.421827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380034+0000","last_change":"2024-09-10T20:38:10.380034+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380034+0000","last_undegraded":"2024-09-10T20:38:10.380034+0000","last_fullsized":"2024-09-10T20:38:10.380034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633539+0000","last_change":"2024-09-10T20:38:11.398330+0000","last_active":"2024-09-10T20:38:13.633539+0000","last_peered":"2024-09-10T20:38:13.633539+0000","last_clean":"2024-09-10T20:38:13.633539+0000","last_became_active":"2024-09-10T20:38:11.398059+0000","last_became_peered":"2024-09-10T20:38:11.398059+0000","last_unstale":"2024-09-10T20:38:13.633539+0000","last_undegraded":"2024-09-10T20:38:13.633539+0000","last_fullsized":"2024-09-10T20:38:13.633539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343446+0000","last_change":"2024-09-10T20:38:12.412484+0000","last_active":"2024-09-10T20:38:14.343446+0000","last_peered":"2024-09-10T20:38:14.343446+0000","last_clean":"2024-09-10T20:38:14.343446+0000","last_became_active":"2024-09-10T20:38:12.412165+0000","last_became_peered":"2024-09-10T20:38:12.412165+0000","last_unstale":"2024-09-10T20:38:14.343446+0000","last_undegraded":"2024-09-10T20:38:14.343446+0000","last_fullsized":"2024-09-10T20:38:14.343446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:09.972659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.378166+0000","last_change":"2024-09-10T20:38:10.378166+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.378166+0000","last_undegraded":"2024-09-10T20:38:10.378166+0000","last_fullsized":"2024-09-10T20:38:10.378166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.380952+0000","last_change":"2024-09-10T20:38:10.380952+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.380952+0000","last_undegraded":"2024-09-10T20:38:10.380952+0000","last_fullsized":"2024-09-10T20:38:10.380952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400564+0000","last_change":"2024-09-10T20:38:11.393414+0000","last_active":"2024-09-10T20:38:13.400564+0000","last_peered":"2024-09-10T20:38:13.400564+0000","last_clean":"2024-09-10T20:38:13.400564+0000","last_became_active":"2024-09-10T20:38:11.393078+0000","last_became_peered":"2024-09-10T20:38:11.393078+0000","last_unstale":"2024-09-10T20:38:13.400564+0000","last_undegraded":"2024-09-10T20:38:13.400564+0000","last_fullsized":"2024-09-10T20:38:13.400564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:58.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381915+0000","last_change":"2024-09-10T20:38:10.381915+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381915+0000","last_undegraded":"2024-09-10T20:38:10.381915+0000","last_fullsized":"2024-09-10T20:38:10.381915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401433+0000","last_change":"2024-09-10T20:38:11.388247+0000","last_active":"2024-09-10T20:38:13.401433+0000","last_peered":"2024-09-10T20:38:13.401433+0000","last_clean":"2024-09-10T20:38:13.401433+0000","last_became_active":"2024-09-10T20:38:11.387924+0000","last_became_peered":"2024-09-10T20:38:11.387924+0000","last_unstale":"2024-09-10T20:38:13.401433+0000","last_undegraded":"2024-09-10T20:38:13.401433+0000","last_fullsized":"2024-09-10T20:38:13.401433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:29.584118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400616+0000","last_change":"2024-09-10T20:38:11.399015+0000","last_active":"2024-09-10T20:38:13.400616+0000","last_peered":"2024-09-10T20:38:13.400616+0000","last_clean":"2024-09-10T20:38:13.400616+0000","last_became_active":"2024-09-10T20:38:11.398826+0000","last_became_peered":"2024-09-10T20:38:11.398826+0000","last_unstale":"2024-09-10T20:38:13.400616+0000","last_undegraded":"2024-09-10T20:38:13.400616+0000","last_fullsized":"2024-09-10T20:38:13.400616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:03.958637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343682+0000","last_change":"2024-09-10T20:38:11.398337+0000","last_active":"2024-09-10T20:38:14.343682+0000","last_peered":"2024-09-10T20:38:14.343682+0000","last_clean":"2024-09-10T20:38:14.343682+0000","last_became_active":"2024-09-10T20:38:11.398068+0000","last_became_peered":"2024-09-10T20:38:11.398068+0000","last_unstale":"2024-09-10T20:38:14.343682+0000","last_undegraded":"2024-09-10T20:38:14.343682+0000","last_fullsized":"2024-09-10T20:38:14.343682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:00:28.236504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":105,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-10T20:38:10.380805+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:10.380805+0000","last_peered":"2024-09-10T20:38:10.380805+0000","last_clean":"2024-09-10T20:38:10.380805+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:10.380805+0000","last_undegraded":"2024-09-10T20:38:10.380805+0000","last_fullsized":"2024-09-10T20:38:10.380805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377085+0000","last_change":"2024-09-10T20:38:10.377085+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377085+0000","last_undegraded":"2024-09-10T20:38:10.377085+0000","last_fullsized":"2024-09-10T20:38:10.377085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400902+0000","last_change":"2024-09-10T20:38:12.407263+0000","last_active":"2024-09-10T20:38:13.400902+0000","last_peered":"2024-09-10T20:38:13.400902+0000","last_clean":"2024-09-10T20:38:13.400902+0000","last_became_active":"2024-09-10T20:38:12.406952+0000","last_became_peered":"2024-09-10T20:38:12.406952+0000","last_unstale":"2024-09-10T20:38:13.400902+0000","last_undegraded":"2024-09-10T20:38:13.400902+0000","last_fullsized":"2024-09-10T20:38:13.400902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.171100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.376990+0000","last_change":"2024-09-10T20:38:10.376990+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.376990+0000","last_undegraded":"2024-09-10T20:38:10.376990+0000","last_fullsized":"2024-09-10T20:38:10.376990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343358+0000","last_change":"2024-09-10T20:38:12.407025+0000","last_active":"2024-09-10T20:38:14.343358+0000","last_peered":"2024-09-10T20:38:14.343358+0000","last_clean":"2024-09-10T20:38:14.343358+0000","last_became_active":"2024-09-10T20:38:12.406124+0000","last_became_peered":"2024-09-10T20:38:12.406124+0000","last_unstale":"2024-09-10T20:38:14.343358+0000","last_undegraded":"2024-09-10T20:38:14.343358+0000","last_fullsized":"2024-09-10T20:38:14.343358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633299+0000","last_change":"2024-09-10T20:38:11.397217+0000","last_active":"2024-09-10T20:38:13.633299+0000","last_peered":"2024-09-10T20:38:13.633299+0000","last_clean":"2024-09-10T20:38:13.633299+0000","last_became_active":"2024-09-10T20:38:11.396928+0000","last_became_peered":"2024-09-10T20:38:11.396928+0000","last_unstale":"2024-09-10T20:38:13.633299+0000","last_undegraded":"2024-09-10T20:38:13.633299+0000","last_fullsized":"2024-09-10T20:38:13.633299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:38.900087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401469+0000","last_change":"2024-09-10T20:38:12.407595+0000","last_active":"2024-09-10T20:38:13.401469+0000","last_peered":"2024-09-10T20:38:13.401469+0000","last_clean":"2024-09-10T20:38:13.401469+0000","last_became_active":"2024-09-10T20:38:12.407355+0000","last_became_peered":"2024-09-10T20:38:12.407355+0000","last_unstale":"2024-09-10T20:38:13.401469+0000","last_undegraded":"2024-09-10T20:38:13.401469+0000","last_fullsized":"2024-09-10T20:38:13.401469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:05.419998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377007+0000","last_change":"2024-09-10T20:38:10.377007+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377007+0000","last_undegraded":"2024-09-10T20:38:10.377007+0000","last_fullsized":"2024-09-10T20:38:10.377007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400255+0000","last_change":"2024-09-10T20:38:11.397463+0000","last_active":"2024-09-10T20:38:13.400255+0000","last_peered":"2024-09-10T20:38:13.400255+0000","last_clean":"2024-09-10T20:38:13.400255+0000","last_became_active":"2024-09-10T20:38:11.397205+0000","last_became_peered":"2024-09-10T20:38:11.397205+0000","last_unstale":"2024-09-10T20:38:13.400255+0000","last_undegraded":"2024-09-10T20:38:13.400255+0000","last_fullsized":"2024-09-10T20:38:13.400255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:53.003074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377042+0000","last_change":"2024-09-10T20:38:10.377042+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377042+0000","last_undegraded":"2024-09-10T20:38:10.377042+0000","last_fullsized":"2024-09-10T20:38:10.377042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343576+0000","last_change":"2024-09-10T20:38:12.408779+0000","last_active":"2024-09-10T20:38:14.343576+0000","last_peered":"2024-09-10T20:38:14.343576+0000","last_clean":"2024-09-10T20:38:14.343576+0000","last_became_active":"2024-09-10T20:38:12.408490+0000","last_became_peered":"2024-09-10T20:38:12.408490+0000","last_unstale":"2024-09-10T20:38:14.343576+0000","last_undegraded":"2024-09-10T20:38:14.343576+0000","last_fullsized":"2024-09-10T20:38:14.343576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:25.888623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407174+0000","last_change":"2024-09-10T20:38:11.394170+0000","last_active":"2024-09-10T20:38:13.407174+0000","last_peered":"2024-09-10T20:38:13.407174+0000","last_clean":"2024-09-10T20:38:13.407174+0000","last_became_active":"2024-09-10T20:38:11.393850+0000","last_became_peered":"2024-09-10T20:38:11.393850+0000","last_unstale":"2024-09-10T20:38:13.407174+0000","last_undegraded":"2024-09-10T20:38:13.407174+0000","last_fullsized":"2024-09-10T20:38:13.407174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:15.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401623+0000","last_change":"2024-09-10T20:38:12.404386+0000","last_active":"2024-09-10T20:38:13.401623+0000","last_peered":"2024-09-10T20:38:13.401623+0000","last_clean":"2024-09-10T20:38:13.401623+0000","last_became_active":"2024-09-10T20:38:12.404045+0000","last_became_peered":"2024-09-10T20:38:12.404045+0000","last_unstale":"2024-09-10T20:38:13.401623+0000","last_undegraded":"2024-09-10T20:38:13.401623+0000","last_fullsized":"2024-09-10T20:38:13.401623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:49.410849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377015+0000","last_change":"2024-09-10T20:38:10.377015+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377015+0000","last_undegraded":"2024-09-10T20:38:10.377015+0000","last_fullsized":"2024-09-10T20:38:10.377015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400970+0000","last_change":"2024-09-10T20:38:12.409091+0000","last_active":"2024-09-10T20:38:13.400970+0000","last_peered":"2024-09-10T20:38:13.400970+0000","last_clean":"2024-09-10T20:38:13.400970+0000","last_became_active":"2024-09-10T20:38:12.408815+0000","last_became_peered":"2024-09-10T20:38:12.408815+0000","last_unstale":"2024-09-10T20:38:13.400970+0000","last_undegraded":"2024-09-10T20:38:13.400970+0000","last_fullsized":"2024-09-10T20:38:13.400970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.870149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.382686+0000","last_change":"2024-09-10T20:38:10.382686+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.382686+0000","last_undegraded":"2024-09-10T20:38:10.382686+0000","last_fullsized":"2024-09-10T20:38:10.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401287+0000","last_change":"2024-09-10T20:38:11.397172+0000","last_active":"2024-09-10T20:38:13.401287+0000","last_peered":"2024-09-10T20:38:13.401287+0000","last_clean":"2024-09-10T20:38:13.401287+0000","last_became_active":"2024-09-10T20:38:11.396670+0000","last_became_peered":"2024-09-10T20:38:11.396670+0000","last_unstale":"2024-09-10T20:38:13.401287+0000","last_undegraded":"2024-09-10T20:38:13.401287+0000","last_fullsized":"2024-09-10T20:38:13.401287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400195+0000","last_change":"2024-09-10T20:38:12.405875+0000","last_active":"2024-09-10T20:38:13.400195+0000","last_peered":"2024-09-10T20:38:13.400195+0000","last_clean":"2024-09-10T20:38:13.400195+0000","last_became_active":"2024-09-10T20:38:12.405583+0000","last_became_peered":"2024-09-10T20:38:12.405583+0000","last_unstale":"2024-09-10T20:38:13.400195+0000","last_undegraded":"2024-09-10T20:38:13.400195+0000","last_fullsized":"2024-09-10T20:38:13.400195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:30.185620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343471+0000","last_change":"2024-09-10T20:38:11.395992+0000","last_active":"2024-09-10T20:38:14.343471+0000","last_peered":"2024-09-10T20:38:14.343471+0000","last_clean":"2024-09-10T20:38:14.343471+0000","last_became_active":"2024-09-10T20:38:11.394935+0000","last_became_peered":"2024-09-10T20:38:11.394935+0000","last_unstale":"2024-09-10T20:38:14.343471+0000","last_undegraded":"2024-09-10T20:38:14.343471+0000","last_fullsized":"2024-09-10T20:38:14.343471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:49.040222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633574+0000","last_change":"2024-09-10T20:38:12.412620+0000","last_active":"2024-09-10T20:38:13.633574+0000","last_peered":"2024-09-10T20:38:13.633574+0000","last_clean":"2024-09-10T20:38:13.633574+0000","last_became_active":"2024-09-10T20:38:12.412329+0000","last_became_peered":"2024-09-10T20:38:12.412329+0000","last_unstale":"2024-09-10T20:38:13.633574+0000","last_undegraded":"2024-09-10T20:38:13.633574+0000","last_fullsized":"2024-09-10T20:38:13.633574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:40.609805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.377791+0000","last_change":"2024-09-10T20:38:10.377791+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.377791+0000","last_undegraded":"2024-09-10T20:38:10.377791+0000","last_fullsized":"2024-09-10T20:38:10.377791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381267+0000","last_change":"2024-09-10T20:38:10.381267+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381267+0000","last_undegraded":"2024-09-10T20:38:10.381267+0000","last_fullsized":"2024-09-10T20:38:10.381267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:11.372745+0000","last_change":"2024-09-10T20:38:11.372745+0000","last_active":"2024-09-10T20:38:11.372745+0000","last_peered":"2024-09-10T20:38:11.372745+0000","last_clean":"2024-09-10T20:38:11.372745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:11.372745+0000","last_undegraded":"2024-09-10T20:38:11.372745+0000","last_fullsized":"2024-09-10T20:38:11.372745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-10T20:38:10.381405+0000","last_change":"2024-09-10T20:38:10.381405+0000","last_active":"2024-09-10T20:38:10.367851+0000","last_peered":"2024-09-10T20:38:10.367851+0000","last_clean":"2024-09-10T20:38:10.367851+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:10.381405+0000","last_undegraded":"2024-09-10T20:38:10.381405+0000","last_fullsized":"2024-09-10T20:38:10.381405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401905+0000","last_change":"2024-09-10T20:38:12.416588+0000","last_active":"2024-09-10T20:38:13.401905+0000","last_peered":"2024-09-10T20:38:13.401905+0000","last_clean":"2024-09-10T20:38:13.401905+0000","last_became_active":"2024-09-10T20:38:12.416297+0000","last_became_peered":"2024-09-10T20:38:12.416297+0000","last_unstale":"2024-09-10T20:38:13.401905+0000","last_undegraded":"2024-09-10T20:38:13.401905+0000","last_fullsized":"2024-09-10T20:38:13.401905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401853+0000","last_change":"2024-09-10T20:38:11.396382+0000","last_active":"2024-09-10T20:38:13.401853+0000","last_peered":"2024-09-10T20:38:13.401853+0000","last_clean":"2024-09-10T20:38:13.401853+0000","last_became_active":"2024-09-10T20:38:11.395934+0000","last_became_peered":"2024-09-10T20:38:11.395934+0000","last_unstale":"2024-09-10T20:38:13.401853+0000","last_undegraded":"2024-09-10T20:38:13.401853+0000","last_fullsized":"2024-09-10T20:38:13.401853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:20:39.553068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401634+0000","last_change":"2024-09-10T20:38:12.402624+0000","last_active":"2024-09-10T20:38:13.401634+0000","last_peered":"2024-09-10T20:38:13.401634+0000","last_clean":"2024-09-10T20:38:13.401634+0000","last_became_active":"2024-09-10T20:38:12.401924+0000","last_became_peered":"2024-09-10T20:38:12.401924+0000","last_unstale":"2024-09-10T20:38:13.401634+0000","last_undegraded":"2024-09-10T20:38:13.401634+0000","last_fullsized":"2024-09-10T20:38:13.401634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:33.683744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401708+0000","last_change":"2024-09-10T20:38:11.401264+0000","last_active":"2024-09-10T20:38:13.401708+0000","last_peered":"2024-09-10T20:38:13.401708+0000","last_clean":"2024-09-10T20:38:13.401708+0000","last_became_active":"2024-09-10T20:38:11.400986+0000","last_became_peered":"2024-09-10T20:38:11.400986+0000","last_unstale":"2024-09-10T20:38:13.401708+0000","last_undegraded":"2024-09-10T20:38:13.401708+0000","last_fullsized":"2024-09-10T20:38:13.401708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:50.668458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":776,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":172032,"data_stored":53177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":753664,"data_stored":637728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":159744,"data_stored":47360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":66227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":662987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":74811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:20.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/1981249830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d0c80 msgr2=0x7fdd6415bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/1981249830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d0c80 0x7fdd6415bc40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdd58005fd0 tx=0x7fdd5802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:38:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/1981249830 shutdown_connections 2024-09-10T20:38:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/1981249830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd64070280 0x7fdd64070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/1981249830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d0c80 0x7fdd6415bc40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/1981249830 >> 172.21.15.28:0/1981249830 conn(0x7fdd6406dbf0 msgr2=0x7fdd6406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/1981249830 shutdown_connections 2024-09-10T20:38:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.987+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/1981249830 wait complete. 2024-09-10T20:38:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.988+0000 7fdd6ab0e640 1 Processor -- start 2024-09-10T20:38:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.988+0000 7fdd6ab0e640 1 -- start start 2024-09-10T20:38:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd6ab0e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd64070280 0x7fdd641f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd6ab0e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd640d0c80 0x7fdd64159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd6ab0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d1060 0x7fdd64159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd6ab0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6415ef70 con 0x7fdd640d1060 2024-09-10T20:38:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd6ab0e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6415edf0 con 0x7fdd64070280 2024-09-10T20:38:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd6ab0e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdd64070bc0 con 0x7fdd640d0c80 2024-09-10T20:38:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd690ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d1060 0x7fdd64159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd690ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d1060 0x7fdd64159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50742/0 (socket says 172.21.15.28:50742) 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd690ab640 1 -- 172.21.15.28:0/4035527314 learned_addr learned my addr 172.21.15.28:0/4035527314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd688aa640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd64070280 0x7fdd641f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd63fff640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd640d0c80 0x7fdd64159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd690ab640 1 -- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd640d0c80 msgr2=0x7fdd64159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd690ab640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd640d0c80 0x7fdd64159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.989+0000 7fdd690ab640 1 -- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd64070280 msgr2=0x7fdd641f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd690ab640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd64070280 0x7fdd641f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd690ab640 1 -- 172.21.15.28:0/4035527314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd640ae770 con 0x7fdd640d1060 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd690ab640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d1060 0x7fdd64159e40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00eb40 tx=0x7fdd5c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd617fa640 1 -- 172.21.15.28:0/4035527314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c004b20 con 0x7fdd640d1060 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd617fa640 1 -- 172.21.15.28:0/4035527314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd5c004ce0 con 0x7fdd640d1060 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd64151400 con 0x7fdd640d1060 2024-09-10T20:38:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd617fa640 1 -- 172.21.15.28:0/4035527314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c005280 con 0x7fdd640d1060 2024-09-10T20:38:20.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.990+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd64151120 con 0x7fdd640d1060 2024-09-10T20:38:20.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.991+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd64070680 con 0x7fdd640d1060 2024-09-10T20:38:20.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.992+0000 7fdd617fa640 1 -- 172.21.15.28:0/4035527314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdd5c005420 con 0x7fdd640d1060 2024-09-10T20:38:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.992+0000 7fdd617fa640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd48077f70 0x7fdd4807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.992+0000 7fdd617fa640 1 -- 172.21.15.28:0/4035527314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdd5c09a950 con 0x7fdd640d1060 2024-09-10T20:38:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.992+0000 7fdd688aa640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd48077f70 0x7fdd4807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.993+0000 7fdd688aa640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd48077f70 0x7fdd4807a430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd58033000 tx=0x7fdd58004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:20.997+0000 7fdd617fa640 1 -- 172.21.15.28:0/4035527314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdd5c062e90 con 0x7fdd640d1060 2024-09-10T20:38:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.177+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd640b4d20 con 0x7fdd48077f70 2024-09-10T20:38:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.181+0000 7fdd617fa640 1 -- 172.21.15.28:0/4035527314 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251930 (secure 0 0 0) 0x7fdd640b4d20 con 0x7fdd48077f70 2024-09-10T20:38:21.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.184+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd48077f70 msgr2=0x7fdd4807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.184+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd48077f70 0x7fdd4807a430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd58033000 tx=0x7fdd58004730 comp rx=0 tx=0).stop 2024-09-10T20:38:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.184+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d1060 msgr2=0x7fdd64159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.184+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d1060 0x7fdd64159e40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00eb40 tx=0x7fdd5c00c2d0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 shutdown_connections 2024-09-10T20:38:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd48077f70 0x7fdd4807a430 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd640d1060 0x7fdd64159e40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd640d0c80 0x7fdd64159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 --2- 172.21.15.28:0/4035527314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd64070280 0x7fdd641f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 >> 172.21.15.28:0/4035527314 conn(0x7fdd6406dbf0 msgr2=0x7fdd6415c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 shutdown_connections 2024-09-10T20:38:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.185+0000 7fdd6ab0e640 1 -- 172.21.15.28:0/4035527314 wait complete. 2024-09-10T20:38:21.199 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-10T20:38:19.977911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235732,"kb_used_data":2844,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738988,"statfs":{"total":773094113280,"available":772852723712,"internally_reserved":0,"allocated":2912256,"data_stored":1759472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001962"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408702+0000","last_change":"2024-09-10T20:38:12.400371+0000","last_active":"2024-09-10T20:38:13.408702+0000","last_peered":"2024-09-10T20:38:13.408702+0000","last_clean":"2024-09-10T20:38:13.408702+0000","last_became_active":"2024-09-10T20:38:12.400049+0000","last_became_peered":"2024-09-10T20:38:12.400049+0000","last_unstale":"2024-09-10T20:38:13.408702+0000","last_undegraded":"2024-09-10T20:38:13.408702+0000","last_fullsized":"2024-09-10T20:38:13.408702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:47.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400675+0000","last_change":"2024-09-10T20:38:12.413418+0000","last_active":"2024-09-10T20:38:13.400675+0000","last_peered":"2024-09-10T20:38:13.400675+0000","last_clean":"2024-09-10T20:38:13.400675+0000","last_became_active":"2024-09-10T20:38:12.413063+0000","last_became_peered":"2024-09-10T20:38:12.413063+0000","last_unstale":"2024-09-10T20:38:13.400675+0000","last_undegraded":"2024-09-10T20:38:13.400675+0000","last_fullsized":"2024-09-10T20:38:13.400675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:12.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401706+0000","last_change":"2024-09-10T20:38:12.404482+0000","last_active":"2024-09-10T20:38:13.401706+0000","last_peered":"2024-09-10T20:38:13.401706+0000","last_clean":"2024-09-10T20:38:13.401706+0000","last_became_active":"2024-09-10T20:38:12.404181+0000","last_became_peered":"2024-09-10T20:38:12.404181+0000","last_unstale":"2024-09-10T20:38:13.401706+0000","last_undegraded":"2024-09-10T20:38:13.401706+0000","last_fullsized":"2024-09-10T20:38:13.401706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:02.847206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343826+0000","last_change":"2024-09-10T20:38:12.399863+0000","last_active":"2024-09-10T20:38:14.343826+0000","last_peered":"2024-09-10T20:38:14.343826+0000","last_clean":"2024-09-10T20:38:14.343826+0000","last_became_active":"2024-09-10T20:38:12.399541+0000","last_became_peered":"2024-09-10T20:38:12.399541+0000","last_unstale":"2024-09-10T20:38:14.343826+0000","last_undegraded":"2024-09-10T20:38:14.343826+0000","last_fullsized":"2024-09-10T20:38:14.343826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:45.518334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408418+0000","last_change":"2024-09-10T20:38:12.403977+0000","last_active":"2024-09-10T20:38:13.408418+0000","last_peered":"2024-09-10T20:38:13.408418+0000","last_clean":"2024-09-10T20:38:13.408418+0000","last_became_active":"2024-09-10T20:38:12.403575+0000","last_became_peered":"2024-09-10T20:38:12.403575+0000","last_unstale":"2024-09-10T20:38:13.408418+0000","last_undegraded":"2024-09-10T20:38:13.408418+0000","last_fullsized":"2024-09-10T20:38:13.408418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:37.571212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344323+0000","last_change":"2024-09-10T20:38:12.399990+0000","last_active":"2024-09-10T20:38:14.344323+0000","last_peered":"2024-09-10T20:38:14.344323+0000","last_clean":"2024-09-10T20:38:14.344323+0000","last_became_active":"2024-09-10T20:38:12.399633+0000","last_became_peered":"2024-09-10T20:38:12.399633+0000","last_unstale":"2024-09-10T20:38:14.344323+0000","last_undegraded":"2024-09-10T20:38:14.344323+0000","last_fullsized":"2024-09-10T20:38:14.344323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:01:29.734867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407809+0000","last_change":"2024-09-10T20:38:12.401934+0000","last_active":"2024-09-10T20:38:13.407809+0000","last_peered":"2024-09-10T20:38:13.407809+0000","last_clean":"2024-09-10T20:38:13.407809+0000","last_became_active":"2024-09-10T20:38:12.401547+0000","last_became_peered":"2024-09-10T20:38:12.401547+0000","last_unstale":"2024-09-10T20:38:13.407809+0000","last_undegraded":"2024-09-10T20:38:13.407809+0000","last_fullsized":"2024-09-10T20:38:13.407809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:21.949782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401300+0000","last_change":"2024-09-10T20:38:12.403203+0000","last_active":"2024-09-10T20:38:13.401300+0000","last_peered":"2024-09-10T20:38:13.401300+0000","last_clean":"2024-09-10T20:38:13.401300+0000","last_became_active":"2024-09-10T20:38:12.402272+0000","last_became_peered":"2024-09-10T20:38:12.402272+0000","last_unstale":"2024-09-10T20:38:13.401300+0000","last_undegraded":"2024-09-10T20:38:13.401300+0000","last_fullsized":"2024-09-10T20:38:13.401300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.399062+0000","last_change":"2024-09-10T20:38:12.420252+0000","last_active":"2024-09-10T20:38:13.399062+0000","last_peered":"2024-09-10T20:38:13.399062+0000","last_clean":"2024-09-10T20:38:13.399062+0000","last_became_active":"2024-09-10T20:38:12.419848+0000","last_became_peered":"2024-09-10T20:38:12.419848+0000","last_unstale":"2024-09-10T20:38:13.399062+0000","last_undegraded":"2024-09-10T20:38:13.399062+0000","last_fullsized":"2024-09-10T20:38:13.399062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:13.317965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408005+0000","last_change":"2024-09-10T20:38:12.398378+0000","last_active":"2024-09-10T20:38:13.408005+0000","last_peered":"2024-09-10T20:38:13.408005+0000","last_clean":"2024-09-10T20:38:13.408005+0000","last_became_active":"2024-09-10T20:38:12.398068+0000","last_became_peered":"2024-09-10T20:38:12.398068+0000","last_unstale":"2024-09-10T20:38:13.408005+0000","last_undegraded":"2024-09-10T20:38:13.408005+0000","last_fullsized":"2024-09-10T20:38:13.408005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408341+0000","last_change":"2024-09-10T20:38:12.401136+0000","last_active":"2024-09-10T20:38:13.408341+0000","last_peered":"2024-09-10T20:38:13.408341+0000","last_clean":"2024-09-10T20:38:13.408341+0000","last_became_active":"2024-09-10T20:38:12.400835+0000","last_became_peered":"2024-09-10T20:38:12.400835+0000","last_unstale":"2024-09-10T20:38:13.408341+0000","last_undegraded":"2024-09-10T20:38:13.408341+0000","last_fullsized":"2024-09-10T20:38:13.408341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:37:58.944607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408006+0000","last_change":"2024-09-10T20:38:12.417280+0000","last_active":"2024-09-10T20:38:13.408006+0000","last_peered":"2024-09-10T20:38:13.408006+0000","last_clean":"2024-09-10T20:38:13.408006+0000","last_became_active":"2024-09-10T20:38:12.416991+0000","last_became_peered":"2024-09-10T20:38:12.416991+0000","last_unstale":"2024-09-10T20:38:13.408006+0000","last_undegraded":"2024-09-10T20:38:13.408006+0000","last_fullsized":"2024-09-10T20:38:13.408006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:36.789848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343562+0000","last_change":"2024-09-10T20:38:12.415360+0000","last_active":"2024-09-10T20:38:14.343562+0000","last_peered":"2024-09-10T20:38:14.343562+0000","last_clean":"2024-09-10T20:38:14.343562+0000","last_became_active":"2024-09-10T20:38:12.415094+0000","last_became_peered":"2024-09-10T20:38:12.415094+0000","last_unstale":"2024-09-10T20:38:14.343562+0000","last_undegraded":"2024-09-10T20:38:14.343562+0000","last_fullsized":"2024-09-10T20:38:14.343562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:36.623506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343324+0000","last_change":"2024-09-10T20:38:12.411600+0000","last_active":"2024-09-10T20:38:14.343324+0000","last_peered":"2024-09-10T20:38:14.343324+0000","last_clean":"2024-09-10T20:38:14.343324+0000","last_became_active":"2024-09-10T20:38:12.411328+0000","last_became_peered":"2024-09-10T20:38:12.411328+0000","last_unstale":"2024-09-10T20:38:14.343324+0000","last_undegraded":"2024-09-10T20:38:14.343324+0000","last_fullsized":"2024-09-10T20:38:14.343324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:30.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400233+0000","last_change":"2024-09-10T20:38:12.404816+0000","last_active":"2024-09-10T20:38:13.400233+0000","last_peered":"2024-09-10T20:38:13.400233+0000","last_clean":"2024-09-10T20:38:13.400233+0000","last_became_active":"2024-09-10T20:38:12.404568+0000","last_became_peered":"2024-09-10T20:38:12.404568+0000","last_unstale":"2024-09-10T20:38:13.400233+0000","last_undegraded":"2024-09-10T20:38:13.400233+0000","last_fullsized":"2024-09-10T20:38:13.400233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:05:49.714066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343559+0000","last_change":"2024-09-10T20:38:12.404745+0000","last_active":"2024-09-10T20:38:14.343559+0000","last_peered":"2024-09-10T20:38:14.343559+0000","last_clean":"2024-09-10T20:38:14.343559+0000","last_became_active":"2024-09-10T20:38:12.404429+0000","last_became_peered":"2024-09-10T20:38:12.404429+0000","last_unstale":"2024-09-10T20:38:14.343559+0000","last_undegraded":"2024-09-10T20:38:14.343559+0000","last_fullsized":"2024-09-10T20:38:14.343559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344641+0000","last_change":"2024-09-10T20:38:12.404923+0000","last_active":"2024-09-10T20:38:14.344641+0000","last_peered":"2024-09-10T20:38:14.344641+0000","last_clean":"2024-09-10T20:38:14.344641+0000","last_became_active":"2024-09-10T20:38:12.404604+0000","last_became_peered":"2024-09-10T20:38:12.404604+0000","last_unstale":"2024-09-10T20:38:14.344641+0000","last_undegraded":"2024-09-10T20:38:14.344641+0000","last_fullsized":"2024-09-10T20:38:14.344641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:27.005436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633628+0000","last_change":"2024-09-10T20:38:12.407265+0000","last_active":"2024-09-10T20:38:13.633628+0000","last_peered":"2024-09-10T20:38:13.633628+0000","last_clean":"2024-09-10T20:38:13.633628+0000","last_became_active":"2024-09-10T20:38:12.406927+0000","last_became_peered":"2024-09-10T20:38:12.406927+0000","last_unstale":"2024-09-10T20:38:13.633628+0000","last_undegraded":"2024-09-10T20:38:13.633628+0000","last_fullsized":"2024-09-10T20:38:13.633628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:14.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344605+0000","last_change":"2024-09-10T20:38:12.411419+0000","last_active":"2024-09-10T20:38:14.344605+0000","last_peered":"2024-09-10T20:38:14.344605+0000","last_clean":"2024-09-10T20:38:14.344605+0000","last_became_active":"2024-09-10T20:38:12.411088+0000","last_became_peered":"2024-09-10T20:38:12.411088+0000","last_unstale":"2024-09-10T20:38:14.344605+0000","last_undegraded":"2024-09-10T20:38:14.344605+0000","last_fullsized":"2024-09-10T20:38:14.344605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:56.228254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408072+0000","last_change":"2024-09-10T20:38:12.416514+0000","last_active":"2024-09-10T20:38:13.408072+0000","last_peered":"2024-09-10T20:38:13.408072+0000","last_clean":"2024-09-10T20:38:13.408072+0000","last_became_active":"2024-09-10T20:38:12.416216+0000","last_became_peered":"2024-09-10T20:38:12.416216+0000","last_unstale":"2024-09-10T20:38:13.408072+0000","last_undegraded":"2024-09-10T20:38:13.408072+0000","last_fullsized":"2024-09-10T20:38:13.408072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:26.545836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400878+0000","last_change":"2024-09-10T20:38:12.407874+0000","last_active":"2024-09-10T20:38:13.400878+0000","last_peered":"2024-09-10T20:38:13.400878+0000","last_clean":"2024-09-10T20:38:13.400878+0000","last_became_active":"2024-09-10T20:38:12.407623+0000","last_became_peered":"2024-09-10T20:38:12.407623+0000","last_unstale":"2024-09-10T20:38:13.400878+0000","last_undegraded":"2024-09-10T20:38:13.400878+0000","last_fullsized":"2024-09-10T20:38:13.400878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:56.990230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398460+0000","last_change":"2024-09-10T20:38:12.414007+0000","last_active":"2024-09-10T20:38:13.398460+0000","last_peered":"2024-09-10T20:38:13.398460+0000","last_clean":"2024-09-10T20:38:13.398460+0000","last_became_active":"2024-09-10T20:38:12.413744+0000","last_became_peered":"2024-09-10T20:38:12.413744+0000","last_unstale":"2024-09-10T20:38:13.398460+0000","last_undegraded":"2024-09-10T20:38:13.398460+0000","last_fullsized":"2024-09-10T20:38:13.398460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.915632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400518+0000","last_change":"2024-09-10T20:38:12.408020+0000","last_active":"2024-09-10T20:38:13.400518+0000","last_peered":"2024-09-10T20:38:13.400518+0000","last_clean":"2024-09-10T20:38:13.400518+0000","last_became_active":"2024-09-10T20:38:12.407757+0000","last_became_peered":"2024-09-10T20:38:12.407757+0000","last_unstale":"2024-09-10T20:38:13.400518+0000","last_undegraded":"2024-09-10T20:38:13.400518+0000","last_fullsized":"2024-09-10T20:38:13.400518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:32.513086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343533+0000","last_change":"2024-09-10T20:38:12.407235+0000","last_active":"2024-09-10T20:38:14.343533+0000","last_peered":"2024-09-10T20:38:14.343533+0000","last_clean":"2024-09-10T20:38:14.343533+0000","last_became_active":"2024-09-10T20:38:12.406414+0000","last_became_peered":"2024-09-10T20:38:12.406414+0000","last_unstale":"2024-09-10T20:38:14.343533+0000","last_undegraded":"2024-09-10T20:38:14.343533+0000","last_fullsized":"2024-09-10T20:38:14.343533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:37.571212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401654+0000","last_change":"2024-09-10T20:38:12.416773+0000","last_active":"2024-09-10T20:38:13.401654+0000","last_peered":"2024-09-10T20:38:13.401654+0000","last_clean":"2024-09-10T20:38:13.401654+0000","last_became_active":"2024-09-10T20:38:12.416498+0000","last_became_peered":"2024-09-10T20:38:12.416498+0000","last_unstale":"2024-09-10T20:38:13.401654+0000","last_undegraded":"2024-09-10T20:38:13.401654+0000","last_fullsized":"2024-09-10T20:38:13.401654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:42.492324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344477+0000","last_change":"2024-09-10T20:38:12.407692+0000","last_active":"2024-09-10T20:38:14.344477+0000","last_peered":"2024-09-10T20:38:14.344477+0000","last_clean":"2024-09-10T20:38:14.344477+0000","last_became_active":"2024-09-10T20:38:12.406817+0000","last_became_peered":"2024-09-10T20:38:12.406817+0000","last_unstale":"2024-09-10T20:38:14.344477+0000","last_undegraded":"2024-09-10T20:38:14.344477+0000","last_fullsized":"2024-09-10T20:38:14.344477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:13.317965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343671+0000","last_change":"2024-09-10T20:38:12.406673+0000","last_active":"2024-09-10T20:38:14.343671+0000","last_peered":"2024-09-10T20:38:14.343671+0000","last_clean":"2024-09-10T20:38:14.343671+0000","last_became_active":"2024-09-10T20:38:12.406401+0000","last_became_peered":"2024-09-10T20:38:12.406401+0000","last_unstale":"2024-09-10T20:38:14.343671+0000","last_undegraded":"2024-09-10T20:38:14.343671+0000","last_fullsized":"2024-09-10T20:38:14.343671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:31.845414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343502+0000","last_change":"2024-09-10T20:38:12.414290+0000","last_active":"2024-09-10T20:38:14.343502+0000","last_peered":"2024-09-10T20:38:14.343502+0000","last_clean":"2024-09-10T20:38:14.343502+0000","last_became_active":"2024-09-10T20:38:12.413712+0000","last_became_peered":"2024-09-10T20:38:12.413712+0000","last_unstale":"2024-09-10T20:38:14.343502+0000","last_undegraded":"2024-09-10T20:38:14.343502+0000","last_fullsized":"2024-09-10T20:38:14.343502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:48.033101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400477+0000","last_change":"2024-09-10T20:38:12.406921+0000","last_active":"2024-09-10T20:38:13.400477+0000","last_peered":"2024-09-10T20:38:13.400477+0000","last_clean":"2024-09-10T20:38:13.400477+0000","last_became_active":"2024-09-10T20:38:12.406642+0000","last_became_peered":"2024-09-10T20:38:12.406642+0000","last_unstale":"2024-09-10T20:38:13.400477+0000","last_undegraded":"2024-09-10T20:38:13.400477+0000","last_fullsized":"2024-09-10T20:38:13.400477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:02.877370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343808+0000","last_change":"2024-09-10T20:38:12.406428+0000","last_active":"2024-09-10T20:38:14.343808+0000","last_peered":"2024-09-10T20:38:14.343808+0000","last_clean":"2024-09-10T20:38:14.343808+0000","last_became_active":"2024-09-10T20:38:12.405117+0000","last_became_peered":"2024-09-10T20:38:12.405117+0000","last_unstale":"2024-09-10T20:38:14.343808+0000","last_undegraded":"2024-09-10T20:38:14.343808+0000","last_fullsized":"2024-09-10T20:38:14.343808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:28.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633368+0000","last_change":"2024-09-10T20:38:12.415776+0000","last_active":"2024-09-10T20:38:13.633368+0000","last_peered":"2024-09-10T20:38:13.633368+0000","last_clean":"2024-09-10T20:38:13.633368+0000","last_became_active":"2024-09-10T20:38:12.415480+0000","last_became_peered":"2024-09-10T20:38:12.415480+0000","last_unstale":"2024-09-10T20:38:13.633368+0000","last_undegraded":"2024-09-10T20:38:13.633368+0000","last_fullsized":"2024-09-10T20:38:13.633368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:13.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343528+0000","last_change":"2024-09-10T20:38:12.405334+0000","last_active":"2024-09-10T20:38:14.343528+0000","last_peered":"2024-09-10T20:38:14.343528+0000","last_clean":"2024-09-10T20:38:14.343528+0000","last_became_active":"2024-09-10T20:38:12.405033+0000","last_became_peered":"2024-09-10T20:38:12.405033+0000","last_unstale":"2024-09-10T20:38:14.343528+0000","last_undegraded":"2024-09-10T20:38:14.343528+0000","last_fullsized":"2024-09-10T20:38:14.343528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:01:45.285223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398727+0000","last_change":"2024-09-10T20:38:12.412798+0000","last_active":"2024-09-10T20:38:13.398727+0000","last_peered":"2024-09-10T20:38:13.398727+0000","last_clean":"2024-09-10T20:38:13.398727+0000","last_became_active":"2024-09-10T20:38:12.412428+0000","last_became_peered":"2024-09-10T20:38:12.412428+0000","last_unstale":"2024-09-10T20:38:13.398727+0000","last_undegraded":"2024-09-10T20:38:13.398727+0000","last_fullsized":"2024-09-10T20:38:13.398727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:45.518334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400454+0000","last_change":"2024-09-10T20:38:11.394869+0000","last_active":"2024-09-10T20:38:13.400454+0000","last_peered":"2024-09-10T20:38:13.400454+0000","last_clean":"2024-09-10T20:38:13.400454+0000","last_became_active":"2024-09-10T20:38:11.394564+0000","last_became_peered":"2024-09-10T20:38:11.394564+0000","last_unstale":"2024-09-10T20:38:13.400454+0000","last_undegraded":"2024-09-10T20:38:13.400454+0000","last_fullsized":"2024-09-10T20:38:13.400454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:19.365550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401125+0000","last_change":"2024-09-10T20:38:12.401521+0000","last_active":"2024-09-10T20:38:13.401125+0000","last_peered":"2024-09-10T20:38:13.401125+0000","last_clean":"2024-09-10T20:38:13.401125+0000","last_became_active":"2024-09-10T20:38:12.401218+0000","last_became_peered":"2024-09-10T20:38:12.401218+0000","last_unstale":"2024-09-10T20:38:13.401125+0000","last_undegraded":"2024-09-10T20:38:13.401125+0000","last_fullsized":"2024-09-10T20:38:13.401125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:33.954998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343882+0000","last_change":"2024-09-10T20:38:11.401029+0000","last_active":"2024-09-10T20:38:14.343882+0000","last_peered":"2024-09-10T20:38:14.343882+0000","last_clean":"2024-09-10T20:38:14.343882+0000","last_became_active":"2024-09-10T20:38:11.400647+0000","last_became_peered":"2024-09-10T20:38:11.400647+0000","last_unstale":"2024-09-10T20:38:14.343882+0000","last_undegraded":"2024-09-10T20:38:14.343882+0000","last_fullsized":"2024-09-10T20:38:14.343882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:38.998000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343167+0000","last_change":"2024-09-10T20:38:11.398400+0000","last_active":"2024-09-10T20:38:14.343167+0000","last_peered":"2024-09-10T20:38:14.343167+0000","last_clean":"2024-09-10T20:38:14.343167+0000","last_became_active":"2024-09-10T20:38:11.398097+0000","last_became_peered":"2024-09-10T20:38:11.398097+0000","last_unstale":"2024-09-10T20:38:14.343167+0000","last_undegraded":"2024-09-10T20:38:14.343167+0000","last_fullsized":"2024-09-10T20:38:14.343167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.599838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401478+0000","last_change":"2024-09-10T20:38:12.413177+0000","last_active":"2024-09-10T20:38:13.401478+0000","last_peered":"2024-09-10T20:38:13.401478+0000","last_clean":"2024-09-10T20:38:13.401478+0000","last_became_active":"2024-09-10T20:38:12.412925+0000","last_became_peered":"2024-09-10T20:38:12.412925+0000","last_unstale":"2024-09-10T20:38:13.401478+0000","last_undegraded":"2024-09-10T20:38:13.401478+0000","last_fullsized":"2024-09-10T20:38:13.401478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:18.693832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343616+0000","last_change":"2024-09-10T20:38:11.399996+0000","last_active":"2024-09-10T20:38:14.343616+0000","last_peered":"2024-09-10T20:38:14.343616+0000","last_clean":"2024-09-10T20:38:14.343616+0000","last_became_active":"2024-09-10T20:38:11.399423+0000","last_became_peered":"2024-09-10T20:38:11.399423+0000","last_unstale":"2024-09-10T20:38:14.343616+0000","last_undegraded":"2024-09-10T20:38:14.343616+0000","last_fullsized":"2024-09-10T20:38:14.343616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:35.628735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401415+0000","last_change":"2024-09-10T20:38:12.415606+0000","last_active":"2024-09-10T20:38:13.401415+0000","last_peered":"2024-09-10T20:38:13.401415+0000","last_clean":"2024-09-10T20:38:13.401415+0000","last_became_active":"2024-09-10T20:38:12.415330+0000","last_became_peered":"2024-09-10T20:38:12.415330+0000","last_unstale":"2024-09-10T20:38:13.401415+0000","last_undegraded":"2024-09-10T20:38:13.401415+0000","last_fullsized":"2024-09-10T20:38:13.401415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:53.050014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398942+0000","last_change":"2024-09-10T20:38:11.395121+0000","last_active":"2024-09-10T20:38:13.398942+0000","last_peered":"2024-09-10T20:38:13.398942+0000","last_clean":"2024-09-10T20:38:13.398942+0000","last_became_active":"2024-09-10T20:38:11.394490+0000","last_became_peered":"2024-09-10T20:38:11.394490+0000","last_unstale":"2024-09-10T20:38:13.398942+0000","last_undegraded":"2024-09-10T20:38:13.398942+0000","last_fullsized":"2024-09-10T20:38:13.398942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:42.485721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344515+0000","last_change":"2024-09-10T20:38:12.402641+0000","last_active":"2024-09-10T20:38:14.344515+0000","last_peered":"2024-09-10T20:38:14.344515+0000","last_clean":"2024-09-10T20:38:14.344515+0000","last_became_active":"2024-09-10T20:38:12.402308+0000","last_became_peered":"2024-09-10T20:38:12.402308+0000","last_unstale":"2024-09-10T20:38:14.344515+0000","last_undegraded":"2024-09-10T20:38:14.344515+0000","last_fullsized":"2024-09-10T20:38:14.344515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408013+0000","last_change":"2024-09-10T20:38:11.396878+0000","last_active":"2024-09-10T20:38:13.408013+0000","last_peered":"2024-09-10T20:38:13.408013+0000","last_clean":"2024-09-10T20:38:13.408013+0000","last_became_active":"2024-09-10T20:38:11.396522+0000","last_became_peered":"2024-09-10T20:38:11.396522+0000","last_unstale":"2024-09-10T20:38:13.408013+0000","last_undegraded":"2024-09-10T20:38:13.408013+0000","last_fullsized":"2024-09-10T20:38:13.408013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:43.005710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401902+0000","last_change":"2024-09-10T20:38:12.410504+0000","last_active":"2024-09-10T20:38:13.401902+0000","last_peered":"2024-09-10T20:38:13.401902+0000","last_clean":"2024-09-10T20:38:13.401902+0000","last_became_active":"2024-09-10T20:38:12.410008+0000","last_became_peered":"2024-09-10T20:38:12.410008+0000","last_unstale":"2024-09-10T20:38:13.401902+0000","last_undegraded":"2024-09-10T20:38:13.401902+0000","last_fullsized":"2024-09-10T20:38:13.401902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:57.544501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633468+0000","last_change":"2024-09-10T20:38:11.395962+0000","last_active":"2024-09-10T20:38:13.633468+0000","last_peered":"2024-09-10T20:38:13.633468+0000","last_clean":"2024-09-10T20:38:13.633468+0000","last_became_active":"2024-09-10T20:38:11.395687+0000","last_became_peered":"2024-09-10T20:38:11.395687+0000","last_unstale":"2024-09-10T20:38:13.633468+0000","last_undegraded":"2024-09-10T20:38:13.633468+0000","last_fullsized":"2024-09-10T20:38:13.633468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:15.421827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344171+0000","last_change":"2024-09-10T20:38:12.405652+0000","last_active":"2024-09-10T20:38:14.344171+0000","last_peered":"2024-09-10T20:38:14.344171+0000","last_clean":"2024-09-10T20:38:14.344171+0000","last_became_active":"2024-09-10T20:38:12.405250+0000","last_became_peered":"2024-09-10T20:38:12.405250+0000","last_unstale":"2024-09-10T20:38:14.344171+0000","last_undegraded":"2024-09-10T20:38:14.344171+0000","last_fullsized":"2024-09-10T20:38:14.344171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:10.251489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343906+0000","last_change":"2024-09-10T20:38:11.387993+0000","last_active":"2024-09-10T20:38:14.343906+0000","last_peered":"2024-09-10T20:38:14.343906+0000","last_clean":"2024-09-10T20:38:14.343906+0000","last_became_active":"2024-09-10T20:38:11.387277+0000","last_became_peered":"2024-09-10T20:38:11.387277+0000","last_unstale":"2024-09-10T20:38:14.343906+0000","last_undegraded":"2024-09-10T20:38:14.343906+0000","last_fullsized":"2024-09-10T20:38:14.343906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:51.949339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408453+0000","last_change":"2024-09-10T20:38:12.412762+0000","last_active":"2024-09-10T20:38:13.408453+0000","last_peered":"2024-09-10T20:38:13.408453+0000","last_clean":"2024-09-10T20:38:13.408453+0000","last_became_active":"2024-09-10T20:38:12.412480+0000","last_became_peered":"2024-09-10T20:38:12.412480+0000","last_unstale":"2024-09-10T20:38:13.408453+0000","last_undegraded":"2024-09-10T20:38:13.408453+0000","last_fullsized":"2024-09-10T20:38:13.408453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633539+0000","last_change":"2024-09-10T20:38:11.398330+0000","last_active":"2024-09-10T20:38:13.633539+0000","last_peered":"2024-09-10T20:38:13.633539+0000","last_clean":"2024-09-10T20:38:13.633539+0000","last_became_active":"2024-09-10T20:38:11.398059+0000","last_became_peered":"2024-09-10T20:38:11.398059+0000","last_unstale":"2024-09-10T20:38:13.633539+0000","last_undegraded":"2024-09-10T20:38:13.633539+0000","last_fullsized":"2024-09-10T20:38:13.633539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343446+0000","last_change":"2024-09-10T20:38:12.412484+0000","last_active":"2024-09-10T20:38:14.343446+0000","last_peered":"2024-09-10T20:38:14.343446+0000","last_clean":"2024-09-10T20:38:14.343446+0000","last_became_active":"2024-09-10T20:38:12.412165+0000","last_became_peered":"2024-09-10T20:38:12.412165+0000","last_unstale":"2024-09-10T20:38:14.343446+0000","last_undegraded":"2024-09-10T20:38:14.343446+0000","last_fullsized":"2024-09-10T20:38:14.343446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:09.972659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344285+0000","last_change":"2024-09-10T20:38:11.400681+0000","last_active":"2024-09-10T20:38:14.344285+0000","last_peered":"2024-09-10T20:38:14.344285+0000","last_clean":"2024-09-10T20:38:14.344285+0000","last_became_active":"2024-09-10T20:38:11.400168+0000","last_became_peered":"2024-09-10T20:38:11.400168+0000","last_unstale":"2024-09-10T20:38:14.344285+0000","last_undegraded":"2024-09-10T20:38:14.344285+0000","last_fullsized":"2024-09-10T20:38:14.344285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:43.005710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344386+0000","last_change":"2024-09-10T20:38:12.413508+0000","last_active":"2024-09-10T20:38:14.344386+0000","last_peered":"2024-09-10T20:38:14.344386+0000","last_clean":"2024-09-10T20:38:14.344386+0000","last_became_active":"2024-09-10T20:38:12.413230+0000","last_became_peered":"2024-09-10T20:38:12.413230+0000","last_unstale":"2024-09-10T20:38:14.344386+0000","last_undegraded":"2024-09-10T20:38:14.344386+0000","last_fullsized":"2024-09-10T20:38:14.344386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:56.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407691+0000","last_change":"2024-09-10T20:38:11.402870+0000","last_active":"2024-09-10T20:38:13.407691+0000","last_peered":"2024-09-10T20:38:13.407691+0000","last_clean":"2024-09-10T20:38:13.407691+0000","last_became_active":"2024-09-10T20:38:11.402580+0000","last_became_peered":"2024-09-10T20:38:11.402580+0000","last_unstale":"2024-09-10T20:38:13.407691+0000","last_undegraded":"2024-09-10T20:38:13.407691+0000","last_fullsized":"2024-09-10T20:38:13.407691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:50:23.981654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398438+0000","last_change":"2024-09-10T20:38:12.416114+0000","last_active":"2024-09-10T20:38:13.398438+0000","last_peered":"2024-09-10T20:38:13.398438+0000","last_clean":"2024-09-10T20:38:13.398438+0000","last_became_active":"2024-09-10T20:38:12.415605+0000","last_became_peered":"2024-09-10T20:38:12.415605+0000","last_unstale":"2024-09-10T20:38:13.398438+0000","last_undegraded":"2024-09-10T20:38:13.398438+0000","last_fullsized":"2024-09-10T20:38:13.398438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:01:45.285223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400564+0000","last_change":"2024-09-10T20:38:11.393414+0000","last_active":"2024-09-10T20:38:13.400564+0000","last_peered":"2024-09-10T20:38:13.400564+0000","last_clean":"2024-09-10T20:38:13.400564+0000","last_became_active":"2024-09-10T20:38:11.393078+0000","last_became_peered":"2024-09-10T20:38:11.393078+0000","last_unstale":"2024-09-10T20:38:13.400564+0000","last_undegraded":"2024-09-10T20:38:13.400564+0000","last_fullsized":"2024-09-10T20:38:13.400564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:58.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343642+0000","last_change":"2024-09-10T20:38:12.407378+0000","last_active":"2024-09-10T20:38:14.343642+0000","last_peered":"2024-09-10T20:38:14.343642+0000","last_clean":"2024-09-10T20:38:14.343642+0000","last_became_active":"2024-09-10T20:38:12.407092+0000","last_became_peered":"2024-09-10T20:38:12.407092+0000","last_unstale":"2024-09-10T20:38:14.343642+0000","last_undegraded":"2024-09-10T20:38:14.343642+0000","last_fullsized":"2024-09-10T20:38:14.343642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:47.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408782+0000","last_change":"2024-09-10T20:38:11.397628+0000","last_active":"2024-09-10T20:38:13.408782+0000","last_peered":"2024-09-10T20:38:13.408782+0000","last_clean":"2024-09-10T20:38:13.408782+0000","last_became_active":"2024-09-10T20:38:11.397316+0000","last_became_peered":"2024-09-10T20:38:11.397316+0000","last_unstale":"2024-09-10T20:38:13.408782+0000","last_undegraded":"2024-09-10T20:38:13.408782+0000","last_fullsized":"2024-09-10T20:38:13.408782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:07.558503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344717+0000","last_change":"2024-09-10T20:38:12.396278+0000","last_active":"2024-09-10T20:38:14.344717+0000","last_peered":"2024-09-10T20:38:14.344717+0000","last_clean":"2024-09-10T20:38:14.344717+0000","last_became_active":"2024-09-10T20:38:12.395937+0000","last_became_peered":"2024-09-10T20:38:12.395937+0000","last_unstale":"2024-09-10T20:38:14.344717+0000","last_undegraded":"2024-09-10T20:38:14.344717+0000","last_fullsized":"2024-09-10T20:38:14.344717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:06.451633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401433+0000","last_change":"2024-09-10T20:38:11.388247+0000","last_active":"2024-09-10T20:38:13.401433+0000","last_peered":"2024-09-10T20:38:13.401433+0000","last_clean":"2024-09-10T20:38:13.401433+0000","last_became_active":"2024-09-10T20:38:11.387924+0000","last_became_peered":"2024-09-10T20:38:11.387924+0000","last_unstale":"2024-09-10T20:38:13.401433+0000","last_undegraded":"2024-09-10T20:38:13.401433+0000","last_fullsized":"2024-09-10T20:38:13.401433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:29.584118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408756+0000","last_change":"2024-09-10T20:38:12.418652+0000","last_active":"2024-09-10T20:38:13.408756+0000","last_peered":"2024-09-10T20:38:13.408756+0000","last_clean":"2024-09-10T20:38:13.408756+0000","last_became_active":"2024-09-10T20:38:12.418400+0000","last_became_peered":"2024-09-10T20:38:12.418400+0000","last_unstale":"2024-09-10T20:38:13.408756+0000","last_undegraded":"2024-09-10T20:38:13.408756+0000","last_fullsized":"2024-09-10T20:38:13.408756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:28.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400616+0000","last_change":"2024-09-10T20:38:11.399015+0000","last_active":"2024-09-10T20:38:13.400616+0000","last_peered":"2024-09-10T20:38:13.400616+0000","last_clean":"2024-09-10T20:38:13.400616+0000","last_became_active":"2024-09-10T20:38:11.398826+0000","last_became_peered":"2024-09-10T20:38:11.398826+0000","last_unstale":"2024-09-10T20:38:13.400616+0000","last_undegraded":"2024-09-10T20:38:13.400616+0000","last_fullsized":"2024-09-10T20:38:13.400616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:03.958637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408168+0000","last_change":"2024-09-10T20:38:12.411663+0000","last_active":"2024-09-10T20:38:13.408168+0000","last_peered":"2024-09-10T20:38:13.408168+0000","last_clean":"2024-09-10T20:38:13.408168+0000","last_became_active":"2024-09-10T20:38:12.411269+0000","last_became_peered":"2024-09-10T20:38:12.411269+0000","last_unstale":"2024-09-10T20:38:13.408168+0000","last_undegraded":"2024-09-10T20:38:13.408168+0000","last_fullsized":"2024-09-10T20:38:13.408168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:20.639033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343682+0000","last_change":"2024-09-10T20:38:11.398337+0000","last_active":"2024-09-10T20:38:14.343682+0000","last_peered":"2024-09-10T20:38:14.343682+0000","last_clean":"2024-09-10T20:38:14.343682+0000","last_became_active":"2024-09-10T20:38:11.398068+0000","last_became_peered":"2024-09-10T20:38:11.398068+0000","last_unstale":"2024-09-10T20:38:14.343682+0000","last_undegraded":"2024-09-10T20:38:14.343682+0000","last_fullsized":"2024-09-10T20:38:14.343682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:00:28.236504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344836+0000","last_change":"2024-09-10T20:38:12.420411+0000","last_active":"2024-09-10T20:38:14.344836+0000","last_peered":"2024-09-10T20:38:14.344836+0000","last_clean":"2024-09-10T20:38:14.344836+0000","last_became_active":"2024-09-10T20:38:12.419107+0000","last_became_peered":"2024-09-10T20:38:12.419107+0000","last_unstale":"2024-09-10T20:38:14.344836+0000","last_undegraded":"2024-09-10T20:38:14.344836+0000","last_fullsized":"2024-09-10T20:38:14.344836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:15.534463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398277+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:13.398277+0000","last_peered":"2024-09-10T20:38:13.398277+0000","last_clean":"2024-09-10T20:38:13.398277+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:13.398277+0000","last_undegraded":"2024-09-10T20:38:13.398277+0000","last_fullsized":"2024-09-10T20:38:13.398277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344545+0000","last_change":"2024-09-10T20:38:11.398527+0000","last_active":"2024-09-10T20:38:14.344545+0000","last_peered":"2024-09-10T20:38:14.344545+0000","last_clean":"2024-09-10T20:38:14.344545+0000","last_became_active":"2024-09-10T20:38:11.395195+0000","last_became_peered":"2024-09-10T20:38:11.395195+0000","last_unstale":"2024-09-10T20:38:14.344545+0000","last_undegraded":"2024-09-10T20:38:14.344545+0000","last_fullsized":"2024-09-10T20:38:14.344545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:35.618612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400902+0000","last_change":"2024-09-10T20:38:12.407263+0000","last_active":"2024-09-10T20:38:13.400902+0000","last_peered":"2024-09-10T20:38:13.400902+0000","last_clean":"2024-09-10T20:38:13.400902+0000","last_became_active":"2024-09-10T20:38:12.406952+0000","last_became_peered":"2024-09-10T20:38:12.406952+0000","last_unstale":"2024-09-10T20:38:13.400902+0000","last_undegraded":"2024-09-10T20:38:13.400902+0000","last_fullsized":"2024-09-10T20:38:13.400902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.171100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343670+0000","last_change":"2024-09-10T20:38:11.399685+0000","last_active":"2024-09-10T20:38:14.343670+0000","last_peered":"2024-09-10T20:38:14.343670+0000","last_clean":"2024-09-10T20:38:14.343670+0000","last_became_active":"2024-09-10T20:38:11.399073+0000","last_became_peered":"2024-09-10T20:38:11.399073+0000","last_unstale":"2024-09-10T20:38:14.343670+0000","last_undegraded":"2024-09-10T20:38:14.343670+0000","last_fullsized":"2024-09-10T20:38:14.343670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:08.967765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343358+0000","last_change":"2024-09-10T20:38:12.407025+0000","last_active":"2024-09-10T20:38:14.343358+0000","last_peered":"2024-09-10T20:38:14.343358+0000","last_clean":"2024-09-10T20:38:14.343358+0000","last_became_active":"2024-09-10T20:38:12.406124+0000","last_became_peered":"2024-09-10T20:38:12.406124+0000","last_unstale":"2024-09-10T20:38:14.343358+0000","last_undegraded":"2024-09-10T20:38:14.343358+0000","last_fullsized":"2024-09-10T20:38:14.343358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633299+0000","last_change":"2024-09-10T20:38:11.397217+0000","last_active":"2024-09-10T20:38:13.633299+0000","last_peered":"2024-09-10T20:38:13.633299+0000","last_clean":"2024-09-10T20:38:13.633299+0000","last_became_active":"2024-09-10T20:38:11.396928+0000","last_became_peered":"2024-09-10T20:38:11.396928+0000","last_unstale":"2024-09-10T20:38:13.633299+0000","last_undegraded":"2024-09-10T20:38:13.633299+0000","last_fullsized":"2024-09-10T20:38:13.633299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:38.900087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401469+0000","last_change":"2024-09-10T20:38:12.407595+0000","last_active":"2024-09-10T20:38:13.401469+0000","last_peered":"2024-09-10T20:38:13.401469+0000","last_clean":"2024-09-10T20:38:13.401469+0000","last_became_active":"2024-09-10T20:38:12.407355+0000","last_became_peered":"2024-09-10T20:38:12.407355+0000","last_unstale":"2024-09-10T20:38:13.401469+0000","last_undegraded":"2024-09-10T20:38:13.401469+0000","last_fullsized":"2024-09-10T20:38:13.401469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:05.419998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344389+0000","last_change":"2024-09-10T20:38:11.401377+0000","last_active":"2024-09-10T20:38:14.344389+0000","last_peered":"2024-09-10T20:38:14.344389+0000","last_clean":"2024-09-10T20:38:14.344389+0000","last_became_active":"2024-09-10T20:38:11.401048+0000","last_became_peered":"2024-09-10T20:38:11.401048+0000","last_unstale":"2024-09-10T20:38:14.344389+0000","last_undegraded":"2024-09-10T20:38:14.344389+0000","last_fullsized":"2024-09-10T20:38:14.344389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:17.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408141+0000","last_change":"2024-09-10T20:38:12.399857+0000","last_active":"2024-09-10T20:38:13.408141+0000","last_peered":"2024-09-10T20:38:13.408141+0000","last_clean":"2024-09-10T20:38:13.408141+0000","last_became_active":"2024-09-10T20:38:12.399121+0000","last_became_peered":"2024-09-10T20:38:12.399121+0000","last_unstale":"2024-09-10T20:38:13.408141+0000","last_undegraded":"2024-09-10T20:38:13.408141+0000","last_fullsized":"2024-09-10T20:38:13.408141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:23.782139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400255+0000","last_change":"2024-09-10T20:38:11.397463+0000","last_active":"2024-09-10T20:38:13.400255+0000","last_peered":"2024-09-10T20:38:13.400255+0000","last_clean":"2024-09-10T20:38:13.400255+0000","last_became_active":"2024-09-10T20:38:11.397205+0000","last_became_peered":"2024-09-10T20:38:11.397205+0000","last_unstale":"2024-09-10T20:38:13.400255+0000","last_undegraded":"2024-09-10T20:38:13.400255+0000","last_fullsized":"2024-09-10T20:38:13.400255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:53.003074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343801+0000","last_change":"2024-09-10T20:38:12.412633+0000","last_active":"2024-09-10T20:38:14.343801+0000","last_peered":"2024-09-10T20:38:14.343801+0000","last_clean":"2024-09-10T20:38:14.343801+0000","last_became_active":"2024-09-10T20:38:12.412350+0000","last_became_peered":"2024-09-10T20:38:12.412350+0000","last_unstale":"2024-09-10T20:38:14.343801+0000","last_undegraded":"2024-09-10T20:38:14.343801+0000","last_fullsized":"2024-09-10T20:38:14.343801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.915632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344655+0000","last_change":"2024-09-10T20:38:11.392177+0000","last_active":"2024-09-10T20:38:14.344655+0000","last_peered":"2024-09-10T20:38:14.344655+0000","last_clean":"2024-09-10T20:38:14.344655+0000","last_became_active":"2024-09-10T20:38:11.390174+0000","last_became_peered":"2024-09-10T20:38:11.390174+0000","last_unstale":"2024-09-10T20:38:14.344655+0000","last_undegraded":"2024-09-10T20:38:14.344655+0000","last_fullsized":"2024-09-10T20:38:14.344655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:29.078056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343576+0000","last_change":"2024-09-10T20:38:12.408779+0000","last_active":"2024-09-10T20:38:14.343576+0000","last_peered":"2024-09-10T20:38:14.343576+0000","last_clean":"2024-09-10T20:38:14.343576+0000","last_became_active":"2024-09-10T20:38:12.408490+0000","last_became_peered":"2024-09-10T20:38:12.408490+0000","last_unstale":"2024-09-10T20:38:14.343576+0000","last_undegraded":"2024-09-10T20:38:14.343576+0000","last_fullsized":"2024-09-10T20:38:14.343576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:25.888623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407174+0000","last_change":"2024-09-10T20:38:11.394170+0000","last_active":"2024-09-10T20:38:13.407174+0000","last_peered":"2024-09-10T20:38:13.407174+0000","last_clean":"2024-09-10T20:38:13.407174+0000","last_became_active":"2024-09-10T20:38:11.393850+0000","last_became_peered":"2024-09-10T20:38:11.393850+0000","last_unstale":"2024-09-10T20:38:13.407174+0000","last_undegraded":"2024-09-10T20:38:13.407174+0000","last_fullsized":"2024-09-10T20:38:13.407174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:15.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401623+0000","last_change":"2024-09-10T20:38:12.404386+0000","last_active":"2024-09-10T20:38:13.401623+0000","last_peered":"2024-09-10T20:38:13.401623+0000","last_clean":"2024-09-10T20:38:13.401623+0000","last_became_active":"2024-09-10T20:38:12.404045+0000","last_became_peered":"2024-09-10T20:38:12.404045+0000","last_unstale":"2024-09-10T20:38:13.401623+0000","last_undegraded":"2024-09-10T20:38:13.401623+0000","last_fullsized":"2024-09-10T20:38:13.401623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:49.410849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344714+0000","last_change":"2024-09-10T20:38:11.399979+0000","last_active":"2024-09-10T20:38:14.344714+0000","last_peered":"2024-09-10T20:38:14.344714+0000","last_clean":"2024-09-10T20:38:14.344714+0000","last_became_active":"2024-09-10T20:38:11.399682+0000","last_became_peered":"2024-09-10T20:38:11.399682+0000","last_unstale":"2024-09-10T20:38:14.344714+0000","last_undegraded":"2024-09-10T20:38:14.344714+0000","last_fullsized":"2024-09-10T20:38:14.344714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:40.818316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400970+0000","last_change":"2024-09-10T20:38:12.409091+0000","last_active":"2024-09-10T20:38:13.400970+0000","last_peered":"2024-09-10T20:38:13.400970+0000","last_clean":"2024-09-10T20:38:13.400970+0000","last_became_active":"2024-09-10T20:38:12.408815+0000","last_became_peered":"2024-09-10T20:38:12.408815+0000","last_unstale":"2024-09-10T20:38:13.400970+0000","last_undegraded":"2024-09-10T20:38:13.400970+0000","last_fullsized":"2024-09-10T20:38:13.400970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.870149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407496+0000","last_change":"2024-09-10T20:38:11.391306+0000","last_active":"2024-09-10T20:38:13.407496+0000","last_peered":"2024-09-10T20:38:13.407496+0000","last_clean":"2024-09-10T20:38:13.407496+0000","last_became_active":"2024-09-10T20:38:11.390956+0000","last_became_peered":"2024-09-10T20:38:11.390956+0000","last_unstale":"2024-09-10T20:38:13.407496+0000","last_undegraded":"2024-09-10T20:38:13.407496+0000","last_fullsized":"2024-09-10T20:38:13.407496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:12:00.829218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344085+0000","last_change":"2024-09-10T20:38:12.402970+0000","last_active":"2024-09-10T20:38:14.344085+0000","last_peered":"2024-09-10T20:38:14.344085+0000","last_clean":"2024-09-10T20:38:14.344085+0000","last_became_active":"2024-09-10T20:38:12.402393+0000","last_became_peered":"2024-09-10T20:38:12.402393+0000","last_unstale":"2024-09-10T20:38:14.344085+0000","last_undegraded":"2024-09-10T20:38:14.344085+0000","last_fullsized":"2024-09-10T20:38:14.344085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:09.528282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401287+0000","last_change":"2024-09-10T20:38:11.397172+0000","last_active":"2024-09-10T20:38:13.401287+0000","last_peered":"2024-09-10T20:38:13.401287+0000","last_clean":"2024-09-10T20:38:13.401287+0000","last_became_active":"2024-09-10T20:38:11.396670+0000","last_became_peered":"2024-09-10T20:38:11.396670+0000","last_unstale":"2024-09-10T20:38:13.401287+0000","last_undegraded":"2024-09-10T20:38:13.401287+0000","last_fullsized":"2024-09-10T20:38:13.401287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400195+0000","last_change":"2024-09-10T20:38:12.405875+0000","last_active":"2024-09-10T20:38:13.400195+0000","last_peered":"2024-09-10T20:38:13.400195+0000","last_clean":"2024-09-10T20:38:13.400195+0000","last_became_active":"2024-09-10T20:38:12.405583+0000","last_became_peered":"2024-09-10T20:38:12.405583+0000","last_unstale":"2024-09-10T20:38:13.400195+0000","last_undegraded":"2024-09-10T20:38:13.400195+0000","last_fullsized":"2024-09-10T20:38:13.400195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:30.185620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343471+0000","last_change":"2024-09-10T20:38:11.395992+0000","last_active":"2024-09-10T20:38:14.343471+0000","last_peered":"2024-09-10T20:38:14.343471+0000","last_clean":"2024-09-10T20:38:14.343471+0000","last_became_active":"2024-09-10T20:38:11.394935+0000","last_became_peered":"2024-09-10T20:38:11.394935+0000","last_unstale":"2024-09-10T20:38:14.343471+0000","last_undegraded":"2024-09-10T20:38:14.343471+0000","last_fullsized":"2024-09-10T20:38:14.343471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:49.040222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633574+0000","last_change":"2024-09-10T20:38:12.412620+0000","last_active":"2024-09-10T20:38:13.633574+0000","last_peered":"2024-09-10T20:38:13.633574+0000","last_clean":"2024-09-10T20:38:13.633574+0000","last_became_active":"2024-09-10T20:38:12.412329+0000","last_became_peered":"2024-09-10T20:38:12.412329+0000","last_unstale":"2024-09-10T20:38:13.633574+0000","last_undegraded":"2024-09-10T20:38:13.633574+0000","last_fullsized":"2024-09-10T20:38:13.633574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:40.609805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344474+0000","last_change":"2024-09-10T20:38:11.398962+0000","last_active":"2024-09-10T20:38:14.344474+0000","last_peered":"2024-09-10T20:38:14.344474+0000","last_clean":"2024-09-10T20:38:14.344474+0000","last_became_active":"2024-09-10T20:38:11.398617+0000","last_became_peered":"2024-09-10T20:38:11.398617+0000","last_unstale":"2024-09-10T20:38:14.344474+0000","last_undegraded":"2024-09-10T20:38:14.344474+0000","last_fullsized":"2024-09-10T20:38:14.344474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:47.028207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344204+0000","last_change":"2024-09-10T20:38:12.418199+0000","last_active":"2024-09-10T20:38:14.344204+0000","last_peered":"2024-09-10T20:38:14.344204+0000","last_clean":"2024-09-10T20:38:14.344204+0000","last_became_active":"2024-09-10T20:38:12.417684+0000","last_became_peered":"2024-09-10T20:38:12.417684+0000","last_unstale":"2024-09-10T20:38:14.344204+0000","last_undegraded":"2024-09-10T20:38:14.344204+0000","last_fullsized":"2024-09-10T20:38:14.344204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:27.561396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344345+0000","last_change":"2024-09-10T20:38:12.406004+0000","last_active":"2024-09-10T20:38:14.344345+0000","last_peered":"2024-09-10T20:38:14.344345+0000","last_clean":"2024-09-10T20:38:14.344345+0000","last_became_active":"2024-09-10T20:38:12.405336+0000","last_became_peered":"2024-09-10T20:38:12.405336+0000","last_unstale":"2024-09-10T20:38:14.344345+0000","last_undegraded":"2024-09-10T20:38:14.344345+0000","last_fullsized":"2024-09-10T20:38:14.344345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:58:35.400897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408136+0000","last_change":"2024-09-10T20:38:11.396962+0000","last_active":"2024-09-10T20:38:13.408136+0000","last_peered":"2024-09-10T20:38:13.408136+0000","last_clean":"2024-09-10T20:38:13.408136+0000","last_became_active":"2024-09-10T20:38:11.396650+0000","last_became_peered":"2024-09-10T20:38:11.396650+0000","last_unstale":"2024-09-10T20:38:13.408136+0000","last_undegraded":"2024-09-10T20:38:13.408136+0000","last_fullsized":"2024-09-10T20:38:13.408136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:40.818316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398946+0000","last_change":"2024-09-10T20:38:12.413236+0000","last_active":"2024-09-10T20:38:13.398946+0000","last_peered":"2024-09-10T20:38:13.398946+0000","last_clean":"2024-09-10T20:38:13.398946+0000","last_became_active":"2024-09-10T20:38:12.412966+0000","last_became_peered":"2024-09-10T20:38:12.412966+0000","last_unstale":"2024-09-10T20:38:13.398946+0000","last_undegraded":"2024-09-10T20:38:13.398946+0000","last_fullsized":"2024-09-10T20:38:13.398946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:01:29.734867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408400+0000","last_change":"2024-09-10T20:38:11.397755+0000","last_active":"2024-09-10T20:38:13.408400+0000","last_peered":"2024-09-10T20:38:13.408400+0000","last_clean":"2024-09-10T20:38:13.408400+0000","last_became_active":"2024-09-10T20:38:11.397450+0000","last_became_peered":"2024-09-10T20:38:11.397450+0000","last_unstale":"2024-09-10T20:38:13.408400+0000","last_undegraded":"2024-09-10T20:38:13.408400+0000","last_fullsized":"2024-09-10T20:38:13.408400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:38.998000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401905+0000","last_change":"2024-09-10T20:38:12.416588+0000","last_active":"2024-09-10T20:38:13.401905+0000","last_peered":"2024-09-10T20:38:13.401905+0000","last_clean":"2024-09-10T20:38:13.401905+0000","last_became_active":"2024-09-10T20:38:12.416297+0000","last_became_peered":"2024-09-10T20:38:12.416297+0000","last_unstale":"2024-09-10T20:38:13.401905+0000","last_undegraded":"2024-09-10T20:38:13.401905+0000","last_fullsized":"2024-09-10T20:38:13.401905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401853+0000","last_change":"2024-09-10T20:38:11.396382+0000","last_active":"2024-09-10T20:38:13.401853+0000","last_peered":"2024-09-10T20:38:13.401853+0000","last_clean":"2024-09-10T20:38:13.401853+0000","last_became_active":"2024-09-10T20:38:11.395934+0000","last_became_peered":"2024-09-10T20:38:11.395934+0000","last_unstale":"2024-09-10T20:38:13.401853+0000","last_undegraded":"2024-09-10T20:38:13.401853+0000","last_fullsized":"2024-09-10T20:38:13.401853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:20:39.553068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401634+0000","last_change":"2024-09-10T20:38:12.402624+0000","last_active":"2024-09-10T20:38:13.401634+0000","last_peered":"2024-09-10T20:38:13.401634+0000","last_clean":"2024-09-10T20:38:13.401634+0000","last_became_active":"2024-09-10T20:38:12.401924+0000","last_became_peered":"2024-09-10T20:38:12.401924+0000","last_unstale":"2024-09-10T20:38:13.401634+0000","last_undegraded":"2024-09-10T20:38:13.401634+0000","last_fullsized":"2024-09-10T20:38:13.401634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:33.683744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401708+0000","last_change":"2024-09-10T20:38:11.401264+0000","last_active":"2024-09-10T20:38:13.401708+0000","last_peered":"2024-09-10T20:38:13.401708+0000","last_clean":"2024-09-10T20:38:13.401708+0000","last_became_active":"2024-09-10T20:38:11.400986+0000","last_became_peered":"2024-09-10T20:38:11.400986+0000","last_unstale":"2024-09-10T20:38:13.401708+0000","last_undegraded":"2024-09-10T20:38:13.401708+0000","last_fullsized":"2024-09-10T20:38:13.401708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:50.668458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":73109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":74139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":806912,"data_stored":662965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":217088,"data_stored":72583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":66227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":662987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":74811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:21.204 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:38:21.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-10T20:38:21.205 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-10T20:38:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.342+0000 7f216051f640 1 -- 172.21.15.28:0/3999226570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 msgr2=0x7f2144004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.342+0000 7f216051f640 1 --2- 172.21.15.28:0/3999226570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 0x7f2144004800 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f21580b9f90 tx=0x7f2144005830 comp rx=0 tx=0).stop 2024-09-10T20:38:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.342+0000 7f215eabc640 1 -- 172.21.15.28:0/3999226570 reap_dead start 2024-09-10T20:38:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.342+0000 7f216051f640 1 -- 172.21.15.28:0/3999226570 shutdown_connections 2024-09-10T20:38:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.342+0000 7f216051f640 1 -- 172.21.15.28:0/3999226570 >> 172.21.15.28:0/3999226570 conn(0x7f21580d4040 msgr2=0x7f21580d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.342+0000 7f216051f640 1 -- 172.21.15.28:0/3999226570 shutdown_connections 2024-09-10T20:38:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.342+0000 7f216051f640 1 -- 172.21.15.28:0/3999226570 wait complete. 2024-09-10T20:38:21.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.343+0000 7f216051f640 1 Processor -- start 2024-09-10T20:38:21.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.343+0000 7f216051f640 1 -- start start 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f216051f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 0x7f21580c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f216051f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21580c9d20 0x7f215815c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f216051f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f215815bbe0 0x7f21581f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f216051f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21580b7c40 con 0x7f2144003f90 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f216051f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f21580d2e90 con 0x7f21580c9d20 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f216051f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f21580b7dc0 con 0x7f215815bbe0 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 0x7f21580c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 0x7f21580c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50762/0 (socket says 172.21.15.28:50762) 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 -- 172.21.15.28:0/2004388910 learned_addr learned my addr 172.21.15.28:0/2004388910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215eabc640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f215815bbe0 0x7f21581f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215daba640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21580c9d20 0x7f215815c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 -- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f215815bbe0 msgr2=0x7f21581f7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f215815bbe0 0x7f21581f7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 -- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21580c9d20 msgr2=0x7f215815c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21580c9d20 0x7f215815c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.344+0000 7f215e2bb640 1 -- 172.21.15.28:0/2004388910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21580c8fa0 con 0x7f2144003f90 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.345+0000 7f215e2bb640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 0x7f21580c8660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2154011b10 tx=0x7f2154009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.345+0000 7f214effd640 1 -- 172.21.15.28:0/2004388910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215400fc80 con 0x7f2144003f90 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.345+0000 7f214effd640 1 -- 172.21.15.28:0/2004388910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f215400fe20 con 0x7f2144003f90 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.345+0000 7f214effd640 1 -- 172.21.15.28:0/2004388910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154004280 con 0x7f2144003f90 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.345+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21580c9240 con 0x7f2144003f90 2024-09-10T20:38:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.345+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21580c9800 con 0x7f2144003f90 2024-09-10T20:38:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.346+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21580ca100 con 0x7f2144003f90 2024-09-10T20:38:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.346+0000 7f214effd640 1 -- 172.21.15.28:0/2004388910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f21540035f0 con 0x7f2144003f90 2024-09-10T20:38:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.347+0000 7f214effd640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f213c077fa0 0x7f213c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.347+0000 7f214effd640 1 -- 172.21.15.28:0/2004388910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f215409a740 con 0x7f2144003f90 2024-09-10T20:38:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.347+0000 7f215daba640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f213c077fa0 0x7f213c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.348+0000 7f215daba640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f213c077fa0 0x7f213c07a460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2148033000 tx=0x7f2148038040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.351+0000 7f214effd640 1 -- 172.21.15.28:0/2004388910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2154062c00 con 0x7f2144003f90 2024-09-10T20:38:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.536+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f21580b8ae0 con 0x7f2144003f90 2024-09-10T20:38:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.537+0000 7f214effd640 1 -- 172.21.15.28:0/2004388910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7f2154067630 con 0x7f2144003f90 2024-09-10T20:38:21.541 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f213c077fa0 msgr2=0x7f213c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f213c077fa0 0x7f213c07a460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2148033000 tx=0x7f2148038040 comp rx=0 tx=0).stop 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 msgr2=0x7f21580c8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 0x7f21580c8660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2154011b10 tx=0x7f2154009f10 comp rx=0 tx=0).stop 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 shutdown_connections 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f213c077fa0 0x7f213c07a460 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f215815bbe0 0x7f21581f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f21580c9d20 0x7f215815c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 --2- 172.21.15.28:0/2004388910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2144003f90 0x7f21580c8660 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 >> 172.21.15.28:0/2004388910 conn(0x7f21580d4040 msgr2=0x7f21580cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 shutdown_connections 2024-09-10T20:38:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.539+0000 7f216051f640 1 -- 172.21.15.28:0/2004388910 wait complete. 2024-09-10T20:38:21.553 INFO:teuthology.orchestra.run.smithi028.stdout:[{"version":27,"timestamp":"2024-09-10T20:37:59.591666+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"3831.945533"}]}] 2024-09-10T20:38:21.553 INFO:tasks.ceph_manager:config epoch is 27 2024-09-10T20:38:21.554 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T20:38:21.554 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-10T20:38:21.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.655+0000 7f5eba49b640 1 -- 172.21.15.28:0/1336080777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 msgr2=0x7f5eb4161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.655+0000 7f5eba49b640 1 --2- 172.21.15.28:0/1336080777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb4161b50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8005fd0 tx=0x7f5ea802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.655+0000 7f5eba49b640 1 -- 172.21.15.28:0/1336080777 shutdown_connections 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.655+0000 7f5eba49b640 1 --2- 172.21.15.28:0/1336080777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb4162090 0x7f5eb40b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.655+0000 7f5eba49b640 1 --2- 172.21.15.28:0/1336080777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb4161b50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.655+0000 7f5eba49b640 1 -- 172.21.15.28:0/1336080777 >> 172.21.15.28:0/1336080777 conn(0x7f5eb40d3fb0 msgr2=0x7f5eb40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.656+0000 7f5eba49b640 1 -- 172.21.15.28:0/1336080777 shutdown_connections 2024-09-10T20:38:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.656+0000 7f5eba49b640 1 -- 172.21.15.28:0/1336080777 wait complete. 2024-09-10T20:38:21.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.657+0000 7f5eba49b640 1 Processor -- start 2024-09-10T20:38:21.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.657+0000 7f5eba49b640 1 -- start start 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eba49b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb415ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eba49b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb415de70 0x7f5eb40a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eba49b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb4162090 0x7f5eb41f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eba49b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb415e3d0 con 0x7f5eb40d2b90 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eba49b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb415e250 con 0x7f5eb415de70 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eba49b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5eb40b7340 con 0x7f5eb4162090 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb415ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb415ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50782/0 (socket says 172.21.15.28:50782) 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 -- 172.21.15.28:0/3162093858 learned_addr learned my addr 172.21.15.28:0/3162093858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb37fe640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb415de70 0x7f5eb40a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb8a38640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb4162090 0x7f5eb41f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 -- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb4162090 msgr2=0x7f5eb41f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb4162090 0x7f5eb41f3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 -- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb415de70 msgr2=0x7f5eb40a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb415de70 0x7f5eb40a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 -- 172.21.15.28:0/3162093858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb40a1bc0 con 0x7f5eb40d2b90 2024-09-10T20:38:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb37fe640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb415de70 0x7f5eb40a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb3fff640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb415ffb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400c500 tx=0x7f5ea400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.658+0000 7f5eb0ff9640 1 -- 172.21.15.28:0/3162093858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea40052d0 con 0x7f5eb40d2b90 2024-09-10T20:38:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.659+0000 7f5eb0ff9640 1 -- 172.21.15.28:0/3162093858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ea4005470 con 0x7f5eb40d2b90 2024-09-10T20:38:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.659+0000 7f5eb0ff9640 1 -- 172.21.15.28:0/3162093858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4012870 con 0x7f5eb40d2b90 2024-09-10T20:38:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.659+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb41606c0 con 0x7f5eb40d2b90 2024-09-10T20:38:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.659+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb4160c80 con 0x7f5eb40d2b90 2024-09-10T20:38:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.660+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb40b9f30 con 0x7f5eb40d2b90 2024-09-10T20:38:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.660+0000 7f5eb0ff9640 1 -- 172.21.15.28:0/3162093858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5ea40035f0 con 0x7f5eb40d2b90 2024-09-10T20:38:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.661+0000 7f5eb0ff9640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ea0077f70 0x7f5ea007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.661+0000 7f5eb0ff9640 1 -- 172.21.15.28:0/3162093858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5ea40998d0 con 0x7f5eb40d2b90 2024-09-10T20:38:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.661+0000 7f5eb37fe640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ea0077f70 0x7f5ea007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.661+0000 7f5eb37fe640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ea0077f70 0x7f5ea007a430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8033000 tx=0x7f5ea8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.665+0000 7f5eb0ff9640 1 -- 172.21.15.28:0/3162093858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ea4062340 con 0x7f5eb40d2b90 2024-09-10T20:38:21.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.897+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f5eb40b8ec0 con 0x7f5eb40d2b90 2024-09-10T20:38:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.899+0000 7f5eb0ff9640 1 -- 172.21.15.28:0/3162093858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+193250 (secure 0 0 0) 0x7f5ea4061c40 con 0x7f5eb40d2b90 2024-09-10T20:38:21.907 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ea0077f70 msgr2=0x7f5ea007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ea0077f70 0x7f5ea007a430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8033000 tx=0x7f5ea8004730 comp rx=0 tx=0).stop 2024-09-10T20:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 msgr2=0x7f5eb415ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb415ffb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400c500 tx=0x7f5ea400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 shutdown_connections 2024-09-10T20:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ea0077f70 0x7f5ea007a430 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eb4162090 0x7f5eb41f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eb415de70 0x7f5eb40a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 --2- 172.21.15.28:0/3162093858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb40d2b90 0x7f5eb415ffb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 >> 172.21.15.28:0/3162093858 conn(0x7f5eb40d3fb0 msgr2=0x7f5eb40b8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 shutdown_connections 2024-09-10T20:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:21.902+0000 7f5eba49b640 1 -- 172.21.15.28:0/3162093858 wait complete. 2024-09-10T20:38:21.917 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4115,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2898596254},{"type":"v1","addr":"172.21.15.159:6801","nonce":2898596254}]},"active_addr":"172.21.15.159:6801/2898596254","active_change":"2024-09-10T20:37:51.965862+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4156,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":2568415012}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3526126208}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1402110084}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1101500009}]}]} 2024-09-10T20:38:21.920 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-10T20:38:21.920 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-10T20:38:21.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:22.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 -- 172.21.15.28:0/3673260769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04580cf370 msgr2=0x7f04580d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 --2- 172.21.15.28:0/3673260769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04580cf370 0x7f04580d0530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0450005fd0 tx=0x7f045002aa00 comp rx=0 tx=0).stop 2024-09-10T20:38:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 -- 172.21.15.28:0/3673260769 shutdown_connections 2024-09-10T20:38:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 --2- 172.21.15.28:0/3673260769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04580cf370 0x7f04580d0530 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 --2- 172.21.15.28:0/3673260769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0458155020 0x7f04580b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 -- 172.21.15.28:0/3673260769 >> 172.21.15.28:0/3673260769 conn(0x7f045806cf00 msgr2=0x7f045806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 -- 172.21.15.28:0/3673260769 shutdown_connections 2024-09-10T20:38:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.021+0000 7f045c859640 1 -- 172.21.15.28:0/3673260769 wait complete. 2024-09-10T20:38:22.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 Processor -- start 2024-09-10T20:38:22.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 -- start start 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580cf370 0x7f04581f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04580d00f0 0x7f0458157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0458155020 0x7f045806f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04580ad0f0 con 0x7f0458155020 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f04580acf70 con 0x7f04580cf370 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.023+0000 7f045c859640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f04580ad270 con 0x7f04580d00f0 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0458155020 0x7f045806f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0458155020 0x7f045806f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50792/0 (socket says 172.21.15.28:50792) 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 -- 172.21.15.28:0/1423539036 learned_addr learned my addr 172.21.15.28:0/1423539036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f045659c640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580cf370 0x7f04581f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f044ffff640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04580d00f0 0x7f0458157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 -- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04580d00f0 msgr2=0x7f0458157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04580d00f0 0x7f0458157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 -- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580cf370 msgr2=0x7f04581f8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580cf370 0x7f04581f8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 -- 172.21.15.28:0/1423539036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0458157680 con 0x7f0458155020 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f044ffff640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04580d00f0 0x7f0458157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f0456d9d640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0458155020 0x7f045806f9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0450006010 tx=0x7f0450003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f044effd640 1 -- 172.21.15.28:0/1423539036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450047070 con 0x7f0458155020 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f044effd640 1 -- 172.21.15.28:0/1423539036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0450004250 con 0x7f0458155020 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04580af940 con 0x7f0458155020 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f044effd640 1 -- 172.21.15.28:0/1423539036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450033050 con 0x7f0458155020 2024-09-10T20:38:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.024+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04580aff00 con 0x7f0458155020 2024-09-10T20:38:22.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.025+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04580b03a0 con 0x7f0458155020 2024-09-10T20:38:22.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.026+0000 7f044effd640 1 -- 172.21.15.28:0/1423539036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f04500046a0 con 0x7f0458155020 2024-09-10T20:38:22.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.026+0000 7f044effd640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0438077f70 0x7f043807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.026+0000 7f044effd640 1 -- 172.21.15.28:0/1423539036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f04500bb270 con 0x7f0458155020 2024-09-10T20:38:22.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.027+0000 7f045659c640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0438077f70 0x7f043807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.027+0000 7f045659c640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0438077f70 0x7f043807a430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0458151730 tx=0x7f044800a250 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.031+0000 7f044effd640 1 -- 172.21.15.28:0/1423539036 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0458157680 con 0x7f0458155020 2024-09-10T20:38:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.216+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04580d0510 con 0x7f0458155020 2024-09-10T20:38:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.217+0000 7f044effd640 1 -- 172.21.15.28:0/1423539036 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16669 (secure 0 0 0) 0x7f0450083730 con 0x7f0458155020 2024-09-10T20:38:22.221 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:22.222 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:13.395359+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0438077f70 msgr2=0x7f043807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0438077f70 0x7f043807a430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0458151730 tx=0x7f044800a250 comp rx=0 tx=0).stop 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0458155020 msgr2=0x7f045806f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0458155020 0x7f045806f9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0450006010 tx=0x7f0450003f10 comp rx=0 tx=0).stop 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 shutdown_connections 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0438077f70 0x7f043807a430 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0458155020 0x7f045806f9c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f04580d00f0 0x7f0458157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 --2- 172.21.15.28:0/1423539036 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f04580cf370 0x7f04581f8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 >> 172.21.15.28:0/1423539036 conn(0x7f045806cf00 msgr2=0x7f04580cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 shutdown_connections 2024-09-10T20:38:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.220+0000 7f045c859640 1 -- 172.21.15.28:0/1423539036 wait complete. 2024-09-10T20:38:22.235 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-10T20:38:22.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 -- 172.21.15.28:0/2423175057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf80b3c20 msgr2=0x7fabf815ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/2423175057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf80b3c20 0x7fabf815ef40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fabf0005fd0 tx=0x7fabf002aa00 comp rx=0 tx=0).stop 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 -- 172.21.15.28:0/2423175057 shutdown_connections 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/2423175057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf80b3c20 0x7fabf815ef40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/2423175057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf8073720 0x7fabf8073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 -- 172.21.15.28:0/2423175057 >> 172.21.15.28:0/2423175057 conn(0x7fabf806e410 msgr2=0x7fabf80b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 -- 172.21.15.28:0/2423175057 shutdown_connections 2024-09-10T20:38:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.337+0000 7fabfe0bd640 1 -- 172.21.15.28:0/2423175057 wait complete. 2024-09-10T20:38:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.338+0000 7fabfe0bd640 1 Processor -- start 2024-09-10T20:38:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.338+0000 7fabfe0bd640 1 -- start start 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabfe0bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf8073720 0x7fabf81e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabfe0bd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf80b3c20 0x7fabf8158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabfe0bd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf815aea0 0x7fabf80b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabfe0bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabf815f600 con 0x7fabf8073720 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabfe0bd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fabf815f480 con 0x7fabf80b3c20 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabfe0bd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fabf8071d80 con 0x7fabf815aea0 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf8073720 0x7fabf81e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf8073720 0x7fabf81e9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50808/0 (socket says 172.21.15.28:50808) 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 -- 172.21.15.28:0/789832601 learned_addr learned my addr 172.21.15.28:0/789832601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf7fff640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf815aea0 0x7fabf80b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf6ffd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf80b3c20 0x7fabf8158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 -- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf815aea0 msgr2=0x7fabf80b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf815aea0 0x7fabf80b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 -- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf80b3c20 msgr2=0x7fabf8158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf80b3c20 0x7fabf8158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 -- 172.21.15.28:0/789832601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf8158e00 con 0x7fabf8073720 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf6ffd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf80b3c20 0x7fabf8158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabf77fe640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf8073720 0x7fabf81e9030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d770 tx=0x7fabe800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.339+0000 7fabdffff640 1 -- 172.21.15.28:0/789832601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8017070 con 0x7fabf8073720 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.340+0000 7fabdffff640 1 -- 172.21.15.28:0/789832601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabe8009d20 con 0x7fabf8073720 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.340+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf81590e0 con 0x7fabf8073720 2024-09-10T20:38:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.340+0000 7fabdffff640 1 -- 172.21.15.28:0/789832601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8005280 con 0x7fabf8073720 2024-09-10T20:38:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.340+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf8145490 con 0x7fabf8073720 2024-09-10T20:38:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.341+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf8073b00 con 0x7fabf8073720 2024-09-10T20:38:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.341+0000 7fabdffff640 1 -- 172.21.15.28:0/789832601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fabe8005420 con 0x7fabf8073720 2024-09-10T20:38:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.342+0000 7fabdffff640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fabe4077f70 0x7fabe407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.342+0000 7fabdffff640 1 -- 172.21.15.28:0/789832601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fabe8099ec0 con 0x7fabf8073720 2024-09-10T20:38:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.342+0000 7fabf6ffd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fabe4077f70 0x7fabe407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.342+0000 7fabf6ffd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fabe4077f70 0x7fabe407a430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fabec006fd0 tx=0x7fabec008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.346+0000 7fabdffff640 1 -- 172.21.15.28:0/789832601 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fabe8062370 con 0x7fabf8073720 2024-09-10T20:38:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.531+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabf80d08d0 con 0x7fabf8073720 2024-09-10T20:38:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.532+0000 7fabdffff640 1 -- 172.21.15.28:0/789832601 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16669 (secure 0 0 0) 0x7fabe806b0e0 con 0x7fabf8073720 2024-09-10T20:38:22.536 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:22.536 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:13.395359+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fabe4077f70 msgr2=0x7fabe407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fabe4077f70 0x7fabe407a430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fabec006fd0 tx=0x7fabec008040 comp rx=0 tx=0).stop 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf8073720 msgr2=0x7fabf81e9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf8073720 0x7fabf81e9030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d770 tx=0x7fabe800dc30 comp rx=0 tx=0).stop 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 shutdown_connections 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fabe4077f70 0x7fabe407a430 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fabf815aea0 0x7fabf80b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fabf80b3c20 0x7fabf8158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 --2- 172.21.15.28:0/789832601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabf8073720 0x7fabf81e9030 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 >> 172.21.15.28:0/789832601 conn(0x7fabf806e410 msgr2=0x7fabf806cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 shutdown_connections 2024-09-10T20:38:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.534+0000 7fabfe0bd640 1 -- 172.21.15.28:0/789832601 wait complete. 2024-09-10T20:38:22.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T20:38:22.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-10T20:38:22.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T20:38:22.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T20:38:22.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-10T20:38:22.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-10T20:38:22.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-10T20:38:22.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-10T20:38:22.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.729+0000 7f667f7fe640 1 -- 172.21.15.28:0/1449264647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66800a6b90 con 0x7f667801c800 2024-09-10T20:38:22.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.729+0000 7f5783fff640 1 -- 172.21.15.28:0/1359342856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c0adab0 con 0x7f5778005210 2024-09-10T20:38:22.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.729+0000 7f6685afb640 1 -- 172.21.15.28:0/1449264647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f667801c800 msgr2=0x7f667801cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.729+0000 7f6685afb640 1 --2- 172.21.15.28:0/1449264647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f667801c800 0x7f667801cc90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6680052430 tx=0x7f66800a5d40 comp rx=0 tx=0).stop 2024-09-10T20:38:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f579375d640 1 -- 172.21.15.28:0/1359342856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5778005210 msgr2=0x7f577800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f579375d640 1 --2- 172.21.15.28:0/1359342856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5778005210 0x7f577800b9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f578c000b60 tx=0x7f5778005830 comp rx=0 tx=0).stop 2024-09-10T20:38:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f6685afb640 1 -- 172.21.15.28:0/1449264647 shutdown_connections 2024-09-10T20:38:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f6685afb640 1 --2- 172.21.15.28:0/1449264647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f667801c800 0x7f667801cc90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f6685afb640 1 --2- 172.21.15.28:0/1449264647 >> v2:172.21.15.159:3300/0 conn(0x7f6678005a60 0x7f6678036520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f6685afb640 1 --2- 172.21.15.28:0/1449264647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f667800ba00 0x7f6678035fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f6685afb640 1 -- 172.21.15.28:0/1449264647 >> 172.21.15.28:0/1449264647 conn(0x7f6678054560 msgr2=0x7f6678054940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f579375d640 1 -- 172.21.15.28:0/1359342856 shutdown_connections 2024-09-10T20:38:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f579375d640 1 --2- 172.21.15.28:0/1359342856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5778005210 0x7f577800b9c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f579375d640 1 -- 172.21.15.28:0/1359342856 >> 172.21.15.28:0/1359342856 conn(0x7f5784056710 msgr2=0x7f5784056b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f6685afb640 1 -- 172.21.15.28:0/1449264647 shutdown_connections 2024-09-10T20:38:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f6685afb640 1 -- 172.21.15.28:0/1449264647 wait complete. 2024-09-10T20:38:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.730+0000 7f579375d640 1 -- 172.21.15.28:0/1359342856 shutdown_connections 2024-09-10T20:38:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.731+0000 7f579375d640 1 -- 172.21.15.28:0/1359342856 wait complete. 2024-09-10T20:38:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fcc596c9640 1 -- 172.21.15.28:0/255966027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc3c0051c0 msgr2=0x7fcc3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fcc596c9640 1 --2- 172.21.15.28:0/255966027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc3c0051c0 0x7fcc3c00b9c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcc540caf70 tx=0x7fcc3c005830 comp rx=0 tx=0).stop 2024-09-10T20:38:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fcc58ec8640 1 -- 172.21.15.28:0/255966027 reap_dead start 2024-09-10T20:38:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fac6b2ae640 1 -- 172.21.15.28:0/2280438977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c01c870 msgr2=0x7fac5c0063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fac6b2ae640 1 --2- 172.21.15.28:0/2280438977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c01c870 0x7fac5c0063d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac60005fd0 tx=0x7fac6000ac00 comp rx=0 tx=0).stop 2024-09-10T20:38:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fcc596c9640 1 -- 172.21.15.28:0/255966027 shutdown_connections 2024-09-10T20:38:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fcc596c9640 1 -- 172.21.15.28:0/255966027 >> 172.21.15.28:0/255966027 conn(0x7fcc540cf990 msgr2=0x7fcc540cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fcc596c9640 1 -- 172.21.15.28:0/255966027 shutdown_connections 2024-09-10T20:38:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fcc596c9640 1 -- 172.21.15.28:0/255966027 wait complete. 2024-09-10T20:38:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7f579375d640 1 Processor -- start 2024-09-10T20:38:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fac6b2ae640 1 -- 172.21.15.28:0/2280438977 shutdown_connections 2024-09-10T20:38:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fac6b2ae640 1 --2- 172.21.15.28:0/2280438977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c01c870 0x7fac5c0063d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fac6b2ae640 1 --2- 172.21.15.28:0/2280438977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5c00a900 0x7fac5c01c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.732+0000 7fac6b2ae640 1 -- 172.21.15.28:0/2280438977 >> 172.21.15.28:0/2280438977 conn(0x7fac5c054530 msgr2=0x7fac5c054910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7fac6b2ae640 1 -- 172.21.15.28:0/2280438977 shutdown_connections 2024-09-10T20:38:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7f579375d640 1 -- start start 2024-09-10T20:38:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 -- 172.21.15.28:0/79983511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580b02b0 msgr2=0x7faa5806cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 --2- 172.21.15.28:0/79983511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580b02b0 0x7faa5806cae0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faa48009830 tx=0x7faa4802a510 comp rx=0 tx=0).stop 2024-09-10T20:38:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7f6685afb640 1 Processor -- start 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 -- 172.21.15.28:0/79983511 shutdown_connections 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 --2- 172.21.15.28:0/79983511 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa580ccb70 0x7faa580cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 --2- 172.21.15.28:0/79983511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580b02b0 0x7faa5806cae0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 -- 172.21.15.28:0/79983511 >> 172.21.15.28:0/79983511 conn(0x7faa5806e010 msgr2=0x7faa5806e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 -- 172.21.15.28:0/79983511 shutdown_connections 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7faa5d472640 1 -- 172.21.15.28:0/79983511 wait complete. 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7f1751d82640 1 -- 172.21.15.28:0/2341762681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f174c0707b0 msgr2=0x7f174c15e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7f1751d82640 1 --2- 172.21.15.28:0/2341762681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f174c0707b0 0x7f174c15e520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f174c0d0110 tx=0x7f173c02a8c0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7f3a9d182640 1 -- 172.21.15.28:0/4139529670 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a940047a0 con 0x7f3a98071c80 2024-09-10T20:38:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7f3aa03e8640 1 -- 172.21.15.28:0/4139529670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a98071c80 msgr2=0x7f3a980ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.733+0000 7f3aa03e8640 1 --2- 172.21.15.28:0/4139529670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a98071c80 0x7f3a980ae780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009880 tx=0x7f3a9402a460 comp rx=0 tx=0).stop 2024-09-10T20:38:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f1751d82640 1 -- 172.21.15.28:0/2341762681 shutdown_connections 2024-09-10T20:38:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f1751d82640 1 --2- 172.21.15.28:0/2341762681 >> v2:172.21.15.28:3300/0 conn(0x7f174c0d0140 0x7f174c0d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f1751d82640 1 --2- 172.21.15.28:0/2341762681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f174c0707b0 0x7f174c15e520 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f1751d82640 1 -- 172.21.15.28:0/2341762681 >> 172.21.15.28:0/2341762681 conn(0x7f174c06cf00 msgr2=0x7f174c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f6685afb640 1 -- start start 2024-09-10T20:38:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f579375d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5778005210 0x7f57840e0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f579375d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57840e1110 0x7f5784180d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f1751d82640 1 -- 172.21.15.28:0/2341762681 shutdown_connections 2024-09-10T20:38:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7fcc596c9640 1 Processor -- start 2024-09-10T20:38:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7f1751d82640 1 -- 172.21.15.28:0/2341762681 wait complete. 2024-09-10T20:38:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7fcc596c9640 1 -- start start 2024-09-10T20:38:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f5790cf8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57840e1110 0x7f5784180d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f5790cf8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57840e1110 0x7f5784180d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50882/0 (socket says 172.21.15.28:50882) 2024-09-10T20:38:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f5790cf8640 1 -- 172.21.15.28:0/221023635 learned_addr learned my addr 172.21.15.28:0/221023635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7faa5d472640 1 Processor -- start 2024-09-10T20:38:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f579375d640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5784181290 0x7f5784181720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f57914f9640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5778005210 0x7f57840e0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7faa5d472640 1 -- start start 2024-09-10T20:38:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57840088b0 con 0x7f57840e1110 2024-09-10T20:38:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5784008730 con 0x7f5778005210 2024-09-10T20:38:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f6685afb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6678005a60 0x7f667804e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f5791cfa640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5784181290 0x7f5784181720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5784056510 con 0x7f5784181290 2024-09-10T20:38:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7f6685afb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f667800ba00 0x7f66780d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.734+0000 7fac6b2ae640 1 -- 172.21.15.28:0/2280438977 wait complete. 2024-09-10T20:38:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.735+0000 7fac6b2ae640 1 Processor -- start 2024-09-10T20:38:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f6685afb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f667801c800 0x7f66780e1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f6685afb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66780e2910 con 0x7f667801c800 2024-09-10T20:38:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7fac6b2ae640 1 -- start start 2024-09-10T20:38:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f6685afb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f66780e2790 con 0x7f667800ba00 2024-09-10T20:38:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f6685afb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f66780e2a90 con 0x7f6678005a60 2024-09-10T20:38:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f667ffff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f667800ba00 0x7f66780d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f667ffff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f667800ba00 0x7f66780d9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41796/0 (socket says 172.21.15.28:41796) 2024-09-10T20:38:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f667ffff640 1 -- 172.21.15.28:0/3836910704 learned_addr learned my addr 172.21.15.28:0/3836910704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f1751d82640 1 Processor -- start 2024-09-10T20:38:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f6684af9640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6678005a60 0x7f667804e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f57914f9640 1 -- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5784181290 msgr2=0x7f5784181720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f57914f9640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5784181290 0x7f5784181720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f57914f9640 1 -- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57840e1110 msgr2=0x7f5784180d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f57914f9640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57840e1110 0x7f5784180d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f57914f9640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5784181c60 con 0x7f5778005210 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f5790cf8640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57840e1110 0x7f5784180d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f3aa03e8640 1 -- 172.21.15.28:0/4139529670 shutdown_connections 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f3aa03e8640 1 --2- 172.21.15.28:0/4139529670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a98071c80 0x7f3a980ae780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f3aa03e8640 1 -- 172.21.15.28:0/4139529670 >> 172.21.15.28:0/4139529670 conn(0x7f3a980b2990 msgr2=0x7f3a980b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f1751d82640 1 -- start start 2024-09-10T20:38:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.736+0000 7f66852fa640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f667801c800 0x7f66780e1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f57914f9640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5778005210 0x7f57840e0bd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f577c00bfd0 tx=0x7f577c00da70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa5d472640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa580b02b0 0x7faa580b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f6684af9640 1 -- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f667800ba00 msgr2=0x7f66780d9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f6684af9640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f667800ba00 0x7f66780d9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f6684af9640 1 -- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f667801c800 msgr2=0x7f66780e1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f6684af9640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f667801c800 0x7f66780e1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f6684af9640 1 -- 172.21.15.28:0/3836910704 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667804abb0 con 0x7f6678005a60 2024-09-10T20:38:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa5d472640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa580ccb70 0x7faa580b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f3aa03e8640 1 -- 172.21.15.28:0/4139529670 shutdown_connections 2024-09-10T20:38:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c017070 con 0x7f5778005210 2024-09-10T20:38:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f577c004310 con 0x7f5778005210 2024-09-10T20:38:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c005280 con 0x7f5778005210 2024-09-10T20:38:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa56ffd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa580b02b0 0x7faa580b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa56ffd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa580b02b0 0x7faa580b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41798/0 (socket says 172.21.15.28:41798) 2024-09-10T20:38:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa56ffd640 1 -- 172.21.15.28:0/3863316830 learned_addr learned my addr 172.21.15.28:0/3863316830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa5d472640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580cd9f0 0x7faa581f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7fac6b2ae640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5c006910 0x7fac5c0487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa580c6610 con 0x7faa580cd9f0 2024-09-10T20:38:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faa58074600 con 0x7faa580b02b0 2024-09-10T20:38:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7fcc596c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c0051c0 0x7fcc5414ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faa580c6790 con 0x7faa580ccb70 2024-09-10T20:38:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f6684af9640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6678005a60 0x7f667804e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f666c00b410 tx=0x7f666c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7fcc596c9640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc54160520 0x7fcc541fcc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7fcc596c9640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc5414b580 0x7fcc54157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7fac6b2ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c00a900 0x7fac5c04af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f1751d82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c0707b0 0x7f174c0af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7faa567fc640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa580ccb70 0x7faa580b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c0051c0 0x7fcc5414ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c0051c0 0x7fcc5414ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50916/0 (socket says 172.21.15.28:50916) 2024-09-10T20:38:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc53fff640 1 -- 172.21.15.28:0/3273389928 learned_addr learned my addr 172.21.15.28:0/3273389928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc540c99c0 con 0x7fcc3c0051c0 2024-09-10T20:38:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcc540c9840 con 0x7fcc5414b580 2024-09-10T20:38:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcc540c9b40 con 0x7fcc54160520 2024-09-10T20:38:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c015030 con 0x7f6678005a60 2024-09-10T20:38:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f666c005a10 con 0x7f6678005a60 2024-09-10T20:38:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c015030 con 0x7f6678005a60 2024-09-10T20:38:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac6a2ac640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5c006910 0x7fac5c0487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac69aab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c00a900 0x7fac5c04af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac69aab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c00a900 0x7fac5c04af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50928/0 (socket says 172.21.15.28:50928) 2024-09-10T20:38:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac6a2ac640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5c006910 0x7fac5c0487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41804/0 (socket says 172.21.15.28:41804) 2024-09-10T20:38:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac69aab640 1 -- 172.21.15.28:0/3592653631 learned_addr learned my addr 172.21.15.28:0/3592653631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa56ffd640 1 -- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa580ccb70 msgr2=0x7faa580b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa56ffd640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa580ccb70 0x7faa580b6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa577fe640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580cd9f0 0x7faa581f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa56ffd640 1 -- 172.21.15.28:0/3863316830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580cd9f0 msgr2=0x7faa581f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac6b2ae640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5c01c870 0x7fac5c0db7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f1751d82640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f174c0d0140 0x7f174c154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f1751d82640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f174c153ec0 0x7f174c0add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f1751d82640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f174c15a9b0 con 0x7f174c0707b0 2024-09-10T20:38:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f1751d82640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f174c15a830 con 0x7f174c153ec0 2024-09-10T20:38:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f1751d82640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f174c15ab30 con 0x7f174c0d0140 2024-09-10T20:38:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57840e15f0 con 0x7f5778005210 2024-09-10T20:38:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57840e1bb0 con 0x7f5778005210 2024-09-10T20:38:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c036840 con 0x7fac5c00a900 2024-09-10T20:38:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c0366c0 con 0x7fac5c006910 2024-09-10T20:38:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fac5c00b9d0 con 0x7fac5c01c870 2024-09-10T20:38:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa56ffd640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580cd9f0 0x7faa581f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa56ffd640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa581f88e0 con 0x7faa580b02b0 2024-09-10T20:38:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f0b7a76b640 1 -- 172.21.15.28:0/3347073982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b60003c10 msgr2=0x7f0b60003ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f0b7a76b640 1 --2- 172.21.15.28:0/3347073982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b60003c10 0x7f0b60003ff0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c01d650 tx=0x7f0b60005830 comp rx=0 tx=0).stop 2024-09-10T20:38:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa567fc640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa580ccb70 0x7faa580b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.737+0000 7f3aa03e8640 1 -- 172.21.15.28:0/4139529670 wait complete. 2024-09-10T20:38:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fac6aaad640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5c01c870 0x7fac5c0db7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f0b78d08640 1 -- 172.21.15.28:0/3347073982 reap_dead start 2024-09-10T20:38:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f0b7a76b640 1 -- 172.21.15.28:0/3347073982 shutdown_connections 2024-09-10T20:38:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f0b7a76b640 1 -- 172.21.15.28:0/3347073982 >> 172.21.15.28:0/3347073982 conn(0x7f0b740cf9c0 msgr2=0x7f0b740cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc537fe640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc54160520 0x7fcc541fcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c0707b0 0x7f174c0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c0707b0 0x7f174c0af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50942/0 (socket says 172.21.15.28:50942) 2024-09-10T20:38:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f174b7fe640 1 -- 172.21.15.28:0/1176129305 learned_addr learned my addr 172.21.15.28:0/1176129305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f174affd640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f174c0d0140 0x7f174c154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f0b7a76b640 1 -- 172.21.15.28:0/3347073982 shutdown_connections 2024-09-10T20:38:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f6685afb640 1 -- 172.21.15.28:0/3836910704 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667804a8d0 con 0x7f6678005a60 2024-09-10T20:38:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7f6685afb640 1 -- 172.21.15.28:0/3836910704 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f667804db70 con 0x7f6678005a60 2024-09-10T20:38:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f174bfff640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f174c153ec0 0x7f174c0add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7fcc58ec8640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc5414b580 0x7fcc54157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fcc537fe640 1 -- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc5414b580 msgr2=0x7fcc54157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fcc537fe640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc5414b580 0x7fcc54157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fcc537fe640 1 -- 172.21.15.28:0/3273389928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c0051c0 msgr2=0x7fcc5414ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fcc537fe640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c0051c0 0x7fcc5414ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fcc537fe640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5414e480 con 0x7fcc54160520 2024-09-10T20:38:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f174b7fe640 1 -- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f174c0d0140 msgr2=0x7f174c154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f174b7fe640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f174c0d0140 0x7f174c154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f174b7fe640 1 -- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f174c153ec0 msgr2=0x7f174c0add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f57840530b0 con 0x7f5778005210 2024-09-10T20:38:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fac69aab640 1 -- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5c01c870 msgr2=0x7fac5c0db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fac69aab640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5c01c870 0x7fac5c0db7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fac69aab640 1 -- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5c006910 msgr2=0x7fac5c0487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fac69aab640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac5c006910 0x7fac5c0487c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fac69aab640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c04b600 con 0x7fac5c00a900 2024-09-10T20:38:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fac6aaad640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac5c01c870 0x7fac5c0db7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f174b7fe640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f174c153ec0 0x7f174c0add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fcc537fe640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc54160520 0x7fcc541fcc60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c005fd0 tx=0x7fcc4c00eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f174b7fe640 1 -- 172.21.15.28:0/1176129305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c0acfb0 con 0x7f174c0707b0 2024-09-10T20:38:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00ce40 con 0x7fcc54160520 2024-09-10T20:38:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc4c004c10 con 0x7fcc54160520 2024-09-10T20:38:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7fac69aab640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c00a900 0x7fac5c04af00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fac600084c0 tx=0x7fac60003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7f174b7fe640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c0707b0 0x7f174c0af810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f173c009b30 tx=0x7f173c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.738+0000 7faa577fe640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa580cd9f0 0x7faa581f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7faa56ffd640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa580b02b0 0x7faa580b4fc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faa48000c00 tx=0x7faa48032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48040070 con 0x7faa580b02b0 2024-09-10T20:38:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa48003af0 con 0x7faa580b02b0 2024-09-10T20:38:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48032e40 con 0x7faa580b02b0 2024-09-10T20:38:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa580b55f0 con 0x7faa580b02b0 2024-09-10T20:38:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa580b5bb0 con 0x7faa580b02b0 2024-09-10T20:38:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7f6685afb640 1 -- 172.21.15.28:0/3836910704 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f667801cc90 con 0x7f6678005a60 2024-09-10T20:38:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.739+0000 7f0b7a76b640 1 -- 172.21.15.28:0/3347073982 wait complete. 2024-09-10T20:38:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c03c420 con 0x7f174c0707b0 2024-09-10T20:38:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f1751d82640 1 -- 172.21.15.28:0/1176129305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174c0ad290 con 0x7f174c0707b0 2024-09-10T20:38:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c003970 con 0x7fcc54160520 2024-09-10T20:38:22.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f1751d82640 1 -- 172.21.15.28:0/1176129305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174c155100 con 0x7f174c0707b0 2024-09-10T20:38:22.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f0b7a76b640 1 Processor -- start 2024-09-10T20:38:22.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f577c005420 con 0x7f5778005210 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7f3aa03e8640 1 Processor -- start 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.740+0000 7f3aa03e8640 1 -- start start 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3aa03e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a98071c80 0x7f3a980710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3aa03e8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a98073e70 0x7f3a981456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3aa03e8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a9815e3e0 0x7f3a98158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3aa03e8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9806d370 con 0x7f3a98071c80 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3aa03e8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9806d1f0 con 0x7f3a98073e70 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3aa03e8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3a98074300 con 0x7f3a9815e3e0 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3a9e985640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a9815e3e0 0x7f3a98158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3a9e985640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a9815e3e0 0x7f3a98158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:53040/0 (socket says 172.21.15.28:53040) 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f3a9e985640 1 -- 172.21.15.28:0/3799829732 learned_addr learned my addr 172.21.15.28:0/3799829732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f3a9d983640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a98073e70 0x7f3a981456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f0b7a76b640 1 -- start start 2024-09-10T20:38:22.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f666c022060 con 0x7f6678005a60 2024-09-10T20:38:22.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5414e760 con 0x7fcc54160520 2024-09-10T20:38:22.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5414ba00 con 0x7fcc54160520 2024-09-10T20:38:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa580cea50 con 0x7faa580b02b0 2024-09-10T20:38:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f173c03c6b0 con 0x7f174c0707b0 2024-09-10T20:38:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f3a9e184640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a98071c80 0x7f3a980710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7f5781ffb640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5770077f70 0x7f577007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f5790cf8640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5770077f70 0x7f577007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f0b7a76b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b740c6450 0x7f0b740a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f3a9e985640 1 -- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a98073e70 msgr2=0x7f3a981456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f3a9e985640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a98073e70 0x7f3a981456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f3a9e985640 1 -- 172.21.15.28:0/3799829732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a98071c80 msgr2=0x7f3a980710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f3a9e985640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a98071c80 0x7f3a980710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f3a9e985640 1 -- 172.21.15.28:0/3799829732 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a981f9300 con 0x7f3a9815e3e0 2024-09-10T20:38:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f3a9e985640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a9815e3e0 0x7f3a98158b00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000caa0 tx=0x7f3a9000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.742+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f577c098650 con 0x7f5778005210 2024-09-10T20:38:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f5781ffb640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7f5770080d20 0x7f57700831c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f5790cf8640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5770077f70 0x7f577007a430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f578403c7c0 tx=0x7f5788009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5770083890 con 0x7f5770080d20 2024-09-10T20:38:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f667d7fa640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6664078370 0x7f666407a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f0b7a76b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b740b5230 0x7f0b74148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b740c6450 0x7f0b740a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b740c6450 0x7f0b740a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41836/0 (socket says 172.21.15.28:41836) 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f0b737fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b740b5230 0x7f0b74148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f0b73fff640 1 -- 172.21.15.28:0/160885078 learned_addr learned my addr 172.21.15.28:0/160885078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b7a76b640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b74148e70 0x7f0b7414a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b7a76b640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b740ca570 con 0x7f0b74148e70 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b7a76b640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0b740ca3f0 con 0x7f0b740c6450 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b7a76b640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0b740ca6f0 con 0x7f0b740b5230 2024-09-10T20:38:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b78d08640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b74148e70 0x7f0b7414a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90003da0 con 0x7f3a9815e3e0 2024-09-10T20:38:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a981f95e0 con 0x7f3a9815e3e0 2024-09-10T20:38:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a9000a6d0 con 0x7f3a9815e3e0 2024-09-10T20:38:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9000ab20 con 0x7f3a9815e3e0 2024-09-10T20:38:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a981f9ba0 con 0x7f3a9815e3e0 2024-09-10T20:38:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f577c05ec00 con 0x7f5778005210 2024-09-10T20:38:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3a980b0a50 con 0x7f3a9815e3e0 2024-09-10T20:38:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7faa48045460 con 0x7faa580b02b0 2024-09-10T20:38:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6002a9f0 con 0x7fac5c00a900 2024-09-10T20:38:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b78d08640 1 -- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b740b5230 msgr2=0x7f0b74148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac60005220 con 0x7fac5c00a900 2024-09-10T20:38:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.741+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac60005650 con 0x7fac5c00a900 2024-09-10T20:38:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c04b8e0 con 0x7fac5c00a900 2024-09-10T20:38:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.743+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c04bea0 con 0x7fac5c00a900 2024-09-10T20:38:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fac5c01c330 con 0x7fac5c00a900 2024-09-10T20:38:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b78d08640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0b740b5230 0x7f0b74148930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b78d08640 1 -- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b740c6450 msgr2=0x7f0b740a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b78d08640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b740c6450 0x7f0b740a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b78d08640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b740b44b0 con 0x7f0b74148e70 2024-09-10T20:38:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f0b73fff640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0b740c6450 0x7f0b740a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f0b78d08640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b74148e70 0x7f0b7414a960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0b740d0a90 tx=0x7f0b5c00a8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7faa3f7fe640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa44077f70 0x7faa4407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faa480bb240 con 0x7faa580b02b0 2024-09-10T20:38:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7faa3f7fe640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7faa44080190 0x7faa440825b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa44082c60 con 0x7faa44080190 2024-09-10T20:38:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7faa480c2050 con 0x7faa580b02b0 2024-09-10T20:38:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fcc4c003b10 con 0x7fcc54160520 2024-09-10T20:38:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f667ffff640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6664078370 0x7f666407a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f666c016070 con 0x7f6678005a60 2024-09-10T20:38:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f667d7fa640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7f6664080570 0x7f6664082990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 --> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f666c01ae50 con 0x7f6664080570 2024-09-10T20:38:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f666c09b620 con 0x7f6678005a60 2024-09-10T20:38:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c00e680 con 0x7f0b74148e70 2024-09-10T20:38:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f66852fa640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7f6664080570 0x7f6664082990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b5c00e930 con 0x7f0b74148e70 2024-09-10T20:38:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c00ed80 con 0x7f0b74148e70 2024-09-10T20:38:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f667ffff640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6664078370 0x7f666407a830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f667804b2a0 tx=0x7f6674009260 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f0b7a76b640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b740c6830 con 0x7f0b74148e70 2024-09-10T20:38:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7faa567fc640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa44077f70 0x7faa4407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7f0b7a76b640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b740b4750 con 0x7f0b74148e70 2024-09-10T20:38:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c03cae0 con 0x7f174c0707b0 2024-09-10T20:38:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.744+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f173c04c030 con 0x7f174c0707b0 2024-09-10T20:38:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f1743fff640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f172c0780a0 0x7f172c07a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.745+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f173c0bb440 con 0x7f174c0707b0 2024-09-10T20:38:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1730001030 con 0x7f174c0707b0 2024-09-10T20:38:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7f1743fff640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7f172c080240 0x7f172c082660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 --> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f172c082d10 con 0x7f172c080240 2024-09-10T20:38:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7f66852fa640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7f6664080570 0x7f6664082990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fac600047a0 con 0x7fac5c00a900 2024-09-10T20:38:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f174affd640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f172c0780a0 0x7f172c07a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7fcc50ff9640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcc40077f70 0x7fcc4007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc4c09a8d0 con 0x7fcc54160520 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc5415a2d0 con 0x7fcc54160520 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7fcc50ff9640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7fcc40080110 0x7fcc40082530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc4c004c10 con 0x7fcc40080110 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcc4c01d070 con 0x7fcc54160520 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0b740cb1a0 con 0x7f0b74148e70 2024-09-10T20:38:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7fcc53fff640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcc40077f70 0x7fcc4007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3a9000acc0 con 0x7f3a9815e3e0 2024-09-10T20:38:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f3a8affd640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a7c077f70 0x7f3a7c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7faa567fc640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa44077f70 0x7faa4407a430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faa4c007a50 tx=0x7faa4c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.746+0000 7faa577fe640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7faa44080190 0x7faa440825b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f173c07ff00 con 0x7f174c0707b0 2024-09-10T20:38:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f5791cfa640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7f5770080d20 0x7f57700831c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3a90099c70 con 0x7f3a9815e3e0 2024-09-10T20:38:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f174bfff640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7f172c080240 0x7f172c082660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7fac5affd640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fac48078040 0x7fac4807a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fac600b9e00 con 0x7fac5c00a900 2024-09-10T20:38:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f3a8affd640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f3a7c080190 0x7f3a7c0825b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 --> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a7c082c60 con 0x7f3a7c080190 2024-09-10T20:38:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.747+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3a9009c250 con 0x7f3a9815e3e0 2024-09-10T20:38:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f3a9e184640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a7c077f70 0x7f3a7c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fac5affd640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7fac48080260 0x7fac48082680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac600081b0 con 0x7fac48080260 2024-09-10T20:38:22.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f3a9d983640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f3a7c080190 0x7f3a7c0825b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fac6a2ac640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fac48078040 0x7fac4807a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fac600ba1f0 con 0x7fac5c00a900 2024-09-10T20:38:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fcc58ec8640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7fcc40080110 0x7fcc40082530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== osd.2 v2:172.21.15.28:6816/4001706199 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f666c01ae50 con 0x7f6664080570 2024-09-10T20:38:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f5791cfa640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7f5770080d20 0x7f57700831c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f174affd640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f172c0780a0 0x7f172c07a560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f174c0b50f0 tx=0x7f1744005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fac6aaad640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7fac48080260 0x7fac48082680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7faa577fe640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7faa44080190 0x7faa440825b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fcc53fff640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcc40077f70 0x7fcc4007a430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc48001860 tx=0x7fcc480014f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f174bfff640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7f172c080240 0x7f172c082660 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fac6a2ac640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fac48078040 0x7fac4807a500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fac64076a80 tx=0x7fac64074040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fac6aaad640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7fac48080260 0x7fac48082680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f3a9d983640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f3a7c080190 0x7f3a7c0825b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7fcc58ec8640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7fcc40080110 0x7fcc40082530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7f3a9e184640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a7c077f70 0x7f3a7c07a430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a94033000 tx=0x7f3a94004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== osd.0 v2:172.21.15.28:6808/2488809808 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5770083890 con 0x7f5770080d20 2024-09-10T20:38:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.748+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0b5c00f450 con 0x7f0b74148e70 2024-09-10T20:38:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== osd.3 v2:172.21.15.28:6824/1557654590 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fac600081b0 con 0x7fac48080260 2024-09-10T20:38:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7f0b70ff9640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b54078070 0x7f0b5407a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== osd.5 v2:172.21.15.159:6826/2363244923 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faa44082c60 con 0x7faa44080190 2024-09-10T20:38:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0b5c099960 con 0x7f0b74148e70 2024-09-10T20:38:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== osd.4 v2:172.21.15.159:6802/1548693517 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3a7c082c60 con 0x7f3a7c080190 2024-09-10T20:38:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== osd.7 v2:172.21.15.159:6818/88942910 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc4c004c10 con 0x7fcc40080110 2024-09-10T20:38:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.749+0000 7f0b70ff9640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7f0b54080290 0x7f0b540826b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:22.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.750+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b54082d60 con 0x7f0b54080290 2024-09-10T20:38:22.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.750+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== osd.1 v2:172.21.15.28:6800/1047588896 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f172c082d10 con 0x7f172c080240 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.750+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0b5c09e2a0 con 0x7f0b74148e70 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.750+0000 7f0b73fff640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b54078070 0x7f0b5407a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.750+0000 7f0b737fe640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7f0b54080290 0x7f0b540826b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.751+0000 7f0b73fff640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b54078070 0x7f0b5407a530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0012e0 tx=0x7f0b6c001730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.751+0000 7f0b737fe640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7f0b54080290 0x7f0b540826b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.752+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== osd.6 v2:172.21.15.159:6810/3584510546 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0b54082d60 con 0x7f0b54080290 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.779+0000 7f6685afb640 1 -- 172.21.15.28:0/3836910704 --> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f667804dd70 con 0x7f6664080570 2024-09-10T20:38:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.780+0000 7f667d7fa640 1 -- 172.21.15.28:0/3836910704 <== osd.2 v2:172.21.15.28:6816/4001706199 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f667804dd70 con 0x7f6664080570 2024-09-10T20:38:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 --> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3a980ae780 con 0x7f3a7c080190 2024-09-10T20:38:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 -- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7f6664080570 msgr2=0x7f6664082990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7f6664080570 0x7f6664082990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 --> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fac5c04f1c0 con 0x7fac48080260 2024-09-10T20:38:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 --> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f57840e2000 con 0x7f5770080d20 2024-09-10T20:38:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 -- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6664078370 msgr2=0x7f666407a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6664078370 0x7f666407a830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f667804b2a0 tx=0x7f6674009260 comp rx=0 tx=0).stop 2024-09-10T20:38:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 -- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6678005a60 msgr2=0x7f667804e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 --2- 172.21.15.28:0/3836910704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6678005a60 0x7f667804e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f666c00b410 tx=0x7f666c00b8d0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f66852fa640 1 -- 172.21.15.28:0/3836910704 reap_dead start 2024-09-10T20:38:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 -- 172.21.15.28:0/3836910704 shutdown_connections 2024-09-10T20:38:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 -- 172.21.15.28:0/3836910704 >> 172.21.15.28:0/3836910704 conn(0x7f6678054560 msgr2=0x7f6678009a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 --> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcc5406eee0 con 0x7fcc40080110 2024-09-10T20:38:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 -- 172.21.15.28:0/3836910704 shutdown_connections 2024-09-10T20:38:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f5781ffb640 1 -- 172.21.15.28:0/221023635 <== osd.0 v2:172.21.15.28:6808/2488809808 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f57840e2000 con 0x7f5770080d20 2024-09-10T20:38:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 --> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faa580b62c0 con 0x7faa44080190 2024-09-10T20:38:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f666affd640 1 -- 172.21.15.28:0/3836910704 wait complete. 2024-09-10T20:38:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7f3a8affd640 1 -- 172.21.15.28:0/3799829732 <== osd.4 v2:172.21.15.159:6802/1548693517 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3a980ae780 con 0x7f3a7c080190 2024-09-10T20:38:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.781+0000 7fcc50ff9640 1 -- 172.21.15.28:0/3273389928 <== osd.7 v2:172.21.15.159:6818/88942910 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcc5406eee0 con 0x7fcc40080110 2024-09-10T20:38:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac5affd640 1 -- 172.21.15.28:0/3592653631 <== osd.3 v2:172.21.15.28:6824/1557654590 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fac5c04f1c0 con 0x7fac48080260 2024-09-10T20:38:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7f5770080d20 msgr2=0x7f57700831c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 --> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1730002dd0 con 0x7f172c080240 2024-09-10T20:38:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f3a7c080190 msgr2=0x7f3a7c0825b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f3a7c080190 0x7f3a7c0825b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 --> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0b7415aea0 con 0x7f0b54080290 2024-09-10T20:38:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f579375d640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7f5770080d20 0x7f57700831c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7fcc40080110 msgr2=0x7fcc40082530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fcc596c9640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7fcc40080110 0x7fcc40082530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7fac48080260 msgr2=0x7fac48082680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6b2ae640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7fac48080260 0x7fac48082680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fac48078040 msgr2=0x7fac4807a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6b2ae640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fac48078040 0x7fac4807a500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fac64076a80 tx=0x7fac64074040 comp rx=0 tx=0).stop 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c00a900 msgr2=0x7fac5c04af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6b2ae640 1 --2- 172.21.15.28:0/3592653631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac5c00a900 0x7fac5c04af00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fac600084c0 tx=0x7fac60003ee0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a7c077f70 msgr2=0x7f3a7c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a7c077f70 0x7f3a7c07a430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a94033000 tx=0x7f3a94004730 comp rx=0 tx=0).stop 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a9815e3e0 msgr2=0x7f3a98158b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 --2- 172.21.15.28:0/3799829732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a9815e3e0 0x7f3a98158b00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000caa0 tx=0x7f3a9000cf60 comp rx=0 tx=0).stop 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3a9e985640 1 -- 172.21.15.28:0/3799829732 reap_dead start 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 shutdown_connections 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 >> 172.21.15.28:0/3799829732 conn(0x7f3a980b2990 msgr2=0x7f3a980b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 shutdown_connections 2024-09-10T20:38:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7faa3f7fe640 1 -- 172.21.15.28:0/3863316830 <== osd.5 v2:172.21.15.159:6826/2363244923 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faa580b62c0 con 0x7faa44080190 2024-09-10T20:38:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6aaad640 1 -- 172.21.15.28:0/3592653631 reap_dead start 2024-09-10T20:38:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f3aa03e8640 1 -- 172.21.15.28:0/3799829732 wait complete. 2024-09-10T20:38:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5770077f70 msgr2=0x7f577007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7f1743fff640 1 -- 172.21.15.28:0/1176129305 <== osd.1 v2:172.21.15.28:6800/1047588896 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1730002dd0 con 0x7f172c080240 2024-09-10T20:38:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 shutdown_connections 2024-09-10T20:38:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 >> 172.21.15.28:0/3592653631 conn(0x7fac5c054530 msgr2=0x7fac5c04f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f579375d640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5770077f70 0x7f577007a430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f578403c7c0 tx=0x7f5788009040 comp rx=0 tx=0).stop 2024-09-10T20:38:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcc40077f70 msgr2=0x7fcc4007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fcc596c9640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcc40077f70 0x7fcc4007a430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc48001860 tx=0x7fcc480014f0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc54160520 msgr2=0x7fcc541fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fcc596c9640 1 --2- 172.21.15.28:0/3273389928 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc54160520 0x7fcc541fcc60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c005fd0 tx=0x7fcc4c00eee0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.782+0000 7fcc58ec8640 1 -- 172.21.15.28:0/3273389928 reap_dead start 2024-09-10T20:38:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7faa44080190 msgr2=0x7faa440825b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7faa44080190 0x7faa440825b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5778005210 msgr2=0x7f57840e0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 shutdown_connections 2024-09-10T20:38:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 >> 172.21.15.28:0/3273389928 conn(0x7fcc540cf990 msgr2=0x7fcc5406e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f579375d640 1 --2- 172.21.15.28:0/221023635 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5778005210 0x7f57840e0bd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f577c00bfd0 tx=0x7f577c00da70 comp rx=0 tx=0).stop 2024-09-10T20:38:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f5791cfa640 1 -- 172.21.15.28:0/221023635 reap_dead start 2024-09-10T20:38:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 shutdown_connections 2024-09-10T20:38:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 >> 172.21.15.28:0/221023635 conn(0x7f5784056710 msgr2=0x7f578400bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa44077f70 msgr2=0x7faa4407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa44077f70 0x7faa4407a430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faa4c007a50 tx=0x7faa4c009040 comp rx=0 tx=0).stop 2024-09-10T20:38:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa580b02b0 msgr2=0x7faa580b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 --2- 172.21.15.28:0/3863316830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa580b02b0 0x7faa580b4fc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faa48000c00 tx=0x7faa48032b90 comp rx=0 tx=0).stop 2024-09-10T20:38:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f0b70ff9640 1 -- 172.21.15.28:0/160885078 <== osd.6 v2:172.21.15.159:6810/3584510546 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0b7415aea0 con 0x7f0b54080290 2024-09-10T20:38:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 shutdown_connections 2024-09-10T20:38:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7fcc596c9640 1 -- 172.21.15.28:0/3273389928 wait complete. 2024-09-10T20:38:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 shutdown_connections 2024-09-10T20:38:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa577fe640 1 -- 172.21.15.28:0/3863316830 reap_dead start 2024-09-10T20:38:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f579375d640 1 -- 172.21.15.28:0/221023635 wait complete. 2024-09-10T20:38:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7f172c080240 msgr2=0x7f172c082660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 shutdown_connections 2024-09-10T20:38:22.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 shutdown_connections 2024-09-10T20:38:22.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f1741ffb640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7f172c080240 0x7f172c082660 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 >> 172.21.15.28:0/3863316830 conn(0x7faa5806e010 msgr2=0x7faa58071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7fac6b2ae640 1 -- 172.21.15.28:0/3592653631 wait complete. 2024-09-10T20:38:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f172c0780a0 msgr2=0x7f172c07a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f1741ffb640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f172c0780a0 0x7f172c07a560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f174c0b50f0 tx=0x7f1744005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c0707b0 msgr2=0x7f174c0af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 shutdown_connections 2024-09-10T20:38:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.783+0000 7f1741ffb640 1 --2- 172.21.15.28:0/1176129305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c0707b0 0x7f174c0af810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f173c009b30 tx=0x7f173c004270 comp rx=0 tx=0).stop 2024-09-10T20:38:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f174bfff640 1 -- 172.21.15.28:0/1176129305 reap_dead start 2024-09-10T20:38:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 shutdown_connections 2024-09-10T20:38:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 >> 172.21.15.28:0/1176129305 conn(0x7f174c06cf00 msgr2=0x7f174c0b3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7f0b54080290 msgr2=0x7f0b540826b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f0b6a7fc640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7f0b54080290 0x7f0b540826b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7faa5d472640 1 -- 172.21.15.28:0/3863316830 wait complete. 2024-09-10T20:38:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 shutdown_connections 2024-09-10T20:38:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f1741ffb640 1 -- 172.21.15.28:0/1176129305 wait complete. 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b54078070 msgr2=0x7f0b5407a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f0b6a7fc640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0b54078070 0x7f0b5407a530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0012e0 tx=0x7f0b6c001730 comp rx=0 tx=0).stop 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b74148e70 msgr2=0x7f0b7414a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f0b6a7fc640 1 --2- 172.21.15.28:0/160885078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b74148e70 0x7f0b7414a960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0b740d0a90 tx=0x7f0b5c00a8d0 comp rx=0 tx=0).stop 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.784+0000 7f0b78d08640 1 -- 172.21.15.28:0/160885078 reap_dead start 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.785+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 shutdown_connections 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.785+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 >> 172.21.15.28:0/160885078 conn(0x7f0b740cf9c0 msgr2=0x7f0b740ae820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.785+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 shutdown_connections 2024-09-10T20:38:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:22.785+0000 7f0b6a7fc640 1 -- 172.21.15.28:0/160885078 wait complete. 2024-09-10T20:38:22.831 INFO:teuthology.orchestra.run.smithi028.stdout:60129542150 2024-09-10T20:38:22.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-10T20:38:22.832 INFO:teuthology.orchestra.run.smithi028.stdout:64424509447 2024-09-10T20:38:22.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:38:22.833 INFO:teuthology.orchestra.run.smithi028.stdout:64424509447 2024-09-10T20:38:22.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:38:22.834 INFO:teuthology.orchestra.run.smithi028.stdout:60129542151 2024-09-10T20:38:22.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-10T20:38:22.835 INFO:teuthology.orchestra.run.smithi028.stdout:64424509448 2024-09-10T20:38:22.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:38:22.836 INFO:teuthology.orchestra.run.smithi028.stdout:64424509447 2024-09-10T20:38:22.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T20:38:22.837 INFO:teuthology.orchestra.run.smithi028.stdout:60129542150 2024-09-10T20:38:22.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-10T20:38:22.838 INFO:teuthology.orchestra.run.smithi028.stdout:60129542152 2024-09-10T20:38:22.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-10T20:38:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.047+0000 7fc377fff640 1 -- 172.21.15.28:0/1848847581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c032890 con 0x7fc3800ae9f0 2024-09-10T20:38:23.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.048+0000 7fc3861e5640 1 -- 172.21.15.28:0/1848847581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800ae9f0 msgr2=0x7fc38015edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.048+0000 7fc3861e5640 1 --2- 172.21.15.28:0/1848847581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800ae9f0 0x7fc38015edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc3800ca440 tx=0x7fc37c02a480 comp rx=0 tx=0).stop 2024-09-10T20:38:23.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.048+0000 7fc3861e5640 1 -- 172.21.15.28:0/1848847581 shutdown_connections 2024-09-10T20:38:23.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.048+0000 7fc3861e5640 1 --2- 172.21.15.28:0/1848847581 >> v2:172.21.15.159:3300/0 conn(0x7fc3800cdde0 0x7fc380161920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.048+0000 7fc3861e5640 1 --2- 172.21.15.28:0/1848847581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800ae9f0 0x7fc38015edb0 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc3800ca440 tx=0x7fc37c02a480 comp rx=0 tx=0).stop 2024-09-10T20:38:23.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.048+0000 7fc3861e5640 1 -- 172.21.15.28:0/1848847581 >> 172.21.15.28:0/1848847581 conn(0x7fc3800cf990 msgr2=0x7fc3800cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.049+0000 7fc3861e5640 1 -- 172.21.15.28:0/1848847581 shutdown_connections 2024-09-10T20:38:23.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.049+0000 7fc3861e5640 1 -- 172.21.15.28:0/1848847581 wait complete. 2024-09-10T20:38:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7fc3861e5640 1 Processor -- start 2024-09-10T20:38:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f5b9719b640 1 -- 172.21.15.28:0/199719926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b900cf1b0 msgr2=0x7f5b900ca550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f5b9719b640 1 --2- 172.21.15.28:0/199719926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b900cf1b0 0x7f5b900ca550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b88005fd0 tx=0x7f5b8802a8a0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f5b9719b640 1 -- 172.21.15.28:0/199719926 shutdown_connections 2024-09-10T20:38:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f5b9719b640 1 --2- 172.21.15.28:0/199719926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b900cf1b0 0x7f5b900ca550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7fc3861e5640 1 -- start start 2024-09-10T20:38:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f5b9719b640 1 --2- 172.21.15.28:0/199719926 >> v2:172.21.15.159:3300/0 conn(0x7f5b900ced40 0x7f5b900ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f5b9719b640 1 -- 172.21.15.28:0/199719926 >> 172.21.15.28:0/199719926 conn(0x7f5b900cf9c0 msgr2=0x7f5b900cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f3520e54640 1 -- 172.21.15.28:0/4069732087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c004bd0 msgr2=0x7f350c004fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.051+0000 7f3520e54640 1 --2- 172.21.15.28:0/4069732087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c004bd0 0x7f350c004fd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f351c000b60 tx=0x7f350c005830 comp rx=0 tx=0).stop 2024-09-10T20:38:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7f3520e54640 1 -- 172.21.15.28:0/4069732087 shutdown_connections 2024-09-10T20:38:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7f3520e54640 1 --2- 172.21.15.28:0/4069732087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c004bd0 0x7f350c004fd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3918177121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd130004de0 msgr2=0x7fd1300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7fd14a9f5640 1 --2- 172.21.15.28:0/3918177121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd130004de0 0x7fd1300051c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd140001440 tx=0x7fd130003810 comp rx=0 tx=0).stop 2024-09-10T20:38:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7fd14a1f4640 1 -- 172.21.15.28:0/3918177121 reap_dead start 2024-09-10T20:38:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7f3520e54640 1 -- 172.21.15.28:0/4069732087 >> 172.21.15.28:0/4069732087 conn(0x7f3514055540 msgr2=0x7f3514055920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7f3520e54640 1 -- 172.21.15.28:0/4069732087 shutdown_connections 2024-09-10T20:38:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7f3520e54640 1 -- 172.21.15.28:0/4069732087 wait complete. 2024-09-10T20:38:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3918177121 shutdown_connections 2024-09-10T20:38:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3918177121 >> 172.21.15.28:0/3918177121 conn(0x7fd1440cf990 msgr2=0x7fd1440cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3918177121 shutdown_connections 2024-09-10T20:38:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.052+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3918177121 wait complete. 2024-09-10T20:38:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3861e5640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3800aedd0 0x7fc3800b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7f5b9719b640 1 -- 172.21.15.28:0/199719926 shutdown_connections 2024-09-10T20:38:23.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3861e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800cdde0 0x7fc3800b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7f5b9719b640 1 -- 172.21.15.28:0/199719926 wait complete. 2024-09-10T20:38:23.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3861e5640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3800cc070 0x7fc3800cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3861e5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3800cadf0 con 0x7fc3800cdde0 2024-09-10T20:38:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3861e5640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc380162780 con 0x7fc3800cc070 2024-09-10T20:38:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7f177db99640 1 -- 172.21.15.28:0/3030369434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770034450 msgr2=0x7f1770050320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7f177db99640 1 --2- 172.21.15.28:0/3030369434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770034450 0x7f1770050320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1770035100 tx=0x7f17780a5e80 comp rx=0 tx=0).stop 2024-09-10T20:38:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3861e5640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc3800caf70 con 0x7fc3800aedd0 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3851e3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3800aedd0 0x7fc3800b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3851e3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3800aedd0 0x7fc3800b0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:53102/0 (socket says 172.21.15.28:53102) 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3851e3640 1 -- 172.21.15.28:0/3858884161 learned_addr learned my addr 172.21.15.28:0/3858884161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3859e4640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3800cc070 0x7fc3800cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7fc3849e2640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800cdde0 0x7fc3800b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7f177db99640 1 -- 172.21.15.28:0/3030369434 shutdown_connections 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7f177db99640 1 --2- 172.21.15.28:0/3030369434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770034450 0x7f1770050320 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.053+0000 7f177db99640 1 -- 172.21.15.28:0/3030369434 >> 172.21.15.28:0/3030369434 conn(0x7f177001e8c0 msgr2=0x7f1770055650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7f3520e54640 1 Processor -- start 2024-09-10T20:38:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc3851e3640 1 -- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3800cc070 msgr2=0x7fc3800cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc3851e3640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3800cc070 0x7fc3800cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc3851e3640 1 -- 172.21.15.28:0/3858884161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800cdde0 msgr2=0x7fc3800b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7f3520e54640 1 -- start start 2024-09-10T20:38:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fd14a9f5640 1 Processor -- start 2024-09-10T20:38:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc3851e3640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800cdde0 0x7fc3800b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fd14a9f5640 1 -- start start 2024-09-10T20:38:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc3851e3640 1 -- 172.21.15.28:0/3858884161 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3801371e0 con 0x7fc3800aedd0 2024-09-10T20:38:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc3859e4640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3800cc070 0x7fc3800cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc3851e3640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3800aedd0 0x7fc3800b0e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009950 tx=0x7fc37c033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.054+0000 7fc375ffb640 1 -- 172.21.15.28:0/3858884161 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c03f020 con 0x7fc3800aedd0 2024-09-10T20:38:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fc375ffb640 1 -- 172.21.15.28:0/3858884161 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc37c0038e0 con 0x7fc3800aedd0 2024-09-10T20:38:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fc375ffb640 1 -- 172.21.15.28:0/3858884161 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c037040 con 0x7fc3800aedd0 2024-09-10T20:38:23.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7f5b9719b640 1 Processor -- start 2024-09-10T20:38:23.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3801374c0 con 0x7fc3800aedd0 2024-09-10T20:38:23.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7f177db99640 1 -- 172.21.15.28:0/3030369434 shutdown_connections 2024-09-10T20:38:23.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7f5b9719b640 1 -- start start 2024-09-10T20:38:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7f177db99640 1 -- 172.21.15.28:0/3030369434 wait complete. 2024-09-10T20:38:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc380137a80 con 0x7fc3800aedd0 2024-09-10T20:38:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7f3520e54640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f350c004bd0 0x7f35140dc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fd14a9f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd130004de0 0x7fd14415a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fd3bead4640 1 -- 172.21.15.28:0/886078730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b80d2a80 msgr2=0x7fd3b80d49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fd3bead4640 1 --2- 172.21.15.28:0/886078730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b80d2a80 0x7fd3b80d49e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd3b000b0a0 tx=0x7fd3b002a830 comp rx=0 tx=0).stop 2024-09-10T20:38:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fd3bead4640 1 -- 172.21.15.28:0/886078730 shutdown_connections 2024-09-10T20:38:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fd3bead4640 1 --2- 172.21.15.28:0/886078730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b80d2a80 0x7fd3b80d49e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.055+0000 7fd3bead4640 1 -- 172.21.15.28:0/886078730 >> 172.21.15.28:0/886078730 conn(0x7fd3b809d230 msgr2=0x7fd3b80d3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd14a9f5640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1440aa590 0x7fd144141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd14a9f5640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1440b44c0 0x7fd1440b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7f3520e54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f351403d040 0x7f351403d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd14a9f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd14415e610 con 0x7fd130004de0 2024-09-10T20:38:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd14a9f5640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd14415e490 con 0x7fd1440aa590 2024-09-10T20:38:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd14a9f5640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd144070630 con 0x7fd1440b44c0 2024-09-10T20:38:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd3bead4640 1 -- 172.21.15.28:0/886078730 shutdown_connections 2024-09-10T20:38:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd3bead4640 1 -- 172.21.15.28:0/886078730 wait complete. 2024-09-10T20:38:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd1499f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd130004de0 0x7fd14415a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd1499f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd130004de0 0x7fd14415a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51006/0 (socket says 172.21.15.28:51006) 2024-09-10T20:38:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fd1499f3640 1 -- 172.21.15.28:0/3952932288 learned_addr learned my addr 172.21.15.28:0/3952932288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3800766a0 con 0x7fc3800aedd0 2024-09-10T20:38:23.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b9719b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b900ced40 0x7f5b901f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7fd1491f2640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1440aa590 0x7fd144141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7fd14a1f4640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1440b44c0 0x7fd1440b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b9719b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b900cf1b0 0x7f5b900c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b9719b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b90157a90 0x7f5b901eff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b9719b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b90070740 con 0x7f5b900ced40 2024-09-10T20:38:23.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7fd1491f2640 1 -- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1440b44c0 msgr2=0x7fd1440b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7fd1491f2640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1440b44c0 0x7fd1440b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7fd1491f2640 1 -- 172.21.15.28:0/3952932288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd130004de0 msgr2=0x7fd14415a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b8ffff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b900cf1b0 0x7f5b900c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b8ffff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b900cf1b0 0x7f5b900c8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41912/0 (socket says 172.21.15.28:41912) 2024-09-10T20:38:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b8ffff640 1 -- 172.21.15.28:0/1026379013 learned_addr learned my addr 172.21.15.28:0/1026379013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b95738640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b90157a90 0x7f5b901eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5b900705c0 con 0x7f5b900cf1b0 2024-09-10T20:38:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5b9006e460 con 0x7f5b90157a90 2024-09-10T20:38:23.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f177db99640 1 Processor -- start 2024-09-10T20:38:23.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f350c004bd0 0x7f35140dc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.056+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f350c004bd0 0x7f35140dc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:53112/0 (socket says 172.21.15.28:53112) 2024-09-10T20:38:23.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f351b7fe640 1 -- 172.21.15.28:0/1622241028 learned_addr learned my addr 172.21.15.28:0/1622241028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f177db99640 1 -- start start 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f3520e54640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35140e08f0 0x7f35140e0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f351403a7b0 con 0x7f351403d040 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f35140586a0 con 0x7f35140e08f0 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f351403a930 con 0x7f350c004bd0 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f351affd640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f351403d040 0x7f351403d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f351b7fe640 1 -- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35140e08f0 msgr2=0x7f35140e0d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f351b7fe640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35140e08f0 0x7f35140e0d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f351b7fe640 1 -- 172.21.15.28:0/1622241028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f351403d040 msgr2=0x7f351403d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f351b7fe640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f351403d040 0x7f351403d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f351b7fe640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35140d8750 con 0x7f350c004bd0 2024-09-10T20:38:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.057+0000 7f5b94f37640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b900ced40 0x7f5b901f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f351b7fe640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f350c004bd0 0x7f35140dc4b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f351c06b780 tx=0x7f351c06be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fd3bead4640 1 Processor -- start 2024-09-10T20:38:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fc375ffb640 1 -- 172.21.15.28:0/3858884161 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc37c03a030 con 0x7fc3800aedd0 2024-09-10T20:38:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fd3bead4640 1 -- start start 2024-09-10T20:38:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b8ffff640 1 -- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b90157a90 msgr2=0x7f5b901eff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b8ffff640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b90157a90 0x7f5b901eff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b8ffff640 1 -- 172.21.15.28:0/1026379013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b900ced40 msgr2=0x7f5b901f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b8ffff640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b900ced40 0x7f5b901f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b8ffff640 1 -- 172.21.15.28:0/1026379013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b900c92c0 con 0x7f5b900cf1b0 2024-09-10T20:38:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b95738640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b90157a90 0x7f5b901eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b94f37640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b900ced40 0x7f5b901f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f3507fff640 1 -- 172.21.15.28:0/1622241028 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c0b8630 con 0x7f350c004bd0 2024-09-10T20:38:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f3507fff640 1 -- 172.21.15.28:0/1622241028 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f351c0b87d0 con 0x7f350c004bd0 2024-09-10T20:38:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3514186090 con 0x7f350c004bd0 2024-09-10T20:38:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fd1491f2640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd130004de0 0x7fd14415a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fd1491f2640 1 -- 172.21.15.28:0/3952932288 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14415abd0 con 0x7fd1440aa590 2024-09-10T20:38:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fd1499f3640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd130004de0 0x7fd14415a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f5b8ffff640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b900cf1b0 0x7f5b900c8d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400c890 tx=0x7f5b8400cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fc375ffb640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc364077fa0 0x7fc36407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7fc375ffb640 1 -- 172.21.15.28:0/3858884161 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc37c0b9e90 con 0x7fc3800aedd0 2024-09-10T20:38:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f3507fff640 1 -- 172.21.15.28:0/1622241028 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c0b8b80 con 0x7f350c004bd0 2024-09-10T20:38:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.058+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35140d8cd0 con 0x7f350c004bd0 2024-09-10T20:38:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7fc3849e2640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc364077fa0 0x7fc36407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f177db99640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1770034450 0x7f17700e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7fd1491f2640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1440aa590 0x7fd144141f90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd134003820 tx=0x7fd134003ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f5b8d7fa640 1 -- 172.21.15.28:0/1026379013 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84004220 con 0x7f5b900cf1b0 2024-09-10T20:38:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f5b8d7fa640 1 -- 172.21.15.28:0/1026379013 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b840079b0 con 0x7f5b900cf1b0 2024-09-10T20:38:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f5b8d7fa640 1 -- 172.21.15.28:0/1026379013 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84007dc0 con 0x7f5b900cf1b0 2024-09-10T20:38:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b901f0860 con 0x7f5b900cf1b0 2024-09-10T20:38:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f177db99640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1770037dd0 0x7f177004c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f56c095c640 1 -- 172.21.15.28:0/2231907806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b4057230 msgr2=0x7f56b4036030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f177db99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770051ca0 0x7f177004ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f56c095c640 1 --2- 172.21.15.28:0/2231907806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b4057230 0x7f56b4036030 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f56bc052430 tx=0x7f56bc0a5d80 comp rx=0 tx=0).stop 2024-09-10T20:38:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b901f0e20 con 0x7f5b900cf1b0 2024-09-10T20:38:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7fd3bead4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3b80d2a80 0x7fd3b80b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7fc3849e2640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc364077fa0 0x7fc36407a460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3800cd000 tx=0x7fc37000a380 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7efd5da85640 1 -- 172.21.15.28:0/2487832179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58162260 msgr2=0x7efd580cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7efd5da85640 1 --2- 172.21.15.28:0/2487832179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58162260 0x7efd580cfbc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efd50005fd0 tx=0x7efd5002a840 comp rx=0 tx=0).stop 2024-09-10T20:38:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd13a7fc640 1 -- 172.21.15.28:0/3952932288 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134015070 con 0x7fd1440aa590 2024-09-10T20:38:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7efd5da85640 1 -- 172.21.15.28:0/2487832179 shutdown_connections 2024-09-10T20:38:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7efd5da85640 1 --2- 172.21.15.28:0/2487832179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58162260 0x7efd580cfbc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7efd5da85640 1 --2- 172.21.15.28:0/2487832179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd5815de70 0x7efd580d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd13a7fc640 1 -- 172.21.15.28:0/3952932288 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd13400fc40 con 0x7fd1440aa590 2024-09-10T20:38:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3514053660 con 0x7f350c004bd0 2024-09-10T20:38:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bc870640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3b80d2a80 0x7fd3b80b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bc870640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3b80d2a80 0x7fd3b80b7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41934/0 (socket says 172.21.15.28:41934) 2024-09-10T20:38:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f56c095c640 1 -- 172.21.15.28:0/2231907806 shutdown_connections 2024-09-10T20:38:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1441f8880 con 0x7fd1440aa590 2024-09-10T20:38:23.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bead4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b8161c00 0x7fd3b80cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f56c095c640 1 --2- 172.21.15.28:0/2231907806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b4057230 0x7f56b4036030 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f56c095c640 1 --2- 172.21.15.28:0/2231907806 >> v2:172.21.15.28:3300/0 conn(0x7f56b40e4310 0x7f56b4053ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f56c095c640 1 -- 172.21.15.28:0/2231907806 >> 172.21.15.28:0/2231907806 conn(0x7f56b4056710 msgr2=0x7f56b4056b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bc870640 1 -- 172.21.15.28:0/2158220835 learned_addr learned my addr 172.21.15.28:0/2158220835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7efd5da85640 1 -- 172.21.15.28:0/2487832179 >> 172.21.15.28:0/2487832179 conn(0x7efd580d3fb0 msgr2=0x7efd580d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1441ed330 con 0x7fd1440aa590 2024-09-10T20:38:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bead4640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b8161fe0 0x7fd3b80ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f177cb97640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1770034450 0x7f17700e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.059+0000 7f177cb97640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1770034450 0x7f17700e4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41924/0 (socket says 172.21.15.28:41924) 2024-09-10T20:38:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f1777fff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1770037dd0 0x7f177004c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f177cb97640 1 -- 172.21.15.28:0/554723423 learned_addr learned my addr 172.21.15.28:0/554723423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1770056480 con 0x7f1770051ca0 2024-09-10T20:38:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1770056300 con 0x7f1770034450 2024-09-10T20:38:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1770050860 con 0x7f1770037dd0 2024-09-10T20:38:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b815ebb0 con 0x7fd3b8161c00 2024-09-10T20:38:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f56c095c640 1 -- 172.21.15.28:0/2231907806 shutdown_connections 2024-09-10T20:38:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd13a7fc640 1 -- 172.21.15.28:0/3952932288 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134011430 con 0x7fd1440aa590 2024-09-10T20:38:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b80d1700 con 0x7fd3b80d2a80 2024-09-10T20:38:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f56c095c640 1 -- 172.21.15.28:0/2231907806 wait complete. 2024-09-10T20:38:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7f177d398640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770051ca0 0x7f177004ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7efd5da85640 1 -- 172.21.15.28:0/2487832179 shutdown_connections 2024-09-10T20:38:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3b7fff640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b8161c00 0x7fd3b80cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b815ed30 con 0x7fd3b8161fe0 2024-09-10T20:38:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7efd5da85640 1 -- 172.21.15.28:0/2487832179 wait complete. 2024-09-10T20:38:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7f177d398640 1 -- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1770037dd0 msgr2=0x7f177004c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7f177d398640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1770037dd0 0x7f177004c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7f177d398640 1 -- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1770034450 msgr2=0x7f17700e4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7f177d398640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1770034450 0x7f17700e4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7f177d398640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17700e4f60 con 0x7f1770051ca0 2024-09-10T20:38:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7fd3b7fff640 1 -- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b8161fe0 msgr2=0x7fd3b80ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b9015a500 con 0x7f5b900cf1b0 2024-09-10T20:38:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.060+0000 7fd3bd071640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b8161fe0 0x7fd3b80ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7fd3b7fff640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b8161fe0 0x7fd3b80ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7fd3b7fff640 1 -- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3b80d2a80 msgr2=0x7fd3b80b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7fd3b7fff640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3b80d2a80 0x7fd3b80b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7fd3b7fff640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b81f7e00 con 0x7fd3b8161c00 2024-09-10T20:38:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.061+0000 7fd3bc870640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3b80d2a80 0x7fd3b80b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7fd3bd071640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b8161fe0 0x7fd3b80ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7f5b8d7fa640 1 -- 172.21.15.28:0/1026379013 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5b84005020 con 0x7f5b900cf1b0 2024-09-10T20:38:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7fd3b7fff640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b8161c00 0x7fd3b80cc6e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b3e0 tx=0x7fd3a400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7fd3b57fa640 1 -- 172.21.15.28:0/2158220835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4015020 con 0x7fd3b8161c00 2024-09-10T20:38:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7fd3b57fa640 1 -- 172.21.15.28:0/2158220835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3a4011920 con 0x7fd3b8161c00 2024-09-10T20:38:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b80b7be0 con 0x7fd3b8161c00 2024-09-10T20:38:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7f5b8d7fa640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5b7c077fc0 0x7f5b7c07a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7f5b94f37640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5b7c077fc0 0x7f5b7c07a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7f5b8d7fa640 1 -- 172.21.15.28:0/1026379013 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5b8409a130 con 0x7f5b900cf1b0 2024-09-10T20:38:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.062+0000 7f56c095c640 1 Processor -- start 2024-09-10T20:38:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7fd13a7fc640 1 -- 172.21.15.28:0/3952932288 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd1340115d0 con 0x7fd1440aa590 2024-09-10T20:38:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b80b81a0 con 0x7fd3b8161c00 2024-09-10T20:38:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7fd3b57fa640 1 -- 172.21.15.28:0/2158220835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4011d70 con 0x7fd3b8161c00 2024-09-10T20:38:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7efd5da85640 1 Processor -- start 2024-09-10T20:38:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7f5b94f37640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5b7c077fc0 0x7f5b7c07a480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5b800046b0 tx=0x7f5b80009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd12c005290 con 0x7fd1440aa590 2024-09-10T20:38:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7efd5da85640 1 -- start start 2024-09-10T20:38:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7f177d398640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770051ca0 0x7f177004ee80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f177806b3a0 tx=0x7f17780ad5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7f56c095c640 1 -- start start 2024-09-10T20:38:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7f17757fa640 1 -- 172.21.15.28:0/554723423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17780bb070 con 0x7f1770051ca0 2024-09-10T20:38:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177004cf70 con 0x7f1770051ca0 2024-09-10T20:38:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177004d530 con 0x7f1770051ca0 2024-09-10T20:38:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.063+0000 7f17757fa640 1 -- 172.21.15.28:0/554723423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1778064310 con 0x7f1770051ca0 2024-09-10T20:38:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.064+0000 7f17757fa640 1 -- 172.21.15.28:0/554723423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17780b3040 con 0x7f1770051ca0 2024-09-10T20:38:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.064+0000 7fd13a7fc640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd128077fc0 0x7fd12807a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.064+0000 7fd13a7fc640 1 -- 172.21.15.28:0/3952932288 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd13409a960 con 0x7fd1440aa590 2024-09-10T20:38:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.064+0000 7f56c095c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b4057230 0x7f56b404a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56c095c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40e3f30 0x7f56b40d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56c095c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b40e4310 0x7f56b4176500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd5da85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd580cf780 0x7efd580a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1758005290 con 0x7f1770051ca0 2024-09-10T20:38:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56c095c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4054610 con 0x7f56b4057230 2024-09-10T20:38:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56c095c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f56b40368c0 con 0x7f56b40e3f30 2024-09-10T20:38:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56c095c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f56b4054790 con 0x7f56b40e4310 2024-09-10T20:38:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7fd1499f3640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd128077fc0 0x7fd12807a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd5da85640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd5815de70 0x7efd581f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd5da85640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd58162260 0x7efd580d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd5da85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd580b7190 con 0x7efd580cf780 2024-09-10T20:38:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd5da85640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efd580b7010 con 0x7efd5815de70 2024-09-10T20:38:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd5da85640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efd580b7310 con 0x7efd58162260 2024-09-10T20:38:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7fd3b57fa640 1 -- 172.21.15.28:0/2158220835 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd3a401a400 con 0x7fd3b8161c00 2024-09-10T20:38:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b815f1e0 con 0x7fd3b8161c00 2024-09-10T20:38:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56ba59c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40e3f30 0x7f56b40d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56ba59c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40e3f30 0x7f56b40d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41948/0 (socket says 172.21.15.28:41948) 2024-09-10T20:38:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56ba59c640 1 -- 172.21.15.28:0/1182364069 learned_addr learned my addr 172.21.15.28:0/1182364069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56bb59e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b40e4310 0x7f56b4176500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd4ffff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd5815de70 0x7efd581f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd4ffff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd5815de70 0x7efd581f44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41956/0 (socket says 172.21.15.28:41956) 2024-09-10T20:38:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd4ffff640 1 -- 172.21.15.28:0/1437216181 learned_addr learned my addr 172.21.15.28:0/1437216181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7fc375ffb640 1 -- 172.21.15.28:0/3858884161 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc37c049030 con 0x7fc3800aedd0 2024-09-10T20:38:23.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd57fff640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd58162260 0x7efd580d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56bad9d640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b4057230 0x7f56b404a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7efd577fe640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd580cf780 0x7efd580a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56ba59c640 1 -- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b40e4310 msgr2=0x7f56b4176500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56ba59c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b40e4310 0x7f56b4176500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.065+0000 7f56ba59c640 1 -- 172.21.15.28:0/1182364069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b4057230 msgr2=0x7f56b404a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7f56ba59c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b4057230 0x7f56b404a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7fd1499f3640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd128077fc0 0x7fd12807a480 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd140001950 tx=0x7fd140001290 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7f56ba59c640 1 -- 172.21.15.28:0/1182364069 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b40d6b40 con 0x7f56b40e3f30 2024-09-10T20:38:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7f56bad9d640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b4057230 0x7f56b404a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7fd3b57fa640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd39c077fa0 0x7fd39c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd57fff640 1 -- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd5815de70 msgr2=0x7efd581f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd57fff640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd5815de70 0x7efd581f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd57fff640 1 -- 172.21.15.28:0/1437216181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd580cf780 msgr2=0x7efd580a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd57fff640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd580cf780 0x7efd580a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7fd3b57fa640 1 -- 172.21.15.28:0/2158220835 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd3a4099dc0 con 0x7fd3b8161c00 2024-09-10T20:38:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7f56ba59c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40e3f30 0x7f56b40d0260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009fd0 tx=0x7f56b000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd57fff640 1 -- 172.21.15.28:0/1437216181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd581f5d30 con 0x7efd58162260 2024-09-10T20:38:23.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd577fe640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd580cf780 0x7efd580a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:23.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd4ffff640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd5815de70 0x7efd581f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:23.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7fd3bc870640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd39c077fa0 0x7fd39c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7f3507fff640 1 -- 172.21.15.28:0/1622241028 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f351c0c93f0 con 0x7f350c004bd0 2024-09-10T20:38:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.066+0000 7efd57fff640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd58162260 0x7efd580d0240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd50009000 tx=0x7efd50032d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.067+0000 7f17757fa640 1 -- 172.21.15.28:0/554723423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f17780b6050 con 0x7f1770051ca0 2024-09-10T20:38:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.067+0000 7fd3bc870640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd39c077fa0 0x7fd39c07a460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac000f60 tx=0x7fd3ac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.067+0000 7f3507fff640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3500078070 0x7f350007a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.067+0000 7f3507fff640 1 -- 172.21.15.28:0/1622241028 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f351c136f20 con 0x7f350c004bd0 2024-09-10T20:38:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.067+0000 7f351affd640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3500078070 0x7f350007a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.067+0000 7f17757fa640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f175c077f70 0x7f175c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.067+0000 7f5b8d7fa640 1 -- 172.21.15.28:0/1026379013 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5b9015a500 con 0x7f5b900cf1b0 2024-09-10T20:38:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.068+0000 7f351affd640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3500078070 0x7f350007a530 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f35140efec0 tx=0x7f3508009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.068+0000 7f17757fa640 1 -- 172.21.15.28:0/554723423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1778135890 con 0x7f1770051ca0 2024-09-10T20:38:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.068+0000 7f177cb97640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f175c077f70 0x7f175c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.068+0000 7f177cb97640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f175c077f70 0x7f175c07a430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1764009a30 tx=0x7f1764009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.070+0000 7efd54ff9640 1 -- 172.21.15.28:0/1437216181 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50040070 con 0x7efd58162260 2024-09-10T20:38:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.070+0000 7efd54ff9640 1 -- 172.21.15.28:0/1437216181 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd50004250 con 0x7efd58162260 2024-09-10T20:38:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.070+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd581f6010 con 0x7efd58162260 2024-09-10T20:38:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.070+0000 7efd54ff9640 1 -- 172.21.15.28:0/1437216181 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd400016d0 con 0x7efd58162260 2024-09-10T20:38:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.070+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd581f65d0 con 0x7efd58162260 2024-09-10T20:38:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.070+0000 7f3507fff640 1 -- 172.21.15.28:0/1622241028 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f351c0bc030 con 0x7f350c004bd0 2024-09-10T20:38:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.071+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd580d3b80 con 0x7efd58162260 2024-09-10T20:38:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.072+0000 7fd13a7fc640 1 -- 172.21.15.28:0/3952932288 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd134062f80 con 0x7fd1440aa590 2024-09-10T20:38:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.072+0000 7fd3b57fa640 1 -- 172.21.15.28:0/2158220835 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd3a401f020 con 0x7fd3b8161c00 2024-09-10T20:38:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.072+0000 7f56a37fe640 1 -- 172.21.15.28:0/1182364069 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b000ccc0 con 0x7f56b40e3f30 2024-09-10T20:38:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.072+0000 7f56a37fe640 1 -- 172.21.15.28:0/1182364069 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56b000ee70 con 0x7f56b40e3f30 2024-09-10T20:38:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.072+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b404c9a0 con 0x7f56b40e3f30 2024-09-10T20:38:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.072+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b404cf60 con 0x7f56b40e3f30 2024-09-10T20:38:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.073+0000 7f56a37fe640 1 -- 172.21.15.28:0/1182364069 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b000ce60 con 0x7f56b40e3f30 2024-09-10T20:38:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.072+0000 7efd54ff9640 1 -- 172.21.15.28:0/1437216181 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efd50004680 con 0x7efd58162260 2024-09-10T20:38:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.073+0000 7efd54ff9640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd38077f70 0x7efd3807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.073+0000 7efd54ff9640 1 -- 172.21.15.28:0/1437216181 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efd500b9530 con 0x7efd58162260 2024-09-10T20:38:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.074+0000 7efd577fe640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd38077f70 0x7efd3807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.074+0000 7efd577fe640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd38077f70 0x7efd3807a430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efd48005fd0 tx=0x7efd48009260 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.075+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b4036030 con 0x7f56b40e3f30 2024-09-10T20:38:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.075+0000 7f56a37fe640 1 -- 172.21.15.28:0/1182364069 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f56b0005220 con 0x7f56b40e3f30 2024-09-10T20:38:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.076+0000 7f56a37fe640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f56a8077f70 0x7f56a807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.076+0000 7f56a37fe640 1 -- 172.21.15.28:0/1182364069 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f56b009a5d0 con 0x7f56b40e3f30 2024-09-10T20:38:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.076+0000 7f56bad9d640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f56a8077f70 0x7f56a807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.077+0000 7f56bad9d640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f56a8077f70 0x7f56a807a430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f56ac001270 tx=0x7f56ac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.078+0000 7efd54ff9640 1 -- 172.21.15.28:0/1437216181 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efd580d3b80 con 0x7efd58162260 2024-09-10T20:38:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.080+0000 7f56a37fe640 1 -- 172.21.15.28:0/1182364069 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f56b00630b0 con 0x7f56b40e3f30 2024-09-10T20:38:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.081+0000 7f17757fa640 1 -- 172.21.15.28:0/554723423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f17780fddd0 con 0x7f1770051ca0 2024-09-10T20:38:23.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.500+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5b9006ea20 con 0x7f5b900cf1b0 2024-09-10T20:38:23.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.501+0000 7f5b8d7fa640 1 -- 172.21.15.28:0/1026379013 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5b84014030 con 0x7f5b900cf1b0 2024-09-10T20:38:23.505 INFO:teuthology.orchestra.run.smithi028.stdout:60129542149 2024-09-10T20:38:23.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.502+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc38015edb0 con 0x7fc3800aedd0 2024-09-10T20:38:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.502+0000 7fc375ffb640 1 -- 172.21.15.28:0/3858884161 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc38015f2f0 con 0x7fc3800aedd0 2024-09-10T20:38:23.506 INFO:teuthology.orchestra.run.smithi028.stdout:60129542150 2024-09-10T20:38:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.503+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3514000e10 con 0x7f350c004bd0 2024-09-10T20:38:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5b7c077fc0 msgr2=0x7f5b7c07a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5b7c077fc0 0x7f5b7c07a480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5b800046b0 tx=0x7f5b80009040 comp rx=0 tx=0).stop 2024-09-10T20:38:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b900cf1b0 msgr2=0x7f5b900c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b900cf1b0 0x7f5b900c8d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400c890 tx=0x7f5b8400cd50 comp rx=0 tx=0).stop 2024-09-10T20:38:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 shutdown_connections 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5b7c077fc0 0x7f5b7c07a480 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5b90157a90 0x7f5b901eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5b900cf1b0 0x7f5b900c8d80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 --2- 172.21.15.28:0/1026379013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b900ced40 0x7f5b901f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 >> 172.21.15.28:0/1026379013 conn(0x7f5b900cf9c0 msgr2=0x7f5b9015cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 shutdown_connections 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.505+0000 7f5b9719b640 1 -- 172.21.15.28:0/1026379013 wait complete. 2024-09-10T20:38:23.510 INFO:teuthology.orchestra.run.smithi028.stdout:64424509446 2024-09-10T20:38:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.503+0000 7f3507fff640 1 -- 172.21.15.28:0/1622241028 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f351c0ffa90 con 0x7f350c004bd0 2024-09-10T20:38:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc364077fa0 msgr2=0x7fc36407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc364077fa0 0x7fc36407a460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3800cd000 tx=0x7fc37000a380 comp rx=0 tx=0).stop 2024-09-10T20:38:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3800aedd0 msgr2=0x7fc3800b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3800aedd0 0x7fc3800b0e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009950 tx=0x7fc37c033c20 comp rx=0 tx=0).stop 2024-09-10T20:38:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd12c005850 con 0x7fd1440aa590 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 shutdown_connections 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc364077fa0 0x7fc36407a460 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3800cc070 0x7fc3800cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3800cdde0 0x7fc3800b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 --2- 172.21.15.28:0/3858884161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3800aedd0 0x7fc3800b0e00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 >> 172.21.15.28:0/3858884161 conn(0x7fc3800cf990 msgr2=0x7fc3801481c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.506+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 shutdown_connections 2024-09-10T20:38:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7fc3861e5640 1 -- 172.21.15.28:0/3858884161 wait complete. 2024-09-10T20:38:23.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7fd13a7fc640 1 -- 172.21.15.28:0/3952932288 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd1340679b0 con 0x7fd1440aa590 2024-09-10T20:38:23.513 INFO:teuthology.orchestra.run.smithi028.stdout:64424509446 2024-09-10T20:38:23.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3500078070 msgr2=0x7f350007a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7f3520e54640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3500078070 0x7f350007a530 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f35140efec0 tx=0x7f3508009040 comp rx=0 tx=0).stop 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f350c004bd0 msgr2=0x7f35140dc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7f3520e54640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f350c004bd0 0x7f35140dc4b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f351c06b780 tx=0x7f351c06be20 comp rx=0 tx=0).stop 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 shutdown_connections 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.507+0000 7f3520e54640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3500078070 0x7f350007a530 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.508+0000 7f3520e54640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35140e08f0 0x7f35140e0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.508+0000 7f3520e54640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f351403d040 0x7f351403d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.508+0000 7f3520e54640 1 --2- 172.21.15.28:0/1622241028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f350c004bd0 0x7f35140dc4b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.508+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 >> 172.21.15.28:0/1622241028 conn(0x7f3514055540 msgr2=0x7f35140e64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.508+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 shutdown_connections 2024-09-10T20:38:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.508+0000 7f3520e54640 1 -- 172.21.15.28:0/1622241028 wait complete. 2024-09-10T20:38:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.508+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd3b80d3d30 con 0x7fd3b8161c00 2024-09-10T20:38:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.509+0000 7fd3b57fa640 1 -- 172.21.15.28:0/2158220835 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd3a4067e20 con 0x7fd3b8161c00 2024-09-10T20:38:23.515 INFO:teuthology.orchestra.run.smithi028.stdout:64424509447 2024-09-10T20:38:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.510+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd128077fc0 msgr2=0x7fd12807a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd128077fc0 0x7fd12807a480 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd140001950 tx=0x7fd140001290 comp rx=0 tx=0).stop 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1440aa590 msgr2=0x7fd144141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1440aa590 0x7fd144141f90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd134003820 tx=0x7fd134003ce0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 shutdown_connections 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd128077fc0 0x7fd12807a480 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1440b44c0 0x7fd1440b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1440aa590 0x7fd144141f90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 --2- 172.21.15.28:0/3952932288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd130004de0 0x7fd14415a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 >> 172.21.15.28:0/3952932288 conn(0x7fd1440cf990 msgr2=0x7fd1440af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 shutdown_connections 2024-09-10T20:38:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.511+0000 7fd14a9f5640 1 -- 172.21.15.28:0/3952932288 wait complete. 2024-09-10T20:38:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.513+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7efd5806a280 con 0x7efd58162260 2024-09-10T20:38:23.518 INFO:teuthology.orchestra.run.smithi028.stdout:60129542151 2024-09-10T20:38:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.514+0000 7efd54ff9640 1 -- 172.21.15.28:0/1437216181 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efd5806a280 con 0x7efd58162260 2024-09-10T20:38:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd39c077fa0 msgr2=0x7fd39c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd39c077fa0 0x7fd39c07a460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac000f60 tx=0x7fd3ac009040 comp rx=0 tx=0).stop 2024-09-10T20:38:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b8161c00 msgr2=0x7fd3b80cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b8161c00 0x7fd3b80cc6e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b3e0 tx=0x7fd3a400b8a0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 shutdown_connections 2024-09-10T20:38:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd39c077fa0 0x7fd39c07a460 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac000f60 tx=0x7fd3ac009040 comp rx=0 tx=0).stop 2024-09-10T20:38:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3b8161fe0 0x7fd3b80ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b8161c00 0x7fd3b80cc6e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 --2- 172.21.15.28:0/2158220835 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3b80d2a80 0x7fd3b80b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 >> 172.21.15.28:0/2158220835 conn(0x7fd3b809d230 msgr2=0x7fd3b80b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f1758005850 con 0x7f1770051ca0 2024-09-10T20:38:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 shutdown_connections 2024-09-10T20:38:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.515+0000 7f17757fa640 1 -- 172.21.15.28:0/554723423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1778102800 con 0x7f1770051ca0 2024-09-10T20:38:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.516+0000 7fd3bead4640 1 -- 172.21.15.28:0/2158220835 wait complete. 2024-09-10T20:38:23.521 INFO:teuthology.orchestra.run.smithi028.stdout:64424509446 2024-09-10T20:38:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.518+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f56b4036ae0 con 0x7f56b40e3f30 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd38077f70 msgr2=0x7efd3807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd38077f70 0x7efd3807a430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efd48005fd0 tx=0x7efd48009260 comp rx=0 tx=0).stop 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd58162260 msgr2=0x7efd580d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd58162260 0x7efd580d0240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd50009000 tx=0x7efd50032d30 comp rx=0 tx=0).stop 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 shutdown_connections 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd38077f70 0x7efd3807a430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd58162260 0x7efd580d0240 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd5815de70 0x7efd581f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 --2- 172.21.15.28:0/1437216181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd580cf780 0x7efd580a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 >> 172.21.15.28:0/1437216181 conn(0x7efd580d3fb0 msgr2=0x7efd580b94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7f56a37fe640 1 -- 172.21.15.28:0/1182364069 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f56b00630b0 con 0x7f56b40e3f30 2024-09-10T20:38:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 shutdown_connections 2024-09-10T20:38:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7efd5da85640 1 -- 172.21.15.28:0/1437216181 wait complete. 2024-09-10T20:38:23.524 INFO:teuthology.orchestra.run.smithi028.stdout:60129542149 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.519+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f175c077f70 msgr2=0x7f175c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.520+0000 7f177db99640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f175c077f70 0x7f175c07a430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1764009a30 tx=0x7f1764009190 comp rx=0 tx=0).stop 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.520+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770051ca0 msgr2=0x7f177004ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.520+0000 7f177db99640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770051ca0 0x7f177004ee80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f177806b3a0 tx=0x7f17780ad5d0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 shutdown_connections 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f175c077f70 0x7f175c07a430 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770051ca0 0x7f177004ee80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1770037dd0 0x7f177004c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 --2- 172.21.15.28:0/554723423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1770034450 0x7f17700e4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 >> 172.21.15.28:0/554723423 conn(0x7f177001e8c0 msgr2=0x7f1770033280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 shutdown_connections 2024-09-10T20:38:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.521+0000 7f177db99640 1 -- 172.21.15.28:0/554723423 wait complete. 2024-09-10T20:38:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f56a8077f70 msgr2=0x7f56a807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f56a8077f70 0x7f56a807a430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f56ac001270 tx=0x7f56ac009040 comp rx=0 tx=0).stop 2024-09-10T20:38:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40e3f30 msgr2=0x7f56b40d0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40e3f30 0x7f56b40d0260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009fd0 tx=0x7f56b000ec20 comp rx=0 tx=0).stop 2024-09-10T20:38:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 shutdown_connections 2024-09-10T20:38:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f56a8077f70 0x7f56a807a430 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f56ac001270 tx=0x7f56ac009040 comp rx=0 tx=0).stop 2024-09-10T20:38:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56b40e4310 0x7f56b4176500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f56b40e3f30 0x7f56b40d0260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 --2- 172.21.15.28:0/1182364069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b4057230 0x7f56b404a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.523+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 >> 172.21.15.28:0/1182364069 conn(0x7f56b4056710 msgr2=0x7f56b4052ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.524+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 shutdown_connections 2024-09-10T20:38:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:23.524+0000 7f56c095c640 1 -- 172.21.15.28:0/1182364069 wait complete. 2024-09-10T20:38:23.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.4 2024-09-10T20:38:23.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2024-09-10T20:38:23.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.0 2024-09-10T20:38:23.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.2 2024-09-10T20:38:23.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.3 2024-09-10T20:38:23.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.6 2024-09-10T20:38:23.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.1 2024-09-10T20:38:23.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2024-09-10T20:38:24.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-10T20:38:24.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-10T20:38:24.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:38:24.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:38:24.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:38:24.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-10T20:38:24.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T20:38:24.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-10T20:38:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.705+0000 7f9307fff640 1 -- 172.21.15.28:0/4193932085 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9308031b80 con 0x7f93100d04b0 2024-09-10T20:38:24.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.706+0000 7f93167de640 1 -- 172.21.15.28:0/4193932085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93100d04b0 msgr2=0x7f93100cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.706+0000 7f93167de640 1 --2- 172.21.15.28:0/4193932085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93100d04b0 0x7f93100cab10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9308005fd0 tx=0x7f930802a720 comp rx=0 tx=0).stop 2024-09-10T20:38:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7f93167de640 1 -- 172.21.15.28:0/4193932085 shutdown_connections 2024-09-10T20:38:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7f93167de640 1 --2- 172.21.15.28:0/4193932085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93100d04b0 0x7f93100cab10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7f93167de640 1 --2- 172.21.15.28:0/4193932085 >> v2:172.21.15.28:3300/0 conn(0x7f931015e2e0 0x7f9310161fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7fc9afdf3640 1 -- 172.21.15.28:0/1903394278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 msgr2=0x7fc9a80cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7fc9afdf3640 1 --2- 172.21.15.28:0/1903394278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 0x7fc9a80cbc80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc998005fd0 tx=0x7fc99802a4d0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7f93167de640 1 -- 172.21.15.28:0/4193932085 >> 172.21.15.28:0/4193932085 conn(0x7f93100cf990 msgr2=0x7f93100cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7f93167de640 1 -- 172.21.15.28:0/4193932085 shutdown_connections 2024-09-10T20:38:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7fc9afdf3640 1 -- 172.21.15.28:0/1903394278 shutdown_connections 2024-09-10T20:38:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7fc9afdf3640 1 --2- 172.21.15.28:0/1903394278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 0x7fc9a80cbc80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7fc9afdf3640 1 --2- 172.21.15.28:0/1903394278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a806eb10 0x7fc9a80ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7fc9afdf3640 1 -- 172.21.15.28:0/1903394278 >> 172.21.15.28:0/1903394278 conn(0x7fc9a80b15b0 msgr2=0x7fc9a80b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.707+0000 7f93167de640 1 -- 172.21.15.28:0/4193932085 wait complete. 2024-09-10T20:38:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.708+0000 7fc9afdf3640 1 -- 172.21.15.28:0/1903394278 shutdown_connections 2024-09-10T20:38:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.708+0000 7fc9afdf3640 1 -- 172.21.15.28:0/1903394278 wait complete. 2024-09-10T20:38:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.709+0000 7f93167de640 1 Processor -- start 2024-09-10T20:38:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.709+0000 7f93167de640 1 -- start start 2024-09-10T20:38:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.710+0000 7f3b7e39b640 1 -- 172.21.15.28:0/4021290471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 msgr2=0x7f3b7806f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.710+0000 7f3b7e39b640 1 --2- 172.21.15.28:0/4021290471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 0x7f3b7806f520 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3b7806ec50 tx=0x7f3b6402a4d0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f3b7e39b640 1 -- 172.21.15.28:0/4021290471 shutdown_connections 2024-09-10T20:38:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f3b7e39b640 1 --2- 172.21.15.28:0/4021290471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 0x7f3b780cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f3b7e39b640 1 --2- 172.21.15.28:0/4021290471 >> v2:172.21.15.28:3300/0 conn(0x7f3b7815ef80 0x7f3b780cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f3b7e39b640 1 --2- 172.21.15.28:0/4021290471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 0x7f3b7806f520 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f3b7e39b640 1 -- 172.21.15.28:0/4021290471 >> 172.21.15.28:0/4021290471 conn(0x7f3b780cfba0 msgr2=0x7f3b780cffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.710+0000 7fc9afdf3640 1 Processor -- start 2024-09-10T20:38:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.710+0000 7fc9afdf3640 1 -- start start 2024-09-10T20:38:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f3b7e39b640 1 -- 172.21.15.28:0/4021290471 shutdown_connections 2024-09-10T20:38:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f3b7e39b640 1 -- 172.21.15.28:0/4021290471 wait complete. 2024-09-10T20:38:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f93167de640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93100ae960 0x7f93101609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.711+0000 7f93167de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93100d04b0 0x7f931014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f9314fdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93100d04b0 0x7f931014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f9314fdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93100d04b0 0x7f931014f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51124/0 (socket says 172.21.15.28:51124) 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f93157dc640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93100ae960 0x7f93101609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f93157dc640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93100ae960 0x7f93101609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:41964/0 (socket says 172.21.15.28:41964) 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f9314fdb640 1 -- 172.21.15.28:0/2622951895 learned_addr learned my addr 172.21.15.28:0/2622951895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f93167de640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f931015e2e0 0x7f9310147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f93167de640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93100ce970 con 0x7f93100d04b0 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f93167de640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f93100ce7f0 con 0x7f93100ae960 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7f93167de640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f93100ceaf0 con 0x7f931015e2e0 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7f9314fdb640 1 -- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f931015e2e0 msgr2=0x7f9310147ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T20:38:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7f9314fdb640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f931015e2e0 0x7f9310147ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.714+0000 7f3b7e39b640 1 Processor -- start 2024-09-10T20:38:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7f9314fdb640 1 -- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93100ae960 msgr2=0x7f93101609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7f9314fdb640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93100ae960 0x7f93101609c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7f9314fdb640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9310072440 con 0x7f93100d04b0 2024-09-10T20:38:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7f9314fdb640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93100d04b0 0x7f931014f7f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f930000e9f0 tx=0x7f930000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9afdf3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a806e730 0x7fc9a815aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9afdf3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a806eb10 0x7fc9a8147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9afdf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 0x7fc9a8148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9afdf3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a80cc690 con 0x7fc9a80cae80 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9afdf3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a80cc510 con 0x7fc9a806e730 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9afdf3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a8150580 con 0x7fc9a806eb10 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9ad38e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a806eb10 0x7fc9a8147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9ad38e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a806eb10 0x7fc9a8147cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:53220/0 (socket says 172.21.15.28:53220) 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9ad38e640 1 -- 172.21.15.28:0/3013210319 learned_addr learned my addr 172.21.15.28:0/3013210319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.712+0000 7fc9ae390640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 0x7fc9a8148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7fc9ae390640 1 -- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a806eb10 msgr2=0x7fc9a8147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7fc9ae390640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a806eb10 0x7fc9a8147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7fc9ae390640 1 -- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a806e730 msgr2=0x7fc9a815aa00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7fc9ae390640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a806e730 0x7fc9a815aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7fc9ae390640 1 -- 172.21.15.28:0/3013210319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a8149160 con 0x7fc9a80cae80 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.713+0000 7fc9ae390640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 0x7fc9a8148200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc9a806efc0 tx=0x7fc998003940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.714+0000 7fc99e7fc640 1 -- 172.21.15.28:0/3013210319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99803e030 con 0x7fc9a80cae80 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.714+0000 7fc99e7fc640 1 -- 172.21.15.28:0/3013210319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc998003e70 con 0x7fc9a80cae80 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.714+0000 7fc9afdf3640 1 -- 172.21.15.28:0/3013210319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a815b460 con 0x7fc9a80cae80 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.714+0000 7fc99e7fc640 1 -- 172.21.15.28:0/3013210319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998035040 con 0x7fc9a80cae80 2024-09-10T20:38:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.714+0000 7fc9afdf3640 1 -- 172.21.15.28:0/3013210319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a8158bc0 con 0x7fc9a80cae80 2024-09-10T20:38:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.714+0000 7f3b7e39b640 1 -- start start 2024-09-10T20:38:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.716+0000 7f3b7e39b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 0x7f3b780c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.716+0000 7f3b7e39b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 0x7f3b781f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.716+0000 7f3b77fff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 0x7f3b780c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.716+0000 7f9305ffb640 1 -- 172.21.15.28:0/2622951895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930000cd80 con 0x7f93100d04b0 2024-09-10T20:38:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f3b77fff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 0x7f3b780c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:53240/0 (socket says 172.21.15.28:53240) 2024-09-10T20:38:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f3b77fff640 1 -- 172.21.15.28:0/1531061254 learned_addr learned my addr 172.21.15.28:0/1531061254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f3b777fe640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 0x7f3b781f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f93167de640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9310160fa0 con 0x7f93100d04b0 2024-09-10T20:38:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f93167de640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93101fc230 con 0x7f93100d04b0 2024-09-10T20:38:24.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f3b7e39b640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7815ef80 0x7f3b780c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f3b7e39b640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b780ada30 con 0x7f3b7815ef80 2024-09-10T20:38:24.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f3b7e39b640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3b780ad8b0 con 0x7f3b780aee50 2024-09-10T20:38:24.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7f3b7c938640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7815ef80 0x7f3b780c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.718+0000 7f3b7e39b640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3b780adbb0 con 0x7f3b7806ff80 2024-09-10T20:38:24.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.718+0000 7f9305ffb640 1 -- 172.21.15.28:0/2622951895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9300004020 con 0x7f93100d04b0 2024-09-10T20:38:24.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.718+0000 7f3b77fff640 1 -- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 msgr2=0x7f3b781f8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7f3b77fff640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 0x7f3b781f8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7f3b77fff640 1 -- 172.21.15.28:0/1531061254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7815ef80 msgr2=0x7f3b780c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7f3b77fff640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7815ef80 0x7f3b780c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7f3b77fff640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b781f90b0 con 0x7f3b7806ff80 2024-09-10T20:38:24.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7f9305ffb640 1 -- 172.21.15.28:0/2622951895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9300004450 con 0x7f93100d04b0 2024-09-10T20:38:24.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7f3b77fff640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 0x7f3b780c8ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3b64005fd0 tx=0x7f3b64004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.717+0000 7fc99e7fc640 1 -- 172.21.15.28:0/3013210319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc9980049e0 con 0x7fc9a80cae80 2024-09-10T20:38:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.718+0000 7fc99e7fc640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc994078190 0x7fc99407a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.718+0000 7fc9afdf3640 1 -- 172.21.15.28:0/3013210319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc990005290 con 0x7fc9a80cae80 2024-09-10T20:38:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.718+0000 7fc9adb8f640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc994078190 0x7fc99407a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.718+0000 7fc99e7fc640 1 -- 172.21.15.28:0/3013210319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc9980ba530 con 0x7fc9a80cae80 2024-09-10T20:38:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7fc9adb8f640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc994078190 0x7fc99407a650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009670 tx=0x7fc9a4006bd0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.720+0000 7f93167de640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93100766a0 con 0x7f93100d04b0 2024-09-10T20:38:24.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.719+0000 7f3b7c938640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7815ef80 0x7f3b780c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:24.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.720+0000 7f3b74ff9640 1 -- 172.21.15.28:0/1531061254 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64040070 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.720+0000 7f3b74ff9640 1 -- 172.21.15.28:0/1531061254 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b640035f0 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.720+0000 7f3b7e39b640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b780c9110 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.720+0000 7f3b7e39b640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b780c96d0 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.720+0000 7f3b74ff9640 1 -- 172.21.15.28:0/1531061254 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64040070 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.722+0000 7f3b7e39b640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b58005290 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.723+0000 7f3b74ff9640 1 -- 172.21.15.28:0/1531061254 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3b6403a040 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.724+0000 7f3b74ff9640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3b5c078370 0x7f3b5c07a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.724+0000 7f3b74ff9640 1 -- 172.21.15.28:0/1531061254 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3b64005020 con 0x7f3b7806ff80 2024-09-10T20:38:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.725+0000 7f3b777fe640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3b5c078370 0x7f3b5c07a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.725+0000 7f3b777fe640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3b5c078370 0x7f3b5c07a830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b70000ba0 tx=0x7f3b70009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.725+0000 7fc99e7fc640 1 -- 172.21.15.28:0/3013210319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc998082a50 con 0x7fc9a80cae80 2024-09-10T20:38:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.728+0000 7f9305ffb640 1 -- 172.21.15.28:0/2622951895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f93000035f0 con 0x7f93100d04b0 2024-09-10T20:38:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.729+0000 7f9305ffb640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f92f4078040 0x7f92f407a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.729+0000 7f9305ffb640 1 -- 172.21.15.28:0/2622951895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f930009ac20 con 0x7f93100d04b0 2024-09-10T20:38:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.729+0000 7f3b74ff9640 1 -- 172.21.15.28:0/1531061254 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3b781f90b0 con 0x7f3b7806ff80 2024-09-10T20:38:24.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.735+0000 7f9305ffb640 1 -- 172.21.15.28:0/2622951895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9300063700 con 0x7f93100d04b0 2024-09-10T20:38:24.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.743+0000 7f93157dc640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f92f4078040 0x7f92f407a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.751+0000 7f93157dc640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f92f4078040 0x7f92f407a500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9310155c30 tx=0x7f930c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 -- 172.21.15.28:0/1575330944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b0070cc0 msgr2=0x7fe6b0074080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/1575330944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b0070cc0 0x7fe6b0074080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0005fd0 tx=0x7fe6a002a5f0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 -- 172.21.15.28:0/1575330944 shutdown_connections 2024-09-10T20:38:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/1575330944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b0070cc0 0x7fe6b0074080 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/1575330944 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b00708e0 0x7fe6b00726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 -- 172.21.15.28:0/1575330944 >> 172.21.15.28:0/1575330944 conn(0x7fe6b00b18a0 msgr2=0x7fe6b00b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 -- 172.21.15.28:0/1575330944 shutdown_connections 2024-09-10T20:38:24.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7f9e7f59e640 1 -- 172.21.15.28:0/1233309834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e780e3d80 msgr2=0x7f9e78052fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/1233309834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e780e3d80 0x7f9e78052fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9e8006bcd0 tx=0x7f9e800a5e00 comp rx=0 tx=0).stop 2024-09-10T20:38:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.767+0000 7f9e7f59e640 1 -- 172.21.15.28:0/1233309834 shutdown_connections 2024-09-10T20:38:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.767+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/1233309834 >> v2:172.21.15.28:3300/0 conn(0x7f9e78037520 0x7f9e78035e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.767+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/1233309834 >> v2:172.21.15.159:3300/0 conn(0x7f9e780e4160 0x7f9e78053520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.767+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/1233309834 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e780e3d80 0x7f9e78052fe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.767+0000 7f9e7f59e640 1 -- 172.21.15.28:0/1233309834 >> 172.21.15.28:0/1233309834 conn(0x7f9e78056710 msgr2=0x7f9e78056b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.767+0000 7fe6f9ffb640 1 -- 172.21.15.28:0/1600626436 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4031c00 con 0x7fe6fc0aedd0 2024-09-10T20:38:24.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.767+0000 7f9e7f59e640 1 -- 172.21.15.28:0/1233309834 shutdown_connections 2024-09-10T20:38:24.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 -- 172.21.15.28:0/1600626436 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0aedd0 msgr2=0x7fe6fc157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/1600626436 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0aedd0 0x7fe6fc157520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009be0 tx=0x7fe6f402a8c0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.766+0000 7fe6b68b6640 1 -- 172.21.15.28:0/1575330944 wait complete. 2024-09-10T20:38:24.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 -- 172.21.15.28:0/1600626436 shutdown_connections 2024-09-10T20:38:24.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/1600626436 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0aedd0 0x7fe6fc157520 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/1600626436 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6fc0cdde0 0x7fe6fc156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 -- 172.21.15.28:0/1600626436 >> 172.21.15.28:0/1600626436 conn(0x7fe6fc0cf990 msgr2=0x7fe6fc0cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6b68b6640 1 Processor -- start 2024-09-10T20:38:24.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 -- 172.21.15.28:0/1600626436 shutdown_connections 2024-09-10T20:38:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fd5a62b5640 1 -- 172.21.15.28:0/2940578788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e75c0 msgr2=0x7fd5980e2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/2940578788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e75c0 0x7fd5980e2fa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd5a006bf50 tx=0x7fd5a00a5d30 comp rx=0 tx=0).stop 2024-09-10T20:38:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fd5a62b5640 1 -- 172.21.15.28:0/2940578788 shutdown_connections 2024-09-10T20:38:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/2940578788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e75c0 0x7fd5980e2fa0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/2940578788 >> v2:172.21.15.159:3300/0 conn(0x7fd5980e7150 0x7fd5980e2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fe6fbfff640 1 -- 172.21.15.28:0/1600626436 wait complete. 2024-09-10T20:38:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.769+0000 7fe6b68b6640 1 -- start start 2024-09-10T20:38:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/2940578788 >> v2:172.21.15.28:3300/0 conn(0x7fd598039b40 0x7fd598039f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7fd5a62b5640 1 -- 172.21.15.28:0/2940578788 >> 172.21.15.28:0/2940578788 conn(0x7fd59801e000 msgr2=0x7fd598054df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.768+0000 7f9e7f59e640 1 -- 172.21.15.28:0/1233309834 wait complete. 2024-09-10T20:38:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7f9e7f59e640 1 Processor -- start 2024-09-10T20:38:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7f0a2cffc640 1 -- 172.21.15.28:0/3652760519 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20036400 con 0x7f0a28155060 2024-09-10T20:38:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fd5a62b5640 1 -- 172.21.15.28:0/2940578788 shutdown_connections 2024-09-10T20:38:24.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7f9e7f59e640 1 -- start start 2024-09-10T20:38:24.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7f0a30262640 1 -- 172.21.15.28:0/3652760519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a28155060 msgr2=0x7f0a2806e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fe6b68b6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b00708e0 0x7fe6b009c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fe6b68b6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b0070cc0 0x7fe6b006f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fe6fbfff640 1 Processor -- start 2024-09-10T20:38:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fe6b68b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b00745c0 0x7fe6b006fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fe6affff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b00708e0 0x7fe6b009c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.771+0000 7fe6affff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b00708e0 0x7fe6b009c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:42022/0 (socket says 172.21.15.28:42022) 2024-09-10T20:38:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.771+0000 7fe6affff640 1 -- 172.21.15.28:0/2610671370 learned_addr learned my addr 172.21.15.28:0/2610671370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.771+0000 7fe6b4e53640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b00745c0 0x7fe6b006fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.771+0000 7fe6af7fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b0070cc0 0x7fe6b006f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.771+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b00cea80 con 0x7fe6b00745c0 2024-09-10T20:38:24.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.771+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b00ce900 con 0x7fe6b00708e0 2024-09-10T20:38:24.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.771+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b01592d0 con 0x7fe6b0070cc0 2024-09-10T20:38:24.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7f0a30262640 1 --2- 172.21.15.28:0/3652760519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a28155060 0x7f0a2806e4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0a20005fd0 tx=0x7f0a2002aaa0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fe6fbfff640 1 -- start start 2024-09-10T20:38:24.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6fbfff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0ae9f0 0x7fe6fc1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7f9e7f59e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e78037520 0x7f9e78185b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6fbfff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6fc0aedd0 0x7fe6fc147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6fc0cdde0 0x7fe6fc147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7f9e7f59e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e780e3d80 0x7f9e78034470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6affff640 1 -- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b0070cc0 msgr2=0x7fe6b006f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6affff640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b0070cc0 0x7fe6b006f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6affff640 1 -- 172.21.15.28:0/2610671370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b00745c0 msgr2=0x7fe6b006fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7f9e7f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780e4160 0x7f9e78051c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6faffd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0ae9f0 0x7fe6fc1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6faffd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0ae9f0 0x7fe6fc1559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:42038/0 (socket says 172.21.15.28:42038) 2024-09-10T20:38:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6faffd640 1 -- 172.21.15.28:0/712136749 learned_addr learned my addr 172.21.15.28:0/712136749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6affff640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b00745c0 0x7fe6b006fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6affff640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b0136f20 con 0x7fe6b00708e0 2024-09-10T20:38:24.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6fb7fe640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6fc0cdde0 0x7fe6fc147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7f9e7e59c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e78037520 0x7f9e78185b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc0cadf0 con 0x7fe6fc0cdde0 2024-09-10T20:38:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc0caf70 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe6fc0cb0f0 con 0x7fe6fc0aedd0 2024-09-10T20:38:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.770+0000 7fd5a62b5640 1 -- 172.21.15.28:0/2940578788 wait complete. 2024-09-10T20:38:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fd5a62b5640 1 Processor -- start 2024-09-10T20:38:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6b4e53640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b00745c0 0x7fe6b006fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7f9e7e59c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e78037520 0x7f9e78185b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:53282/0 (socket says 172.21.15.28:53282) 2024-09-10T20:38:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7f9e7dd9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e780e3d80 0x7f9e78034470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fd5a62b5640 1 -- start start 2024-09-10T20:38:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7f9e7dd9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e780e3d80 0x7f9e78034470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:42046/0 (socket says 172.21.15.28:42046) 2024-09-10T20:38:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.772+0000 7fe6affff640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b00708e0 0x7fe6b009c750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe69800b450 tx=0x7fe69800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f9e7e59c640 1 -- 172.21.15.28:0/3619422729 learned_addr learned my addr 172.21.15.28:0/3619422729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e780367a0 con 0x7f9e780e4160 2024-09-10T20:38:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f9e7ed9d640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780e4160 0x7f9e78051c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6faffd640 1 -- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6fc0aedd0 msgr2=0x7fe6fc147350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:38:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6faffd640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6fc0aedd0 0x7fe6fc147350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6faffd640 1 -- 172.21.15.28:0/712136749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6fc0cdde0 msgr2=0x7fe6fc147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9e78036620 con 0x7f9e780e3d80 2024-09-10T20:38:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9e78036920 con 0x7f9e78037520 2024-09-10T20:38:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6faffd640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6fc0cdde0 0x7fe6fc147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6faffd640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6fc156610 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6fb7fe640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6fc0cdde0 0x7fe6fc147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:38:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f0a30262640 1 -- 172.21.15.28:0/3652760519 shutdown_connections 2024-09-10T20:38:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f0a30262640 1 --2- 172.21.15.28:0/3652760519 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a28155060 0x7f0a2806e4c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f0a30262640 1 --2- 172.21.15.28:0/3652760519 >> v2:172.21.15.159:3300/0 conn(0x7f0a2806ea00 0x7f0a280ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7f0a30262640 1 -- 172.21.15.28:0/3652760519 >> 172.21.15.28:0/3652760519 conn(0x7f0a280b15b0 msgr2=0x7f0a280b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f0a30262640 1 -- 172.21.15.28:0/3652760519 shutdown_connections 2024-09-10T20:38:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7fd5a62b5640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd598039b40 0x7fd59801e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6acff9640 1 -- 172.21.15.28:0/2610671370 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698015020 con 0x7fe6b00708e0 2024-09-10T20:38:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6acff9640 1 -- 172.21.15.28:0/2610671370 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6980119a0 con 0x7fe6b00708e0 2024-09-10T20:38:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6acff9640 1 -- 172.21.15.28:0/2610671370 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698011df0 con 0x7fe6b00708e0 2024-09-10T20:38:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b009ab60 con 0x7fe6b00708e0 2024-09-10T20:38:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.773+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b009b120 con 0x7fe6b00708e0 2024-09-10T20:38:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7fd5a62b5640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e7150 0x7fd59818a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7fe6faffd640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0ae9f0 0x7fe6fc1559b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00c500 tx=0x7fe6ec00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7fd5a52b3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd598039b40 0x7fd59801e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7fd5a52b3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd598039b40 0x7fd59801e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:42052/0 (socket says 172.21.15.28:42052) 2024-09-10T20:38:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f0a30262640 1 -- 172.21.15.28:0/3652760519 wait complete. 2024-09-10T20:38:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7fd5a52b3640 1 -- 172.21.15.28:0/4274710271 learned_addr learned my addr 172.21.15.28:0/4274710271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980e75c0 0x7fd5981861c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f9e7ed9d640 1 -- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e78037520 msgr2=0x7f9e78185b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7fd5a4ab2640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e7150 0x7fd59818a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f9e7ed9d640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e78037520 0x7f9e78185b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f9e7ed9d640 1 -- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e780e3d80 msgr2=0x7f9e78034470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f9e7ed9d640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e780e3d80 0x7f9e78034470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f9e7ed9d640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e780333d0 con 0x7f9e780e4160 2024-09-10T20:38:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.774+0000 7f9e7ed9d640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780e4160 0x7f9e78051c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000bfd0 tx=0x7f9e7000da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7f9e7e59c640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e78037520 0x7f9e78185b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7fd5a5ab4640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980e75c0 0x7fd5981861c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7f9e6effd640 1 -- 172.21.15.28:0/3619422729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70017070 con 0x7f9e780e4160 2024-09-10T20:38:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7f9e6effd640 1 -- 172.21.15.28:0/3619422729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e70005310 con 0x7f9e780e4160 2024-09-10T20:38:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd598054030 con 0x7fd5980e75c0 2024-09-10T20:38:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd598053eb0 con 0x7fd598039b40 2024-09-10T20:38:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.775+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd5980541b0 con 0x7fd5980e7150 2024-09-10T20:38:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b00726c0 con 0x7fe6b00708e0 2024-09-10T20:38:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7f0a30262640 1 Processor -- start 2024-09-10T20:38:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7f9e6effd640 1 -- 172.21.15.28:0/3619422729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e700047a0 con 0x7f9e780e4160 2024-09-10T20:38:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e780336b0 con 0x7f9e780e4160 2024-09-10T20:38:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7f9e7dd9b640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e780e3d80 0x7f9e78034470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7f0a30262640 1 -- start start 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7fe6acff9640 1 -- 172.21.15.28:0/2610671370 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe69801a400 con 0x7fe6b00708e0 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fe6acff9640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe690077f70 0x7fe69007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fe6acff9640 1 -- 172.21.15.28:0/2610671370 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe69809a110 con 0x7fe6b00708e0 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7fd5a5ab4640 1 -- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e7150 msgr2=0x7fd59818a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7fd5a5ab4640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e7150 0x7fd59818a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7fd5a5ab4640 1 -- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd598039b40 msgr2=0x7fd59801e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7fd5a5ab4640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd598039b40 0x7fd59801e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.776+0000 7fd5a5ab4640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd598186d30 con 0x7fd5980e75c0 2024-09-10T20:38:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e78033c70 con 0x7f9e780e4160 2024-09-10T20:38:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fe6af7fe640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe690077f70 0x7fe69007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fd5a5ab4640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980e75c0 0x7fd5981861c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5a006bcd0 tx=0x7fd5a00aed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fe6e37fe640 1 -- 172.21.15.28:0/712136749 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec016020 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fe6e37fe640 1 -- 172.21.15.28:0/712136749 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6ec012920 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fe6e37fe640 1 -- 172.21.15.28:0/712136749 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec012d30 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.777+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6fc14b680 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6fc14bc40 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a30262640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2806ea00 0x7f0a28153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a30262640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a28154c80 0x7f0a28153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a30262640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a28155060 0x7f0a280cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a30262640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a280ada30 con 0x7f0a2806ea00 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a30262640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0a280ad8b0 con 0x7f0a28155060 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a30262640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0a280adbb0 con 0x7f0a28154c80 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a2dffe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2806ea00 0x7f0a28153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a2dffe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2806ea00 0x7f0a28153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51228/0 (socket says 172.21.15.28:51228) 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f0a2dffe640 1 -- 172.21.15.28:0/3898647499 learned_addr learned my addr 172.21.15.28:0/3898647499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7fd595ffb640 1 -- 172.21.15.28:0/4274710271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00b3050 con 0x7fd5980e75c0 2024-09-10T20:38:24.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7fe6af7fe640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe690077f70 0x7fe69007a430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0033000 tx=0x7fe6a0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.778+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e78036eb0 con 0x7f9e780e4160 2024-09-10T20:38:24.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7f0a2dffe640 1 -- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a28154c80 msgr2=0x7f0a28153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:24.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7f0a2dffe640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a28154c80 0x7f0a28153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7f0a2dffe640 1 -- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a28155060 msgr2=0x7f0a280cd8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T20:38:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7f0a2dffe640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a28155060 0x7f0a280cd8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7f0a2dffe640 1 -- 172.21.15.28:0/3898647499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a280b4f30 con 0x7f0a2806ea00 2024-09-10T20:38:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd598187010 con 0x7fd5980e75c0 2024-09-10T20:38:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd598021720 con 0x7fd5980e75c0 2024-09-10T20:38:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.779+0000 7fd595ffb640 1 -- 172.21.15.28:0/4274710271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5a00b4040 con 0x7fd5980e75c0 2024-09-10T20:38:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.780+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6fc0766a0 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.780+0000 7fd595ffb640 1 -- 172.21.15.28:0/4274710271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00ba450 con 0x7fd5980e75c0 2024-09-10T20:38:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.780+0000 7f9e6effd640 1 -- 172.21.15.28:0/3619422729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9e70004940 con 0x7f9e780e4160 2024-09-10T20:38:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.780+0000 7f0a2dffe640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2806ea00 0x7f0a28153220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800c530 tx=0x7f0a1800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.780+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd598021230 con 0x7fd5980e75c0 2024-09-10T20:38:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.781+0000 7fe6acff9640 1 -- 172.21.15.28:0/2610671370 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe69801f020 con 0x7fe6b00708e0 2024-09-10T20:38:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.780+0000 7f9e6effd640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e60078040 0x7f9e6007a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.781+0000 7f9e6effd640 1 -- 172.21.15.28:0/3619422729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9e70099d50 con 0x7f9e780e4160 2024-09-10T20:38:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.781+0000 7f9e7e59c640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e60078040 0x7f9e6007a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.782+0000 7fd595ffb640 1 -- 172.21.15.28:0/4274710271 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd5a00ba5f0 con 0x7fd5980e75c0 2024-09-10T20:38:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.783+0000 7f9e7e59c640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e60078040 0x7f9e6007a500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e80052150 tx=0x7f9e800a8070 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.783+0000 7fd595ffb640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd584077f70 0x7fd58407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.783+0000 7fd595ffb640 1 -- 172.21.15.28:0/4274710271 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd5a01368d0 con 0x7fd5980e75c0 2024-09-10T20:38:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.783+0000 7fd5a52b3640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd584077f70 0x7fd58407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.783+0000 7fe6e37fe640 1 -- 172.21.15.28:0/712136749 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe6ec01b400 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.784+0000 7fe6e37fe640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe6e8078040 0x7fe6e807a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.784+0000 7fd5a52b3640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd584077f70 0x7fd58407a430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd59817d310 tx=0x7fd59c009260 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.784+0000 7fe6e37fe640 1 -- 172.21.15.28:0/712136749 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe6ec09a2b0 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.784+0000 7fe6fa7fc640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe6e8078040 0x7fe6e807a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.785+0000 7fe6fa7fc640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe6e8078040 0x7fe6e807a500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0005fd0 tx=0x7fe6f0005c70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.786+0000 7fe6e37fe640 1 -- 172.21.15.28:0/712136749 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe6ec020030 con 0x7fe6fc0ae9f0 2024-09-10T20:38:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.787+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/3898647499 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a180052d0 con 0x7f0a2806ea00 2024-09-10T20:38:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.787+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a280b51f0 con 0x7f0a2806ea00 2024-09-10T20:38:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.787+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a28153f70 con 0x7f0a2806ea00 2024-09-10T20:38:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.787+0000 7f9e6effd640 1 -- 172.21.15.28:0/3619422729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9e700628c0 con 0x7f9e780e4160 2024-09-10T20:38:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.789+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a280b53c0 con 0x7f0a2806ea00 2024-09-10T20:38:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.793+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/3898647499 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a180011c0 con 0x7f0a2806ea00 2024-09-10T20:38:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.794+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/3898647499 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18012850 con 0x7f0a2806ea00 2024-09-10T20:38:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.793+0000 7fd595ffb640 1 -- 172.21.15.28:0/4274710271 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd5a00ff450 con 0x7fd5980e75c0 2024-09-10T20:38:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.795+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/3898647499 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0a180129f0 con 0x7f0a2806ea00 2024-09-10T20:38:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.796+0000 7f0a1e7fc640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0a0c077fa0 0x7f0a0c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.796+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/3898647499 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0a1809a360 con 0x7f0a2806ea00 2024-09-10T20:38:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.801+0000 7f0a2d7fd640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0a0c077fa0 0x7f0a0c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.802+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/3898647499 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0a18005470 con 0x7f0a2806ea00 2024-09-10T20:38:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:24.802+0000 7f0a2d7fd640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0a0c077fa0 0x7f0a0c07a460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a20035000 tx=0x7f0a20004a40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.143+0000 7fc9afdf3640 1 -- 172.21.15.28:0/3013210319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc990005580 con 0x7fc9a80cae80 2024-09-10T20:38:25.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.144+0000 7fc99e7fc640 1 -- 172.21.15.28:0/3013210319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc998087480 con 0x7fc9a80cae80 2024-09-10T20:38:25.147 INFO:teuthology.orchestra.run.smithi028.stdout:64424509447 2024-09-10T20:38:25.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.147+0000 7fc98bfff640 1 -- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc994078190 msgr2=0x7fc99407a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.147+0000 7fc98bfff640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc994078190 0x7fc99407a650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009670 tx=0x7fc9a4006bd0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 -- 172.21.15.28:0/3013210319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 msgr2=0x7fc9a8148200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 0x7fc9a8148200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc9a806efc0 tx=0x7fc998003940 comp rx=0 tx=0).stop 2024-09-10T20:38:25.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 -- 172.21.15.28:0/3013210319 shutdown_connections 2024-09-10T20:38:25.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc994078190 0x7fc99407a650 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a80cae80 0x7fc9a8148200 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc9a806eb10 0x7fc9a8147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 --2- 172.21.15.28:0/3013210319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc9a806e730 0x7fc9a815aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7f3b7e39b640 1 -- 172.21.15.28:0/1531061254 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3b58005580 con 0x7f3b7806ff80 2024-09-10T20:38:25.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.148+0000 7fc98bfff640 1 -- 172.21.15.28:0/3013210319 >> 172.21.15.28:0/3013210319 conn(0x7fc9a80b15b0 msgr2=0x7fc9a815bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.149+0000 7fc98bfff640 1 -- 172.21.15.28:0/3013210319 shutdown_connections 2024-09-10T20:38:25.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.149+0000 7fc98bfff640 1 -- 172.21.15.28:0/3013210319 wait complete. 2024-09-10T20:38:25.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.149+0000 7f3b74ff9640 1 -- 172.21.15.28:0/1531061254 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3b781f90b0 con 0x7f3b7806ff80 2024-09-10T20:38:25.154 INFO:teuthology.orchestra.run.smithi028.stdout:60129542151 2024-09-10T20:38:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 -- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3b5c078370 msgr2=0x7f3b5c07a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3b5c078370 0x7f3b5c07a830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b70000ba0 tx=0x7f3b70009040 comp rx=0 tx=0).stop 2024-09-10T20:38:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 -- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 msgr2=0x7f3b780c8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 0x7f3b780c8ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3b64005fd0 tx=0x7f3b64004a50 comp rx=0 tx=0).stop 2024-09-10T20:38:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 -- 172.21.15.28:0/1531061254 shutdown_connections 2024-09-10T20:38:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3b5c078370 0x7f3b5c07a830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7815ef80 0x7f3b780c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.153+0000 7f3b627fc640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b780aee50 0x7f3b781f8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.154+0000 7f3b627fc640 1 --2- 172.21.15.28:0/1531061254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b7806ff80 0x7f3b780c8ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.154+0000 7f3b627fc640 1 -- 172.21.15.28:0/1531061254 >> 172.21.15.28:0/1531061254 conn(0x7f3b780cfba0 msgr2=0x7f3b780cd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.154+0000 7f3b627fc640 1 -- 172.21.15.28:0/1531061254 shutdown_connections 2024-09-10T20:38:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.154+0000 7f3b627fc640 1 -- 172.21.15.28:0/1531061254 wait complete. 2024-09-10T20:38:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.160+0000 7f93167de640 1 -- 172.21.15.28:0/2622951895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f93100cab10 con 0x7f93100d04b0 2024-09-10T20:38:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.161+0000 7f9305ffb640 1 -- 172.21.15.28:0/2622951895 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f93100cab10 con 0x7f93100d04b0 2024-09-10T20:38:25.165 INFO:teuthology.orchestra.run.smithi028.stdout:60129542150 2024-09-10T20:38:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 -- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f92f4078040 msgr2=0x7f92f407a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f92f4078040 0x7f92f407a500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9310155c30 tx=0x7f930c009040 comp rx=0 tx=0).stop 2024-09-10T20:38:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 -- 172.21.15.28:0/2622951895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93100d04b0 msgr2=0x7f931014f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93100d04b0 0x7f931014f7f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f930000e9f0 tx=0x7f930000eeb0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 -- 172.21.15.28:0/2622951895 shutdown_connections 2024-09-10T20:38:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f92f4078040 0x7f92f407a500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f931015e2e0 0x7f9310147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93100d04b0 0x7f931014f7f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.165+0000 7f92fb7fe640 1 --2- 172.21.15.28:0/2622951895 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93100ae960 0x7f93101609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.166+0000 7f92fb7fe640 1 -- 172.21.15.28:0/2622951895 >> 172.21.15.28:0/2622951895 conn(0x7f93100cf990 msgr2=0x7f931006f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.166+0000 7f92fb7fe640 1 -- 172.21.15.28:0/2622951895 shutdown_connections 2024-09-10T20:38:25.171 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.0 2024-09-10T20:38:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.166+0000 7f92fb7fe640 1 -- 172.21.15.28:0/2622951895 wait complete. 2024-09-10T20:38:25.171 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.175 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2024-09-10T20:38:25.175 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.4 2024-09-10T20:38:25.186 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.186+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe6b0154790 con 0x7fe6b00708e0 2024-09-10T20:38:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.186+0000 7fe6acff9640 1 -- 172.21.15.28:0/2610671370 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe69800d040 con 0x7fe6b00708e0 2024-09-10T20:38:25.190 INFO:teuthology.orchestra.run.smithi028.stdout:64424509448 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe690077f70 msgr2=0x7fe69007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe690077f70 0x7fe69007a430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0033000 tx=0x7fe6a0004730 comp rx=0 tx=0).stop 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b00708e0 msgr2=0x7fe6b009c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b00708e0 0x7fe6b009c750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe69800b450 tx=0x7fe69800b910 comp rx=0 tx=0).stop 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 shutdown_connections 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe690077f70 0x7fe69007a430 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6b00745c0 0x7fe6b006fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b0070cc0 0x7fe6b006f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 --2- 172.21.15.28:0/2610671370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b00708e0 0x7fe6b009c750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 >> 172.21.15.28:0/2610671370 conn(0x7fe6b00b18a0 msgr2=0x7fe6b0158080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 shutdown_connections 2024-09-10T20:38:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.190+0000 7fe6b68b6640 1 -- 172.21.15.28:0/2610671370 wait complete. 2024-09-10T20:38:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.191+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe6fc0b0710 con 0x7fe6fc0ae9f0 2024-09-10T20:38:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.192+0000 7fe6e37fe640 1 -- 172.21.15.28:0/712136749 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe6fc0b0710 con 0x7fe6fc0ae9f0 2024-09-10T20:38:25.196 INFO:teuthology.orchestra.run.smithi028.stdout:60129542150 2024-09-10T20:38:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe6e8078040 msgr2=0x7fe6e807a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe6e8078040 0x7fe6e807a500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0005fd0 tx=0x7fe6f0005c70 comp rx=0 tx=0).stop 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0ae9f0 msgr2=0x7fe6fc1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0ae9f0 0x7fe6fc1559b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00c500 tx=0x7fe6ec00c9c0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 shutdown_connections 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe6e8078040 0x7fe6e807a500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6fc0cdde0 0x7fe6fc147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6fc0aedd0 0x7fe6fc147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 --2- 172.21.15.28:0/712136749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6fc0ae9f0 0x7fe6fc1559b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.194+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 >> 172.21.15.28:0/712136749 conn(0x7fe6fc0cf990 msgr2=0x7fe6fc06eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.195+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 shutdown_connections 2024-09-10T20:38:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.195+0000 7fe6fbfff640 1 -- 172.21.15.28:0/712136749 wait complete. 2024-09-10T20:38:25.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.195+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9e7800c350 con 0x7f9e780e4160 2024-09-10T20:38:25.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.195+0000 7f9e6effd640 1 -- 172.21.15.28:0/3619422729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9e700621c0 con 0x7f9e780e4160 2024-09-10T20:38:25.200 INFO:teuthology.orchestra.run.smithi028.stdout:64424509447 2024-09-10T20:38:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e60078040 msgr2=0x7f9e6007a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e60078040 0x7f9e6007a500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e80052150 tx=0x7f9e800a8070 comp rx=0 tx=0).stop 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780e4160 msgr2=0x7f9e78051c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780e4160 0x7f9e78051c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000bfd0 tx=0x7f9e7000da70 comp rx=0 tx=0).stop 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 shutdown_connections 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e60078040 0x7f9e6007a500 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780e4160 0x7f9e78051c80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e780e3d80 0x7f9e78034470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.197+0000 7f9e7f59e640 1 --2- 172.21.15.28:0/3619422729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e78037520 0x7f9e78185b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.198+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 >> 172.21.15.28:0/3619422729 conn(0x7f9e78056710 msgr2=0x7f9e7800d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.198+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 shutdown_connections 2024-09-10T20:38:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.198+0000 7f9e7f59e640 1 -- 172.21.15.28:0/3619422729 wait complete. 2024-09-10T20:38:25.210 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.3 2024-09-10T20:38:25.210 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.5 2024-09-10T20:38:25.212 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.215 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.2 2024-09-10T20:38:25.215 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.228+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd5981871e0 con 0x7fd5980e75c0 2024-09-10T20:38:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.228+0000 7fd595ffb640 1 -- 172.21.15.28:0/4274710271 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd5981871e0 con 0x7fd5980e75c0 2024-09-10T20:38:25.232 INFO:teuthology.orchestra.run.smithi028.stdout:64424509447 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd584077f70 msgr2=0x7fd58407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd584077f70 0x7fd58407a430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd59817d310 tx=0x7fd59c009260 comp rx=0 tx=0).stop 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980e75c0 msgr2=0x7fd5981861c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980e75c0 0x7fd5981861c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5a006bcd0 tx=0x7fd5a00aed10 comp rx=0 tx=0).stop 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 shutdown_connections 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd584077f70 0x7fd58407a430 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980e75c0 0x7fd5981861c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5980e7150 0x7fd59818a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 --2- 172.21.15.28:0/4274710271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd598039b40 0x7fd59801e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 >> 172.21.15.28:0/4274710271 conn(0x7fd59801e000 msgr2=0x7fd5980e5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 shutdown_connections 2024-09-10T20:38:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.232+0000 7fd5a62b5640 1 -- 172.21.15.28:0/4274710271 wait complete. 2024-09-10T20:38:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.234+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0a280ae0d0 con 0x7f0a2806ea00 2024-09-10T20:38:25.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.234+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/3898647499 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0a18062ed0 con 0x7f0a2806ea00 2024-09-10T20:38:25.238 INFO:teuthology.orchestra.run.smithi028.stdout:60129542152 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.236+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0a0c077fa0 msgr2=0x7f0a0c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.236+0000 7f0a30262640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0a0c077fa0 0x7f0a0c07a460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a20035000 tx=0x7f0a20004a40 comp rx=0 tx=0).stop 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2806ea00 msgr2=0x7f0a28153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2806ea00 0x7f0a28153220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800c530 tx=0x7f0a1800c9f0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 shutdown_connections 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0a0c077fa0 0x7f0a0c07a460 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0a28155060 0x7f0a280cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0a28154c80 0x7f0a28153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 --2- 172.21.15.28:0/3898647499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2806ea00 0x7f0a28153220 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 >> 172.21.15.28:0/3898647499 conn(0x7f0a280b15b0 msgr2=0x7f0a280b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 shutdown_connections 2024-09-10T20:38:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.237+0000 7f0a30262640 1 -- 172.21.15.28:0/3898647499 wait complete. 2024-09-10T20:38:25.248 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.1 2024-09-10T20:38:25.248 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.6 2024-09-10T20:38:25.252 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:25.252 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:38:25.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 -- 172.21.15.28:0/2004802994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2960 msgr2=0x7f83400d1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 --2- 172.21.15.28:0/2004802994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2960 0x7f83400d1d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8330005fd0 tx=0x7f833002aa90 comp rx=0 tx=0).stop 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 -- 172.21.15.28:0/2004802994 shutdown_connections 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 --2- 172.21.15.28:0/2004802994 >> v2:172.21.15.159:3300/0 conn(0x7f83400d1730 0x7f8340162780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 --2- 172.21.15.28:0/2004802994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2960 0x7f83400d1d00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 -- 172.21.15.28:0/2004802994 >> 172.21.15.28:0/2004802994 conn(0x7f83400d4130 msgr2=0x7f83400d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 -- 172.21.15.28:0/2004802994 shutdown_connections 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.394+0000 7f83458cf640 1 -- 172.21.15.28:0/2004802994 wait complete. 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.395+0000 7f83458cf640 1 Processor -- start 2024-09-10T20:38:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.396+0000 7f83458cf640 1 -- start start 2024-09-10T20:38:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.396+0000 7f83458cf640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83400d1730 0x7f83401f8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.396+0000 7f83458cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83400d2960 0x7f83401624a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.396+0000 7f83458cf640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2d40 0x7f83400b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.396+0000 7f83458cf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83400cf3e0 con 0x7f83400d2960 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.396+0000 7f83458cf640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f834015f5c0 con 0x7f83400d1730 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.396+0000 7f83458cf640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f83400cf560 con 0x7f83400d2d40 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83400d2960 0x7f83401624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83400d2960 0x7f83401624a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51240/0 (socket says 172.21.15.28:51240) 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 -- 172.21.15.28:0/1375383296 learned_addr learned my addr 172.21.15.28:0/1375383296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833f7fe640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2d40 0x7f83400b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833effd640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83400d1730 0x7f83401f8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 -- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2d40 msgr2=0x7f83400b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2d40 0x7f83400b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 -- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83400d1730 msgr2=0x7f83401f8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83400d1730 0x7f83401f8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 -- 172.21.15.28:0/1375383296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83401f9200 con 0x7f83400d2960 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f833e7fc640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83400d2960 0x7f83401624a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f832800c3a0 tx=0x7f832800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f8337fff640 1 -- 172.21.15.28:0/1375383296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328017070 con 0x7f83400d2960 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f8337fff640 1 -- 172.21.15.28:0/1375383296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83280053c0 con 0x7f83400d2960 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83400b8a60 con 0x7f83400d2960 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f8337fff640 1 -- 172.21.15.28:0/1375383296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328003850 con 0x7f83400d2960 2024-09-10T20:38:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.397+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83400b7800 con 0x7f83400d2960 2024-09-10T20:38:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.398+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83400cfa80 con 0x7f83400d2960 2024-09-10T20:38:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.399+0000 7f8337fff640 1 -- 172.21.15.28:0/1375383296 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f83280039f0 con 0x7f83400d2960 2024-09-10T20:38:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.399+0000 7f8337fff640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8320077f70 0x7f832007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.399+0000 7f8337fff640 1 -- 172.21.15.28:0/1375383296 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8328099440 con 0x7f83400d2960 2024-09-10T20:38:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.400+0000 7f833effd640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8320077f70 0x7f832007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.400+0000 7f833effd640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8320077f70 0x7f832007a430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8330035000 tx=0x7f8330039040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:25.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.403+0000 7f8337fff640 1 -- 172.21.15.28:0/1375383296 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f83401f9200 con 0x7f83400d2960 2024-09-10T20:38:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.584+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8340069680 con 0x7f8320077f70 2024-09-10T20:38:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.589+0000 7f8337fff640 1 -- 172.21.15.28:0/1375383296 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251822 (secure 0 0 0) 0x7f83300847c0 con 0x7f8320077f70 2024-09-10T20:38:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8320077f70 msgr2=0x7f832007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8320077f70 0x7f832007a430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8330035000 tx=0x7f8330039040 comp rx=0 tx=0).stop 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83400d2960 msgr2=0x7f83401624a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83400d2960 0x7f83401624a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f832800c3a0 tx=0x7f832800c860 comp rx=0 tx=0).stop 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 shutdown_connections 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8320077f70 0x7f832007a430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83400d2d40 0x7f83400b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83400d2960 0x7f83401624a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 --2- 172.21.15.28:0/1375383296 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83400d1730 0x7f83401f8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 >> 172.21.15.28:0/1375383296 conn(0x7f83400d4130 msgr2=0x7f83400b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 shutdown_connections 2024-09-10T20:38:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.593+0000 7f83458cf640 1 -- 172.21.15.28:0/1375383296 wait complete. 2024-09-10T20:38:25.607 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-10T20:38:23.978833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235740,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738980,"statfs":{"total":773094113280,"available":772852715520,"internally_reserved":0,"allocated":2924544,"data_stored":1765772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.580611"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408702+0000","last_change":"2024-09-10T20:38:12.400371+0000","last_active":"2024-09-10T20:38:13.408702+0000","last_peered":"2024-09-10T20:38:13.408702+0000","last_clean":"2024-09-10T20:38:13.408702+0000","last_became_active":"2024-09-10T20:38:12.400049+0000","last_became_peered":"2024-09-10T20:38:12.400049+0000","last_unstale":"2024-09-10T20:38:13.408702+0000","last_undegraded":"2024-09-10T20:38:13.408702+0000","last_fullsized":"2024-09-10T20:38:13.408702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:47.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400675+0000","last_change":"2024-09-10T20:38:12.413418+0000","last_active":"2024-09-10T20:38:13.400675+0000","last_peered":"2024-09-10T20:38:13.400675+0000","last_clean":"2024-09-10T20:38:13.400675+0000","last_became_active":"2024-09-10T20:38:12.413063+0000","last_became_peered":"2024-09-10T20:38:12.413063+0000","last_unstale":"2024-09-10T20:38:13.400675+0000","last_undegraded":"2024-09-10T20:38:13.400675+0000","last_fullsized":"2024-09-10T20:38:13.400675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:12.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401706+0000","last_change":"2024-09-10T20:38:12.404482+0000","last_active":"2024-09-10T20:38:13.401706+0000","last_peered":"2024-09-10T20:38:13.401706+0000","last_clean":"2024-09-10T20:38:13.401706+0000","last_became_active":"2024-09-10T20:38:12.404181+0000","last_became_peered":"2024-09-10T20:38:12.404181+0000","last_unstale":"2024-09-10T20:38:13.401706+0000","last_undegraded":"2024-09-10T20:38:13.401706+0000","last_fullsized":"2024-09-10T20:38:13.401706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:02.847206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343826+0000","last_change":"2024-09-10T20:38:12.399863+0000","last_active":"2024-09-10T20:38:14.343826+0000","last_peered":"2024-09-10T20:38:14.343826+0000","last_clean":"2024-09-10T20:38:14.343826+0000","last_became_active":"2024-09-10T20:38:12.399541+0000","last_became_peered":"2024-09-10T20:38:12.399541+0000","last_unstale":"2024-09-10T20:38:14.343826+0000","last_undegraded":"2024-09-10T20:38:14.343826+0000","last_fullsized":"2024-09-10T20:38:14.343826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:45.518334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408418+0000","last_change":"2024-09-10T20:38:12.403977+0000","last_active":"2024-09-10T20:38:13.408418+0000","last_peered":"2024-09-10T20:38:13.408418+0000","last_clean":"2024-09-10T20:38:13.408418+0000","last_became_active":"2024-09-10T20:38:12.403575+0000","last_became_peered":"2024-09-10T20:38:12.403575+0000","last_unstale":"2024-09-10T20:38:13.408418+0000","last_undegraded":"2024-09-10T20:38:13.408418+0000","last_fullsized":"2024-09-10T20:38:13.408418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:37.571212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344323+0000","last_change":"2024-09-10T20:38:12.399990+0000","last_active":"2024-09-10T20:38:14.344323+0000","last_peered":"2024-09-10T20:38:14.344323+0000","last_clean":"2024-09-10T20:38:14.344323+0000","last_became_active":"2024-09-10T20:38:12.399633+0000","last_became_peered":"2024-09-10T20:38:12.399633+0000","last_unstale":"2024-09-10T20:38:14.344323+0000","last_undegraded":"2024-09-10T20:38:14.344323+0000","last_fullsized":"2024-09-10T20:38:14.344323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:01:29.734867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407809+0000","last_change":"2024-09-10T20:38:12.401934+0000","last_active":"2024-09-10T20:38:13.407809+0000","last_peered":"2024-09-10T20:38:13.407809+0000","last_clean":"2024-09-10T20:38:13.407809+0000","last_became_active":"2024-09-10T20:38:12.401547+0000","last_became_peered":"2024-09-10T20:38:12.401547+0000","last_unstale":"2024-09-10T20:38:13.407809+0000","last_undegraded":"2024-09-10T20:38:13.407809+0000","last_fullsized":"2024-09-10T20:38:13.407809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:21.949782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401300+0000","last_change":"2024-09-10T20:38:12.403203+0000","last_active":"2024-09-10T20:38:13.401300+0000","last_peered":"2024-09-10T20:38:13.401300+0000","last_clean":"2024-09-10T20:38:13.401300+0000","last_became_active":"2024-09-10T20:38:12.402272+0000","last_became_peered":"2024-09-10T20:38:12.402272+0000","last_unstale":"2024-09-10T20:38:13.401300+0000","last_undegraded":"2024-09-10T20:38:13.401300+0000","last_fullsized":"2024-09-10T20:38:13.401300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.399062+0000","last_change":"2024-09-10T20:38:12.420252+0000","last_active":"2024-09-10T20:38:13.399062+0000","last_peered":"2024-09-10T20:38:13.399062+0000","last_clean":"2024-09-10T20:38:13.399062+0000","last_became_active":"2024-09-10T20:38:12.419848+0000","last_became_peered":"2024-09-10T20:38:12.419848+0000","last_unstale":"2024-09-10T20:38:13.399062+0000","last_undegraded":"2024-09-10T20:38:13.399062+0000","last_fullsized":"2024-09-10T20:38:13.399062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:13.317965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408005+0000","last_change":"2024-09-10T20:38:12.398378+0000","last_active":"2024-09-10T20:38:13.408005+0000","last_peered":"2024-09-10T20:38:13.408005+0000","last_clean":"2024-09-10T20:38:13.408005+0000","last_became_active":"2024-09-10T20:38:12.398068+0000","last_became_peered":"2024-09-10T20:38:12.398068+0000","last_unstale":"2024-09-10T20:38:13.408005+0000","last_undegraded":"2024-09-10T20:38:13.408005+0000","last_fullsized":"2024-09-10T20:38:13.408005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408341+0000","last_change":"2024-09-10T20:38:12.401136+0000","last_active":"2024-09-10T20:38:13.408341+0000","last_peered":"2024-09-10T20:38:13.408341+0000","last_clean":"2024-09-10T20:38:13.408341+0000","last_became_active":"2024-09-10T20:38:12.400835+0000","last_became_peered":"2024-09-10T20:38:12.400835+0000","last_unstale":"2024-09-10T20:38:13.408341+0000","last_undegraded":"2024-09-10T20:38:13.408341+0000","last_fullsized":"2024-09-10T20:38:13.408341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:37:58.944607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408006+0000","last_change":"2024-09-10T20:38:12.417280+0000","last_active":"2024-09-10T20:38:13.408006+0000","last_peered":"2024-09-10T20:38:13.408006+0000","last_clean":"2024-09-10T20:38:13.408006+0000","last_became_active":"2024-09-10T20:38:12.416991+0000","last_became_peered":"2024-09-10T20:38:12.416991+0000","last_unstale":"2024-09-10T20:38:13.408006+0000","last_undegraded":"2024-09-10T20:38:13.408006+0000","last_fullsized":"2024-09-10T20:38:13.408006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:36.789848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343562+0000","last_change":"2024-09-10T20:38:12.415360+0000","last_active":"2024-09-10T20:38:14.343562+0000","last_peered":"2024-09-10T20:38:14.343562+0000","last_clean":"2024-09-10T20:38:14.343562+0000","last_became_active":"2024-09-10T20:38:12.415094+0000","last_became_peered":"2024-09-10T20:38:12.415094+0000","last_unstale":"2024-09-10T20:38:14.343562+0000","last_undegraded":"2024-09-10T20:38:14.343562+0000","last_fullsized":"2024-09-10T20:38:14.343562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:36.623506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343324+0000","last_change":"2024-09-10T20:38:12.411600+0000","last_active":"2024-09-10T20:38:14.343324+0000","last_peered":"2024-09-10T20:38:14.343324+0000","last_clean":"2024-09-10T20:38:14.343324+0000","last_became_active":"2024-09-10T20:38:12.411328+0000","last_became_peered":"2024-09-10T20:38:12.411328+0000","last_unstale":"2024-09-10T20:38:14.343324+0000","last_undegraded":"2024-09-10T20:38:14.343324+0000","last_fullsized":"2024-09-10T20:38:14.343324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:30.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400233+0000","last_change":"2024-09-10T20:38:12.404816+0000","last_active":"2024-09-10T20:38:13.400233+0000","last_peered":"2024-09-10T20:38:13.400233+0000","last_clean":"2024-09-10T20:38:13.400233+0000","last_became_active":"2024-09-10T20:38:12.404568+0000","last_became_peered":"2024-09-10T20:38:12.404568+0000","last_unstale":"2024-09-10T20:38:13.400233+0000","last_undegraded":"2024-09-10T20:38:13.400233+0000","last_fullsized":"2024-09-10T20:38:13.400233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:05:49.714066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343559+0000","last_change":"2024-09-10T20:38:12.404745+0000","last_active":"2024-09-10T20:38:14.343559+0000","last_peered":"2024-09-10T20:38:14.343559+0000","last_clean":"2024-09-10T20:38:14.343559+0000","last_became_active":"2024-09-10T20:38:12.404429+0000","last_became_peered":"2024-09-10T20:38:12.404429+0000","last_unstale":"2024-09-10T20:38:14.343559+0000","last_undegraded":"2024-09-10T20:38:14.343559+0000","last_fullsized":"2024-09-10T20:38:14.343559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344641+0000","last_change":"2024-09-10T20:38:12.404923+0000","last_active":"2024-09-10T20:38:14.344641+0000","last_peered":"2024-09-10T20:38:14.344641+0000","last_clean":"2024-09-10T20:38:14.344641+0000","last_became_active":"2024-09-10T20:38:12.404604+0000","last_became_peered":"2024-09-10T20:38:12.404604+0000","last_unstale":"2024-09-10T20:38:14.344641+0000","last_undegraded":"2024-09-10T20:38:14.344641+0000","last_fullsized":"2024-09-10T20:38:14.344641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:27.005436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633628+0000","last_change":"2024-09-10T20:38:12.407265+0000","last_active":"2024-09-10T20:38:13.633628+0000","last_peered":"2024-09-10T20:38:13.633628+0000","last_clean":"2024-09-10T20:38:13.633628+0000","last_became_active":"2024-09-10T20:38:12.406927+0000","last_became_peered":"2024-09-10T20:38:12.406927+0000","last_unstale":"2024-09-10T20:38:13.633628+0000","last_undegraded":"2024-09-10T20:38:13.633628+0000","last_fullsized":"2024-09-10T20:38:13.633628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:14.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344605+0000","last_change":"2024-09-10T20:38:12.411419+0000","last_active":"2024-09-10T20:38:14.344605+0000","last_peered":"2024-09-10T20:38:14.344605+0000","last_clean":"2024-09-10T20:38:14.344605+0000","last_became_active":"2024-09-10T20:38:12.411088+0000","last_became_peered":"2024-09-10T20:38:12.411088+0000","last_unstale":"2024-09-10T20:38:14.344605+0000","last_undegraded":"2024-09-10T20:38:14.344605+0000","last_fullsized":"2024-09-10T20:38:14.344605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:56.228254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408072+0000","last_change":"2024-09-10T20:38:12.416514+0000","last_active":"2024-09-10T20:38:13.408072+0000","last_peered":"2024-09-10T20:38:13.408072+0000","last_clean":"2024-09-10T20:38:13.408072+0000","last_became_active":"2024-09-10T20:38:12.416216+0000","last_became_peered":"2024-09-10T20:38:12.416216+0000","last_unstale":"2024-09-10T20:38:13.408072+0000","last_undegraded":"2024-09-10T20:38:13.408072+0000","last_fullsized":"2024-09-10T20:38:13.408072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:26.545836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400878+0000","last_change":"2024-09-10T20:38:12.407874+0000","last_active":"2024-09-10T20:38:13.400878+0000","last_peered":"2024-09-10T20:38:13.400878+0000","last_clean":"2024-09-10T20:38:13.400878+0000","last_became_active":"2024-09-10T20:38:12.407623+0000","last_became_peered":"2024-09-10T20:38:12.407623+0000","last_unstale":"2024-09-10T20:38:13.400878+0000","last_undegraded":"2024-09-10T20:38:13.400878+0000","last_fullsized":"2024-09-10T20:38:13.400878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:56.990230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398460+0000","last_change":"2024-09-10T20:38:12.414007+0000","last_active":"2024-09-10T20:38:13.398460+0000","last_peered":"2024-09-10T20:38:13.398460+0000","last_clean":"2024-09-10T20:38:13.398460+0000","last_became_active":"2024-09-10T20:38:12.413744+0000","last_became_peered":"2024-09-10T20:38:12.413744+0000","last_unstale":"2024-09-10T20:38:13.398460+0000","last_undegraded":"2024-09-10T20:38:13.398460+0000","last_fullsized":"2024-09-10T20:38:13.398460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.915632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400518+0000","last_change":"2024-09-10T20:38:12.408020+0000","last_active":"2024-09-10T20:38:13.400518+0000","last_peered":"2024-09-10T20:38:13.400518+0000","last_clean":"2024-09-10T20:38:13.400518+0000","last_became_active":"2024-09-10T20:38:12.407757+0000","last_became_peered":"2024-09-10T20:38:12.407757+0000","last_unstale":"2024-09-10T20:38:13.400518+0000","last_undegraded":"2024-09-10T20:38:13.400518+0000","last_fullsized":"2024-09-10T20:38:13.400518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:32.513086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343533+0000","last_change":"2024-09-10T20:38:12.407235+0000","last_active":"2024-09-10T20:38:14.343533+0000","last_peered":"2024-09-10T20:38:14.343533+0000","last_clean":"2024-09-10T20:38:14.343533+0000","last_became_active":"2024-09-10T20:38:12.406414+0000","last_became_peered":"2024-09-10T20:38:12.406414+0000","last_unstale":"2024-09-10T20:38:14.343533+0000","last_undegraded":"2024-09-10T20:38:14.343533+0000","last_fullsized":"2024-09-10T20:38:14.343533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:37.571212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401654+0000","last_change":"2024-09-10T20:38:12.416773+0000","last_active":"2024-09-10T20:38:13.401654+0000","last_peered":"2024-09-10T20:38:13.401654+0000","last_clean":"2024-09-10T20:38:13.401654+0000","last_became_active":"2024-09-10T20:38:12.416498+0000","last_became_peered":"2024-09-10T20:38:12.416498+0000","last_unstale":"2024-09-10T20:38:13.401654+0000","last_undegraded":"2024-09-10T20:38:13.401654+0000","last_fullsized":"2024-09-10T20:38:13.401654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:42.492324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344477+0000","last_change":"2024-09-10T20:38:12.407692+0000","last_active":"2024-09-10T20:38:14.344477+0000","last_peered":"2024-09-10T20:38:14.344477+0000","last_clean":"2024-09-10T20:38:14.344477+0000","last_became_active":"2024-09-10T20:38:12.406817+0000","last_became_peered":"2024-09-10T20:38:12.406817+0000","last_unstale":"2024-09-10T20:38:14.344477+0000","last_undegraded":"2024-09-10T20:38:14.344477+0000","last_fullsized":"2024-09-10T20:38:14.344477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:13.317965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343671+0000","last_change":"2024-09-10T20:38:12.406673+0000","last_active":"2024-09-10T20:38:14.343671+0000","last_peered":"2024-09-10T20:38:14.343671+0000","last_clean":"2024-09-10T20:38:14.343671+0000","last_became_active":"2024-09-10T20:38:12.406401+0000","last_became_peered":"2024-09-10T20:38:12.406401+0000","last_unstale":"2024-09-10T20:38:14.343671+0000","last_undegraded":"2024-09-10T20:38:14.343671+0000","last_fullsized":"2024-09-10T20:38:14.343671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:31.845414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343502+0000","last_change":"2024-09-10T20:38:12.414290+0000","last_active":"2024-09-10T20:38:14.343502+0000","last_peered":"2024-09-10T20:38:14.343502+0000","last_clean":"2024-09-10T20:38:14.343502+0000","last_became_active":"2024-09-10T20:38:12.413712+0000","last_became_peered":"2024-09-10T20:38:12.413712+0000","last_unstale":"2024-09-10T20:38:14.343502+0000","last_undegraded":"2024-09-10T20:38:14.343502+0000","last_fullsized":"2024-09-10T20:38:14.343502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:48.033101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400477+0000","last_change":"2024-09-10T20:38:12.406921+0000","last_active":"2024-09-10T20:38:13.400477+0000","last_peered":"2024-09-10T20:38:13.400477+0000","last_clean":"2024-09-10T20:38:13.400477+0000","last_became_active":"2024-09-10T20:38:12.406642+0000","last_became_peered":"2024-09-10T20:38:12.406642+0000","last_unstale":"2024-09-10T20:38:13.400477+0000","last_undegraded":"2024-09-10T20:38:13.400477+0000","last_fullsized":"2024-09-10T20:38:13.400477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:02.877370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343808+0000","last_change":"2024-09-10T20:38:12.406428+0000","last_active":"2024-09-10T20:38:14.343808+0000","last_peered":"2024-09-10T20:38:14.343808+0000","last_clean":"2024-09-10T20:38:14.343808+0000","last_became_active":"2024-09-10T20:38:12.405117+0000","last_became_peered":"2024-09-10T20:38:12.405117+0000","last_unstale":"2024-09-10T20:38:14.343808+0000","last_undegraded":"2024-09-10T20:38:14.343808+0000","last_fullsized":"2024-09-10T20:38:14.343808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:28.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633368+0000","last_change":"2024-09-10T20:38:12.415776+0000","last_active":"2024-09-10T20:38:13.633368+0000","last_peered":"2024-09-10T20:38:13.633368+0000","last_clean":"2024-09-10T20:38:13.633368+0000","last_became_active":"2024-09-10T20:38:12.415480+0000","last_became_peered":"2024-09-10T20:38:12.415480+0000","last_unstale":"2024-09-10T20:38:13.633368+0000","last_undegraded":"2024-09-10T20:38:13.633368+0000","last_fullsized":"2024-09-10T20:38:13.633368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:13.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343528+0000","last_change":"2024-09-10T20:38:12.405334+0000","last_active":"2024-09-10T20:38:14.343528+0000","last_peered":"2024-09-10T20:38:14.343528+0000","last_clean":"2024-09-10T20:38:14.343528+0000","last_became_active":"2024-09-10T20:38:12.405033+0000","last_became_peered":"2024-09-10T20:38:12.405033+0000","last_unstale":"2024-09-10T20:38:14.343528+0000","last_undegraded":"2024-09-10T20:38:14.343528+0000","last_fullsized":"2024-09-10T20:38:14.343528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:01:45.285223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398727+0000","last_change":"2024-09-10T20:38:12.412798+0000","last_active":"2024-09-10T20:38:13.398727+0000","last_peered":"2024-09-10T20:38:13.398727+0000","last_clean":"2024-09-10T20:38:13.398727+0000","last_became_active":"2024-09-10T20:38:12.412428+0000","last_became_peered":"2024-09-10T20:38:12.412428+0000","last_unstale":"2024-09-10T20:38:13.398727+0000","last_undegraded":"2024-09-10T20:38:13.398727+0000","last_fullsized":"2024-09-10T20:38:13.398727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:45.518334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400454+0000","last_change":"2024-09-10T20:38:11.394869+0000","last_active":"2024-09-10T20:38:13.400454+0000","last_peered":"2024-09-10T20:38:13.400454+0000","last_clean":"2024-09-10T20:38:13.400454+0000","last_became_active":"2024-09-10T20:38:11.394564+0000","last_became_peered":"2024-09-10T20:38:11.394564+0000","last_unstale":"2024-09-10T20:38:13.400454+0000","last_undegraded":"2024-09-10T20:38:13.400454+0000","last_fullsized":"2024-09-10T20:38:13.400454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:19.365550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401125+0000","last_change":"2024-09-10T20:38:12.401521+0000","last_active":"2024-09-10T20:38:13.401125+0000","last_peered":"2024-09-10T20:38:13.401125+0000","last_clean":"2024-09-10T20:38:13.401125+0000","last_became_active":"2024-09-10T20:38:12.401218+0000","last_became_peered":"2024-09-10T20:38:12.401218+0000","last_unstale":"2024-09-10T20:38:13.401125+0000","last_undegraded":"2024-09-10T20:38:13.401125+0000","last_fullsized":"2024-09-10T20:38:13.401125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:33.954998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343882+0000","last_change":"2024-09-10T20:38:11.401029+0000","last_active":"2024-09-10T20:38:14.343882+0000","last_peered":"2024-09-10T20:38:14.343882+0000","last_clean":"2024-09-10T20:38:14.343882+0000","last_became_active":"2024-09-10T20:38:11.400647+0000","last_became_peered":"2024-09-10T20:38:11.400647+0000","last_unstale":"2024-09-10T20:38:14.343882+0000","last_undegraded":"2024-09-10T20:38:14.343882+0000","last_fullsized":"2024-09-10T20:38:14.343882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:38.998000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343167+0000","last_change":"2024-09-10T20:38:11.398400+0000","last_active":"2024-09-10T20:38:14.343167+0000","last_peered":"2024-09-10T20:38:14.343167+0000","last_clean":"2024-09-10T20:38:14.343167+0000","last_became_active":"2024-09-10T20:38:11.398097+0000","last_became_peered":"2024-09-10T20:38:11.398097+0000","last_unstale":"2024-09-10T20:38:14.343167+0000","last_undegraded":"2024-09-10T20:38:14.343167+0000","last_fullsized":"2024-09-10T20:38:14.343167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.599838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401478+0000","last_change":"2024-09-10T20:38:12.413177+0000","last_active":"2024-09-10T20:38:13.401478+0000","last_peered":"2024-09-10T20:38:13.401478+0000","last_clean":"2024-09-10T20:38:13.401478+0000","last_became_active":"2024-09-10T20:38:12.412925+0000","last_became_peered":"2024-09-10T20:38:12.412925+0000","last_unstale":"2024-09-10T20:38:13.401478+0000","last_undegraded":"2024-09-10T20:38:13.401478+0000","last_fullsized":"2024-09-10T20:38:13.401478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:18.693832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343616+0000","last_change":"2024-09-10T20:38:11.399996+0000","last_active":"2024-09-10T20:38:14.343616+0000","last_peered":"2024-09-10T20:38:14.343616+0000","last_clean":"2024-09-10T20:38:14.343616+0000","last_became_active":"2024-09-10T20:38:11.399423+0000","last_became_peered":"2024-09-10T20:38:11.399423+0000","last_unstale":"2024-09-10T20:38:14.343616+0000","last_undegraded":"2024-09-10T20:38:14.343616+0000","last_fullsized":"2024-09-10T20:38:14.343616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:35.628735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401415+0000","last_change":"2024-09-10T20:38:12.415606+0000","last_active":"2024-09-10T20:38:13.401415+0000","last_peered":"2024-09-10T20:38:13.401415+0000","last_clean":"2024-09-10T20:38:13.401415+0000","last_became_active":"2024-09-10T20:38:12.415330+0000","last_became_peered":"2024-09-10T20:38:12.415330+0000","last_unstale":"2024-09-10T20:38:13.401415+0000","last_undegraded":"2024-09-10T20:38:13.401415+0000","last_fullsized":"2024-09-10T20:38:13.401415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:53.050014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398942+0000","last_change":"2024-09-10T20:38:11.395121+0000","last_active":"2024-09-10T20:38:13.398942+0000","last_peered":"2024-09-10T20:38:13.398942+0000","last_clean":"2024-09-10T20:38:13.398942+0000","last_became_active":"2024-09-10T20:38:11.394490+0000","last_became_peered":"2024-09-10T20:38:11.394490+0000","last_unstale":"2024-09-10T20:38:13.398942+0000","last_undegraded":"2024-09-10T20:38:13.398942+0000","last_fullsized":"2024-09-10T20:38:13.398942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:42.485721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344515+0000","last_change":"2024-09-10T20:38:12.402641+0000","last_active":"2024-09-10T20:38:14.344515+0000","last_peered":"2024-09-10T20:38:14.344515+0000","last_clean":"2024-09-10T20:38:14.344515+0000","last_became_active":"2024-09-10T20:38:12.402308+0000","last_became_peered":"2024-09-10T20:38:12.402308+0000","last_unstale":"2024-09-10T20:38:14.344515+0000","last_undegraded":"2024-09-10T20:38:14.344515+0000","last_fullsized":"2024-09-10T20:38:14.344515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408013+0000","last_change":"2024-09-10T20:38:11.396878+0000","last_active":"2024-09-10T20:38:13.408013+0000","last_peered":"2024-09-10T20:38:13.408013+0000","last_clean":"2024-09-10T20:38:13.408013+0000","last_became_active":"2024-09-10T20:38:11.396522+0000","last_became_peered":"2024-09-10T20:38:11.396522+0000","last_unstale":"2024-09-10T20:38:13.408013+0000","last_undegraded":"2024-09-10T20:38:13.408013+0000","last_fullsized":"2024-09-10T20:38:13.408013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:43.005710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401902+0000","last_change":"2024-09-10T20:38:12.410504+0000","last_active":"2024-09-10T20:38:13.401902+0000","last_peered":"2024-09-10T20:38:13.401902+0000","last_clean":"2024-09-10T20:38:13.401902+0000","last_became_active":"2024-09-10T20:38:12.410008+0000","last_became_peered":"2024-09-10T20:38:12.410008+0000","last_unstale":"2024-09-10T20:38:13.401902+0000","last_undegraded":"2024-09-10T20:38:13.401902+0000","last_fullsized":"2024-09-10T20:38:13.401902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:57.544501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633468+0000","last_change":"2024-09-10T20:38:11.395962+0000","last_active":"2024-09-10T20:38:13.633468+0000","last_peered":"2024-09-10T20:38:13.633468+0000","last_clean":"2024-09-10T20:38:13.633468+0000","last_became_active":"2024-09-10T20:38:11.395687+0000","last_became_peered":"2024-09-10T20:38:11.395687+0000","last_unstale":"2024-09-10T20:38:13.633468+0000","last_undegraded":"2024-09-10T20:38:13.633468+0000","last_fullsized":"2024-09-10T20:38:13.633468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:15.421827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344171+0000","last_change":"2024-09-10T20:38:12.405652+0000","last_active":"2024-09-10T20:38:14.344171+0000","last_peered":"2024-09-10T20:38:14.344171+0000","last_clean":"2024-09-10T20:38:14.344171+0000","last_became_active":"2024-09-10T20:38:12.405250+0000","last_became_peered":"2024-09-10T20:38:12.405250+0000","last_unstale":"2024-09-10T20:38:14.344171+0000","last_undegraded":"2024-09-10T20:38:14.344171+0000","last_fullsized":"2024-09-10T20:38:14.344171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:10.251489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343906+0000","last_change":"2024-09-10T20:38:11.387993+0000","last_active":"2024-09-10T20:38:14.343906+0000","last_peered":"2024-09-10T20:38:14.343906+0000","last_clean":"2024-09-10T20:38:14.343906+0000","last_became_active":"2024-09-10T20:38:11.387277+0000","last_became_peered":"2024-09-10T20:38:11.387277+0000","last_unstale":"2024-09-10T20:38:14.343906+0000","last_undegraded":"2024-09-10T20:38:14.343906+0000","last_fullsized":"2024-09-10T20:38:14.343906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:51.949339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408453+0000","last_change":"2024-09-10T20:38:12.412762+0000","last_active":"2024-09-10T20:38:13.408453+0000","last_peered":"2024-09-10T20:38:13.408453+0000","last_clean":"2024-09-10T20:38:13.408453+0000","last_became_active":"2024-09-10T20:38:12.412480+0000","last_became_peered":"2024-09-10T20:38:12.412480+0000","last_unstale":"2024-09-10T20:38:13.408453+0000","last_undegraded":"2024-09-10T20:38:13.408453+0000","last_fullsized":"2024-09-10T20:38:13.408453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633539+0000","last_change":"2024-09-10T20:38:11.398330+0000","last_active":"2024-09-10T20:38:13.633539+0000","last_peered":"2024-09-10T20:38:13.633539+0000","last_clean":"2024-09-10T20:38:13.633539+0000","last_became_active":"2024-09-10T20:38:11.398059+0000","last_became_peered":"2024-09-10T20:38:11.398059+0000","last_unstale":"2024-09-10T20:38:13.633539+0000","last_undegraded":"2024-09-10T20:38:13.633539+0000","last_fullsized":"2024-09-10T20:38:13.633539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343446+0000","last_change":"2024-09-10T20:38:12.412484+0000","last_active":"2024-09-10T20:38:14.343446+0000","last_peered":"2024-09-10T20:38:14.343446+0000","last_clean":"2024-09-10T20:38:14.343446+0000","last_became_active":"2024-09-10T20:38:12.412165+0000","last_became_peered":"2024-09-10T20:38:12.412165+0000","last_unstale":"2024-09-10T20:38:14.343446+0000","last_undegraded":"2024-09-10T20:38:14.343446+0000","last_fullsized":"2024-09-10T20:38:14.343446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:09.972659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344285+0000","last_change":"2024-09-10T20:38:11.400681+0000","last_active":"2024-09-10T20:38:14.344285+0000","last_peered":"2024-09-10T20:38:14.344285+0000","last_clean":"2024-09-10T20:38:14.344285+0000","last_became_active":"2024-09-10T20:38:11.400168+0000","last_became_peered":"2024-09-10T20:38:11.400168+0000","last_unstale":"2024-09-10T20:38:14.344285+0000","last_undegraded":"2024-09-10T20:38:14.344285+0000","last_fullsized":"2024-09-10T20:38:14.344285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:43.005710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344386+0000","last_change":"2024-09-10T20:38:12.413508+0000","last_active":"2024-09-10T20:38:14.344386+0000","last_peered":"2024-09-10T20:38:14.344386+0000","last_clean":"2024-09-10T20:38:14.344386+0000","last_became_active":"2024-09-10T20:38:12.413230+0000","last_became_peered":"2024-09-10T20:38:12.413230+0000","last_unstale":"2024-09-10T20:38:14.344386+0000","last_undegraded":"2024-09-10T20:38:14.344386+0000","last_fullsized":"2024-09-10T20:38:14.344386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:56.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407691+0000","last_change":"2024-09-10T20:38:11.402870+0000","last_active":"2024-09-10T20:38:13.407691+0000","last_peered":"2024-09-10T20:38:13.407691+0000","last_clean":"2024-09-10T20:38:13.407691+0000","last_became_active":"2024-09-10T20:38:11.402580+0000","last_became_peered":"2024-09-10T20:38:11.402580+0000","last_unstale":"2024-09-10T20:38:13.407691+0000","last_undegraded":"2024-09-10T20:38:13.407691+0000","last_fullsized":"2024-09-10T20:38:13.407691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:50:23.981654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398438+0000","last_change":"2024-09-10T20:38:12.416114+0000","last_active":"2024-09-10T20:38:13.398438+0000","last_peered":"2024-09-10T20:38:13.398438+0000","last_clean":"2024-09-10T20:38:13.398438+0000","last_became_active":"2024-09-10T20:38:12.415605+0000","last_became_peered":"2024-09-10T20:38:12.415605+0000","last_unstale":"2024-09-10T20:38:13.398438+0000","last_undegraded":"2024-09-10T20:38:13.398438+0000","last_fullsized":"2024-09-10T20:38:13.398438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:01:45.285223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400564+0000","last_change":"2024-09-10T20:38:11.393414+0000","last_active":"2024-09-10T20:38:13.400564+0000","last_peered":"2024-09-10T20:38:13.400564+0000","last_clean":"2024-09-10T20:38:13.400564+0000","last_became_active":"2024-09-10T20:38:11.393078+0000","last_became_peered":"2024-09-10T20:38:11.393078+0000","last_unstale":"2024-09-10T20:38:13.400564+0000","last_undegraded":"2024-09-10T20:38:13.400564+0000","last_fullsized":"2024-09-10T20:38:13.400564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:58.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343642+0000","last_change":"2024-09-10T20:38:12.407378+0000","last_active":"2024-09-10T20:38:14.343642+0000","last_peered":"2024-09-10T20:38:14.343642+0000","last_clean":"2024-09-10T20:38:14.343642+0000","last_became_active":"2024-09-10T20:38:12.407092+0000","last_became_peered":"2024-09-10T20:38:12.407092+0000","last_unstale":"2024-09-10T20:38:14.343642+0000","last_undegraded":"2024-09-10T20:38:14.343642+0000","last_fullsized":"2024-09-10T20:38:14.343642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:47.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408782+0000","last_change":"2024-09-10T20:38:11.397628+0000","last_active":"2024-09-10T20:38:13.408782+0000","last_peered":"2024-09-10T20:38:13.408782+0000","last_clean":"2024-09-10T20:38:13.408782+0000","last_became_active":"2024-09-10T20:38:11.397316+0000","last_became_peered":"2024-09-10T20:38:11.397316+0000","last_unstale":"2024-09-10T20:38:13.408782+0000","last_undegraded":"2024-09-10T20:38:13.408782+0000","last_fullsized":"2024-09-10T20:38:13.408782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:07.558503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344717+0000","last_change":"2024-09-10T20:38:12.396278+0000","last_active":"2024-09-10T20:38:14.344717+0000","last_peered":"2024-09-10T20:38:14.344717+0000","last_clean":"2024-09-10T20:38:14.344717+0000","last_became_active":"2024-09-10T20:38:12.395937+0000","last_became_peered":"2024-09-10T20:38:12.395937+0000","last_unstale":"2024-09-10T20:38:14.344717+0000","last_undegraded":"2024-09-10T20:38:14.344717+0000","last_fullsized":"2024-09-10T20:38:14.344717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:06.451633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401433+0000","last_change":"2024-09-10T20:38:11.388247+0000","last_active":"2024-09-10T20:38:13.401433+0000","last_peered":"2024-09-10T20:38:13.401433+0000","last_clean":"2024-09-10T20:38:13.401433+0000","last_became_active":"2024-09-10T20:38:11.387924+0000","last_became_peered":"2024-09-10T20:38:11.387924+0000","last_unstale":"2024-09-10T20:38:13.401433+0000","last_undegraded":"2024-09-10T20:38:13.401433+0000","last_fullsized":"2024-09-10T20:38:13.401433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:29.584118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408756+0000","last_change":"2024-09-10T20:38:12.418652+0000","last_active":"2024-09-10T20:38:13.408756+0000","last_peered":"2024-09-10T20:38:13.408756+0000","last_clean":"2024-09-10T20:38:13.408756+0000","last_became_active":"2024-09-10T20:38:12.418400+0000","last_became_peered":"2024-09-10T20:38:12.418400+0000","last_unstale":"2024-09-10T20:38:13.408756+0000","last_undegraded":"2024-09-10T20:38:13.408756+0000","last_fullsized":"2024-09-10T20:38:13.408756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:28.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400616+0000","last_change":"2024-09-10T20:38:11.399015+0000","last_active":"2024-09-10T20:38:13.400616+0000","last_peered":"2024-09-10T20:38:13.400616+0000","last_clean":"2024-09-10T20:38:13.400616+0000","last_became_active":"2024-09-10T20:38:11.398826+0000","last_became_peered":"2024-09-10T20:38:11.398826+0000","last_unstale":"2024-09-10T20:38:13.400616+0000","last_undegraded":"2024-09-10T20:38:13.400616+0000","last_fullsized":"2024-09-10T20:38:13.400616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:03.958637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408168+0000","last_change":"2024-09-10T20:38:12.411663+0000","last_active":"2024-09-10T20:38:13.408168+0000","last_peered":"2024-09-10T20:38:13.408168+0000","last_clean":"2024-09-10T20:38:13.408168+0000","last_became_active":"2024-09-10T20:38:12.411269+0000","last_became_peered":"2024-09-10T20:38:12.411269+0000","last_unstale":"2024-09-10T20:38:13.408168+0000","last_undegraded":"2024-09-10T20:38:13.408168+0000","last_fullsized":"2024-09-10T20:38:13.408168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:20.639033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343682+0000","last_change":"2024-09-10T20:38:11.398337+0000","last_active":"2024-09-10T20:38:14.343682+0000","last_peered":"2024-09-10T20:38:14.343682+0000","last_clean":"2024-09-10T20:38:14.343682+0000","last_became_active":"2024-09-10T20:38:11.398068+0000","last_became_peered":"2024-09-10T20:38:11.398068+0000","last_unstale":"2024-09-10T20:38:14.343682+0000","last_undegraded":"2024-09-10T20:38:14.343682+0000","last_fullsized":"2024-09-10T20:38:14.343682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:00:28.236504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344836+0000","last_change":"2024-09-10T20:38:12.420411+0000","last_active":"2024-09-10T20:38:14.344836+0000","last_peered":"2024-09-10T20:38:14.344836+0000","last_clean":"2024-09-10T20:38:14.344836+0000","last_became_active":"2024-09-10T20:38:12.419107+0000","last_became_peered":"2024-09-10T20:38:12.419107+0000","last_unstale":"2024-09-10T20:38:14.344836+0000","last_undegraded":"2024-09-10T20:38:14.344836+0000","last_fullsized":"2024-09-10T20:38:14.344836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:15.534463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398277+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:13.398277+0000","last_peered":"2024-09-10T20:38:13.398277+0000","last_clean":"2024-09-10T20:38:13.398277+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:13.398277+0000","last_undegraded":"2024-09-10T20:38:13.398277+0000","last_fullsized":"2024-09-10T20:38:13.398277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344545+0000","last_change":"2024-09-10T20:38:11.398527+0000","last_active":"2024-09-10T20:38:14.344545+0000","last_peered":"2024-09-10T20:38:14.344545+0000","last_clean":"2024-09-10T20:38:14.344545+0000","last_became_active":"2024-09-10T20:38:11.395195+0000","last_became_peered":"2024-09-10T20:38:11.395195+0000","last_unstale":"2024-09-10T20:38:14.344545+0000","last_undegraded":"2024-09-10T20:38:14.344545+0000","last_fullsized":"2024-09-10T20:38:14.344545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:35.618612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400902+0000","last_change":"2024-09-10T20:38:12.407263+0000","last_active":"2024-09-10T20:38:13.400902+0000","last_peered":"2024-09-10T20:38:13.400902+0000","last_clean":"2024-09-10T20:38:13.400902+0000","last_became_active":"2024-09-10T20:38:12.406952+0000","last_became_peered":"2024-09-10T20:38:12.406952+0000","last_unstale":"2024-09-10T20:38:13.400902+0000","last_undegraded":"2024-09-10T20:38:13.400902+0000","last_fullsized":"2024-09-10T20:38:13.400902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.171100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343670+0000","last_change":"2024-09-10T20:38:11.399685+0000","last_active":"2024-09-10T20:38:14.343670+0000","last_peered":"2024-09-10T20:38:14.343670+0000","last_clean":"2024-09-10T20:38:14.343670+0000","last_became_active":"2024-09-10T20:38:11.399073+0000","last_became_peered":"2024-09-10T20:38:11.399073+0000","last_unstale":"2024-09-10T20:38:14.343670+0000","last_undegraded":"2024-09-10T20:38:14.343670+0000","last_fullsized":"2024-09-10T20:38:14.343670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:08.967765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343358+0000","last_change":"2024-09-10T20:38:12.407025+0000","last_active":"2024-09-10T20:38:14.343358+0000","last_peered":"2024-09-10T20:38:14.343358+0000","last_clean":"2024-09-10T20:38:14.343358+0000","last_became_active":"2024-09-10T20:38:12.406124+0000","last_became_peered":"2024-09-10T20:38:12.406124+0000","last_unstale":"2024-09-10T20:38:14.343358+0000","last_undegraded":"2024-09-10T20:38:14.343358+0000","last_fullsized":"2024-09-10T20:38:14.343358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633299+0000","last_change":"2024-09-10T20:38:11.397217+0000","last_active":"2024-09-10T20:38:13.633299+0000","last_peered":"2024-09-10T20:38:13.633299+0000","last_clean":"2024-09-10T20:38:13.633299+0000","last_became_active":"2024-09-10T20:38:11.396928+0000","last_became_peered":"2024-09-10T20:38:11.396928+0000","last_unstale":"2024-09-10T20:38:13.633299+0000","last_undegraded":"2024-09-10T20:38:13.633299+0000","last_fullsized":"2024-09-10T20:38:13.633299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:38.900087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401469+0000","last_change":"2024-09-10T20:38:12.407595+0000","last_active":"2024-09-10T20:38:13.401469+0000","last_peered":"2024-09-10T20:38:13.401469+0000","last_clean":"2024-09-10T20:38:13.401469+0000","last_became_active":"2024-09-10T20:38:12.407355+0000","last_became_peered":"2024-09-10T20:38:12.407355+0000","last_unstale":"2024-09-10T20:38:13.401469+0000","last_undegraded":"2024-09-10T20:38:13.401469+0000","last_fullsized":"2024-09-10T20:38:13.401469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:05.419998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344389+0000","last_change":"2024-09-10T20:38:11.401377+0000","last_active":"2024-09-10T20:38:14.344389+0000","last_peered":"2024-09-10T20:38:14.344389+0000","last_clean":"2024-09-10T20:38:14.344389+0000","last_became_active":"2024-09-10T20:38:11.401048+0000","last_became_peered":"2024-09-10T20:38:11.401048+0000","last_unstale":"2024-09-10T20:38:14.344389+0000","last_undegraded":"2024-09-10T20:38:14.344389+0000","last_fullsized":"2024-09-10T20:38:14.344389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:17.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408141+0000","last_change":"2024-09-10T20:38:12.399857+0000","last_active":"2024-09-10T20:38:13.408141+0000","last_peered":"2024-09-10T20:38:13.408141+0000","last_clean":"2024-09-10T20:38:13.408141+0000","last_became_active":"2024-09-10T20:38:12.399121+0000","last_became_peered":"2024-09-10T20:38:12.399121+0000","last_unstale":"2024-09-10T20:38:13.408141+0000","last_undegraded":"2024-09-10T20:38:13.408141+0000","last_fullsized":"2024-09-10T20:38:13.408141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:23.782139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400255+0000","last_change":"2024-09-10T20:38:11.397463+0000","last_active":"2024-09-10T20:38:13.400255+0000","last_peered":"2024-09-10T20:38:13.400255+0000","last_clean":"2024-09-10T20:38:13.400255+0000","last_became_active":"2024-09-10T20:38:11.397205+0000","last_became_peered":"2024-09-10T20:38:11.397205+0000","last_unstale":"2024-09-10T20:38:13.400255+0000","last_undegraded":"2024-09-10T20:38:13.400255+0000","last_fullsized":"2024-09-10T20:38:13.400255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:53.003074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343801+0000","last_change":"2024-09-10T20:38:12.412633+0000","last_active":"2024-09-10T20:38:14.343801+0000","last_peered":"2024-09-10T20:38:14.343801+0000","last_clean":"2024-09-10T20:38:14.343801+0000","last_became_active":"2024-09-10T20:38:12.412350+0000","last_became_peered":"2024-09-10T20:38:12.412350+0000","last_unstale":"2024-09-10T20:38:14.343801+0000","last_undegraded":"2024-09-10T20:38:14.343801+0000","last_fullsized":"2024-09-10T20:38:14.343801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.915632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344655+0000","last_change":"2024-09-10T20:38:11.392177+0000","last_active":"2024-09-10T20:38:14.344655+0000","last_peered":"2024-09-10T20:38:14.344655+0000","last_clean":"2024-09-10T20:38:14.344655+0000","last_became_active":"2024-09-10T20:38:11.390174+0000","last_became_peered":"2024-09-10T20:38:11.390174+0000","last_unstale":"2024-09-10T20:38:14.344655+0000","last_undegraded":"2024-09-10T20:38:14.344655+0000","last_fullsized":"2024-09-10T20:38:14.344655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:29.078056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343576+0000","last_change":"2024-09-10T20:38:12.408779+0000","last_active":"2024-09-10T20:38:14.343576+0000","last_peered":"2024-09-10T20:38:14.343576+0000","last_clean":"2024-09-10T20:38:14.343576+0000","last_became_active":"2024-09-10T20:38:12.408490+0000","last_became_peered":"2024-09-10T20:38:12.408490+0000","last_unstale":"2024-09-10T20:38:14.343576+0000","last_undegraded":"2024-09-10T20:38:14.343576+0000","last_fullsized":"2024-09-10T20:38:14.343576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:25.888623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407174+0000","last_change":"2024-09-10T20:38:11.394170+0000","last_active":"2024-09-10T20:38:13.407174+0000","last_peered":"2024-09-10T20:38:13.407174+0000","last_clean":"2024-09-10T20:38:13.407174+0000","last_became_active":"2024-09-10T20:38:11.393850+0000","last_became_peered":"2024-09-10T20:38:11.393850+0000","last_unstale":"2024-09-10T20:38:13.407174+0000","last_undegraded":"2024-09-10T20:38:13.407174+0000","last_fullsized":"2024-09-10T20:38:13.407174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:15.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401623+0000","last_change":"2024-09-10T20:38:12.404386+0000","last_active":"2024-09-10T20:38:13.401623+0000","last_peered":"2024-09-10T20:38:13.401623+0000","last_clean":"2024-09-10T20:38:13.401623+0000","last_became_active":"2024-09-10T20:38:12.404045+0000","last_became_peered":"2024-09-10T20:38:12.404045+0000","last_unstale":"2024-09-10T20:38:13.401623+0000","last_undegraded":"2024-09-10T20:38:13.401623+0000","last_fullsized":"2024-09-10T20:38:13.401623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:49.410849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344714+0000","last_change":"2024-09-10T20:38:11.399979+0000","last_active":"2024-09-10T20:38:14.344714+0000","last_peered":"2024-09-10T20:38:14.344714+0000","last_clean":"2024-09-10T20:38:14.344714+0000","last_became_active":"2024-09-10T20:38:11.399682+0000","last_became_peered":"2024-09-10T20:38:11.399682+0000","last_unstale":"2024-09-10T20:38:14.344714+0000","last_undegraded":"2024-09-10T20:38:14.344714+0000","last_fullsized":"2024-09-10T20:38:14.344714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:40.818316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400970+0000","last_change":"2024-09-10T20:38:12.409091+0000","last_active":"2024-09-10T20:38:13.400970+0000","last_peered":"2024-09-10T20:38:13.400970+0000","last_clean":"2024-09-10T20:38:13.400970+0000","last_became_active":"2024-09-10T20:38:12.408815+0000","last_became_peered":"2024-09-10T20:38:12.408815+0000","last_unstale":"2024-09-10T20:38:13.400970+0000","last_undegraded":"2024-09-10T20:38:13.400970+0000","last_fullsized":"2024-09-10T20:38:13.400970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.870149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407496+0000","last_change":"2024-09-10T20:38:11.391306+0000","last_active":"2024-09-10T20:38:13.407496+0000","last_peered":"2024-09-10T20:38:13.407496+0000","last_clean":"2024-09-10T20:38:13.407496+0000","last_became_active":"2024-09-10T20:38:11.390956+0000","last_became_peered":"2024-09-10T20:38:11.390956+0000","last_unstale":"2024-09-10T20:38:13.407496+0000","last_undegraded":"2024-09-10T20:38:13.407496+0000","last_fullsized":"2024-09-10T20:38:13.407496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:12:00.829218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344085+0000","last_change":"2024-09-10T20:38:12.402970+0000","last_active":"2024-09-10T20:38:14.344085+0000","last_peered":"2024-09-10T20:38:14.344085+0000","last_clean":"2024-09-10T20:38:14.344085+0000","last_became_active":"2024-09-10T20:38:12.402393+0000","last_became_peered":"2024-09-10T20:38:12.402393+0000","last_unstale":"2024-09-10T20:38:14.344085+0000","last_undegraded":"2024-09-10T20:38:14.344085+0000","last_fullsized":"2024-09-10T20:38:14.344085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:09.528282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401287+0000","last_change":"2024-09-10T20:38:11.397172+0000","last_active":"2024-09-10T20:38:13.401287+0000","last_peered":"2024-09-10T20:38:13.401287+0000","last_clean":"2024-09-10T20:38:13.401287+0000","last_became_active":"2024-09-10T20:38:11.396670+0000","last_became_peered":"2024-09-10T20:38:11.396670+0000","last_unstale":"2024-09-10T20:38:13.401287+0000","last_undegraded":"2024-09-10T20:38:13.401287+0000","last_fullsized":"2024-09-10T20:38:13.401287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400195+0000","last_change":"2024-09-10T20:38:12.405875+0000","last_active":"2024-09-10T20:38:13.400195+0000","last_peered":"2024-09-10T20:38:13.400195+0000","last_clean":"2024-09-10T20:38:13.400195+0000","last_became_active":"2024-09-10T20:38:12.405583+0000","last_became_peered":"2024-09-10T20:38:12.405583+0000","last_unstale":"2024-09-10T20:38:13.400195+0000","last_undegraded":"2024-09-10T20:38:13.400195+0000","last_fullsized":"2024-09-10T20:38:13.400195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:30.185620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343471+0000","last_change":"2024-09-10T20:38:11.395992+0000","last_active":"2024-09-10T20:38:14.343471+0000","last_peered":"2024-09-10T20:38:14.343471+0000","last_clean":"2024-09-10T20:38:14.343471+0000","last_became_active":"2024-09-10T20:38:11.394935+0000","last_became_peered":"2024-09-10T20:38:11.394935+0000","last_unstale":"2024-09-10T20:38:14.343471+0000","last_undegraded":"2024-09-10T20:38:14.343471+0000","last_fullsized":"2024-09-10T20:38:14.343471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:49.040222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633574+0000","last_change":"2024-09-10T20:38:12.412620+0000","last_active":"2024-09-10T20:38:13.633574+0000","last_peered":"2024-09-10T20:38:13.633574+0000","last_clean":"2024-09-10T20:38:13.633574+0000","last_became_active":"2024-09-10T20:38:12.412329+0000","last_became_peered":"2024-09-10T20:38:12.412329+0000","last_unstale":"2024-09-10T20:38:13.633574+0000","last_undegraded":"2024-09-10T20:38:13.633574+0000","last_fullsized":"2024-09-10T20:38:13.633574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:40.609805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344474+0000","last_change":"2024-09-10T20:38:11.398962+0000","last_active":"2024-09-10T20:38:14.344474+0000","last_peered":"2024-09-10T20:38:14.344474+0000","last_clean":"2024-09-10T20:38:14.344474+0000","last_became_active":"2024-09-10T20:38:11.398617+0000","last_became_peered":"2024-09-10T20:38:11.398617+0000","last_unstale":"2024-09-10T20:38:14.344474+0000","last_undegraded":"2024-09-10T20:38:14.344474+0000","last_fullsized":"2024-09-10T20:38:14.344474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:47.028207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344204+0000","last_change":"2024-09-10T20:38:12.418199+0000","last_active":"2024-09-10T20:38:14.344204+0000","last_peered":"2024-09-10T20:38:14.344204+0000","last_clean":"2024-09-10T20:38:14.344204+0000","last_became_active":"2024-09-10T20:38:12.417684+0000","last_became_peered":"2024-09-10T20:38:12.417684+0000","last_unstale":"2024-09-10T20:38:14.344204+0000","last_undegraded":"2024-09-10T20:38:14.344204+0000","last_fullsized":"2024-09-10T20:38:14.344204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:27.561396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344345+0000","last_change":"2024-09-10T20:38:12.406004+0000","last_active":"2024-09-10T20:38:14.344345+0000","last_peered":"2024-09-10T20:38:14.344345+0000","last_clean":"2024-09-10T20:38:14.344345+0000","last_became_active":"2024-09-10T20:38:12.405336+0000","last_became_peered":"2024-09-10T20:38:12.405336+0000","last_unstale":"2024-09-10T20:38:14.344345+0000","last_undegraded":"2024-09-10T20:38:14.344345+0000","last_fullsized":"2024-09-10T20:38:14.344345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:58:35.400897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408136+0000","last_change":"2024-09-10T20:38:11.396962+0000","last_active":"2024-09-10T20:38:13.408136+0000","last_peered":"2024-09-10T20:38:13.408136+0000","last_clean":"2024-09-10T20:38:13.408136+0000","last_became_active":"2024-09-10T20:38:11.396650+0000","last_became_peered":"2024-09-10T20:38:11.396650+0000","last_unstale":"2024-09-10T20:38:13.408136+0000","last_undegraded":"2024-09-10T20:38:13.408136+0000","last_fullsized":"2024-09-10T20:38:13.408136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:40.818316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398946+0000","last_change":"2024-09-10T20:38:12.413236+0000","last_active":"2024-09-10T20:38:13.398946+0000","last_peered":"2024-09-10T20:38:13.398946+0000","last_clean":"2024-09-10T20:38:13.398946+0000","last_became_active":"2024-09-10T20:38:12.412966+0000","last_became_peered":"2024-09-10T20:38:12.412966+0000","last_unstale":"2024-09-10T20:38:13.398946+0000","last_undegraded":"2024-09-10T20:38:13.398946+0000","last_fullsized":"2024-09-10T20:38:13.398946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:01:29.734867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408400+0000","last_change":"2024-09-10T20:38:11.397755+0000","last_active":"2024-09-10T20:38:13.408400+0000","last_peered":"2024-09-10T20:38:13.408400+0000","last_clean":"2024-09-10T20:38:13.408400+0000","last_became_active":"2024-09-10T20:38:11.397450+0000","last_became_peered":"2024-09-10T20:38:11.397450+0000","last_unstale":"2024-09-10T20:38:13.408400+0000","last_undegraded":"2024-09-10T20:38:13.408400+0000","last_fullsized":"2024-09-10T20:38:13.408400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:38.998000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401905+0000","last_change":"2024-09-10T20:38:12.416588+0000","last_active":"2024-09-10T20:38:13.401905+0000","last_peered":"2024-09-10T20:38:13.401905+0000","last_clean":"2024-09-10T20:38:13.401905+0000","last_became_active":"2024-09-10T20:38:12.416297+0000","last_became_peered":"2024-09-10T20:38:12.416297+0000","last_unstale":"2024-09-10T20:38:13.401905+0000","last_undegraded":"2024-09-10T20:38:13.401905+0000","last_fullsized":"2024-09-10T20:38:13.401905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401853+0000","last_change":"2024-09-10T20:38:11.396382+0000","last_active":"2024-09-10T20:38:13.401853+0000","last_peered":"2024-09-10T20:38:13.401853+0000","last_clean":"2024-09-10T20:38:13.401853+0000","last_became_active":"2024-09-10T20:38:11.395934+0000","last_became_peered":"2024-09-10T20:38:11.395934+0000","last_unstale":"2024-09-10T20:38:13.401853+0000","last_undegraded":"2024-09-10T20:38:13.401853+0000","last_fullsized":"2024-09-10T20:38:13.401853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:20:39.553068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401634+0000","last_change":"2024-09-10T20:38:12.402624+0000","last_active":"2024-09-10T20:38:13.401634+0000","last_peered":"2024-09-10T20:38:13.401634+0000","last_clean":"2024-09-10T20:38:13.401634+0000","last_became_active":"2024-09-10T20:38:12.401924+0000","last_became_peered":"2024-09-10T20:38:12.401924+0000","last_unstale":"2024-09-10T20:38:13.401634+0000","last_undegraded":"2024-09-10T20:38:13.401634+0000","last_fullsized":"2024-09-10T20:38:13.401634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:33.683744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401708+0000","last_change":"2024-09-10T20:38:11.401264+0000","last_active":"2024-09-10T20:38:13.401708+0000","last_peered":"2024-09-10T20:38:13.401708+0000","last_clean":"2024-09-10T20:38:13.401708+0000","last_became_active":"2024-09-10T20:38:11.400986+0000","last_became_peered":"2024-09-10T20:38:11.400986+0000","last_unstale":"2024-09-10T20:38:13.401708+0000","last_undegraded":"2024-09-10T20:38:13.401708+0000","last_fullsized":"2024-09-10T20:38:13.401708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:50.668458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":73109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":74139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":806912,"data_stored":662965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":217088,"data_stored":72583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":72527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":662987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":74811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:25.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:25.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 -- 172.21.15.28:0/1484415530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 msgr2=0x7f89540ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 --2- 172.21.15.28:0/1484415530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540ce850 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f89540b7ed0 tx=0x7f894802a560 comp rx=0 tx=0).stop 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 -- 172.21.15.28:0/1484415530 shutdown_connections 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 --2- 172.21.15.28:0/1484415530 >> v2:172.21.15.159:3300/0 conn(0x7f89540cd9b0 0x7f89540ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 --2- 172.21.15.28:0/1484415530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540ce850 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 --2- 172.21.15.28:0/1484415530 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f895415c050 0x7f89540cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 -- 172.21.15.28:0/1484415530 >> 172.21.15.28:0/1484415530 conn(0x7f89540d1e40 msgr2=0x7f89540d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 -- 172.21.15.28:0/1484415530 shutdown_connections 2024-09-10T20:38:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.748+0000 7f8959924640 1 -- 172.21.15.28:0/1484415530 wait complete. 2024-09-10T20:38:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.750+0000 7f8959924640 1 Processor -- start 2024-09-10T20:38:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.750+0000 7f8959924640 1 -- start start 2024-09-10T20:38:25.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540cd9b0 0x7f89540b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895415c050 0x7f89540cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89540b1820 con 0x7f89540d2960 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f89540b16a0 con 0x7f895415c050 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f89540b19a0 con 0x7f89540cd9b0 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51264/0 (socket says 172.21.15.28:51264) 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 -- 172.21.15.28:0/131760341 learned_addr learned my addr 172.21.15.28:0/131760341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89537fe640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895415c050 0x7f89540cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8952ffd640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540cd9b0 0x7f89540b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 -- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540cd9b0 msgr2=0x7f89540b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540cd9b0 0x7f89540b73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 -- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895415c050 msgr2=0x7f89540cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895415c050 0x7f89540cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 -- 172.21.15.28:0/131760341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89540b7a20 con 0x7f89540d2960 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89537fe640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895415c050 0x7f89540cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8952ffd640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540cd9b0 0x7f89540b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f89527fc640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540b67b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f89480099c0 tx=0x7f89480039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f893bfff640 1 -- 172.21.15.28:0/131760341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894803f070 con 0x7f89540d2960 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f893bfff640 1 -- 172.21.15.28:0/131760341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8948004890 con 0x7f89540d2960 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f893bfff640 1 -- 172.21.15.28:0/131760341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948035030 con 0x7f89540d2960 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89541f2a70 con 0x7f89540d2960 2024-09-10T20:38:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.751+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89541f3310 con 0x7f89540d2960 2024-09-10T20:38:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.753+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89541f2e00 con 0x7f89540d2960 2024-09-10T20:38:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.753+0000 7f893bfff640 1 -- 172.21.15.28:0/131760341 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8948004ce0 con 0x7f89540d2960 2024-09-10T20:38:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.754+0000 7f893bfff640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8940077f70 0x7f894007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.754+0000 7f893bfff640 1 -- 172.21.15.28:0/131760341 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f89480bb170 con 0x7f89540d2960 2024-09-10T20:38:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.754+0000 7f8952ffd640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8940077f70 0x7f894007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.754+0000 7f8952ffd640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8940077f70 0x7f894007a430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8944009fa0 tx=0x7f8944009570 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.758+0000 7f893bfff640 1 -- 172.21.15.28:0/131760341 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8948083bd0 con 0x7f89540d2960 2024-09-10T20:38:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.938+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89541f34e0 con 0x7f8940077f70 2024-09-10T20:38:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.942+0000 7f893bfff640 1 -- 172.21.15.28:0/131760341 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251822 (secure 0 0 0) 0x7f89541f34e0 con 0x7f8940077f70 2024-09-10T20:38:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:25.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8940077f70 msgr2=0x7f894007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8940077f70 0x7f894007a430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8944009fa0 tx=0x7f8944009570 comp rx=0 tx=0).stop 2024-09-10T20:38:25.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 msgr2=0x7f89540b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540b67b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f89480099c0 tx=0x7f89480039e0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 shutdown_connections 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8940077f70 0x7f894007a430 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895415c050 0x7f89540cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540d2960 0x7f89540b67b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 --2- 172.21.15.28:0/131760341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540cd9b0 0x7f89540b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 >> 172.21.15.28:0/131760341 conn(0x7f89540d1e40 msgr2=0x7f8954071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 shutdown_connections 2024-09-10T20:38:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:25.945+0000 7f8959924640 1 -- 172.21.15.28:0/131760341 wait complete. 2024-09-10T20:38:25.960 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-10T20:38:23.978833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235740,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738980,"statfs":{"total":773094113280,"available":772852715520,"internally_reserved":0,"allocated":2924544,"data_stored":1765772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.580611"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408702+0000","last_change":"2024-09-10T20:38:12.400371+0000","last_active":"2024-09-10T20:38:13.408702+0000","last_peered":"2024-09-10T20:38:13.408702+0000","last_clean":"2024-09-10T20:38:13.408702+0000","last_became_active":"2024-09-10T20:38:12.400049+0000","last_became_peered":"2024-09-10T20:38:12.400049+0000","last_unstale":"2024-09-10T20:38:13.408702+0000","last_undegraded":"2024-09-10T20:38:13.408702+0000","last_fullsized":"2024-09-10T20:38:13.408702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:47.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400675+0000","last_change":"2024-09-10T20:38:12.413418+0000","last_active":"2024-09-10T20:38:13.400675+0000","last_peered":"2024-09-10T20:38:13.400675+0000","last_clean":"2024-09-10T20:38:13.400675+0000","last_became_active":"2024-09-10T20:38:12.413063+0000","last_became_peered":"2024-09-10T20:38:12.413063+0000","last_unstale":"2024-09-10T20:38:13.400675+0000","last_undegraded":"2024-09-10T20:38:13.400675+0000","last_fullsized":"2024-09-10T20:38:13.400675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:12.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401706+0000","last_change":"2024-09-10T20:38:12.404482+0000","last_active":"2024-09-10T20:38:13.401706+0000","last_peered":"2024-09-10T20:38:13.401706+0000","last_clean":"2024-09-10T20:38:13.401706+0000","last_became_active":"2024-09-10T20:38:12.404181+0000","last_became_peered":"2024-09-10T20:38:12.404181+0000","last_unstale":"2024-09-10T20:38:13.401706+0000","last_undegraded":"2024-09-10T20:38:13.401706+0000","last_fullsized":"2024-09-10T20:38:13.401706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:02.847206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343826+0000","last_change":"2024-09-10T20:38:12.399863+0000","last_active":"2024-09-10T20:38:14.343826+0000","last_peered":"2024-09-10T20:38:14.343826+0000","last_clean":"2024-09-10T20:38:14.343826+0000","last_became_active":"2024-09-10T20:38:12.399541+0000","last_became_peered":"2024-09-10T20:38:12.399541+0000","last_unstale":"2024-09-10T20:38:14.343826+0000","last_undegraded":"2024-09-10T20:38:14.343826+0000","last_fullsized":"2024-09-10T20:38:14.343826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:45.518334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408418+0000","last_change":"2024-09-10T20:38:12.403977+0000","last_active":"2024-09-10T20:38:13.408418+0000","last_peered":"2024-09-10T20:38:13.408418+0000","last_clean":"2024-09-10T20:38:13.408418+0000","last_became_active":"2024-09-10T20:38:12.403575+0000","last_became_peered":"2024-09-10T20:38:12.403575+0000","last_unstale":"2024-09-10T20:38:13.408418+0000","last_undegraded":"2024-09-10T20:38:13.408418+0000","last_fullsized":"2024-09-10T20:38:13.408418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:37.571212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344323+0000","last_change":"2024-09-10T20:38:12.399990+0000","last_active":"2024-09-10T20:38:14.344323+0000","last_peered":"2024-09-10T20:38:14.344323+0000","last_clean":"2024-09-10T20:38:14.344323+0000","last_became_active":"2024-09-10T20:38:12.399633+0000","last_became_peered":"2024-09-10T20:38:12.399633+0000","last_unstale":"2024-09-10T20:38:14.344323+0000","last_undegraded":"2024-09-10T20:38:14.344323+0000","last_fullsized":"2024-09-10T20:38:14.344323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:01:29.734867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407809+0000","last_change":"2024-09-10T20:38:12.401934+0000","last_active":"2024-09-10T20:38:13.407809+0000","last_peered":"2024-09-10T20:38:13.407809+0000","last_clean":"2024-09-10T20:38:13.407809+0000","last_became_active":"2024-09-10T20:38:12.401547+0000","last_became_peered":"2024-09-10T20:38:12.401547+0000","last_unstale":"2024-09-10T20:38:13.407809+0000","last_undegraded":"2024-09-10T20:38:13.407809+0000","last_fullsized":"2024-09-10T20:38:13.407809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:21.949782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401300+0000","last_change":"2024-09-10T20:38:12.403203+0000","last_active":"2024-09-10T20:38:13.401300+0000","last_peered":"2024-09-10T20:38:13.401300+0000","last_clean":"2024-09-10T20:38:13.401300+0000","last_became_active":"2024-09-10T20:38:12.402272+0000","last_became_peered":"2024-09-10T20:38:12.402272+0000","last_unstale":"2024-09-10T20:38:13.401300+0000","last_undegraded":"2024-09-10T20:38:13.401300+0000","last_fullsized":"2024-09-10T20:38:13.401300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.399062+0000","last_change":"2024-09-10T20:38:12.420252+0000","last_active":"2024-09-10T20:38:13.399062+0000","last_peered":"2024-09-10T20:38:13.399062+0000","last_clean":"2024-09-10T20:38:13.399062+0000","last_became_active":"2024-09-10T20:38:12.419848+0000","last_became_peered":"2024-09-10T20:38:12.419848+0000","last_unstale":"2024-09-10T20:38:13.399062+0000","last_undegraded":"2024-09-10T20:38:13.399062+0000","last_fullsized":"2024-09-10T20:38:13.399062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:13.317965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408005+0000","last_change":"2024-09-10T20:38:12.398378+0000","last_active":"2024-09-10T20:38:13.408005+0000","last_peered":"2024-09-10T20:38:13.408005+0000","last_clean":"2024-09-10T20:38:13.408005+0000","last_became_active":"2024-09-10T20:38:12.398068+0000","last_became_peered":"2024-09-10T20:38:12.398068+0000","last_unstale":"2024-09-10T20:38:13.408005+0000","last_undegraded":"2024-09-10T20:38:13.408005+0000","last_fullsized":"2024-09-10T20:38:13.408005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:52.400247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408341+0000","last_change":"2024-09-10T20:38:12.401136+0000","last_active":"2024-09-10T20:38:13.408341+0000","last_peered":"2024-09-10T20:38:13.408341+0000","last_clean":"2024-09-10T20:38:13.408341+0000","last_became_active":"2024-09-10T20:38:12.400835+0000","last_became_peered":"2024-09-10T20:38:12.400835+0000","last_unstale":"2024-09-10T20:38:13.408341+0000","last_undegraded":"2024-09-10T20:38:13.408341+0000","last_fullsized":"2024-09-10T20:38:13.408341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:37:58.944607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408006+0000","last_change":"2024-09-10T20:38:12.417280+0000","last_active":"2024-09-10T20:38:13.408006+0000","last_peered":"2024-09-10T20:38:13.408006+0000","last_clean":"2024-09-10T20:38:13.408006+0000","last_became_active":"2024-09-10T20:38:12.416991+0000","last_became_peered":"2024-09-10T20:38:12.416991+0000","last_unstale":"2024-09-10T20:38:13.408006+0000","last_undegraded":"2024-09-10T20:38:13.408006+0000","last_fullsized":"2024-09-10T20:38:13.408006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:36.789848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343562+0000","last_change":"2024-09-10T20:38:12.415360+0000","last_active":"2024-09-10T20:38:14.343562+0000","last_peered":"2024-09-10T20:38:14.343562+0000","last_clean":"2024-09-10T20:38:14.343562+0000","last_became_active":"2024-09-10T20:38:12.415094+0000","last_became_peered":"2024-09-10T20:38:12.415094+0000","last_unstale":"2024-09-10T20:38:14.343562+0000","last_undegraded":"2024-09-10T20:38:14.343562+0000","last_fullsized":"2024-09-10T20:38:14.343562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:36.623506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343324+0000","last_change":"2024-09-10T20:38:12.411600+0000","last_active":"2024-09-10T20:38:14.343324+0000","last_peered":"2024-09-10T20:38:14.343324+0000","last_clean":"2024-09-10T20:38:14.343324+0000","last_became_active":"2024-09-10T20:38:12.411328+0000","last_became_peered":"2024-09-10T20:38:12.411328+0000","last_unstale":"2024-09-10T20:38:14.343324+0000","last_undegraded":"2024-09-10T20:38:14.343324+0000","last_fullsized":"2024-09-10T20:38:14.343324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:30.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400233+0000","last_change":"2024-09-10T20:38:12.404816+0000","last_active":"2024-09-10T20:38:13.400233+0000","last_peered":"2024-09-10T20:38:13.400233+0000","last_clean":"2024-09-10T20:38:13.400233+0000","last_became_active":"2024-09-10T20:38:12.404568+0000","last_became_peered":"2024-09-10T20:38:12.404568+0000","last_unstale":"2024-09-10T20:38:13.400233+0000","last_undegraded":"2024-09-10T20:38:13.400233+0000","last_fullsized":"2024-09-10T20:38:13.400233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:05:49.714066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343559+0000","last_change":"2024-09-10T20:38:12.404745+0000","last_active":"2024-09-10T20:38:14.343559+0000","last_peered":"2024-09-10T20:38:14.343559+0000","last_clean":"2024-09-10T20:38:14.343559+0000","last_became_active":"2024-09-10T20:38:12.404429+0000","last_became_peered":"2024-09-10T20:38:12.404429+0000","last_unstale":"2024-09-10T20:38:14.343559+0000","last_undegraded":"2024-09-10T20:38:14.343559+0000","last_fullsized":"2024-09-10T20:38:14.343559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344641+0000","last_change":"2024-09-10T20:38:12.404923+0000","last_active":"2024-09-10T20:38:14.344641+0000","last_peered":"2024-09-10T20:38:14.344641+0000","last_clean":"2024-09-10T20:38:14.344641+0000","last_became_active":"2024-09-10T20:38:12.404604+0000","last_became_peered":"2024-09-10T20:38:12.404604+0000","last_unstale":"2024-09-10T20:38:14.344641+0000","last_undegraded":"2024-09-10T20:38:14.344641+0000","last_fullsized":"2024-09-10T20:38:14.344641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:27.005436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633628+0000","last_change":"2024-09-10T20:38:12.407265+0000","last_active":"2024-09-10T20:38:13.633628+0000","last_peered":"2024-09-10T20:38:13.633628+0000","last_clean":"2024-09-10T20:38:13.633628+0000","last_became_active":"2024-09-10T20:38:12.406927+0000","last_became_peered":"2024-09-10T20:38:12.406927+0000","last_unstale":"2024-09-10T20:38:13.633628+0000","last_undegraded":"2024-09-10T20:38:13.633628+0000","last_fullsized":"2024-09-10T20:38:13.633628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:14.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344605+0000","last_change":"2024-09-10T20:38:12.411419+0000","last_active":"2024-09-10T20:38:14.344605+0000","last_peered":"2024-09-10T20:38:14.344605+0000","last_clean":"2024-09-10T20:38:14.344605+0000","last_became_active":"2024-09-10T20:38:12.411088+0000","last_became_peered":"2024-09-10T20:38:12.411088+0000","last_unstale":"2024-09-10T20:38:14.344605+0000","last_undegraded":"2024-09-10T20:38:14.344605+0000","last_fullsized":"2024-09-10T20:38:14.344605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:56.228254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408072+0000","last_change":"2024-09-10T20:38:12.416514+0000","last_active":"2024-09-10T20:38:13.408072+0000","last_peered":"2024-09-10T20:38:13.408072+0000","last_clean":"2024-09-10T20:38:13.408072+0000","last_became_active":"2024-09-10T20:38:12.416216+0000","last_became_peered":"2024-09-10T20:38:12.416216+0000","last_unstale":"2024-09-10T20:38:13.408072+0000","last_undegraded":"2024-09-10T20:38:13.408072+0000","last_fullsized":"2024-09-10T20:38:13.408072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:26.545836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400878+0000","last_change":"2024-09-10T20:38:12.407874+0000","last_active":"2024-09-10T20:38:13.400878+0000","last_peered":"2024-09-10T20:38:13.400878+0000","last_clean":"2024-09-10T20:38:13.400878+0000","last_became_active":"2024-09-10T20:38:12.407623+0000","last_became_peered":"2024-09-10T20:38:12.407623+0000","last_unstale":"2024-09-10T20:38:13.400878+0000","last_undegraded":"2024-09-10T20:38:13.400878+0000","last_fullsized":"2024-09-10T20:38:13.400878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:56.990230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398460+0000","last_change":"2024-09-10T20:38:12.414007+0000","last_active":"2024-09-10T20:38:13.398460+0000","last_peered":"2024-09-10T20:38:13.398460+0000","last_clean":"2024-09-10T20:38:13.398460+0000","last_became_active":"2024-09-10T20:38:12.413744+0000","last_became_peered":"2024-09-10T20:38:12.413744+0000","last_unstale":"2024-09-10T20:38:13.398460+0000","last_undegraded":"2024-09-10T20:38:13.398460+0000","last_fullsized":"2024-09-10T20:38:13.398460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.915632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400518+0000","last_change":"2024-09-10T20:38:12.408020+0000","last_active":"2024-09-10T20:38:13.400518+0000","last_peered":"2024-09-10T20:38:13.400518+0000","last_clean":"2024-09-10T20:38:13.400518+0000","last_became_active":"2024-09-10T20:38:12.407757+0000","last_became_peered":"2024-09-10T20:38:12.407757+0000","last_unstale":"2024-09-10T20:38:13.400518+0000","last_undegraded":"2024-09-10T20:38:13.400518+0000","last_fullsized":"2024-09-10T20:38:13.400518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:32.513086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343533+0000","last_change":"2024-09-10T20:38:12.407235+0000","last_active":"2024-09-10T20:38:14.343533+0000","last_peered":"2024-09-10T20:38:14.343533+0000","last_clean":"2024-09-10T20:38:14.343533+0000","last_became_active":"2024-09-10T20:38:12.406414+0000","last_became_peered":"2024-09-10T20:38:12.406414+0000","last_unstale":"2024-09-10T20:38:14.343533+0000","last_undegraded":"2024-09-10T20:38:14.343533+0000","last_fullsized":"2024-09-10T20:38:14.343533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:37.571212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401654+0000","last_change":"2024-09-10T20:38:12.416773+0000","last_active":"2024-09-10T20:38:13.401654+0000","last_peered":"2024-09-10T20:38:13.401654+0000","last_clean":"2024-09-10T20:38:13.401654+0000","last_became_active":"2024-09-10T20:38:12.416498+0000","last_became_peered":"2024-09-10T20:38:12.416498+0000","last_unstale":"2024-09-10T20:38:13.401654+0000","last_undegraded":"2024-09-10T20:38:13.401654+0000","last_fullsized":"2024-09-10T20:38:13.401654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:42.492324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344477+0000","last_change":"2024-09-10T20:38:12.407692+0000","last_active":"2024-09-10T20:38:14.344477+0000","last_peered":"2024-09-10T20:38:14.344477+0000","last_clean":"2024-09-10T20:38:14.344477+0000","last_became_active":"2024-09-10T20:38:12.406817+0000","last_became_peered":"2024-09-10T20:38:12.406817+0000","last_unstale":"2024-09-10T20:38:14.344477+0000","last_undegraded":"2024-09-10T20:38:14.344477+0000","last_fullsized":"2024-09-10T20:38:14.344477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:13.317965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343671+0000","last_change":"2024-09-10T20:38:12.406673+0000","last_active":"2024-09-10T20:38:14.343671+0000","last_peered":"2024-09-10T20:38:14.343671+0000","last_clean":"2024-09-10T20:38:14.343671+0000","last_became_active":"2024-09-10T20:38:12.406401+0000","last_became_peered":"2024-09-10T20:38:12.406401+0000","last_unstale":"2024-09-10T20:38:14.343671+0000","last_undegraded":"2024-09-10T20:38:14.343671+0000","last_fullsized":"2024-09-10T20:38:14.343671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:31.845414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343502+0000","last_change":"2024-09-10T20:38:12.414290+0000","last_active":"2024-09-10T20:38:14.343502+0000","last_peered":"2024-09-10T20:38:14.343502+0000","last_clean":"2024-09-10T20:38:14.343502+0000","last_became_active":"2024-09-10T20:38:12.413712+0000","last_became_peered":"2024-09-10T20:38:12.413712+0000","last_unstale":"2024-09-10T20:38:14.343502+0000","last_undegraded":"2024-09-10T20:38:14.343502+0000","last_fullsized":"2024-09-10T20:38:14.343502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:48.033101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400477+0000","last_change":"2024-09-10T20:38:12.406921+0000","last_active":"2024-09-10T20:38:13.400477+0000","last_peered":"2024-09-10T20:38:13.400477+0000","last_clean":"2024-09-10T20:38:13.400477+0000","last_became_active":"2024-09-10T20:38:12.406642+0000","last_became_peered":"2024-09-10T20:38:12.406642+0000","last_unstale":"2024-09-10T20:38:13.400477+0000","last_undegraded":"2024-09-10T20:38:13.400477+0000","last_fullsized":"2024-09-10T20:38:13.400477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:02.877370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343808+0000","last_change":"2024-09-10T20:38:12.406428+0000","last_active":"2024-09-10T20:38:14.343808+0000","last_peered":"2024-09-10T20:38:14.343808+0000","last_clean":"2024-09-10T20:38:14.343808+0000","last_became_active":"2024-09-10T20:38:12.405117+0000","last_became_peered":"2024-09-10T20:38:12.405117+0000","last_unstale":"2024-09-10T20:38:14.343808+0000","last_undegraded":"2024-09-10T20:38:14.343808+0000","last_fullsized":"2024-09-10T20:38:14.343808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:28.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633368+0000","last_change":"2024-09-10T20:38:12.415776+0000","last_active":"2024-09-10T20:38:13.633368+0000","last_peered":"2024-09-10T20:38:13.633368+0000","last_clean":"2024-09-10T20:38:13.633368+0000","last_became_active":"2024-09-10T20:38:12.415480+0000","last_became_peered":"2024-09-10T20:38:12.415480+0000","last_unstale":"2024-09-10T20:38:13.633368+0000","last_undegraded":"2024-09-10T20:38:13.633368+0000","last_fullsized":"2024-09-10T20:38:13.633368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:13.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343528+0000","last_change":"2024-09-10T20:38:12.405334+0000","last_active":"2024-09-10T20:38:14.343528+0000","last_peered":"2024-09-10T20:38:14.343528+0000","last_clean":"2024-09-10T20:38:14.343528+0000","last_became_active":"2024-09-10T20:38:12.405033+0000","last_became_peered":"2024-09-10T20:38:12.405033+0000","last_unstale":"2024-09-10T20:38:14.343528+0000","last_undegraded":"2024-09-10T20:38:14.343528+0000","last_fullsized":"2024-09-10T20:38:14.343528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:01:45.285223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398727+0000","last_change":"2024-09-10T20:38:12.412798+0000","last_active":"2024-09-10T20:38:13.398727+0000","last_peered":"2024-09-10T20:38:13.398727+0000","last_clean":"2024-09-10T20:38:13.398727+0000","last_became_active":"2024-09-10T20:38:12.412428+0000","last_became_peered":"2024-09-10T20:38:12.412428+0000","last_unstale":"2024-09-10T20:38:13.398727+0000","last_undegraded":"2024-09-10T20:38:13.398727+0000","last_fullsized":"2024-09-10T20:38:13.398727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:45.518334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400454+0000","last_change":"2024-09-10T20:38:11.394869+0000","last_active":"2024-09-10T20:38:13.400454+0000","last_peered":"2024-09-10T20:38:13.400454+0000","last_clean":"2024-09-10T20:38:13.400454+0000","last_became_active":"2024-09-10T20:38:11.394564+0000","last_became_peered":"2024-09-10T20:38:11.394564+0000","last_unstale":"2024-09-10T20:38:13.400454+0000","last_undegraded":"2024-09-10T20:38:13.400454+0000","last_fullsized":"2024-09-10T20:38:13.400454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:19.365550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401125+0000","last_change":"2024-09-10T20:38:12.401521+0000","last_active":"2024-09-10T20:38:13.401125+0000","last_peered":"2024-09-10T20:38:13.401125+0000","last_clean":"2024-09-10T20:38:13.401125+0000","last_became_active":"2024-09-10T20:38:12.401218+0000","last_became_peered":"2024-09-10T20:38:12.401218+0000","last_unstale":"2024-09-10T20:38:13.401125+0000","last_undegraded":"2024-09-10T20:38:13.401125+0000","last_fullsized":"2024-09-10T20:38:13.401125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:33.954998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343882+0000","last_change":"2024-09-10T20:38:11.401029+0000","last_active":"2024-09-10T20:38:14.343882+0000","last_peered":"2024-09-10T20:38:14.343882+0000","last_clean":"2024-09-10T20:38:14.343882+0000","last_became_active":"2024-09-10T20:38:11.400647+0000","last_became_peered":"2024-09-10T20:38:11.400647+0000","last_unstale":"2024-09-10T20:38:14.343882+0000","last_undegraded":"2024-09-10T20:38:14.343882+0000","last_fullsized":"2024-09-10T20:38:14.343882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:38.998000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343167+0000","last_change":"2024-09-10T20:38:11.398400+0000","last_active":"2024-09-10T20:38:14.343167+0000","last_peered":"2024-09-10T20:38:14.343167+0000","last_clean":"2024-09-10T20:38:14.343167+0000","last_became_active":"2024-09-10T20:38:11.398097+0000","last_became_peered":"2024-09-10T20:38:11.398097+0000","last_unstale":"2024-09-10T20:38:14.343167+0000","last_undegraded":"2024-09-10T20:38:14.343167+0000","last_fullsized":"2024-09-10T20:38:14.343167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.599838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401478+0000","last_change":"2024-09-10T20:38:12.413177+0000","last_active":"2024-09-10T20:38:13.401478+0000","last_peered":"2024-09-10T20:38:13.401478+0000","last_clean":"2024-09-10T20:38:13.401478+0000","last_became_active":"2024-09-10T20:38:12.412925+0000","last_became_peered":"2024-09-10T20:38:12.412925+0000","last_unstale":"2024-09-10T20:38:13.401478+0000","last_undegraded":"2024-09-10T20:38:13.401478+0000","last_fullsized":"2024-09-10T20:38:13.401478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:18.693832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343616+0000","last_change":"2024-09-10T20:38:11.399996+0000","last_active":"2024-09-10T20:38:14.343616+0000","last_peered":"2024-09-10T20:38:14.343616+0000","last_clean":"2024-09-10T20:38:14.343616+0000","last_became_active":"2024-09-10T20:38:11.399423+0000","last_became_peered":"2024-09-10T20:38:11.399423+0000","last_unstale":"2024-09-10T20:38:14.343616+0000","last_undegraded":"2024-09-10T20:38:14.343616+0000","last_fullsized":"2024-09-10T20:38:14.343616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:35.628735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401415+0000","last_change":"2024-09-10T20:38:12.415606+0000","last_active":"2024-09-10T20:38:13.401415+0000","last_peered":"2024-09-10T20:38:13.401415+0000","last_clean":"2024-09-10T20:38:13.401415+0000","last_became_active":"2024-09-10T20:38:12.415330+0000","last_became_peered":"2024-09-10T20:38:12.415330+0000","last_unstale":"2024-09-10T20:38:13.401415+0000","last_undegraded":"2024-09-10T20:38:13.401415+0000","last_fullsized":"2024-09-10T20:38:13.401415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:53.050014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398942+0000","last_change":"2024-09-10T20:38:11.395121+0000","last_active":"2024-09-10T20:38:13.398942+0000","last_peered":"2024-09-10T20:38:13.398942+0000","last_clean":"2024-09-10T20:38:13.398942+0000","last_became_active":"2024-09-10T20:38:11.394490+0000","last_became_peered":"2024-09-10T20:38:11.394490+0000","last_unstale":"2024-09-10T20:38:13.398942+0000","last_undegraded":"2024-09-10T20:38:13.398942+0000","last_fullsized":"2024-09-10T20:38:13.398942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:42.485721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344515+0000","last_change":"2024-09-10T20:38:12.402641+0000","last_active":"2024-09-10T20:38:14.344515+0000","last_peered":"2024-09-10T20:38:14.344515+0000","last_clean":"2024-09-10T20:38:14.344515+0000","last_became_active":"2024-09-10T20:38:12.402308+0000","last_became_peered":"2024-09-10T20:38:12.402308+0000","last_unstale":"2024-09-10T20:38:14.344515+0000","last_undegraded":"2024-09-10T20:38:14.344515+0000","last_fullsized":"2024-09-10T20:38:14.344515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408013+0000","last_change":"2024-09-10T20:38:11.396878+0000","last_active":"2024-09-10T20:38:13.408013+0000","last_peered":"2024-09-10T20:38:13.408013+0000","last_clean":"2024-09-10T20:38:13.408013+0000","last_became_active":"2024-09-10T20:38:11.396522+0000","last_became_peered":"2024-09-10T20:38:11.396522+0000","last_unstale":"2024-09-10T20:38:13.408013+0000","last_undegraded":"2024-09-10T20:38:13.408013+0000","last_fullsized":"2024-09-10T20:38:13.408013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:43.005710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401902+0000","last_change":"2024-09-10T20:38:12.410504+0000","last_active":"2024-09-10T20:38:13.401902+0000","last_peered":"2024-09-10T20:38:13.401902+0000","last_clean":"2024-09-10T20:38:13.401902+0000","last_became_active":"2024-09-10T20:38:12.410008+0000","last_became_peered":"2024-09-10T20:38:12.410008+0000","last_unstale":"2024-09-10T20:38:13.401902+0000","last_undegraded":"2024-09-10T20:38:13.401902+0000","last_fullsized":"2024-09-10T20:38:13.401902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:57.544501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633468+0000","last_change":"2024-09-10T20:38:11.395962+0000","last_active":"2024-09-10T20:38:13.633468+0000","last_peered":"2024-09-10T20:38:13.633468+0000","last_clean":"2024-09-10T20:38:13.633468+0000","last_became_active":"2024-09-10T20:38:11.395687+0000","last_became_peered":"2024-09-10T20:38:11.395687+0000","last_unstale":"2024-09-10T20:38:13.633468+0000","last_undegraded":"2024-09-10T20:38:13.633468+0000","last_fullsized":"2024-09-10T20:38:13.633468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:15.421827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344171+0000","last_change":"2024-09-10T20:38:12.405652+0000","last_active":"2024-09-10T20:38:14.344171+0000","last_peered":"2024-09-10T20:38:14.344171+0000","last_clean":"2024-09-10T20:38:14.344171+0000","last_became_active":"2024-09-10T20:38:12.405250+0000","last_became_peered":"2024-09-10T20:38:12.405250+0000","last_unstale":"2024-09-10T20:38:14.344171+0000","last_undegraded":"2024-09-10T20:38:14.344171+0000","last_fullsized":"2024-09-10T20:38:14.344171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:10.251489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343906+0000","last_change":"2024-09-10T20:38:11.387993+0000","last_active":"2024-09-10T20:38:14.343906+0000","last_peered":"2024-09-10T20:38:14.343906+0000","last_clean":"2024-09-10T20:38:14.343906+0000","last_became_active":"2024-09-10T20:38:11.387277+0000","last_became_peered":"2024-09-10T20:38:11.387277+0000","last_unstale":"2024-09-10T20:38:14.343906+0000","last_undegraded":"2024-09-10T20:38:14.343906+0000","last_fullsized":"2024-09-10T20:38:14.343906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:51.949339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408453+0000","last_change":"2024-09-10T20:38:12.412762+0000","last_active":"2024-09-10T20:38:13.408453+0000","last_peered":"2024-09-10T20:38:13.408453+0000","last_clean":"2024-09-10T20:38:13.408453+0000","last_became_active":"2024-09-10T20:38:12.412480+0000","last_became_peered":"2024-09-10T20:38:12.412480+0000","last_unstale":"2024-09-10T20:38:13.408453+0000","last_undegraded":"2024-09-10T20:38:13.408453+0000","last_fullsized":"2024-09-10T20:38:13.408453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633539+0000","last_change":"2024-09-10T20:38:11.398330+0000","last_active":"2024-09-10T20:38:13.633539+0000","last_peered":"2024-09-10T20:38:13.633539+0000","last_clean":"2024-09-10T20:38:13.633539+0000","last_became_active":"2024-09-10T20:38:11.398059+0000","last_became_peered":"2024-09-10T20:38:11.398059+0000","last_unstale":"2024-09-10T20:38:13.633539+0000","last_undegraded":"2024-09-10T20:38:13.633539+0000","last_fullsized":"2024-09-10T20:38:13.633539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343446+0000","last_change":"2024-09-10T20:38:12.412484+0000","last_active":"2024-09-10T20:38:14.343446+0000","last_peered":"2024-09-10T20:38:14.343446+0000","last_clean":"2024-09-10T20:38:14.343446+0000","last_became_active":"2024-09-10T20:38:12.412165+0000","last_became_peered":"2024-09-10T20:38:12.412165+0000","last_unstale":"2024-09-10T20:38:14.343446+0000","last_undegraded":"2024-09-10T20:38:14.343446+0000","last_fullsized":"2024-09-10T20:38:14.343446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:09.972659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344285+0000","last_change":"2024-09-10T20:38:11.400681+0000","last_active":"2024-09-10T20:38:14.344285+0000","last_peered":"2024-09-10T20:38:14.344285+0000","last_clean":"2024-09-10T20:38:14.344285+0000","last_became_active":"2024-09-10T20:38:11.400168+0000","last_became_peered":"2024-09-10T20:38:11.400168+0000","last_unstale":"2024-09-10T20:38:14.344285+0000","last_undegraded":"2024-09-10T20:38:14.344285+0000","last_fullsized":"2024-09-10T20:38:14.344285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:43.005710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344386+0000","last_change":"2024-09-10T20:38:12.413508+0000","last_active":"2024-09-10T20:38:14.344386+0000","last_peered":"2024-09-10T20:38:14.344386+0000","last_clean":"2024-09-10T20:38:14.344386+0000","last_became_active":"2024-09-10T20:38:12.413230+0000","last_became_peered":"2024-09-10T20:38:12.413230+0000","last_unstale":"2024-09-10T20:38:14.344386+0000","last_undegraded":"2024-09-10T20:38:14.344386+0000","last_fullsized":"2024-09-10T20:38:14.344386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:56.070087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407691+0000","last_change":"2024-09-10T20:38:11.402870+0000","last_active":"2024-09-10T20:38:13.407691+0000","last_peered":"2024-09-10T20:38:13.407691+0000","last_clean":"2024-09-10T20:38:13.407691+0000","last_became_active":"2024-09-10T20:38:11.402580+0000","last_became_peered":"2024-09-10T20:38:11.402580+0000","last_unstale":"2024-09-10T20:38:13.407691+0000","last_undegraded":"2024-09-10T20:38:13.407691+0000","last_fullsized":"2024-09-10T20:38:13.407691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:50:23.981654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398438+0000","last_change":"2024-09-10T20:38:12.416114+0000","last_active":"2024-09-10T20:38:13.398438+0000","last_peered":"2024-09-10T20:38:13.398438+0000","last_clean":"2024-09-10T20:38:13.398438+0000","last_became_active":"2024-09-10T20:38:12.415605+0000","last_became_peered":"2024-09-10T20:38:12.415605+0000","last_unstale":"2024-09-10T20:38:13.398438+0000","last_undegraded":"2024-09-10T20:38:13.398438+0000","last_fullsized":"2024-09-10T20:38:13.398438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:01:45.285223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400564+0000","last_change":"2024-09-10T20:38:11.393414+0000","last_active":"2024-09-10T20:38:13.400564+0000","last_peered":"2024-09-10T20:38:13.400564+0000","last_clean":"2024-09-10T20:38:13.400564+0000","last_became_active":"2024-09-10T20:38:11.393078+0000","last_became_peered":"2024-09-10T20:38:11.393078+0000","last_unstale":"2024-09-10T20:38:13.400564+0000","last_undegraded":"2024-09-10T20:38:13.400564+0000","last_fullsized":"2024-09-10T20:38:13.400564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:58.100556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343642+0000","last_change":"2024-09-10T20:38:12.407378+0000","last_active":"2024-09-10T20:38:14.343642+0000","last_peered":"2024-09-10T20:38:14.343642+0000","last_clean":"2024-09-10T20:38:14.343642+0000","last_became_active":"2024-09-10T20:38:12.407092+0000","last_became_peered":"2024-09-10T20:38:12.407092+0000","last_unstale":"2024-09-10T20:38:14.343642+0000","last_undegraded":"2024-09-10T20:38:14.343642+0000","last_fullsized":"2024-09-10T20:38:14.343642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:47.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408782+0000","last_change":"2024-09-10T20:38:11.397628+0000","last_active":"2024-09-10T20:38:13.408782+0000","last_peered":"2024-09-10T20:38:13.408782+0000","last_clean":"2024-09-10T20:38:13.408782+0000","last_became_active":"2024-09-10T20:38:11.397316+0000","last_became_peered":"2024-09-10T20:38:11.397316+0000","last_unstale":"2024-09-10T20:38:13.408782+0000","last_undegraded":"2024-09-10T20:38:13.408782+0000","last_fullsized":"2024-09-10T20:38:13.408782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:07.558503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344717+0000","last_change":"2024-09-10T20:38:12.396278+0000","last_active":"2024-09-10T20:38:14.344717+0000","last_peered":"2024-09-10T20:38:14.344717+0000","last_clean":"2024-09-10T20:38:14.344717+0000","last_became_active":"2024-09-10T20:38:12.395937+0000","last_became_peered":"2024-09-10T20:38:12.395937+0000","last_unstale":"2024-09-10T20:38:14.344717+0000","last_undegraded":"2024-09-10T20:38:14.344717+0000","last_fullsized":"2024-09-10T20:38:14.344717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:06.451633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401433+0000","last_change":"2024-09-10T20:38:11.388247+0000","last_active":"2024-09-10T20:38:13.401433+0000","last_peered":"2024-09-10T20:38:13.401433+0000","last_clean":"2024-09-10T20:38:13.401433+0000","last_became_active":"2024-09-10T20:38:11.387924+0000","last_became_peered":"2024-09-10T20:38:11.387924+0000","last_unstale":"2024-09-10T20:38:13.401433+0000","last_undegraded":"2024-09-10T20:38:13.401433+0000","last_fullsized":"2024-09-10T20:38:13.401433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:29.584118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408756+0000","last_change":"2024-09-10T20:38:12.418652+0000","last_active":"2024-09-10T20:38:13.408756+0000","last_peered":"2024-09-10T20:38:13.408756+0000","last_clean":"2024-09-10T20:38:13.408756+0000","last_became_active":"2024-09-10T20:38:12.418400+0000","last_became_peered":"2024-09-10T20:38:12.418400+0000","last_unstale":"2024-09-10T20:38:13.408756+0000","last_undegraded":"2024-09-10T20:38:13.408756+0000","last_fullsized":"2024-09-10T20:38:13.408756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:28.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400616+0000","last_change":"2024-09-10T20:38:11.399015+0000","last_active":"2024-09-10T20:38:13.400616+0000","last_peered":"2024-09-10T20:38:13.400616+0000","last_clean":"2024-09-10T20:38:13.400616+0000","last_became_active":"2024-09-10T20:38:11.398826+0000","last_became_peered":"2024-09-10T20:38:11.398826+0000","last_unstale":"2024-09-10T20:38:13.400616+0000","last_undegraded":"2024-09-10T20:38:13.400616+0000","last_fullsized":"2024-09-10T20:38:13.400616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:03.958637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408168+0000","last_change":"2024-09-10T20:38:12.411663+0000","last_active":"2024-09-10T20:38:13.408168+0000","last_peered":"2024-09-10T20:38:13.408168+0000","last_clean":"2024-09-10T20:38:13.408168+0000","last_became_active":"2024-09-10T20:38:12.411269+0000","last_became_peered":"2024-09-10T20:38:12.411269+0000","last_unstale":"2024-09-10T20:38:13.408168+0000","last_undegraded":"2024-09-10T20:38:13.408168+0000","last_fullsized":"2024-09-10T20:38:13.408168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:20.639033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343682+0000","last_change":"2024-09-10T20:38:11.398337+0000","last_active":"2024-09-10T20:38:14.343682+0000","last_peered":"2024-09-10T20:38:14.343682+0000","last_clean":"2024-09-10T20:38:14.343682+0000","last_became_active":"2024-09-10T20:38:11.398068+0000","last_became_peered":"2024-09-10T20:38:11.398068+0000","last_unstale":"2024-09-10T20:38:14.343682+0000","last_undegraded":"2024-09-10T20:38:14.343682+0000","last_fullsized":"2024-09-10T20:38:14.343682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:00:28.236504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344836+0000","last_change":"2024-09-10T20:38:12.420411+0000","last_active":"2024-09-10T20:38:14.344836+0000","last_peered":"2024-09-10T20:38:14.344836+0000","last_clean":"2024-09-10T20:38:14.344836+0000","last_became_active":"2024-09-10T20:38:12.419107+0000","last_became_peered":"2024-09-10T20:38:12.419107+0000","last_unstale":"2024-09-10T20:38:14.344836+0000","last_undegraded":"2024-09-10T20:38:14.344836+0000","last_fullsized":"2024-09-10T20:38:14.344836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:15.534463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398277+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:13.398277+0000","last_peered":"2024-09-10T20:38:13.398277+0000","last_clean":"2024-09-10T20:38:13.398277+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:13.398277+0000","last_undegraded":"2024-09-10T20:38:13.398277+0000","last_fullsized":"2024-09-10T20:38:13.398277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344545+0000","last_change":"2024-09-10T20:38:11.398527+0000","last_active":"2024-09-10T20:38:14.344545+0000","last_peered":"2024-09-10T20:38:14.344545+0000","last_clean":"2024-09-10T20:38:14.344545+0000","last_became_active":"2024-09-10T20:38:11.395195+0000","last_became_peered":"2024-09-10T20:38:11.395195+0000","last_unstale":"2024-09-10T20:38:14.344545+0000","last_undegraded":"2024-09-10T20:38:14.344545+0000","last_fullsized":"2024-09-10T20:38:14.344545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:35.618612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400902+0000","last_change":"2024-09-10T20:38:12.407263+0000","last_active":"2024-09-10T20:38:13.400902+0000","last_peered":"2024-09-10T20:38:13.400902+0000","last_clean":"2024-09-10T20:38:13.400902+0000","last_became_active":"2024-09-10T20:38:12.406952+0000","last_became_peered":"2024-09-10T20:38:12.406952+0000","last_unstale":"2024-09-10T20:38:13.400902+0000","last_undegraded":"2024-09-10T20:38:13.400902+0000","last_fullsized":"2024-09-10T20:38:13.400902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.171100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343670+0000","last_change":"2024-09-10T20:38:11.399685+0000","last_active":"2024-09-10T20:38:14.343670+0000","last_peered":"2024-09-10T20:38:14.343670+0000","last_clean":"2024-09-10T20:38:14.343670+0000","last_became_active":"2024-09-10T20:38:11.399073+0000","last_became_peered":"2024-09-10T20:38:11.399073+0000","last_unstale":"2024-09-10T20:38:14.343670+0000","last_undegraded":"2024-09-10T20:38:14.343670+0000","last_fullsized":"2024-09-10T20:38:14.343670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:08.967765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343358+0000","last_change":"2024-09-10T20:38:12.407025+0000","last_active":"2024-09-10T20:38:14.343358+0000","last_peered":"2024-09-10T20:38:14.343358+0000","last_clean":"2024-09-10T20:38:14.343358+0000","last_became_active":"2024-09-10T20:38:12.406124+0000","last_became_peered":"2024-09-10T20:38:12.406124+0000","last_unstale":"2024-09-10T20:38:14.343358+0000","last_undegraded":"2024-09-10T20:38:14.343358+0000","last_fullsized":"2024-09-10T20:38:14.343358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:36.633629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633299+0000","last_change":"2024-09-10T20:38:11.397217+0000","last_active":"2024-09-10T20:38:13.633299+0000","last_peered":"2024-09-10T20:38:13.633299+0000","last_clean":"2024-09-10T20:38:13.633299+0000","last_became_active":"2024-09-10T20:38:11.396928+0000","last_became_peered":"2024-09-10T20:38:11.396928+0000","last_unstale":"2024-09-10T20:38:13.633299+0000","last_undegraded":"2024-09-10T20:38:13.633299+0000","last_fullsized":"2024-09-10T20:38:13.633299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:38.900087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401469+0000","last_change":"2024-09-10T20:38:12.407595+0000","last_active":"2024-09-10T20:38:13.401469+0000","last_peered":"2024-09-10T20:38:13.401469+0000","last_clean":"2024-09-10T20:38:13.401469+0000","last_became_active":"2024-09-10T20:38:12.407355+0000","last_became_peered":"2024-09-10T20:38:12.407355+0000","last_unstale":"2024-09-10T20:38:13.401469+0000","last_undegraded":"2024-09-10T20:38:13.401469+0000","last_fullsized":"2024-09-10T20:38:13.401469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:05.419998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344389+0000","last_change":"2024-09-10T20:38:11.401377+0000","last_active":"2024-09-10T20:38:14.344389+0000","last_peered":"2024-09-10T20:38:14.344389+0000","last_clean":"2024-09-10T20:38:14.344389+0000","last_became_active":"2024-09-10T20:38:11.401048+0000","last_became_peered":"2024-09-10T20:38:11.401048+0000","last_unstale":"2024-09-10T20:38:14.344389+0000","last_undegraded":"2024-09-10T20:38:14.344389+0000","last_fullsized":"2024-09-10T20:38:14.344389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:17.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408141+0000","last_change":"2024-09-10T20:38:12.399857+0000","last_active":"2024-09-10T20:38:13.408141+0000","last_peered":"2024-09-10T20:38:13.408141+0000","last_clean":"2024-09-10T20:38:13.408141+0000","last_became_active":"2024-09-10T20:38:12.399121+0000","last_became_peered":"2024-09-10T20:38:12.399121+0000","last_unstale":"2024-09-10T20:38:13.408141+0000","last_undegraded":"2024-09-10T20:38:13.408141+0000","last_fullsized":"2024-09-10T20:38:13.408141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:23.782139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400255+0000","last_change":"2024-09-10T20:38:11.397463+0000","last_active":"2024-09-10T20:38:13.400255+0000","last_peered":"2024-09-10T20:38:13.400255+0000","last_clean":"2024-09-10T20:38:13.400255+0000","last_became_active":"2024-09-10T20:38:11.397205+0000","last_became_peered":"2024-09-10T20:38:11.397205+0000","last_unstale":"2024-09-10T20:38:13.400255+0000","last_undegraded":"2024-09-10T20:38:13.400255+0000","last_fullsized":"2024-09-10T20:38:13.400255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:53.003074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343801+0000","last_change":"2024-09-10T20:38:12.412633+0000","last_active":"2024-09-10T20:38:14.343801+0000","last_peered":"2024-09-10T20:38:14.343801+0000","last_clean":"2024-09-10T20:38:14.343801+0000","last_became_active":"2024-09-10T20:38:12.412350+0000","last_became_peered":"2024-09-10T20:38:12.412350+0000","last_unstale":"2024-09-10T20:38:14.343801+0000","last_undegraded":"2024-09-10T20:38:14.343801+0000","last_fullsized":"2024-09-10T20:38:14.343801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.915632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344655+0000","last_change":"2024-09-10T20:38:11.392177+0000","last_active":"2024-09-10T20:38:14.344655+0000","last_peered":"2024-09-10T20:38:14.344655+0000","last_clean":"2024-09-10T20:38:14.344655+0000","last_became_active":"2024-09-10T20:38:11.390174+0000","last_became_peered":"2024-09-10T20:38:11.390174+0000","last_unstale":"2024-09-10T20:38:14.344655+0000","last_undegraded":"2024-09-10T20:38:14.344655+0000","last_fullsized":"2024-09-10T20:38:14.344655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:29.078056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343576+0000","last_change":"2024-09-10T20:38:12.408779+0000","last_active":"2024-09-10T20:38:14.343576+0000","last_peered":"2024-09-10T20:38:14.343576+0000","last_clean":"2024-09-10T20:38:14.343576+0000","last_became_active":"2024-09-10T20:38:12.408490+0000","last_became_peered":"2024-09-10T20:38:12.408490+0000","last_unstale":"2024-09-10T20:38:14.343576+0000","last_undegraded":"2024-09-10T20:38:14.343576+0000","last_fullsized":"2024-09-10T20:38:14.343576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:25.888623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407174+0000","last_change":"2024-09-10T20:38:11.394170+0000","last_active":"2024-09-10T20:38:13.407174+0000","last_peered":"2024-09-10T20:38:13.407174+0000","last_clean":"2024-09-10T20:38:13.407174+0000","last_became_active":"2024-09-10T20:38:11.393850+0000","last_became_peered":"2024-09-10T20:38:11.393850+0000","last_unstale":"2024-09-10T20:38:13.407174+0000","last_undegraded":"2024-09-10T20:38:13.407174+0000","last_fullsized":"2024-09-10T20:38:13.407174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:15.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401623+0000","last_change":"2024-09-10T20:38:12.404386+0000","last_active":"2024-09-10T20:38:13.401623+0000","last_peered":"2024-09-10T20:38:13.401623+0000","last_clean":"2024-09-10T20:38:13.401623+0000","last_became_active":"2024-09-10T20:38:12.404045+0000","last_became_peered":"2024-09-10T20:38:12.404045+0000","last_unstale":"2024-09-10T20:38:13.401623+0000","last_undegraded":"2024-09-10T20:38:13.401623+0000","last_fullsized":"2024-09-10T20:38:13.401623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:49.410849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344714+0000","last_change":"2024-09-10T20:38:11.399979+0000","last_active":"2024-09-10T20:38:14.344714+0000","last_peered":"2024-09-10T20:38:14.344714+0000","last_clean":"2024-09-10T20:38:14.344714+0000","last_became_active":"2024-09-10T20:38:11.399682+0000","last_became_peered":"2024-09-10T20:38:11.399682+0000","last_unstale":"2024-09-10T20:38:14.344714+0000","last_undegraded":"2024-09-10T20:38:14.344714+0000","last_fullsized":"2024-09-10T20:38:14.344714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:40.818316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400970+0000","last_change":"2024-09-10T20:38:12.409091+0000","last_active":"2024-09-10T20:38:13.400970+0000","last_peered":"2024-09-10T20:38:13.400970+0000","last_clean":"2024-09-10T20:38:13.400970+0000","last_became_active":"2024-09-10T20:38:12.408815+0000","last_became_peered":"2024-09-10T20:38:12.408815+0000","last_unstale":"2024-09-10T20:38:13.400970+0000","last_undegraded":"2024-09-10T20:38:13.400970+0000","last_fullsized":"2024-09-10T20:38:13.400970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.870149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.407496+0000","last_change":"2024-09-10T20:38:11.391306+0000","last_active":"2024-09-10T20:38:13.407496+0000","last_peered":"2024-09-10T20:38:13.407496+0000","last_clean":"2024-09-10T20:38:13.407496+0000","last_became_active":"2024-09-10T20:38:11.390956+0000","last_became_peered":"2024-09-10T20:38:11.390956+0000","last_unstale":"2024-09-10T20:38:13.407496+0000","last_undegraded":"2024-09-10T20:38:13.407496+0000","last_fullsized":"2024-09-10T20:38:13.407496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:12:00.829218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344085+0000","last_change":"2024-09-10T20:38:12.402970+0000","last_active":"2024-09-10T20:38:14.344085+0000","last_peered":"2024-09-10T20:38:14.344085+0000","last_clean":"2024-09-10T20:38:14.344085+0000","last_became_active":"2024-09-10T20:38:12.402393+0000","last_became_peered":"2024-09-10T20:38:12.402393+0000","last_unstale":"2024-09-10T20:38:14.344085+0000","last_undegraded":"2024-09-10T20:38:14.344085+0000","last_fullsized":"2024-09-10T20:38:14.344085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:09.528282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401287+0000","last_change":"2024-09-10T20:38:11.397172+0000","last_active":"2024-09-10T20:38:13.401287+0000","last_peered":"2024-09-10T20:38:13.401287+0000","last_clean":"2024-09-10T20:38:13.401287+0000","last_became_active":"2024-09-10T20:38:11.396670+0000","last_became_peered":"2024-09-10T20:38:11.396670+0000","last_unstale":"2024-09-10T20:38:13.401287+0000","last_undegraded":"2024-09-10T20:38:13.401287+0000","last_fullsized":"2024-09-10T20:38:13.401287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:14.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.400195+0000","last_change":"2024-09-10T20:38:12.405875+0000","last_active":"2024-09-10T20:38:13.400195+0000","last_peered":"2024-09-10T20:38:13.400195+0000","last_clean":"2024-09-10T20:38:13.400195+0000","last_became_active":"2024-09-10T20:38:12.405583+0000","last_became_peered":"2024-09-10T20:38:12.405583+0000","last_unstale":"2024-09-10T20:38:13.400195+0000","last_undegraded":"2024-09-10T20:38:13.400195+0000","last_fullsized":"2024-09-10T20:38:13.400195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:30.185620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.343471+0000","last_change":"2024-09-10T20:38:11.395992+0000","last_active":"2024-09-10T20:38:14.343471+0000","last_peered":"2024-09-10T20:38:14.343471+0000","last_clean":"2024-09-10T20:38:14.343471+0000","last_became_active":"2024-09-10T20:38:11.394935+0000","last_became_peered":"2024-09-10T20:38:11.394935+0000","last_unstale":"2024-09-10T20:38:14.343471+0000","last_undegraded":"2024-09-10T20:38:14.343471+0000","last_fullsized":"2024-09-10T20:38:14.343471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:49.040222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.633574+0000","last_change":"2024-09-10T20:38:12.412620+0000","last_active":"2024-09-10T20:38:13.633574+0000","last_peered":"2024-09-10T20:38:13.633574+0000","last_clean":"2024-09-10T20:38:13.633574+0000","last_became_active":"2024-09-10T20:38:12.412329+0000","last_became_peered":"2024-09-10T20:38:12.412329+0000","last_unstale":"2024-09-10T20:38:13.633574+0000","last_undegraded":"2024-09-10T20:38:13.633574+0000","last_fullsized":"2024-09-10T20:38:13.633574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:40.609805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344474+0000","last_change":"2024-09-10T20:38:11.398962+0000","last_active":"2024-09-10T20:38:14.344474+0000","last_peered":"2024-09-10T20:38:14.344474+0000","last_clean":"2024-09-10T20:38:14.344474+0000","last_became_active":"2024-09-10T20:38:11.398617+0000","last_became_peered":"2024-09-10T20:38:11.398617+0000","last_unstale":"2024-09-10T20:38:14.344474+0000","last_undegraded":"2024-09-10T20:38:14.344474+0000","last_fullsized":"2024-09-10T20:38:14.344474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:47.028207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344204+0000","last_change":"2024-09-10T20:38:12.418199+0000","last_active":"2024-09-10T20:38:14.344204+0000","last_peered":"2024-09-10T20:38:14.344204+0000","last_clean":"2024-09-10T20:38:14.344204+0000","last_became_active":"2024-09-10T20:38:12.417684+0000","last_became_peered":"2024-09-10T20:38:12.417684+0000","last_unstale":"2024-09-10T20:38:14.344204+0000","last_undegraded":"2024-09-10T20:38:14.344204+0000","last_fullsized":"2024-09-10T20:38:14.344204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:27.561396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:14.344345+0000","last_change":"2024-09-10T20:38:12.406004+0000","last_active":"2024-09-10T20:38:14.344345+0000","last_peered":"2024-09-10T20:38:14.344345+0000","last_clean":"2024-09-10T20:38:14.344345+0000","last_became_active":"2024-09-10T20:38:12.405336+0000","last_became_peered":"2024-09-10T20:38:12.405336+0000","last_unstale":"2024-09-10T20:38:14.344345+0000","last_undegraded":"2024-09-10T20:38:14.344345+0000","last_fullsized":"2024-09-10T20:38:14.344345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:58:35.400897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408136+0000","last_change":"2024-09-10T20:38:11.396962+0000","last_active":"2024-09-10T20:38:13.408136+0000","last_peered":"2024-09-10T20:38:13.408136+0000","last_clean":"2024-09-10T20:38:13.408136+0000","last_became_active":"2024-09-10T20:38:11.396650+0000","last_became_peered":"2024-09-10T20:38:11.396650+0000","last_unstale":"2024-09-10T20:38:13.408136+0000","last_undegraded":"2024-09-10T20:38:13.408136+0000","last_fullsized":"2024-09-10T20:38:13.408136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:40.818316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.398946+0000","last_change":"2024-09-10T20:38:12.413236+0000","last_active":"2024-09-10T20:38:13.398946+0000","last_peered":"2024-09-10T20:38:13.398946+0000","last_clean":"2024-09-10T20:38:13.398946+0000","last_became_active":"2024-09-10T20:38:12.412966+0000","last_became_peered":"2024-09-10T20:38:12.412966+0000","last_unstale":"2024-09-10T20:38:13.398946+0000","last_undegraded":"2024-09-10T20:38:13.398946+0000","last_fullsized":"2024-09-10T20:38:13.398946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:01:29.734867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.408400+0000","last_change":"2024-09-10T20:38:11.397755+0000","last_active":"2024-09-10T20:38:13.408400+0000","last_peered":"2024-09-10T20:38:13.408400+0000","last_clean":"2024-09-10T20:38:13.408400+0000","last_became_active":"2024-09-10T20:38:11.397450+0000","last_became_peered":"2024-09-10T20:38:11.397450+0000","last_unstale":"2024-09-10T20:38:13.408400+0000","last_undegraded":"2024-09-10T20:38:13.408400+0000","last_fullsized":"2024-09-10T20:38:13.408400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:38.998000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401905+0000","last_change":"2024-09-10T20:38:12.416588+0000","last_active":"2024-09-10T20:38:13.401905+0000","last_peered":"2024-09-10T20:38:13.401905+0000","last_clean":"2024-09-10T20:38:13.401905+0000","last_became_active":"2024-09-10T20:38:12.416297+0000","last_became_peered":"2024-09-10T20:38:12.416297+0000","last_unstale":"2024-09-10T20:38:13.401905+0000","last_undegraded":"2024-09-10T20:38:13.401905+0000","last_fullsized":"2024-09-10T20:38:13.401905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:28.970080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401853+0000","last_change":"2024-09-10T20:38:11.396382+0000","last_active":"2024-09-10T20:38:13.401853+0000","last_peered":"2024-09-10T20:38:13.401853+0000","last_clean":"2024-09-10T20:38:13.401853+0000","last_became_active":"2024-09-10T20:38:11.395934+0000","last_became_peered":"2024-09-10T20:38:11.395934+0000","last_unstale":"2024-09-10T20:38:13.401853+0000","last_undegraded":"2024-09-10T20:38:13.401853+0000","last_fullsized":"2024-09-10T20:38:13.401853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:20:39.553068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401634+0000","last_change":"2024-09-10T20:38:12.402624+0000","last_active":"2024-09-10T20:38:13.401634+0000","last_peered":"2024-09-10T20:38:13.401634+0000","last_clean":"2024-09-10T20:38:13.401634+0000","last_became_active":"2024-09-10T20:38:12.401924+0000","last_became_peered":"2024-09-10T20:38:12.401924+0000","last_unstale":"2024-09-10T20:38:13.401634+0000","last_undegraded":"2024-09-10T20:38:13.401634+0000","last_fullsized":"2024-09-10T20:38:13.401634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:11.372745+0000","last_clean_scrub_stamp":"2024-09-10T20:38:11.372745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:33.683744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T20:38:13.401708+0000","last_change":"2024-09-10T20:38:11.401264+0000","last_active":"2024-09-10T20:38:13.401708+0000","last_peered":"2024-09-10T20:38:13.401708+0000","last_clean":"2024-09-10T20:38:13.401708+0000","last_became_active":"2024-09-10T20:38:11.400986+0000","last_became_peered":"2024-09-10T20:38:11.400986+0000","last_unstale":"2024-09-10T20:38:13.401708+0000","last_undegraded":"2024-09-10T20:38:13.401708+0000","last_fullsized":"2024-09-10T20:38:13.401708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:10.367851+0000","last_clean_scrub_stamp":"2024-09-10T20:38:10.367851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:50.668458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":73109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":74139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":806912,"data_stored":662965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":217088,"data_stored":72583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":72527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":662987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":74811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:25.964 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:38:25.964 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T20:38:25.964 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-10T20:38:25.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-10T20:38:26.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 -- 172.21.15.28:0/1362425787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f780d4bb0 msgr2=0x7f8f780d2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1362425787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f780d4bb0 0x7f8f780d2640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8f7806e9e0 tx=0x7f8f6002a440 comp rx=0 tx=0).stop 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 -- 172.21.15.28:0/1362425787 shutdown_connections 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1362425787 >> v2:172.21.15.159:3300/0 conn(0x7f8f7806e530 0x7f8f7806d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1362425787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f780d4bb0 0x7f8f780d2640 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1362425787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f78161790 0x7f8f780d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 -- 172.21.15.28:0/1362425787 >> 172.21.15.28:0/1362425787 conn(0x7f8f780d4090 msgr2=0x7f8f780d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 -- 172.21.15.28:0/1362425787 shutdown_connections 2024-09-10T20:38:26.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.100+0000 7f8f7d964640 1 -- 172.21.15.28:0/1362425787 wait complete. 2024-09-10T20:38:26.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.101+0000 7f8f7d964640 1 Processor -- start 2024-09-10T20:38:26.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.102+0000 7f8f7d964640 1 -- start start 2024-09-10T20:38:26.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.102+0000 7f8f7d964640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f7806e530 0x7f8f7815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.102+0000 7f8f7d964640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f780d4bb0 0x7f8f7815dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.102+0000 7f8f7d964640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f78161790 0x7f8f780b7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.102+0000 7f8f7d964640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7806dd60 con 0x7f8f78161790 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.102+0000 7f8f7d964640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7806dbe0 con 0x7f8f780d4bb0 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.102+0000 7f8f7d964640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8f7806dee0 con 0x7f8f7806e530 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f78161790 0x7f8f780b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f78161790 0x7f8f780b7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51288/0 (socket says 172.21.15.28:51288) 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 -- 172.21.15.28:0/1122032443 learned_addr learned my addr 172.21.15.28:0/1122032443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f76ffd640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f7806e530 0x7f8f7815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f767fc640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f780d4bb0 0x7f8f7815dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 -- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f7806e530 msgr2=0x7f8f7815bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f7806e530 0x7f8f7815bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 -- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f780d4bb0 msgr2=0x7f8f7815dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f780d4bb0 0x7f8f7815dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 -- 172.21.15.28:0/1122032443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f7815c120 con 0x7f8f78161790 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f777fe640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f78161790 0x7f8f780b7860 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c760 tx=0x7f8f7000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f6ffff640 1 -- 172.21.15.28:0/1122032443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70007bb0 con 0x7f8f78161790 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f6ffff640 1 -- 172.21.15.28:0/1122032443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f70005020 con 0x7f8f78161790 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f6ffff640 1 -- 172.21.15.28:0/1122032443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70007e60 con 0x7f8f78161790 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f7815c400 con 0x7f8f78161790 2024-09-10T20:38:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.103+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f7815e2b0 con 0x7f8f78161790 2024-09-10T20:38:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.104+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f7806d490 con 0x7f8f78161790 2024-09-10T20:38:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.105+0000 7f8f6ffff640 1 -- 172.21.15.28:0/1122032443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8f700054b0 con 0x7f8f78161790 2024-09-10T20:38:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.105+0000 7f8f6ffff640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8f58077f70 0x7f8f5807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.105+0000 7f8f6ffff640 1 -- 172.21.15.28:0/1122032443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8f7009a080 con 0x7f8f78161790 2024-09-10T20:38:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.105+0000 7f8f76ffd640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8f58077f70 0x7f8f5807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.106+0000 7f8f76ffd640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8f58077f70 0x7f8f5807a430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8f68005c40 tx=0x7f8f6800a530 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.110+0000 7f8f6ffff640 1 -- 172.21.15.28:0/1122032443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8f70014030 con 0x7f8f78161790 2024-09-10T20:38:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.333+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8f780d18f0 con 0x7f8f78161790 2024-09-10T20:38:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.334+0000 7f8f6ffff640 1 -- 172.21.15.28:0/1122032443 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8f700625c0 con 0x7f8f78161790 2024-09-10T20:38:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:26.338 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8f58077f70 msgr2=0x7f8f5807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8f58077f70 0x7f8f5807a430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8f68005c40 tx=0x7f8f6800a530 comp rx=0 tx=0).stop 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f78161790 msgr2=0x7f8f780b7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f78161790 0x7f8f780b7860 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c760 tx=0x7f8f7000cc20 comp rx=0 tx=0).stop 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 shutdown_connections 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8f58077f70 0x7f8f5807a430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f78161790 0x7f8f780b7860 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8f780d4bb0 0x7f8f7815dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 --2- 172.21.15.28:0/1122032443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8f7806e530 0x7f8f7815bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 >> 172.21.15.28:0/1122032443 conn(0x7f8f780d4090 msgr2=0x7f8f780b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 shutdown_connections 2024-09-10T20:38:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.336+0000 7f8f7d964640 1 -- 172.21.15.28:0/1122032443 wait complete. 2024-09-10T20:38:26.350 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-10T20:38:26.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:26.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.452+0000 7fe11828c640 1 -- 172.21.15.28:0/2179307781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0f8003a50 msgr2=0x7fe0f8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.452+0000 7fe11828c640 1 --2- 172.21.15.28:0/2179307781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0f8003a50 0x7fe0f8003e70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe10000c610 tx=0x7fe0f8005830 comp rx=0 tx=0).stop 2024-09-10T20:38:26.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.452+0000 7fe11828c640 1 -- 172.21.15.28:0/2179307781 shutdown_connections 2024-09-10T20:38:26.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.452+0000 7fe11828c640 1 --2- 172.21.15.28:0/2179307781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0f8003a50 0x7fe0f8003e70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.452+0000 7fe11828c640 1 -- 172.21.15.28:0/2179307781 >> 172.21.15.28:0/2179307781 conn(0x7fe1100d4040 msgr2=0x7fe1100d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:26.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.452+0000 7fe11828c640 1 -- 172.21.15.28:0/2179307781 shutdown_connections 2024-09-10T20:38:26.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.452+0000 7fe11828c640 1 -- 172.21.15.28:0/2179307781 wait complete. 2024-09-10T20:38:26.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.453+0000 7fe11828c640 1 Processor -- start 2024-09-10T20:38:26.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.453+0000 7fe11828c640 1 -- start start 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe11828c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0f8003a50 0x7fe110153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe11828c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1100cd800 0x7fe1100cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe11828c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1100c9400 0x7fe1100c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe11828c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1100d21b0 con 0x7fe1100cd800 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe11828c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe1100d2030 con 0x7fe0f8003a50 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe11828c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe1100d2330 con 0x7fe1100c9400 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe115827640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1100cd800 0x7fe1100cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe115827640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1100cd800 0x7fe1100cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51296/0 (socket says 172.21.15.28:51296) 2024-09-10T20:38:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe115827640 1 -- 172.21.15.28:0/3774915363 learned_addr learned my addr 172.21.15.28:0/3774915363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe116028640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0f8003a50 0x7fe110153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.454+0000 7fe116829640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1100c9400 0x7fe1100c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe115827640 1 -- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1100c9400 msgr2=0x7fe1100c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe115827640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1100c9400 0x7fe1100c9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe115827640 1 -- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0f8003a50 msgr2=0x7fe110153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe115827640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0f8003a50 0x7fe110153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe115827640 1 -- 172.21.15.28:0/3774915363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1100cc200 con 0x7fe1100cd800 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe115827640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1100cd800 0x7fe1100cba90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe104004d90 tx=0x7fe104001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe10effd640 1 -- 172.21.15.28:0/3774915363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104001d40 con 0x7fe1100cd800 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe10effd640 1 -- 172.21.15.28:0/3774915363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1040108c0 con 0x7fe1100cd800 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe10effd640 1 -- 172.21.15.28:0/3774915363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104010cb0 con 0x7fe1100cd800 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1100cc4e0 con 0x7fe1100cd800 2024-09-10T20:38:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.455+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1100ccaa0 con 0x7fe1100cd800 2024-09-10T20:38:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.456+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1100d2890 con 0x7fe1100cd800 2024-09-10T20:38:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.457+0000 7fe10effd640 1 -- 172.21.15.28:0/3774915363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe10401c470 con 0x7fe1100cd800 2024-09-10T20:38:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.457+0000 7fe10effd640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe0fc077f70 0x7fe0fc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.457+0000 7fe10effd640 1 -- 172.21.15.28:0/3774915363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe10409a550 con 0x7fe1100cd800 2024-09-10T20:38:26.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.458+0000 7fe116028640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe0fc077f70 0x7fe0fc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.458+0000 7fe116028640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe0fc077f70 0x7fe0fc07a430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe100033000 tx=0x7fe100001360 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:26.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.461+0000 7fe10effd640 1 -- 172.21.15.28:0/3774915363 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe104062a00 con 0x7fe1100cd800 2024-09-10T20:38:26.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.684+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1100b9fc0 con 0x7fe1100cd800 2024-09-10T20:38:26.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.684+0000 7fe10effd640 1 -- 172.21.15.28:0/3774915363 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7fe104067430 con 0x7fe1100cd800 2024-09-10T20:38:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:26.689 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.686+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe0fc077f70 msgr2=0x7fe0fc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.686+0000 7fe11828c640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe0fc077f70 0x7fe0fc07a430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe100033000 tx=0x7fe100001360 comp rx=0 tx=0).stop 2024-09-10T20:38:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.686+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1100cd800 msgr2=0x7fe1100cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1100cd800 0x7fe1100cba90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe104004d90 tx=0x7fe104001630 comp rx=0 tx=0).stop 2024-09-10T20:38:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 shutdown_connections 2024-09-10T20:38:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe0fc077f70 0x7fe0fc07a430 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe1100c9400 0x7fe1100c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1100cd800 0x7fe1100cba90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 --2- 172.21.15.28:0/3774915363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0f8003a50 0x7fe110153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 >> 172.21.15.28:0/3774915363 conn(0x7fe1100d4040 msgr2=0x7fe11006d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 shutdown_connections 2024-09-10T20:38:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.687+0000 7fe11828c640 1 -- 172.21.15.28:0/3774915363 wait complete. 2024-09-10T20:38:26.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1623958289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40cd3d0 msgr2=0x7f2dd40cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1623958289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40cd3d0 0x7f2dd40cc570 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40ce0d0 tx=0x7f2dd002a530 comp rx=0 tx=0).stop 2024-09-10T20:38:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1623958289 shutdown_connections 2024-09-10T20:38:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1623958289 >> v2:172.21.15.159:3300/0 conn(0x7f2dd40d07a0 0x7f2dd40b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1623958289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40cd3d0 0x7f2dd40cc570 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1623958289 >> 172.21.15.28:0/1623958289 conn(0x7f2dd406cf00 msgr2=0x7f2dd406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1623958289 shutdown_connections 2024-09-10T20:38:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.803+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1623958289 wait complete. 2024-09-10T20:38:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.804+0000 7f2ddc7f3640 1 Processor -- start 2024-09-10T20:38:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.804+0000 7f2ddc7f3640 1 -- start start 2024-09-10T20:38:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddc7f3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd40b4340 0x7f2dd41f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddc7f3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd40cd3d0 0x7f2dd41f8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddc7f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40d07a0 0x7f2dd414e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddc7f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd40b33c0 con 0x7f2dd40d07a0 2024-09-10T20:38:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddc7f3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd40b4aa0 con 0x7f2dd40b4340 2024-09-10T20:38:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddc7f3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd40b3540 con 0x7f2dd40cd3d0 2024-09-10T20:38:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40d07a0 0x7f2dd414e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40d07a0 0x7f2dd414e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51314/0 (socket says 172.21.15.28:51314) 2024-09-10T20:38:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 -- 172.21.15.28:0/1411987582 learned_addr learned my addr 172.21.15.28:0/1411987582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2dda58f640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd40b4340 0x7f2dd41f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2dd9d8e640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd40cd3d0 0x7f2dd41f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 -- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd40cd3d0 msgr2=0x7f2dd41f8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd40cd3d0 0x7f2dd41f8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 -- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd40b4340 msgr2=0x7f2dd41f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd40b4340 0x7f2dd41f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.805+0000 7f2ddad90640 1 -- 172.21.15.28:0/1411987582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd41f8cd0 con 0x7f2dd40d07a0 2024-09-10T20:38:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.806+0000 7f2ddad90640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40d07a0 0x7f2dd414e930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00cab0 tx=0x7f2dcc00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.806+0000 7f2dcaffd640 1 -- 172.21.15.28:0/1411987582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc007d90 con 0x7f2dd40d07a0 2024-09-10T20:38:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.806+0000 7f2dcaffd640 1 -- 172.21.15.28:0/1411987582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2dcc003990 con 0x7f2dd40d07a0 2024-09-10T20:38:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.806+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd40cb9a0 con 0x7f2dd40d07a0 2024-09-10T20:38:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.806+0000 7f2dcaffd640 1 -- 172.21.15.28:0/1411987582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc003dc0 con 0x7f2dd40d07a0 2024-09-10T20:38:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.806+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd40b0ab0 con 0x7f2dd40d07a0 2024-09-10T20:38:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.807+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd40b39f0 con 0x7f2dd40d07a0 2024-09-10T20:38:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.808+0000 7f2dcaffd640 1 -- 172.21.15.28:0/1411987582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2dcc003f60 con 0x7f2dd40d07a0 2024-09-10T20:38:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.808+0000 7f2dcaffd640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2db8077f70 0x7f2db807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.808+0000 7f2dcaffd640 1 -- 172.21.15.28:0/1411987582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2dcc09a3e0 con 0x7f2dd40d07a0 2024-09-10T20:38:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.808+0000 7f2dda58f640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2db8077f70 0x7f2db807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.809+0000 7f2dda58f640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2db8077f70 0x7f2db807a430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0034000 tx=0x7f2dd0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:26.812+0000 7f2dcaffd640 1 -- 172.21.15.28:0/1411987582 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2dcc014030 con 0x7f2dd40d07a0 2024-09-10T20:38:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.033+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dd40cce90 con 0x7f2dd40d07a0 2024-09-10T20:38:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.034+0000 7f2dcaffd640 1 -- 172.21.15.28:0/1411987582 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f2dcc062890 con 0x7f2dd40d07a0 2024-09-10T20:38:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2db8077f70 msgr2=0x7f2db807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2db8077f70 0x7f2db807a430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0034000 tx=0x7f2dd0004730 comp rx=0 tx=0).stop 2024-09-10T20:38:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40d07a0 msgr2=0x7f2dd414e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40d07a0 0x7f2dd414e930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00cab0 tx=0x7f2dcc00cf70 comp rx=0 tx=0).stop 2024-09-10T20:38:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 shutdown_connections 2024-09-10T20:38:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2db8077f70 0x7f2db807a430 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd40d07a0 0x7f2dd414e930 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2dd40cd3d0 0x7f2dd41f8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 --2- 172.21.15.28:0/1411987582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2dd40b4340 0x7f2dd41f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 >> 172.21.15.28:0/1411987582 conn(0x7f2dd406cf00 msgr2=0x7f2dd409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 shutdown_connections 2024-09-10T20:38:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.036+0000 7f2ddc7f3640 1 -- 172.21.15.28:0/1411987582 wait complete. 2024-09-10T20:38:27.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 -- 172.21.15.28:0/3247761698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 msgr2=0x7fa5e80cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/3247761698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 0x7fa5e80cc570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009830 tx=0x7fa5e402a4a0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 -- 172.21.15.28:0/3247761698 shutdown_connections 2024-09-10T20:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/3247761698 >> v2:172.21.15.159:3300/0 conn(0x7fa5e80d07a0 0x7fa5e80b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/3247761698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 0x7fa5e80cc570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 -- 172.21.15.28:0/3247761698 >> 172.21.15.28:0/3247761698 conn(0x7fa5e806cf00 msgr2=0x7fa5e806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 -- 172.21.15.28:0/3247761698 shutdown_connections 2024-09-10T20:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.151+0000 7fa5f01a0640 1 -- 172.21.15.28:0/3247761698 wait complete. 2024-09-10T20:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 Processor -- start 2024-09-10T20:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 -- start start 2024-09-10T20:38:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5e80b4340 0x7fa5e81f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 0x7fa5e8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5e80d07a0 0x7fa5e81528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e80b3320 con 0x7fa5e80b4340 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e80b31a0 con 0x7fa5e80d07a0 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5f01a0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e80b34a0 con 0x7fa5e80cd3d0 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5edf3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5e80b4340 0x7fa5e81f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5edf3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5e80b4340 0x7fa5e81f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51318/0 (socket says 172.21.15.28:51318) 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.153+0000 7fa5edf3c640 1 -- 172.21.15.28:0/1183223460 learned_addr learned my addr 172.21.15.28:0/1183223460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5ed73b640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 0x7fa5e8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5ee73d640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5e80d07a0 0x7fa5e81528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5edf3c640 1 -- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 msgr2=0x7fa5e8073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5edf3c640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 0x7fa5e8073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5edf3c640 1 -- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5e80d07a0 msgr2=0x7fa5e81528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5edf3c640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5e80d07a0 0x7fa5e81528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5edf3c640 1 -- 172.21.15.28:0/1183223460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e8073bc0 con 0x7fa5e80b4340 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5edf3c640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5e80b4340 0x7fa5e81f4570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009950 tx=0x7fa5e4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5de7fc640 1 -- 172.21.15.28:0/1183223460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4005c90 con 0x7fa5e80b4340 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5de7fc640 1 -- 172.21.15.28:0/1183223460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5e4005020 con 0x7fa5e80b4340 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e81f4c70 con 0x7fa5e80b4340 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5de7fc640 1 -- 172.21.15.28:0/1183223460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4005450 con 0x7fa5e80b4340 2024-09-10T20:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.154+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e81f5230 con 0x7fa5e80b4340 2024-09-10T20:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.155+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5e80b39b0 con 0x7fa5e80b4340 2024-09-10T20:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.156+0000 7fa5de7fc640 1 -- 172.21.15.28:0/1183223460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa5e40317c0 con 0x7fa5e80b4340 2024-09-10T20:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.157+0000 7fa5de7fc640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa5cc077fa0 0x7fa5cc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.157+0000 7fa5de7fc640 1 -- 172.21.15.28:0/1183223460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa5e40baf20 con 0x7fa5e80b4340 2024-09-10T20:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.157+0000 7fa5ed73b640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa5cc077fa0 0x7fa5cc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.157+0000 7fa5ed73b640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa5cc077fa0 0x7fa5cc07a460 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8159140 tx=0x7fa5d8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.161+0000 7fa5de7fc640 1 -- 172.21.15.28:0/1183223460 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa5e4040030 con 0x7fa5e80b4340 2024-09-10T20:38:27.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.346+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5e80b2170 con 0x7fa5e80b4340 2024-09-10T20:38:27.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.347+0000 7fa5de7fc640 1 -- 172.21.15.28:0/1183223460 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16669 (secure 0 0 0) 0x7fa5e4083a20 con 0x7fa5e80b4340 2024-09-10T20:38:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:27.351 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:13.395359+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.349+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa5cc077fa0 msgr2=0x7fa5cc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.349+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa5cc077fa0 0x7fa5cc07a460 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8159140 tx=0x7fa5d8009040 comp rx=0 tx=0).stop 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.349+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5e80b4340 msgr2=0x7fa5e81f4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.349+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5e80b4340 0x7fa5e81f4570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009950 tx=0x7fa5e4003c10 comp rx=0 tx=0).stop 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 shutdown_connections 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa5cc077fa0 0x7fa5cc07a460 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5e80d07a0 0x7fa5e81528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5e80cd3d0 0x7fa5e8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 --2- 172.21.15.28:0/1183223460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5e80b4340 0x7fa5e81f4570 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 >> 172.21.15.28:0/1183223460 conn(0x7fa5e806cf00 msgr2=0x7fa5e8070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 shutdown_connections 2024-09-10T20:38:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.350+0000 7fa5f01a0640 1 -- 172.21.15.28:0/1183223460 wait complete. 2024-09-10T20:38:27.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:27.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.467+0000 7f583ca93640 1 -- 172.21.15.28:0/4209937319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380cd3d0 msgr2=0x7f58380cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.467+0000 7f583ca93640 1 --2- 172.21.15.28:0/4209937319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380cd3d0 0x7f58380cc570 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5828009830 tx=0x7f582802a510 comp rx=0 tx=0).stop 2024-09-10T20:38:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.467+0000 7f583ca93640 1 -- 172.21.15.28:0/4209937319 shutdown_connections 2024-09-10T20:38:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.467+0000 7f583ca93640 1 --2- 172.21.15.28:0/4209937319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380cd3d0 0x7f58380cc570 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.467+0000 7f583ca93640 1 -- 172.21.15.28:0/4209937319 >> 172.21.15.28:0/4209937319 conn(0x7f583806cf00 msgr2=0x7f583806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.467+0000 7f583ca93640 1 -- 172.21.15.28:0/4209937319 shutdown_connections 2024-09-10T20:38:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.467+0000 7f583ca93640 1 -- 172.21.15.28:0/4209937319 wait complete. 2024-09-10T20:38:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.469+0000 7f583ca93640 1 Processor -- start 2024-09-10T20:38:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.469+0000 7f583ca93640 1 -- start start 2024-09-10T20:38:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583ca93640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58380b4340 0x7f58381542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583ca93640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58380cd3d0 0x7f5838144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583ca93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380d07a0 0x7f5838145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583ca93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58380b3590 con 0x7f58380d07a0 2024-09-10T20:38:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583ca93640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f58380b4af0 con 0x7f58380b4340 2024-09-10T20:38:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583ca93640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f58380b3710 con 0x7f58380cd3d0 2024-09-10T20:38:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380d07a0 0x7f5838145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380d07a0 0x7f5838145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51328/0 (socket says 172.21.15.28:51328) 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 -- 172.21.15.28:0/117364552 learned_addr learned my addr 172.21.15.28:0/117364552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5835d9b640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58380cd3d0 0x7f5838144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583659c640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58380b4340 0x7f58381542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 -- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58380cd3d0 msgr2=0x7f5838144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58380cd3d0 0x7f5838144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 -- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58380b4340 msgr2=0x7f58381542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58380b4340 0x7f58381542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 -- 172.21.15.28:0/117364552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58380b0a00 con 0x7f58380d07a0 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5836d9d640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380d07a0 0x7f5838145770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f583000ca80 tx=0x7f583000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5826ffd640 1 -- 172.21.15.28:0/117364552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830016020 con 0x7f58380d07a0 2024-09-10T20:38:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5826ffd640 1 -- 172.21.15.28:0/117364552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f583000ad20 con 0x7f58380d07a0 2024-09-10T20:38:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58380b0ce0 con 0x7f58380d07a0 2024-09-10T20:38:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.470+0000 7f5826ffd640 1 -- 172.21.15.28:0/117364552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830012680 con 0x7f58380d07a0 2024-09-10T20:38:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.471+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58380b12a0 con 0x7f58380d07a0 2024-09-10T20:38:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.471+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58380cc570 con 0x7f58380d07a0 2024-09-10T20:38:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.472+0000 7f5826ffd640 1 -- 172.21.15.28:0/117364552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f583001a3b0 con 0x7f58380d07a0 2024-09-10T20:38:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.473+0000 7f5826ffd640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5818077f70 0x7f581807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.473+0000 7f5826ffd640 1 -- 172.21.15.28:0/117364552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f583009a1f0 con 0x7f58380d07a0 2024-09-10T20:38:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.473+0000 7f583659c640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5818077f70 0x7f581807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.473+0000 7f583659c640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5818077f70 0x7f581807a430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5828034000 tx=0x7f5828004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.477+0000 7f5826ffd640 1 -- 172.21.15.28:0/117364552 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5830020030 con 0x7f58380d07a0 2024-09-10T20:38:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.698+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f583806d480 con 0x7f58380d07a0 2024-09-10T20:38:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.699+0000 7f5826ffd640 1 -- 172.21.15.28:0/117364552 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f583006b370 con 0x7f58380d07a0 2024-09-10T20:38:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:27.703 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5818077f70 msgr2=0x7f581807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5818077f70 0x7f581807a430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5828034000 tx=0x7f5828004730 comp rx=0 tx=0).stop 2024-09-10T20:38:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380d07a0 msgr2=0x7f5838145770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380d07a0 0x7f5838145770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f583000ca80 tx=0x7f583000cf40 comp rx=0 tx=0).stop 2024-09-10T20:38:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 shutdown_connections 2024-09-10T20:38:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5818077f70 0x7f581807a430 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58380d07a0 0x7f5838145770 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58380cd3d0 0x7f5838144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 --2- 172.21.15.28:0/117364552 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58380b4340 0x7f58381542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 >> 172.21.15.28:0/117364552 conn(0x7f583806cf00 msgr2=0x7f58380cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 shutdown_connections 2024-09-10T20:38:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:27.701+0000 7f583ca93640 1 -- 172.21.15.28:0/117364552 wait complete. 2024-09-10T20:38:28.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:38:11.751747+0000', 'modified': '2024-09-10T20:38:14.774063+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4357}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4357': {'gid': 4357, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.159:6835/2925750660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6834', 'nonce': 2925750660}, {'type': 'v1', 'addr': '172.21.15.159:6835', 'nonce': 2925750660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4357, 'qdb_cluster': [4357]} 2024-09-10T20:38:28.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:38:28.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-10T20:38:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.230+0000 7f79d222a640 1 -- 172.21.15.28:0/3951043158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b8004d40 msgr2=0x7f79b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.230+0000 7f79d222a640 1 --2- 172.21.15.28:0/3951043158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b8004d40 0x7f79b80051c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c770 tx=0x7f79b8005830 comp rx=0 tx=0).stop 2024-09-10T20:38:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.230+0000 7f79cbfff640 1 -- 172.21.15.28:0/3951043158 reap_dead start 2024-09-10T20:38:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.230+0000 7f79d222a640 1 -- 172.21.15.28:0/3951043158 shutdown_connections 2024-09-10T20:38:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.230+0000 7f79d222a640 1 -- 172.21.15.28:0/3951043158 >> 172.21.15.28:0/3951043158 conn(0x7f79cc06cf00 msgr2=0x7f79cc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.230+0000 7f79d222a640 1 -- 172.21.15.28:0/3951043158 shutdown_connections 2024-09-10T20:38:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.230+0000 7f79d222a640 1 -- 172.21.15.28:0/3951043158 wait complete. 2024-09-10T20:38:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.232+0000 7f79d222a640 1 Processor -- start 2024-09-10T20:38:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.232+0000 7f79d222a640 1 -- start start 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.232+0000 7f79d222a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79b8004d40 0x7f79cc1efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79d222a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79cc146d50 0x7f79cc09cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79d222a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79cc09d2b0 0x7f79cc1411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79d222a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc0d05d0 con 0x7f79cc146d50 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79d222a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc15bfb0 con 0x7f79cc09d2b0 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79d222a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f79cc0d0750 con 0x7f79b8004d40 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79cc146d50 0x7f79cc09cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79cc146d50 0x7f79cc09cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51346/0 (socket says 172.21.15.28:51346) 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 -- 172.21.15.28:0/2877215157 learned_addr learned my addr 172.21.15.28:0/2877215157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79b8004d40 0x7f79cc1efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79cbfff640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79cc09d2b0 0x7f79cc1411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 -- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79b8004d40 msgr2=0x7f79cc1efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79b8004d40 0x7f79cc1efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 -- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79cc09d2b0 msgr2=0x7f79cc1411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79cc09d2b0 0x7f79cc1411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 -- 172.21.15.28:0/2877215157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79cc09bcd0 con 0x7f79cc146d50 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c3fff640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79cc146d50 0x7f79cc09cd70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f79b4003f50 tx=0x7f79b400f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c8ff9640 1 -- 172.21.15.28:0/2877215157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4015070 con 0x7f79cc146d50 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c8ff9640 1 -- 172.21.15.28:0/2877215157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79b4011840 con 0x7f79cc146d50 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79cc09bfb0 con 0x7f79cc146d50 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79c8ff9640 1 -- 172.21.15.28:0/2877215157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4011c70 con 0x7f79cc146d50 2024-09-10T20:38:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.233+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79cc09c570 con 0x7f79cc146d50 2024-09-10T20:38:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.235+0000 7f79c8ff9640 1 -- 172.21.15.28:0/2877215157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f79b40103c0 con 0x7f79cc146d50 2024-09-10T20:38:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.236+0000 7f79c8ff9640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f79ac077fa0 0x7f79ac07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.236+0000 7f79c8ff9640 1 -- 172.21.15.28:0/2877215157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f79b409a720 con 0x7f79cc146d50 2024-09-10T20:38:28.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.236+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f79ac077fa0 0x7f79ac07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f79ac077fa0 0x7f79ac07a460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009c50 tx=0x7f79bc010040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 Processor -- start 2024-09-10T20:38:28.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 -- start start 2024-09-10T20:38:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7bab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a802df0 0x55b75a8031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7b85f0 0x55b75a7b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b75a72f820 con 0x55b75a802df0 2024-09-10T20:38:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b75a71ec50 con 0x55b75a7b85f0 2024-09-10T20:38:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.237+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b75a722a60 con 0x55b75a7e31b0 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a802df0 0x55b75a8031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a802df0 0x55b75a8031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51354/0 (socket says 172.21.15.28:51354) 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 -- 172.21.15.28:0/1036546961 learned_addr learned my addr 172.21.15.28:0/1036546961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7bab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79cbfff640 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7b85f0 0x55b75a7b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 -- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 msgr2=0x55b75a7bab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7bab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 -- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7b85f0 msgr2=0x55b75a7b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7b85f0 0x55b75a7b8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 -- 172.21.15.28:0/1036546961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b75a480850 con 0x55b75a802df0 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c3fff640 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a802df0 0x55b75a8031f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f79b4098bf0 tx=0x7f79b4062400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c95a2a0d59ded0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c17fa640 1 -- 172.21.15.28:0/1036546961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4020070 con 0x55b75a802df0 2024-09-10T20:38:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c17fa640 1 -- 172.21.15.28:0/1036546961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79b4063e50 con 0x55b75a802df0 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.238+0000 7f79c17fa640 1 -- 172.21.15.28:0/1036546961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4067600 con 0x55b75a802df0 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1036546961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a802df0 msgr2=0x55b75a8031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a802df0 0x55b75a8031f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f79b4098bf0 tx=0x7f79b4062400 comp rx=0 tx=0).stop 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1036546961 shutdown_connections 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7b85f0 0x55b75a7b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a802df0 0x55b75a8031f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1036546961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7bab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1036546961 >> 172.21.15.28:0/1036546961 conn(0x55b75a7b96a0 msgr2=0x55b75a7bd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1036546961 shutdown_connections 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.239+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1036546961 wait complete. 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.240+0000 7f79d4ffe740 1 -- start start 2024-09-10T20:38:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.240+0000 7f79d4ffe740 1 Processor -- start 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.240+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7e24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.240+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a802df0 0x55b75a7fc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.240+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7fc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.240+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b75a48f620 con 0x55b75a7cda80 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.240+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b75a6c45f0 con 0x55b75a802df0 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b75a7b7b30 con 0x55b75a7e31b0 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79d4ffe740 10 client.0 _ll_register_callbacks cb 0x55b75a6c80a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7fc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7fc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51370/0 (socket says 172.21.15.28:51370) 2024-09-10T20:38:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 -- 172.21.15.28:0/1891147814 learned_addr learned my addr 172.21.15.28:0/1891147814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7e24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79c3fff640 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a802df0 0x55b75a7fc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 -- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 msgr2=0x55b75a7e24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7e24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 -- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a802df0 msgr2=0x55b75a7fc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a802df0 0x55b75a7fc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 -- 172.21.15.28:0/1891147814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b75a4f76b0 con 0x55b75a7cda80 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79cbfff640 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7fc710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c890 tx=0x7f79c400c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cca79114e7f5cc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79aa7fc640 1 -- 172.21.15.28:0/1891147814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4043060 con 0x55b75a7cda80 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79aa7fc640 1 -- 172.21.15.28:0/1891147814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f79c403eab0 con 0x55b75a7cda80 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79b4063e50 con 0x55b75a7cda80 2024-09-10T20:38:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.241+0000 7f79aa7fc640 1 -- 172.21.15.28:0/1891147814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f79c4039040 con 0x55b75a7cda80 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 10 client.4644 fetch_fsmap learned FSMap version 8 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55b75a7d0140 con 0x55b75a7cda80 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79aa7fc640 1 -- 172.21.15.28:0/1891147814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79b4063e50 con 0x55b75a7cda80 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79aa7fc640 1 -- 172.21.15.28:0/1891147814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f79c403ecb0 con 0x55b75a7cda80 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 10 client.4644 fetch_fsmap finished waiting for FSMap version 8 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 10 client.4644 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 20 client.4644 populate_metadata read hostname 'smithi028' 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7cf380 0x55b75a8151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 4 client.4644 mds_command: new command op to 4357 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 --> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b75a731770 con 0x55b75a7cf380 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7cf380 0x55b75a8151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79aa7fc640 10 client.4644 ms_handle_connect on v2:172.21.15.159:6834/2925750660 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.242+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7cf380 0x55b75a8151d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e3cd5933ee14c57 server_cookie=310d59b282c2fab6 in_seq=0 out_seq=0 2024-09-10T20:38:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79aa7fc640 1 -- 172.21.15.28:0/1891147814 <== mds.0 v2:172.21.15.159:6834/2925750660 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b75a731770 con 0x55b75a7cf380 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79aa7fc640 10 client.4644 handle_command_reply: tid=0 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 client.4644 shutdown 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 msgr2=0x55b75a7fc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7fc710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c890 tx=0x7f79c400c4b0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 shutdown_connections 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7cf380 0x55b75a8151d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7fc710 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a802df0 0x55b75a7fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79aa7fc640 0 client.4644 ms_handle_reset on v2:172.21.15.159:6834/2925750660 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/1891147814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a7e31b0 0x55b75a7e24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 >> 172.21.15.28:0/1891147814 conn(0x55b75a7b85f0 msgr2=0x55b75a7bd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.243+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 shutdown_connections 2024-09-10T20:38:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.244+0000 7f79d4ffe740 1 -- 172.21.15.28:0/1891147814 wait complete. 2024-09-10T20:38:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.244+0000 7f79d4ffe740 20 client.4644 trim_cache size 0 max 16384 2024-09-10T20:38:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.262+0000 7f79d4ffe740 1 Processor -- start 2024-09-10T20:38:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.262+0000 7f79d4ffe740 1 -- start start 2024-09-10T20:38:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a8014b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a7dafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7db510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1fc770 con 0x55b75a7cda80 2024-09-10T20:38:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1f48d0 con 0x55b75a7e31b0 2024-09-10T20:38:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f79cc1fc120 con 0x55b75a802df0 2024-09-10T20:38:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a8014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a8014b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51378/0 (socket says 172.21.15.28:51378) 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2108448706 learned_addr learned my addr 172.21.15.28:0/2108448706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79c3fff640 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a7dafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 msgr2=0x55b75a7db510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cbfff640 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7db510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7db510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 msgr2=0x55b75a7dafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a7dafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2108448706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b75a803fb0 con 0x55b75a7cda80 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cbfff640 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7db510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.263+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a8014b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00efd0 tx=0x7f79bc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c073db22eab4009 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79c0ff9640 1 -- 172.21.15.28:0/2108448706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc01bde0 con 0x55b75a7cda80 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79c0ff9640 1 -- 172.21.15.28:0/2108448706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79bc02f3f0 con 0x55b75a7cda80 2024-09-10T20:38:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79c0ff9640 1 -- 172.21.15.28:0/2108448706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc02f7c0 con 0x55b75a7cda80 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2108448706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 msgr2=0x55b75a8014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a8014b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00efd0 tx=0x7f79bc018000 comp rx=0 tx=0).stop 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2108448706 shutdown_connections 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7db510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a7dafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2108448706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a8014b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2108448706 >> 172.21.15.28:0/2108448706 conn(0x55b75a7b85f0 msgr2=0x55b75a803480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2108448706 shutdown_connections 2024-09-10T20:38:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.264+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2108448706 wait complete. 2024-09-10T20:38:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.265+0000 7f79d4ffe740 1 -- start start 2024-09-10T20:38:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.265+0000 7f79d4ffe740 1 Processor -- start 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7e3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a8115a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7dbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b75a7cf200 con 0x55b75a7cda80 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b75a7e2a20 con 0x55b75a7e31b0 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b75a7cf850 con 0x55b75a802df0 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7e3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 10 client.0 _ll_register_callbacks cb 0x7f79bc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7e3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51380/0 (socket says 172.21.15.28:51380) 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2781418589 learned_addr learned my addr 172.21.15.28:0/2781418589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cbfff640 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7dbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79c3fff640 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a8115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 msgr2=0x55b75a7dbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7dbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 msgr2=0x55b75a8115a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a8115a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 -- 172.21.15.28:0/2781418589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b75a804840 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79cb7fe640 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7e3e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00ed40 tx=0x7f79bc01bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431124672896862a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79aa7fc640 1 -- 172.21.15.28:0/2781418589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc03c070 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79aa7fc640 1 -- 172.21.15.28:0/2781418589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f79bc02f7a0 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.266+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79bc02f3f0 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79aa7fc640 1 -- 172.21.15.28:0/2781418589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f79bc02f9a0 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 10 client.4650 fetch_fsmap learned FSMap version 8 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55b75a812560 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79aa7fc640 1 -- 172.21.15.28:0/2781418589 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79bc02f3f0 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79aa7fc640 1 -- 172.21.15.28:0/2781418589 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f79bc04a080 con 0x55b75a7cda80 2024-09-10T20:38:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 10 client.4650 fetch_fsmap finished waiting for FSMap version 8 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 10 client.4650 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 20 client.4650 populate_metadata read hostname 'smithi028' 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7fa1e0 0x55b75a7fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 4 client.4650 mds_command: new command op to 4357 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 --> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] -- command(tid 0: {"prefix": "status"}) -- 0x55b75a72d220 con 0x55b75a7fa1e0 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79c3fff640 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7fa1e0 0x55b75a7fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79c3fff640 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7fa1e0 0x55b75a7fa5c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25c217983c2e43f7 server_cookie=e0f689baafc1f84d in_seq=0 out_seq=0 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.267+0000 7f79aa7fc640 10 client.4650 ms_handle_connect on v2:172.21.15.159:6834/2925750660 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79aa7fc640 1 -- 172.21.15.28:0/2781418589 <== mds.0 v2:172.21.15.159:6834/2925750660 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55b75a72d220 con 0x55b75a7fa1e0 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79aa7fc640 10 client.4650 handle_command_reply: tid=0 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 client.4650 shutdown 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 msgr2=0x55b75a7e3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7e3e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00ed40 tx=0x7f79bc01bbe0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 shutdown_connections 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:6834/2925750660,v1:172.21.15.159:6835/2925750660] conn(0x55b75a7fa1e0 0x55b75a7fa5c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b75a802df0 0x55b75a7dbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b75a7e31b0 0x55b75a8115a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 --2- 172.21.15.28:0/2781418589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b75a7cda80 0x55b75a7e3e50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79aa7fc640 0 client.4650 ms_handle_reset on v2:172.21.15.159:6834/2925750660 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 >> 172.21.15.28:0/2781418589 conn(0x55b75a7b85f0 msgr2=0x55b75a7fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 shutdown_connections 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 1 -- 172.21.15.28:0/2781418589 wait complete. 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d4ffe740 20 client.4650 trim_cache size 0 max 16384 2024-09-10T20:38:28.277 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4357, 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 15.869060448000001, 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 8, 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 21, 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 21, 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 21.926135474999999 2024-09-10T20:38:28.278 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:38:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f79ac077fa0 msgr2=0x7f79ac07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f79ac077fa0 0x7f79ac07a460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009c50 tx=0x7f79bc010040 comp rx=0 tx=0).stop 2024-09-10T20:38:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79cc146d50 msgr2=0x7f79cc09cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79cc146d50 0x7f79cc09cd70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f79b4003f50 tx=0x7f79b400f720 comp rx=0 tx=0).stop 2024-09-10T20:38:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 shutdown_connections 2024-09-10T20:38:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f79ac077fa0 0x7f79ac07a460 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79cc09d2b0 0x7f79cc1411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79cc146d50 0x7f79cc09cd70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 --2- 172.21.15.28:0/2877215157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79b8004d40 0x7f79cc1efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 >> 172.21.15.28:0/2877215157 conn(0x7f79cc06cf00 msgr2=0x7f79cc06e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.268+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 shutdown_connections 2024-09-10T20:38:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.269+0000 7f79d222a640 1 -- 172.21.15.28:0/2877215157 wait complete. 2024-09-10T20:38:28.283 INFO:teuthology.run_tasks:Running task kclient... 2024-09-10T20:38:28.309 INFO:tasks.kclient:Mounting kernel clients... 2024-09-10T20:38:28.309 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-09-10T20:38:28.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.309 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:38:28.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-10T20:38:28.393 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-10T20:38:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:38:28.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.395 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:38:28.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-10T20:38:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-10T20:38:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:38:28.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.521 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:38:28.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-09-10T20:38:28.613 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-09-10T20:38:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:38:28.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.615 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:38:28.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:28.675 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-09-10T20:38:28.744 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-09-10T20:38:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:38:28.746 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-09-10T20:38:28.746 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-10T20:38:28.746 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:38:28.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:38:28.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 -- 172.21.15.28:0/2186309443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500b5610 msgr2=0x7f89500b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 --2- 172.21.15.28:0/2186309443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500b5610 0x7f89500b5a60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f89500cc420 tx=0x7f894802aa30 comp rx=0 tx=0).stop 2024-09-10T20:38:28.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 -- 172.21.15.28:0/2186309443 shutdown_connections 2024-09-10T20:38:28.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 --2- 172.21.15.28:0/2186309443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500b5610 0x7f89500b5a60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 --2- 172.21.15.28:0/2186309443 >> v2:172.21.15.159:3300/0 conn(0x7f89500d05d0 0x7f89500d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 -- 172.21.15.28:0/2186309443 >> 172.21.15.28:0/2186309443 conn(0x7f89500d1e40 msgr2=0x7f89500d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:28.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 -- 172.21.15.28:0/2186309443 shutdown_connections 2024-09-10T20:38:28.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.849+0000 7f895849b640 1 -- 172.21.15.28:0/2186309443 wait complete. 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.850+0000 7f895849b640 1 Processor -- start 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.850+0000 7f895849b640 1 -- start start 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f895849b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500b5610 0x7f8950152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f895849b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500d05d0 0x7f89501f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f895849b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015bf30 0x7f89501f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f895849b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89500b5fa0 con 0x7f89500d05d0 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f895849b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f89500d0d50 con 0x7f895015bf30 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f895849b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f89500b6120 con 0x7f89500b5610 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500d05d0 0x7f89501f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500d05d0 0x7f89501f2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51398/0 (socket says 172.21.15.28:51398) 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 -- 172.21.15.28:0/1159652797 learned_addr learned my addr 172.21.15.28:0/1159652797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8956a38640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015bf30 0x7f89501f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8956237640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500b5610 0x7f8950152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 -- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500b5610 msgr2=0x7f8950152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500b5610 0x7f8950152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 -- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015bf30 msgr2=0x7f89501f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015bf30 0x7f89501f28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 -- 172.21.15.28:0/1159652797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8950152890 con 0x7f89500d05d0 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8956237640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500b5610 0x7f8950152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8955a36640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500d05d0 0x7f89501f2370 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f894000c970 tx=0x7f894000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8946ffd640 1 -- 172.21.15.28:0/1159652797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940007bb0 con 0x7f89500d05d0 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8946ffd640 1 -- 172.21.15.28:0/1159652797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8940005020 con 0x7f89500d05d0 2024-09-10T20:38:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89501586e0 con 0x7f89500d05d0 2024-09-10T20:38:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.851+0000 7f8946ffd640 1 -- 172.21.15.28:0/1159652797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940007e60 con 0x7f89500d05d0 2024-09-10T20:38:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.852+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895015ca50 con 0x7f89500d05d0 2024-09-10T20:38:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.852+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895015c840 con 0x7f89500d05d0 2024-09-10T20:38:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.853+0000 7f8946ffd640 1 -- 172.21.15.28:0/1159652797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8940005470 con 0x7f89500d05d0 2024-09-10T20:38:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.854+0000 7f8946ffd640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8934077f70 0x7f893407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.854+0000 7f8946ffd640 1 -- 172.21.15.28:0/1159652797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f894009a240 con 0x7f89500d05d0 2024-09-10T20:38:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.854+0000 7f8956237640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8934077f70 0x7f893407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.855+0000 7f8956237640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8934077f70 0x7f893407a430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f894c00a810 tx=0x7f894c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:28.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:28.858+0000 7f8946ffd640 1 -- 172.21.15.28:0/1159652797 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8940014030 con 0x7f89500d05d0 2024-09-10T20:38:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.079+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f895015f7e0 con 0x7f89500d05d0 2024-09-10T20:38:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.079+0000 7f8946ffd640 1 -- 172.21.15.28:0/1159652797 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f8940062d40 con 0x7f89500d05d0 2024-09-10T20:38:29.083 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:38:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8934077f70 msgr2=0x7f893407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8934077f70 0x7f893407a430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f894c00a810 tx=0x7f894c008040 comp rx=0 tx=0).stop 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500d05d0 msgr2=0x7f89501f2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500d05d0 0x7f89501f2370 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f894000c970 tx=0x7f894000ce30 comp rx=0 tx=0).stop 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 shutdown_connections 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8934077f70 0x7f893407a430 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f895015bf30 0x7f89501f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89500d05d0 0x7f89501f2370 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 --2- 172.21.15.28:0/1159652797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89500b5610 0x7f8950152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 >> 172.21.15.28:0/1159652797 conn(0x7f89500d1e40 msgr2=0x7f8950070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.081+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 shutdown_connections 2024-09-10T20:38:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29.082+0000 7f895849b640 1 -- 172.21.15.28:0/1159652797 wait complete. 2024-09-10T20:38:29.096 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-10T20:38:29.096 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:38:29.096 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:38:29.096 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:38:29.096 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:38:29.097 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-10T20:38:29.097 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-10T20:38:29.097 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:38:29.097 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:38:29.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:29.097 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:38:29.113 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:38:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:38:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42722sec preferred_lft 42722sec 2024-09-10T20:38:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:38:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:29.115 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:38:29.115 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-10T20:38:29.116 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:29.116 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:29.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-09-10T20:38:29.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-09-10T20:38:29.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-09-10T20:38:29.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-10T20:38:29.116 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:29.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:29.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:29.327 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:38:29.354 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:38:29.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:29.356 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:38:29.420 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:38:29.420 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:38:29.420 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:38:29.420 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:38:29.421 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:38:29.421 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:29.421 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:29.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:38:29.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:38:29.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:38:29.422 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:29.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:29.736 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:38:29.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:29.753 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-10T20:38:29.809 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:29.809 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:29.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:38:29.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-10T20:38:29.809 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:29.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:29.929 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-10T20:38:29.929 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:30.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:30.061 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:30.061 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:30.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:38:30.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-09-10T20:38:30.061 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:30.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:30.140 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:30.140 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:38:30.164 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:38:30.165 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:30.165 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:38:30.220 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:38:30.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:30.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:38:30.331 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:38:30.331 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:38:30.332 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:38:30.451 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:38:30.451 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:38:30.451 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:38:30.451 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:30.451 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:38:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:38:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:38:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:38:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:38:30.460 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:38:30.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:30.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:38:30.488 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:30.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:38:30.574 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:38:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:30.595 INFO:teuthology.orchestra.run.smithi028.stdout:client4659 2024-09-10T20:38:30.595 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:30.595 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:30.679 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:30.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:30.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:30.784 INFO:teuthology.orchestra.run.smithi028.stdout:client4659 2024-09-10T20:38:30.784 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:30.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:30.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:client4659 2024-09-10T20:38:30.848 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/943b2354-acb0-43ad-b06a-e7974842506f.client4659/status') 2024-09-10T20:38:30.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:30.915 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4659 (3)192.168.0.1:0/18117490 2024-09-10T20:38:30.915 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-09-10T20:38:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-09-10T20:38:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-09-10T20:38:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:67 bytes copied, 2.3907e-05 s, 2.8 MB/s 2024-09-10T20:38:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:30.922 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-09-10T20:38:30.922 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-10T20:38:30.923 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-10T20:38:30.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:38:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3496895757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 msgr2=0x7f71e00cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3496895757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e00cfbc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f71d8005fd0 tx=0x7f71d802aa00 comp rx=0 tx=0).stop 2024-09-10T20:38:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3496895757 shutdown_connections 2024-09-10T20:38:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3496895757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e00cfbc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3496895757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e015de70 0x7f71e00d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3496895757 >> 172.21.15.28:0/3496895757 conn(0x7f71e00d3fb0 msgr2=0x7f71e00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3496895757 shutdown_connections 2024-09-10T20:38:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.026+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3496895757 wait complete. 2024-09-10T20:38:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.028+0000 7f71e5ef6640 1 Processor -- start 2024-09-10T20:38:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.028+0000 7f71e5ef6640 1 -- start start 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71e5ef6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e00cf780 0x7f71e015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71e5ef6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e015de70 0x7f71e01f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71e5ef6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e01f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71e5ef6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71e00b7190 con 0x7f71e0162260 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71e5ef6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f71e00b7010 con 0x7f71e00cf780 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71e5ef6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f71e00b7310 con 0x7f71e015de70 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e01f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e01f80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48652/0 (socket says 172.21.15.28:48652) 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 -- 172.21.15.28:0/3728119031 learned_addr learned my addr 172.21.15.28:0/3728119031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71df7fe640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e00cf780 0x7f71e015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71deffd640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e015de70 0x7f71e01f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 -- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e015de70 msgr2=0x7f71e01f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e015de70 0x7f71e01f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 -- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e00cf780 msgr2=0x7f71e015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e00cf780 0x7f71e015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.029+0000 7f71dffff640 1 -- 172.21.15.28:0/3728119031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e014d100 con 0x7f71e0162260 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.030+0000 7f71dffff640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e01f80b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006010 tx=0x7f71d802f6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.030+0000 7f71c7fff640 1 -- 172.21.15.28:0/3728119031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8046070 con 0x7f71e0162260 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.030+0000 7f71c7fff640 1 -- 172.21.15.28:0/3728119031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71d80045c0 con 0x7f71e0162260 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.030+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e0160760 con 0x7f71e0162260 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.030+0000 7f71c7fff640 1 -- 172.21.15.28:0/3728119031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8033070 con 0x7f71e0162260 2024-09-10T20:38:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.030+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e014d370 con 0x7f71e0162260 2024-09-10T20:38:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.031+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e00b7850 con 0x7f71e0162260 2024-09-10T20:38:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.032+0000 7f71c7fff640 1 -- 172.21.15.28:0/3728119031 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f71d8040030 con 0x7f71e0162260 2024-09-10T20:38:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.032+0000 7f71c7fff640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71cc077f70 0x7f71cc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.032+0000 7f71c7fff640 1 -- 172.21.15.28:0/3728119031 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71d80bad30 con 0x7f71e0162260 2024-09-10T20:38:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.032+0000 7f71df7fe640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71cc077f70 0x7f71cc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.033+0000 7f71df7fe640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71cc077f70 0x7f71cc07a430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f71d00058c0 tx=0x7f71d000a2d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.040+0000 7f71c7fff640 1 -- 172.21.15.28:0/3728119031 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f71d803e070 con 0x7f71e0162260 2024-09-10T20:38:31.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.262+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f71e00b9f30 con 0x7f71e0162260 2024-09-10T20:38:31.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.263+0000 7f71c7fff640 1 -- 172.21.15.28:0/3728119031 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f71d80831e0 con 0x7f71e0162260 2024-09-10T20:38:31.267 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:38:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71cc077f70 msgr2=0x7f71cc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71cc077f70 0x7f71cc07a430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f71d00058c0 tx=0x7f71d000a2d0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 msgr2=0x7f71e01f80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e01f80b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006010 tx=0x7f71d802f6d0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 shutdown_connections 2024-09-10T20:38:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71cc077f70 0x7f71cc07a430 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71e0162260 0x7f71e01f80b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71e015de70 0x7f71e01f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 --2- 172.21.15.28:0/3728119031 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71e00cf780 0x7f71e015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 >> 172.21.15.28:0/3728119031 conn(0x7f71e00d3fb0 msgr2=0x7f71e00ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 shutdown_connections 2024-09-10T20:38:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31.265+0000 7f71e5ef6640 1 -- 172.21.15.28:0/3728119031 wait complete. 2024-09-10T20:38:31.279 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:38:31.280 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-09-10T20:38:31.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:31.281 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:38:31.298 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42719sec preferred_lft 42719sec 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:31.299 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link tentative 2024-09-10T20:38:31.300 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:31.301 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:38:31.363 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:38:31.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:31.365 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:38:31.422 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:38:31.422 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:38:31.422 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:38:31.423 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:38:31.423 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:38:31.423 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:31.423 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:31.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:38:31.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:38:31.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:38:31.423 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:31.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:31.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:31.554 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:38:31.570 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:38:31.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:31.571 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-10T20:38:31.627 INFO:teuthology.orchestra.run.smithi028.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-09-10T20:38:31.628 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:31.628 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:31.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:38:31.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-09-10T20:38:31.629 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:31.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:31.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:31.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 06:9a:b1:bd:af:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-10T20:38:31.773 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-10T20:38:31.774 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.774 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::49a:b1ff:febd:afac/64 scope link 2024-09-10T20:38:31.774 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:31.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-10T20:38:31.840 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 06:9a:b1:bd:af:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-10T20:38:31.841 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-10T20:38:31.842 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.842 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::49a:b1ff:febd:afac/64 scope link 2024-09-10T20:38:31.842 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:31.843 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-09-10T20:38:31.843 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:31.843 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:31.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-09-10T20:38:31.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-10T20:38:31.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-09-10T20:38:31.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-09-10T20:38:31.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-09-10T20:38:31.844 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:31.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:31.999 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:31.999 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:38:32.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.1 up 2024-09-10T20:38:32.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.1 master ceph-brx 2024-09-10T20:38:32.000 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:38:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:32.081 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:32.081 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-09-10T20:38:32.096 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-09-10T20:38:32.097 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:32.097 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-09-10T20:38:32.152 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:38:32.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:32.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-10T20:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-10T20:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-10T20:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-10T20:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-10T20:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-09-10T20:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:38:32.255 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:38:32.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:32.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-09-10T20:38:32.321 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:client4668 2024-09-10T20:38:32.420 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:32.420 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:32.437 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:32.534 INFO:teuthology.orchestra.run.smithi028.stdout:client4668 2024-09-10T20:38:32.534 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:32.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:32.591 INFO:teuthology.orchestra.run.smithi028.stdout:client4668 2024-09-10T20:38:32.591 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/943b2354-acb0-43ad-b06a-e7974842506f.client4668/status') 2024-09-10T20:38:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:32.651 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4668 (3)192.168.0.2:0/3244232928 2024-09-10T20:38:32.651 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-09-10T20:38:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-09-10T20:38:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-09-10T20:38:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:69 bytes copied, 2.3254e-05 s, 3.0 MB/s 2024-09-10T20:38:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:32.658 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-09-10T20:38:32.658 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-10T20:38:32.658 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:38:32.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:38:32.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.759+0000 7f6b5e317640 1 -- 172.21.15.28:0/1012731739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 msgr2=0x7f6b580cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.759+0000 7f6b5e317640 1 --2- 172.21.15.28:0/1012731739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580cfbf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6b5808c020 tx=0x7f6b5002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.759+0000 7f6b5e317640 1 -- 172.21.15.28:0/1012731739 shutdown_connections 2024-09-10T20:38:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.759+0000 7f6b5e317640 1 --2- 172.21.15.28:0/1012731739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580cfbf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.759+0000 7f6b5e317640 1 --2- 172.21.15.28:0/1012731739 >> v2:172.21.15.159:3300/0 conn(0x7f6b5815de70 0x7f6b580d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.759+0000 7f6b5e317640 1 -- 172.21.15.28:0/1012731739 >> 172.21.15.28:0/1012731739 conn(0x7f6b580d3fb0 msgr2=0x7f6b580d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.759+0000 7f6b5e317640 1 -- 172.21.15.28:0/1012731739 shutdown_connections 2024-09-10T20:38:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.760+0000 7f6b5e317640 1 -- 172.21.15.28:0/1012731739 wait complete. 2024-09-10T20:38:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.761+0000 7f6b5e317640 1 Processor -- start 2024-09-10T20:38:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.761+0000 7f6b5e317640 1 -- start start 2024-09-10T20:38:32.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.761+0000 7f6b5e317640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b580cf780 0x7f6b581f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:32.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.761+0000 7f6b5e317640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5815de70 0x7f6b580ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.761+0000 7f6b5e317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.761+0000 7f6b5e317640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b580b6f80 con 0x7f6b58162260 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5e317640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6b580b6e00 con 0x7f6b580cf780 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5e317640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6b580b7100 con 0x7f6b5815de70 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580a1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48672/0 (socket says 172.21.15.28:48672) 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 -- 172.21.15.28:0/2497127934 learned_addr learned my addr 172.21.15.28:0/2497127934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b57fff640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b580cf780 0x7f6b581f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b4ffff640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5815de70 0x7f6b580ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 -- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5815de70 msgr2=0x7f6b580ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5815de70 0x7f6b580ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 -- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b580cf780 msgr2=0x7f6b581f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b580cf780 0x7f6b581f57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 -- 172.21.15.28:0/2497127934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b581f5d30 con 0x7f6b58162260 2024-09-10T20:38:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5c8b4640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580a1380 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6b5002aea0 tx=0x7f6b50003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b557fa640 1 -- 172.21.15.28:0/2497127934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50031640 con 0x7f6b58162260 2024-09-10T20:38:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b557fa640 1 -- 172.21.15.28:0/2497127934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b50003e10 con 0x7f6b58162260 2024-09-10T20:38:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b557fa640 1 -- 172.21.15.28:0/2497127934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50033330 con 0x7f6b58162260 2024-09-10T20:38:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b5809df40 con 0x7f6b58162260 2024-09-10T20:38:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.762+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b5809e500 con 0x7f6b58162260 2024-09-10T20:38:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.763+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b580d2b90 con 0x7f6b58162260 2024-09-10T20:38:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.764+0000 7f6b557fa640 1 -- 172.21.15.28:0/2497127934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6b5004b030 con 0x7f6b58162260 2024-09-10T20:38:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.765+0000 7f6b557fa640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6b38077fa0 0x7f6b3807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.765+0000 7f6b557fa640 1 -- 172.21.15.28:0/2497127934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6b500baf60 con 0x7f6b58162260 2024-09-10T20:38:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.765+0000 7f6b57fff640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6b38077fa0 0x7f6b3807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.765+0000 7f6b57fff640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6b38077fa0 0x7f6b3807a460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009890 tx=0x7f6b48008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.769+0000 7f6b557fa640 1 -- 172.21.15.28:0/2497127934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6b5003f030 con 0x7f6b58162260 2024-09-10T20:38:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.992+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6b580b9f30 con 0x7f6b58162260 2024-09-10T20:38:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.992+0000 7f6b557fa640 1 -- 172.21.15.28:0/2497127934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f6b50083a60 con 0x7f6b58162260 2024-09-10T20:38:32.996 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:38:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.994+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6b38077fa0 msgr2=0x7f6b3807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6b38077fa0 0x7f6b3807a460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009890 tx=0x7f6b48008040 comp rx=0 tx=0).stop 2024-09-10T20:38:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 msgr2=0x7f6b580a1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580a1380 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6b5002aea0 tx=0x7f6b50003c20 comp rx=0 tx=0).stop 2024-09-10T20:38:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 shutdown_connections 2024-09-10T20:38:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6b38077fa0 0x7f6b3807a460 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58162260 0x7f6b580a1380 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5815de70 0x7f6b580ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 --2- 172.21.15.28:0/2497127934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b580cf780 0x7f6b581f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 >> 172.21.15.28:0/2497127934 conn(0x7f6b580d3fb0 msgr2=0x7f6b580d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 shutdown_connections 2024-09-10T20:38:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:32.995+0000 7f6b5e317640 1 -- 172.21.15.28:0/2497127934 wait complete. 2024-09-10T20:38:33.010 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-10T20:38:33.010 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:38:33.010 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:38:33.010 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:38:33.010 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:38:33.011 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-10T20:38:33.011 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-10T20:38:33.011 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:38:33.011 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:38:33.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:33.012 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:38:33.031 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:33.031 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:33.031 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:38:33.032 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:33.033 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:38:33.033 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42736sec preferred_lft 42736sec 2024-09-10T20:38:33.033 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:38:33.033 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:33.033 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-10T20:38:33.033 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:33.033 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:33.033 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add name ceph-brx type bridge 2024-09-10T20:38:33.033 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip addr flush dev ceph-brx 2024-09-10T20:38:33.034 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set ceph-brx up 2024-09-10T20:38:33.034 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-10T20:38:33.034 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:33.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:33.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:33.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:33.260 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:38:33.286 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:38:33.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:33.288 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:38:33.355 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:38:33.355 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:38:33.355 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:38:33.355 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:38:33.356 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:38:33.356 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:33.356 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:33.356 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:38:33.356 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:38:33.356 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:38:33.357 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:33.443 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:33.636 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:33.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:33.642 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:38:33.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:33.657 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list-id 2024-09-10T20:38:33.713 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:33.713 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:33.713 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:38:33.713 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-10T20:38:33.713 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:33.800 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:33.823 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:33.830 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-10T20:38:33.830 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:33.878 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:33.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:33.951 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:33.951 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:33.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:38:33.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set dev brx.0 master ceph-brx 2024-09-10T20:38:33.952 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:33.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:34.021 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:34.028 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:34.028 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:38:34.043 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:38:34.043 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:34.043 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:38:34.098 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:38:34.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:34.100 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:38:34.189 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:38:34.189 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:38:34.189 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:38:34.312 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:38:34.313 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:38:34.313 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:38:34.313 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:34.313 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:38:34.321 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:38:34.322 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:38:34.322 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:38:34.322 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:38:34.322 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:38:34.322 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:38:34.324 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:38:34.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:34.324 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:38:34.348 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:34.437 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:34.438 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:34.444 INFO:teuthology.orchestra.run.smithi159.stdout:client4663 2024-09-10T20:38:34.444 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:34.445 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:34.461 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:34.549 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:34.550 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:34.556 INFO:teuthology.orchestra.run.smithi159.stdout:client4663 2024-09-10T20:38:34.556 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:34.605 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:34.606 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:34.612 INFO:teuthology.orchestra.run.smithi159.stdout:client4663 2024-09-10T20:38:34.612 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/943b2354-acb0-43ad-b06a-e7974842506f.client4663/status') 2024-09-10T20:38:34.659 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:34.669 INFO:teuthology.orchestra.run.smithi159.stdout:instance: client.4663 (3)192.168.0.1:0/3811128951 2024-09-10T20:38:34.670 INFO:teuthology.orchestra.run.smithi159.stdout:blocklisted: false 2024-09-10T20:38:34.670 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records in 2024-09-10T20:38:34.670 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records out 2024-09-10T20:38:34.670 INFO:teuthology.orchestra.run.smithi159.stderr:69 bytes copied, 2.3304e-05 s, 3.0 MB/s 2024-09-10T20:38:34.670 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:34.677 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-09-10T20:38:34.677 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-10T20:38:34.677 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-10T20:38:34.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:38:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.792+0000 7f4586ff3640 1 -- 172.21.15.28:0/1176022003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bb40 msgr2=0x7f45800af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.792+0000 7f4586ff3640 1 --2- 172.21.15.28:0/1176022003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bb40 0x7f45800af660 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f458015c8d0 tx=0x7f457002a510 comp rx=0 tx=0).stop 2024-09-10T20:38:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.792+0000 7f4586ff3640 1 -- 172.21.15.28:0/1176022003 shutdown_connections 2024-09-10T20:38:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.792+0000 7f4586ff3640 1 --2- 172.21.15.28:0/1176022003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bb40 0x7f45800af660 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.792+0000 7f4586ff3640 1 -- 172.21.15.28:0/1176022003 >> 172.21.15.28:0/1176022003 conn(0x7f458006dbf0 msgr2=0x7f458006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.792+0000 7f4586ff3640 1 -- 172.21.15.28:0/1176022003 shutdown_connections 2024-09-10T20:38:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.792+0000 7f4586ff3640 1 -- 172.21.15.28:0/1176022003 wait complete. 2024-09-10T20:38:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.794+0000 7f4586ff3640 1 Processor -- start 2024-09-10T20:38:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.794+0000 7f4586ff3640 1 -- start start 2024-09-10T20:38:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4586ff3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f458006d1f0 0x7f45801448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4586ff3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f458015bb40 0x7f4580070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4586ff3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bf20 0x7f458015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4586ff3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f458015d680 con 0x7f458015bf20 2024-09-10T20:38:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4586ff3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f458015d500 con 0x7f458006d1f0 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4586ff3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f458015d800 con 0x7f458015bb40 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bf20 0x7f458015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bf20 0x7f458015a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48690/0 (socket says 172.21.15.28:48690) 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 -- 172.21.15.28:0/469112151 learned_addr learned my addr 172.21.15.28:0/469112151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4584d8f640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f458006d1f0 0x7f45801448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f457ffff640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f458015bb40 0x7f4580070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 -- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f458015bb40 msgr2=0x7f4580070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f458015bb40 0x7f4580070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 -- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f458006d1f0 msgr2=0x7f45801448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f458006d1f0 0x7f45801448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 -- 172.21.15.28:0/469112151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45801f7c40 con 0x7f458015bf20 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4585590640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bf20 0x7f458015a7f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f457800c7a0 tx=0x7f457800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f457d7fa640 1 -- 172.21.15.28:0/469112151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578017070 con 0x7f458015bf20 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f457d7fa640 1 -- 172.21.15.28:0/469112151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4578005070 con 0x7f458015bf20 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.795+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45801f7ee0 con 0x7f458015bf20 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.796+0000 7f457d7fa640 1 -- 172.21.15.28:0/469112151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45780054c0 con 0x7f458015bf20 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.796+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45801f84a0 con 0x7f458015bf20 2024-09-10T20:38:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.796+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f458015dd20 con 0x7f458015bf20 2024-09-10T20:38:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.797+0000 7f457d7fa640 1 -- 172.21.15.28:0/469112151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4578005660 con 0x7f458015bf20 2024-09-10T20:38:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.798+0000 7f457d7fa640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f456c077f70 0x7f456c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.798+0000 7f457d7fa640 1 -- 172.21.15.28:0/469112151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f45780996d0 con 0x7f458015bf20 2024-09-10T20:38:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.798+0000 7f4584d8f640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f456c077f70 0x7f456c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.798+0000 7f4584d8f640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f456c077f70 0x7f456c07a430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4570034000 tx=0x7f4570004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:34.802+0000 7f457d7fa640 1 -- 172.21.15.28:0/469112151 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f45780621d0 con 0x7f458015bf20 2024-09-10T20:38:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.024+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f45800b5590 con 0x7f458015bf20 2024-09-10T20:38:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.024+0000 7f457d7fa640 1 -- 172.21.15.28:0/469112151 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f4578061ad0 con 0x7f458015bf20 2024-09-10T20:38:35.029 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.026+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f456c077f70 msgr2=0x7f456c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.026+0000 7f4586ff3640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f456c077f70 0x7f456c07a430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4570034000 tx=0x7f4570004730 comp rx=0 tx=0).stop 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.026+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bf20 msgr2=0x7f458015a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.026+0000 7f4586ff3640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bf20 0x7f458015a7f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f457800c7a0 tx=0x7f457800cc60 comp rx=0 tx=0).stop 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.026+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 shutdown_connections 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.026+0000 7f4586ff3640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f456c077f70 0x7f456c07a430 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.027+0000 7f4586ff3640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f458015bf20 0x7f458015a7f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.027+0000 7f4586ff3640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f458015bb40 0x7f4580070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.027+0000 7f4586ff3640 1 --2- 172.21.15.28:0/469112151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f458006d1f0 0x7f45801448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.027+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 >> 172.21.15.28:0/469112151 conn(0x7f458006dbf0 msgr2=0x7f4580159010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.027+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 shutdown_connections 2024-09-10T20:38:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:35.027+0000 7f4586ff3640 1 -- 172.21.15.28:0/469112151 wait complete. 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:38:35.042 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-09-10T20:38:35.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.043 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:38:35.059 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:35.060 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42734sec preferred_lft 42734sec 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:38:35.061 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link tentative 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.062 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:38:35.125 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:38:35.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.127 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:38:35.184 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:38:35.184 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:38:35.184 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:38:35.184 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:38:35.184 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:38:35.185 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:35.185 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:35.185 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:38:35.185 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:38:35.185 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:38:35.185 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:35.273 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:35.304 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:35.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.311 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:38:35.326 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:38:35.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.327 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list-id 2024-09-10T20:38:35.382 INFO:teuthology.orchestra.run.smithi159.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-09-10T20:38:35.382 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:35.382 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:35.383 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:38:35.383 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-09-10T20:38:35.383 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:35.471 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:35.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:35.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.500 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ea:c6:70:58:27:50 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-10T20:38:35.525 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-10T20:38:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e8c6:70ff:fe58:2750/64 scope link 2024-09-10T20:38:35.526 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.527 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ea:c6:70:58:27:50 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-10T20:38:35.592 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.593 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e8c6:70ff:fe58:2750/64 scope link 2024-09-10T20:38:35.593 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:38:35.593 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-09-10T20:38:35.594 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:35.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:35.740 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:35.745 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:38:35.745 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:38:35.746 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.1 up 2024-09-10T20:38:35.746 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set dev brx.1 master ceph-brx 2024-09-10T20:38:35.746 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:38:35.793 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:35.815 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:35.821 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:35.821 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-09-10T20:38:35.836 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-09-10T20:38:35.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:38:35.837 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-09-10T20:38:35.892 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:38:35.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.892 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-10T20:38:35.975 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-10T20:38:35.976 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-10T20:38:35.976 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:38:35.984 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-10T20:38:35.984 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-10T20:38:35.984 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:38:35.984 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:35.985 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:35.993 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-10T20:38:35.993 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-10T20:38:35.993 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:38:35.993 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:38:35.993 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:38:35.994 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:38:35.996 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:38:35.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:35.997 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-09-10T20:38:36.062 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:36.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:36.178 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:36.178 INFO:teuthology.orchestra.run.smithi159.stdout:client4562 2024-09-10T20:38:36.178 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:36.179 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:36.194 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:36.283 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:36.284 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:36.290 INFO:teuthology.orchestra.run.smithi159.stdout:client4562 2024-09-10T20:38:36.291 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:38:36.338 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:38:36.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:36.346 INFO:teuthology.orchestra.run.smithi159.stdout:client4562 2024-09-10T20:38:36.347 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/943b2354-acb0-43ad-b06a-e7974842506f.client4562/status') 2024-09-10T20:38:36.395 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:38:36.405 INFO:teuthology.orchestra.run.smithi159.stdout:instance: client.4562 (3)192.168.0.2:0/232062633 2024-09-10T20:38:36.405 INFO:teuthology.orchestra.run.smithi159.stdout:blocklisted: false 2024-09-10T20:38:36.405 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records in 2024-09-10T20:38:36.405 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records out 2024-09-10T20:38:36.406 INFO:teuthology.orchestra.run.smithi159.stderr:68 bytes copied, 2.3307e-05 s, 2.9 MB/s 2024-09-10T20:38:36.406 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:38:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:38:36.413 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-09-10T20:38:36.535 INFO:tasks.cephfs_test_runner:Starting test: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate binary .meta file on legacy subvol root doesn't break the system 2024-09-10T20:38:36.535 INFO:tasks.cephfs_test_runner:test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:38:36.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.649+0000 7f93bddc2640 1 -- 172.21.15.28:0/2187123368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 msgr2=0x7f93b80d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.649+0000 7f93bddc2640 1 --2- 172.21.15.28:0/2187123368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 0x7f93b80d2060 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009830 tx=0x7f93ac02a530 comp rx=0 tx=0).stop 2024-09-10T20:38:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.649+0000 7f93bddc2640 1 -- 172.21.15.28:0/2187123368 shutdown_connections 2024-09-10T20:38:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.649+0000 7f93bddc2640 1 --2- 172.21.15.28:0/2187123368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 0x7f93b80d2060 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.649+0000 7f93bddc2640 1 -- 172.21.15.28:0/2187123368 >> 172.21.15.28:0/2187123368 conn(0x7f93b809d2d0 msgr2=0x7f93b80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.649+0000 7f93bddc2640 1 -- 172.21.15.28:0/2187123368 shutdown_connections 2024-09-10T20:38:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.649+0000 7f93bddc2640 1 -- 172.21.15.28:0/2187123368 wait complete. 2024-09-10T20:38:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.650+0000 7f93bddc2640 1 Processor -- start 2024-09-10T20:38:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.651+0000 7f93bddc2640 1 -- start start 2024-09-10T20:38:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.651+0000 7f93bddc2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b80d07e0 0x7f93b81f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.651+0000 7f93bddc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b80d0bc0 0x7f93b80b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.651+0000 7f93bddc2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 0x7f93b80b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.651+0000 7f93bddc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80d0030 con 0x7f93b80d0bc0 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.651+0000 7f93bddc2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80cfeb0 con 0x7f93b80d4950 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.651+0000 7f93bddc2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f93b81696f0 con 0x7f93b80d07e0 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b80d0bc0 0x7f93b80b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b80d0bc0 0x7f93b80b6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48700/0 (socket says 172.21.15.28:48700) 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 -- 172.21.15.28:0/1769196661 learned_addr learned my addr 172.21.15.28:0/1769196661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b77fe640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b80d07e0 0x7f93b81f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b7fff640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 0x7f93b80b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 -- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b80d07e0 msgr2=0x7f93b81f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b80d07e0 0x7f93b81f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 -- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 msgr2=0x7f93b80b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 0x7f93b80b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 -- 172.21.15.28:0/1769196661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b8154f10 con 0x7f93b80d0bc0 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93b6ffd640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b80d0bc0 0x7f93b80b6f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009950 tx=0x7f93ac033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f939ffff640 1 -- 172.21.15.28:0/1769196661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac031d60 con 0x7f93b80d0bc0 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f939ffff640 1 -- 172.21.15.28:0/1769196661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93ac033e10 con 0x7f93b80d0bc0 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b81551f0 con 0x7f93b80d0bc0 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f939ffff640 1 -- 172.21.15.28:0/1769196661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac033310 con 0x7f93b80d0bc0 2024-09-10T20:38:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.652+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b81557b0 con 0x7f93b80d0bc0 2024-09-10T20:38:36.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.653+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b80d2060 con 0x7f93b80d0bc0 2024-09-10T20:38:36.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.654+0000 7f939ffff640 1 -- 172.21.15.28:0/1769196661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f93ac04b030 con 0x7f93b80d0bc0 2024-09-10T20:38:36.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.654+0000 7f939ffff640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f93a4077f70 0x7f93a407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.655+0000 7f939ffff640 1 -- 172.21.15.28:0/1769196661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f93ac0bad60 con 0x7f93b80d0bc0 2024-09-10T20:38:36.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.655+0000 7f93b77fe640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f93a4077f70 0x7f93a407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.655+0000 7f93b77fe640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f93a4077f70 0x7f93a407a430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f93a80016b0 tx=0x7f93a8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.659+0000 7f939ffff640 1 -- 172.21.15.28:0/1769196661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f93ac03f030 con 0x7f93b80d0bc0 2024-09-10T20:38:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.847+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93b80d1a30 con 0x7f93b80d0bc0 2024-09-10T20:38:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.848+0000 7f939ffff640 1 -- 172.21.15.28:0/1769196661 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16669 (secure 0 0 0) 0x7f93ac083860 con 0x7f93b80d0bc0 2024-09-10T20:38:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:36.852 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:13.395359+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f93a4077f70 msgr2=0x7f93a407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f93a4077f70 0x7f93a407a430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f93a80016b0 tx=0x7f93a8009040 comp rx=0 tx=0).stop 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b80d0bc0 msgr2=0x7f93b80b6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b80d0bc0 0x7f93b80b6f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009950 tx=0x7f93ac033c20 comp rx=0 tx=0).stop 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 shutdown_connections 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f93a4077f70 0x7f93a407a430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93b80d4950 0x7f93b80b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b80d0bc0 0x7f93b80b6f20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 --2- 172.21.15.28:0/1769196661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93b80d07e0 0x7f93b81f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 >> 172.21.15.28:0/1769196661 conn(0x7f93b809d2d0 msgr2=0x7f93b806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.850+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 shutdown_connections 2024-09-10T20:38:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.851+0000 7f93bddc2640 1 -- 172.21.15.28:0/1769196661 wait complete. 2024-09-10T20:38:36.866 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-10T20:38:09.596930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-10T20:38:10.711700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:38:36.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:38:36.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/2727782797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3806e940 msgr2=0x7f0d380ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/2727782797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3806e940 0x7f0d380ced20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0d30005fd0 tx=0x7f0d3002aa10 comp rx=0 tx=0).stop 2024-09-10T20:38:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/2727782797 shutdown_connections 2024-09-10T20:38:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/2727782797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3806e940 0x7f0d380ced20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/2727782797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d380b3580 0x7f0d3806e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/2727782797 >> 172.21.15.28:0/2727782797 conn(0x7f0d3806d110 msgr2=0x7f0d380b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/2727782797 shutdown_connections 2024-09-10T20:38:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.971+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/2727782797 wait complete. 2024-09-10T20:38:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 Processor -- start 2024-09-10T20:38:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 -- start start 2024-09-10T20:38:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d3806e940 0x7f0d3813e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d38152e00 0x7f0d381f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d38156410 0x7f0d38153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3815f820 con 0x7f0d38156410 2024-09-10T20:38:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0d380cf630 con 0x7f0d3806e940 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.973+0000 7f0d3f8ed640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0d3815f9a0 con 0x7f0d38152e00 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d38156410 0x7f0d38153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d38156410 0x7f0d38153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48720/0 (socket says 172.21.15.28:48720) 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 -- 172.21.15.28:0/4066465103 learned_addr learned my addr 172.21.15.28:0/4066465103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3ce88640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d38152e00 0x7f0d381f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3d689640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d3806e940 0x7f0d3813e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 -- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d38152e00 msgr2=0x7f0d381f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d38152e00 0x7f0d381f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 -- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d3806e940 msgr2=0x7f0d3813e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d3806e940 0x7f0d3813e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 -- 172.21.15.28:0/4066465103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d381553e0 con 0x7f0d38156410 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3de8a640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d38156410 0x7f0d38153350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009d90 tx=0x7f0d30003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d2dffb640 1 -- 172.21.15.28:0/4066465103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30047070 con 0x7f0d38156410 2024-09-10T20:38:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d2dffb640 1 -- 172.21.15.28:0/4066465103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d30004250 con 0x7f0d38156410 2024-09-10T20:38:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d38155650 con 0x7f0d38156410 2024-09-10T20:38:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d2dffb640 1 -- 172.21.15.28:0/4066465103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30033050 con 0x7f0d38156410 2024-09-10T20:38:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.974+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d38155c10 con 0x7f0d38156410 2024-09-10T20:38:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.975+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3815ff30 con 0x7f0d38156410 2024-09-10T20:38:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.976+0000 7f0d2dffb640 1 -- 172.21.15.28:0/4066465103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0d30004660 con 0x7f0d38156410 2024-09-10T20:38:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.977+0000 7f0d2dffb640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0d1c077f70 0x7f0d1c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.977+0000 7f0d2dffb640 1 -- 172.21.15.28:0/4066465103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0d300bb230 con 0x7f0d38156410 2024-09-10T20:38:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.977+0000 7f0d3d689640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0d1c077f70 0x7f0d1c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.977+0000 7f0d3d689640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0d1c077f70 0x7f0d1c07a430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d34001720 tx=0x7f0d34009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:36.981+0000 7f0d2dffb640 1 -- 172.21.15.28:0/4066465103 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0d300836e0 con 0x7f0d38156410 2024-09-10T20:38:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.164+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0d3813e910 con 0x7f0d38156410 2024-09-10T20:38:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.164+0000 7f0d2dffb640 1 -- 172.21.15.28:0/4066465103 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f0d30088110 con 0x7f0d38156410 2024-09-10T20:38:37.168 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:38:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0d1c077f70 msgr2=0x7f0d1c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0d1c077f70 0x7f0d1c07a430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d34001720 tx=0x7f0d34009040 comp rx=0 tx=0).stop 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d38156410 msgr2=0x7f0d38153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d38156410 0x7f0d38153350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009d90 tx=0x7f0d30003f10 comp rx=0 tx=0).stop 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 shutdown_connections 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0d1c077f70 0x7f0d1c07a430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d38156410 0x7f0d38153350 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d38152e00 0x7f0d381f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.166+0000 7f0d3f8ed640 1 --2- 172.21.15.28:0/4066465103 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d3806e940 0x7f0d3813e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.167+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 >> 172.21.15.28:0/4066465103 conn(0x7f0d3806d110 msgr2=0x7f0d380d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.167+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 shutdown_connections 2024-09-10T20:38:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.167+0000 7f0d3f8ed640 1 -- 172.21.15.28:0/4066465103 wait complete. 2024-09-10T20:38:37.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-10T20:38:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.287+0000 7f71f63c6640 1 -- 172.21.15.28:0/458343424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d8004800 msgr2=0x7f71d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.287+0000 7f71f63c6640 1 --2- 172.21.15.28:0/458343424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d8004800 0x7f71d8004c80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009b50 tx=0x7f71d8005830 comp rx=0 tx=0).stop 2024-09-10T20:38:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.287+0000 7f71f4963640 1 -- 172.21.15.28:0/458343424 reap_dead start 2024-09-10T20:38:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.287+0000 7f71f63c6640 1 -- 172.21.15.28:0/458343424 shutdown_connections 2024-09-10T20:38:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.287+0000 7f71f63c6640 1 -- 172.21.15.28:0/458343424 >> 172.21.15.28:0/458343424 conn(0x7f71f00d1f30 msgr2=0x7f71f00d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.287+0000 7f71f63c6640 1 -- 172.21.15.28:0/458343424 shutdown_connections 2024-09-10T20:38:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.287+0000 7f71f63c6640 1 -- 172.21.15.28:0/458343424 wait complete. 2024-09-10T20:38:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.288+0000 7f71f63c6640 1 Processor -- start 2024-09-10T20:38:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.288+0000 7f71f63c6640 1 -- start start 2024-09-10T20:38:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71f63c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d8004800 0x7f71f01f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71f63c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f01f45c0 0x7f71f01564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71f63c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f015a010 0x7f71f0156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71f63c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71f00d1030 con 0x7f71f01f45c0 2024-09-10T20:38:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71f63c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f71f00d0eb0 con 0x7f71f015a010 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71f63c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f71f00ccb20 con 0x7f71d8004800 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f01f45c0 0x7f71f01564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f01f45c0 0x7f71f01564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48752/0 (socket says 172.21.15.28:48752) 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71ef7fe640 1 -- 172.21.15.28:0/1556882848 learned_addr learned my addr 172.21.15.28:0/1556882848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71effff640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d8004800 0x7f71f01f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71f4963640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f015a010 0x7f71f0156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71ef7fe640 1 -- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d8004800 msgr2=0x7f71f01f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71ef7fe640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d8004800 0x7f71f01f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71ef7fe640 1 -- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f015a010 msgr2=0x7f71f0156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.289+0000 7f71ef7fe640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f015a010 0x7f71f0156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.290+0000 7f71ef7fe640 1 -- 172.21.15.28:0/1556882848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f0153fc0 con 0x7f71f01f45c0 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.290+0000 7f71ef7fe640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f01f45c0 0x7f71f01564c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009a70 tx=0x7f71e40336d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.290+0000 7f71ecff9640 1 -- 172.21.15.28:0/1556882848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e4043070 con 0x7f71f01f45c0 2024-09-10T20:38:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.290+0000 7f71ecff9640 1 -- 172.21.15.28:0/1556882848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71e4004120 con 0x7f71f01f45c0 2024-09-10T20:38:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.290+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f01f4a60 con 0x7f71f01f45c0 2024-09-10T20:38:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.290+0000 7f71ecff9640 1 -- 172.21.15.28:0/1556882848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e4038040 con 0x7f71f01f45c0 2024-09-10T20:38:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.290+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f01f5020 con 0x7f71f01f45c0 2024-09-10T20:38:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.291+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71f01f57d0 con 0x7f71f01f45c0 2024-09-10T20:38:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.292+0000 7f71ecff9640 1 -- 172.21.15.28:0/1556882848 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f71e403b030 con 0x7f71f01f45c0 2024-09-10T20:38:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.292+0000 7f71ecff9640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71dc077fa0 0x7f71dc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.293+0000 7f71ecff9640 1 -- 172.21.15.28:0/1556882848 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71e40ba040 con 0x7f71f01f45c0 2024-09-10T20:38:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.293+0000 7f71effff640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71dc077fa0 0x7f71dc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.293+0000 7f71effff640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71dc077fa0 0x7f71dc07a460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f71e0001220 tx=0x7f71e00067a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.297+0000 7f71ecff9640 1 -- 172.21.15.28:0/1556882848 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f71e40824f0 con 0x7f71f01f45c0 2024-09-10T20:38:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.481+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f71f00ce750 con 0x7f71f01f45c0 2024-09-10T20:38:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.481+0000 7f71ecff9640 1 -- 172.21.15.28:0/1556882848 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f71e4086f20 con 0x7f71f01f45c0 2024-09-10T20:38:37.485 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-10T20:38:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71dc077fa0 msgr2=0x7f71dc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71dc077fa0 0x7f71dc07a460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f71e0001220 tx=0x7f71e00067a0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f01f45c0 msgr2=0x7f71f01564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f01f45c0 0x7f71f01564c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009a70 tx=0x7f71e40336d0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 shutdown_connections 2024-09-10T20:38:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f71dc077fa0 0x7f71dc07a460 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71f015a010 0x7f71f0156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f01f45c0 0x7f71f01564c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 --2- 172.21.15.28:0/1556882848 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d8004800 0x7f71f01f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 >> 172.21.15.28:0/1556882848 conn(0x7f71f00d1f30 msgr2=0x7f71f015c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 shutdown_connections 2024-09-10T20:38:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.483+0000 7f71f63c6640 1 -- 172.21.15.28:0/1556882848 wait complete. 2024-09-10T20:38:37.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-10T20:38:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 -- 172.21.15.28:0/621714986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd81587a0 msgr2=0x7fadd815b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 --2- 172.21.15.28:0/621714986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd81587a0 0x7fadd815b170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009830 tx=0x7fadc802a480 comp rx=0 tx=0).stop 2024-09-10T20:38:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 -- 172.21.15.28:0/621714986 shutdown_connections 2024-09-10T20:38:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 --2- 172.21.15.28:0/621714986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadd815e7d0 0x7fadd806f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 --2- 172.21.15.28:0/621714986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd81587a0 0x7fadd815b170 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 -- 172.21.15.28:0/621714986 >> 172.21.15.28:0/621714986 conn(0x7fadd806d110 msgr2=0x7fadd80b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 -- 172.21.15.28:0/621714986 shutdown_connections 2024-09-10T20:38:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.603+0000 7faddf262640 1 -- 172.21.15.28:0/621714986 wait complete. 2024-09-10T20:38:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.604+0000 7faddf262640 1 Processor -- start 2024-09-10T20:38:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.604+0000 7faddf262640 1 -- start start 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddf262640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd806fa50 0x7fadd80a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddf262640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd81587a0 0x7fadd8144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddf262640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadd815e7d0 0x7fadd80cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddf262640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd815dd60 con 0x7fadd81587a0 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddf262640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fadd815dbe0 con 0x7fadd806fa50 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddf262640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fadd815dee0 con 0x7fadd815e7d0 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7fadd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd81587a0 0x7fadd8144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7fadd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd81587a0 0x7fadd8144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48758/0 (socket says 172.21.15.28:48758) 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7fadd7fff640 1 -- 172.21.15.28:0/3718445391 learned_addr learned my addr 172.21.15.28:0/3718445391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddcffe640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd806fa50 0x7fadd80a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.605+0000 7faddd7ff640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadd815e7d0 0x7fadd80cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd7fff640 1 -- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadd815e7d0 msgr2=0x7fadd80cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd7fff640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadd815e7d0 0x7fadd80cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd7fff640 1 -- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd806fa50 msgr2=0x7fadd80a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd7fff640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd806fa50 0x7fadd80a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd7fff640 1 -- 172.21.15.28:0/3718445391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd80a9b20 con 0x7fadd81587a0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd7fff640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd81587a0 0x7fadd8144f90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e9f0 tx=0x7fadcc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd57fa640 1 -- 172.21.15.28:0/3718445391 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00cd20 con 0x7fadd81587a0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd57fa640 1 -- 172.21.15.28:0/3718445391 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadcc004020 con 0x7fadd81587a0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd81e4640 con 0x7fadd81587a0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7fadd57fa640 1 -- 172.21.15.28:0/3718445391 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc004450 con 0x7fadd81587a0 2024-09-10T20:38:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.606+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd81e4c00 con 0x7fadd81587a0 2024-09-10T20:38:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.607+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd806f840 con 0x7fadd81587a0 2024-09-10T20:38:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.608+0000 7fadd57fa640 1 -- 172.21.15.28:0/3718445391 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fadcc0035f0 con 0x7fadd81587a0 2024-09-10T20:38:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.608+0000 7fadd57fa640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fadc4077f70 0x7fadc407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.608+0000 7fadd57fa640 1 -- 172.21.15.28:0/3718445391 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fadcc09a4f0 con 0x7fadd81587a0 2024-09-10T20:38:37.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.608+0000 7faddcffe640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fadc4077f70 0x7fadc407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.609+0000 7faddcffe640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fadc4077f70 0x7fadc407a430 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fadc8034000 tx=0x7fadc8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.612+0000 7fadd57fa640 1 -- 172.21.15.28:0/3718445391 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fadd806f840 con 0x7fadd81587a0 2024-09-10T20:38:37.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.797+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fadd80b4440 con 0x7fadd81587a0 2024-09-10T20:38:37.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.797+0000 7fadd57fa640 1 -- 172.21.15.28:0/3718445391 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fadcc0629a0 con 0x7fadd81587a0 2024-09-10T20:38:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fadc4077f70 msgr2=0x7fadc407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fadc4077f70 0x7fadc407a430 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fadc8034000 tx=0x7fadc8004730 comp rx=0 tx=0).stop 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd81587a0 msgr2=0x7fadd8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd81587a0 0x7fadd8144f90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e9f0 tx=0x7fadcc00eeb0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 shutdown_connections 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fadc4077f70 0x7fadc407a430 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadd815e7d0 0x7fadd80cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd81587a0 0x7fadd8144f90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 --2- 172.21.15.28:0/3718445391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadd806fa50 0x7fadd80a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 >> 172.21.15.28:0/3718445391 conn(0x7fadd806d110 msgr2=0x7fadd806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 shutdown_connections 2024-09-10T20:38:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.800+0000 7faddf262640 1 -- 172.21.15.28:0/3718445391 wait complete. 2024-09-10T20:38:37.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-09-10T20:38:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/4002653322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a015de70 msgr2=0x7fb6a00d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/4002653322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a015de70 0x7fb6a00d2b90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00d4d80 tx=0x7fb69802a530 comp rx=0 tx=0).stop 2024-09-10T20:38:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/4002653322 shutdown_connections 2024-09-10T20:38:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/4002653322 >> v2:172.21.15.159:3300/0 conn(0x7fb6a00cf780 0x7fb6a00cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/4002653322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a015de70 0x7fb6a00d2b90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/4002653322 >> 172.21.15.28:0/4002653322 conn(0x7fb6a00d3fb0 msgr2=0x7fb6a00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/4002653322 shutdown_connections 2024-09-10T20:38:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.915+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/4002653322 wait complete. 2024-09-10T20:38:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.917+0000 7fb6a6ab6640 1 Processor -- start 2024-09-10T20:38:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.917+0000 7fb6a6ab6640 1 -- start start 2024-09-10T20:38:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a6ab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a00cf780 0x7fb6a01f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a6ab6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015de70 0x7fb6a00a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a6ab6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a0161b50 0x7fb6a00b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a6ab6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a00b7690 con 0x7fb6a00cf780 2024-09-10T20:38:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a6ab6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a00b7510 con 0x7fb6a015de70 2024-09-10T20:38:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a6ab6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a00b7810 con 0x7fb6a0161b50 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a00cf780 0x7fb6a01f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a00cf780 0x7fb6a01f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48776/0 (socket says 172.21.15.28:48776) 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 -- 172.21.15.28:0/1536723900 learned_addr learned my addr 172.21.15.28:0/1536723900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb69ffff640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015de70 0x7fb6a00a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a5053640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a0161b50 0x7fb6a00b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 -- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a0161b50 msgr2=0x7fb6a00b8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a0161b50 0x7fb6a00b8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 -- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015de70 msgr2=0x7fb6a00a1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015de70 0x7fb6a00a1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 -- 172.21.15.28:0/1536723900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a01f7eb0 con 0x7fb6a00cf780 2024-09-10T20:38:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb6a4852640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a00cf780 0x7fb6a01f7560 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb698009780 tx=0x7fb698032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.918+0000 7fb69d7fa640 1 -- 172.21.15.28:0/1536723900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698040070 con 0x7fb6a00cf780 2024-09-10T20:38:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.919+0000 7fb69d7fa640 1 -- 172.21.15.28:0/1536723900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb698005310 con 0x7fb6a00cf780 2024-09-10T20:38:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.919+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a01f8190 con 0x7fb6a00cf780 2024-09-10T20:38:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.919+0000 7fb69d7fa640 1 -- 172.21.15.28:0/1536723900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698003800 con 0x7fb6a00cf780 2024-09-10T20:38:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.919+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a01f8750 con 0x7fb6a00cf780 2024-09-10T20:38:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.919+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a00cfbf0 con 0x7fb6a00cf780 2024-09-10T20:38:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.920+0000 7fb69d7fa640 1 -- 172.21.15.28:0/1536723900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb69804c030 con 0x7fb6a00cf780 2024-09-10T20:38:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.921+0000 7fb69d7fa640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb680077f70 0x7fb68007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.921+0000 7fb69d7fa640 1 -- 172.21.15.28:0/1536723900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb6980bae70 con 0x7fb6a00cf780 2024-09-10T20:38:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.921+0000 7fb69ffff640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb680077f70 0x7fb68007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.921+0000 7fb69ffff640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb680077f70 0x7fb68007a430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00cad80 tx=0x7fb69000d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:37.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:37.925+0000 7fb69d7fa640 1 -- 172.21.15.28:0/1536723900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb6980833b0 con 0x7fb6a00cf780 2024-09-10T20:38:38.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.185+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fb6a00b9f30 con 0x7fb6a00cf780 2024-09-10T20:38:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.461+0000 7fb69d7fa640 1 -- 172.21.15.28:0/1536723900 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v45) ==== 161+0+0 (secure 0 0 0) 0x7fb698087de0 con 0x7fb6a00cf780 2024-09-10T20:38:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb680077f70 msgr2=0x7fb68007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb680077f70 0x7fb68007a430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00cad80 tx=0x7fb69000d040 comp rx=0 tx=0).stop 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a00cf780 msgr2=0x7fb6a01f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a00cf780 0x7fb6a01f7560 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb698009780 tx=0x7fb698032d50 comp rx=0 tx=0).stop 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 shutdown_connections 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb680077f70 0x7fb68007a430 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6a0161b50 0x7fb6a00b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6a015de70 0x7fb6a00a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 --2- 172.21.15.28:0/1536723900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a00cf780 0x7fb6a01f7560 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 >> 172.21.15.28:0/1536723900 conn(0x7fb6a00d3fb0 msgr2=0x7fb6a00b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 shutdown_connections 2024-09-10T20:38:38.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.464+0000 7fb6a6ab6640 1 -- 172.21.15.28:0/1536723900 wait complete. 2024-09-10T20:38:38.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:38:38.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.583+0000 7f5a35d9f640 1 -- 172.21.15.28:0/3504990583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 msgr2=0x7f5a1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.583+0000 7f5a35d9f640 1 --2- 172.21.15.28:0/3504990583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 0x7f5a1c004c80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009b20 tx=0x7f5a1c005830 comp rx=0 tx=0).stop 2024-09-10T20:38:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.583+0000 7f5a3559e640 1 -- 172.21.15.28:0/3504990583 reap_dead start 2024-09-10T20:38:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.583+0000 7f5a35d9f640 1 -- 172.21.15.28:0/3504990583 shutdown_connections 2024-09-10T20:38:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.583+0000 7f5a35d9f640 1 -- 172.21.15.28:0/3504990583 >> 172.21.15.28:0/3504990583 conn(0x7f5a300d3e80 msgr2=0x7f5a300d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.583+0000 7f5a35d9f640 1 -- 172.21.15.28:0/3504990583 shutdown_connections 2024-09-10T20:38:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.583+0000 7f5a35d9f640 1 -- 172.21.15.28:0/3504990583 wait complete. 2024-09-10T20:38:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.584+0000 7f5a35d9f640 1 Processor -- start 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.584+0000 7f5a35d9f640 1 -- start start 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a35d9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 0x7f5a301f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a35d9f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a30157e10 0x7f5a30158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a35d9f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a3015e680 0x7f5a3015eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a35d9f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a300b2b70 con 0x7f5a1c004800 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a35d9f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5a300d4bb0 con 0x7f5a3015e680 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a35d9f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5a300b2cf0 con 0x7f5a30157e10 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a34d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 0x7f5a301f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a34d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 0x7f5a301f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48792/0 (socket says 172.21.15.28:48792) 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a34d9d640 1 -- 172.21.15.28:0/2678044624 learned_addr learned my addr 172.21.15.28:0/2678044624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a2ffff640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a30157e10 0x7f5a30158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.585+0000 7f5a3559e640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a3015e680 0x7f5a3015eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a34d9d640 1 -- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a30157e10 msgr2=0x7f5a30158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a34d9d640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a30157e10 0x7f5a30158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a34d9d640 1 -- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a3015e680 msgr2=0x7f5a3015eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a34d9d640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a3015e680 0x7f5a3015eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a34d9d640 1 -- 172.21.15.28:0/2678044624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a30157370 con 0x7f5a1c004800 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a34d9d640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 0x7f5a301f3d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001be0 tx=0x7f5a20006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a2d7fa640 1 -- 172.21.15.28:0/2678044624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20015070 con 0x7f5a1c004800 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a2d7fa640 1 -- 172.21.15.28:0/2678044624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a2000fb30 con 0x7f5a1c004800 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a301f8a20 con 0x7f5a1c004800 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a2d7fa640 1 -- 172.21.15.28:0/2678044624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20011430 con 0x7f5a1c004800 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.586+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a30157610 con 0x7f5a1c004800 2024-09-10T20:38:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.587+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a300ccc70 con 0x7f5a1c004800 2024-09-10T20:38:38.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.587+0000 7f5a2d7fa640 1 -- 172.21.15.28:0/2678044624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5a200115d0 con 0x7f5a1c004800 2024-09-10T20:38:38.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.588+0000 7f5a2d7fa640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a10077f70 0x7f5a1007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.588+0000 7f5a2d7fa640 1 -- 172.21.15.28:0/2678044624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5a2009adf0 con 0x7f5a1c004800 2024-09-10T20:38:38.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.588+0000 7f5a2ffff640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a10077f70 0x7f5a1007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.589+0000 7f5a2ffff640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a10077f70 0x7f5a1007a430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a18035000 tx=0x7f5a1802d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:38.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.592+0000 7f5a2d7fa640 1 -- 172.21.15.28:0/2678044624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5a20063850 con 0x7f5a1c004800 2024-09-10T20:38:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.778+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f5a300d0770 con 0x7f5a1c004800 2024-09-10T20:38:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.779+0000 7f5a2d7fa640 1 -- 172.21.15.28:0/2678044624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1998 (secure 0 0 0) 0x7f5a20063150 con 0x7f5a1c004800 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:38:38.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:38:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:38:38.785 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:38:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.780+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a10077f70 msgr2=0x7f5a1007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.780+0000 7f5a35d9f640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a10077f70 0x7f5a1007a430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a18035000 tx=0x7f5a1802d040 comp rx=0 tx=0).stop 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.780+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 msgr2=0x7f5a301f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.780+0000 7f5a35d9f640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 0x7f5a301f3d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001be0 tx=0x7f5a20006720 comp rx=0 tx=0).stop 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 shutdown_connections 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a10077f70 0x7f5a1007a430 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a3015e680 0x7f5a3015eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a30157e10 0x7f5a30158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 --2- 172.21.15.28:0/2678044624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c004800 0x7f5a301f3d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 >> 172.21.15.28:0/2678044624 conn(0x7f5a300d3e80 msgr2=0x7f5a3006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 shutdown_connections 2024-09-10T20:38:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.781+0000 7f5a35d9f640 1 -- 172.21.15.28:0/2678044624 wait complete. 2024-09-10T20:38:38.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:38:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.901+0000 7f439fb17640 1 -- 172.21.15.28:0/888492882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 msgr2=0x7f4384003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.901+0000 7f439fb17640 1 --2- 172.21.15.28:0/888492882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 0x7f4384003e20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f439400c290 tx=0x7f4384005830 comp rx=0 tx=0).stop 2024-09-10T20:38:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.901+0000 7f439e0b4640 1 -- 172.21.15.28:0/888492882 reap_dead start 2024-09-10T20:38:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.901+0000 7f439fb17640 1 -- 172.21.15.28:0/888492882 shutdown_connections 2024-09-10T20:38:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.901+0000 7f439fb17640 1 -- 172.21.15.28:0/888492882 >> 172.21.15.28:0/888492882 conn(0x7f43980d4130 msgr2=0x7f43980d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.901+0000 7f439fb17640 1 -- 172.21.15.28:0/888492882 shutdown_connections 2024-09-10T20:38:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.901+0000 7f439fb17640 1 -- 172.21.15.28:0/888492882 wait complete. 2024-09-10T20:38:38.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.902+0000 7f439fb17640 1 Processor -- start 2024-09-10T20:38:38.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.902+0000 7f439fb17640 1 -- start start 2024-09-10T20:38:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439fb17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 0x7f43980ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439fb17640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f439815efc0 0x7f4398150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439fb17640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4398160810 0x7f4398160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439fb17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43980b1db0 con 0x7f4384003a00 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439fb17640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f43980d28f0 con 0x7f4398160810 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439fb17640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f43980b1f30 con 0x7f439815efc0 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 0x7f43980ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 0x7f43980ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48806/0 (socket says 172.21.15.28:48806) 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 -- 172.21.15.28:0/3898864832 learned_addr learned my addr 172.21.15.28:0/3898864832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d0b2640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f439815efc0 0x7f4398150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439e0b4640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4398160810 0x7f4398160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 -- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f439815efc0 msgr2=0x7f4398150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f439815efc0 0x7f4398150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 -- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4398160810 msgr2=0x7f4398160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4398160810 0x7f4398160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 -- 172.21.15.28:0/3898864832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439815f510 con 0x7f4384003a00 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f439d8b3640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 0x7f43980ca690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f439400c3b0 tx=0x7f4394001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.903+0000 7f438e7fc640 1 -- 172.21.15.28:0/3898864832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394031040 con 0x7f4384003a00 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.904+0000 7f438e7fc640 1 -- 172.21.15.28:0/3898864832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43940035f0 con 0x7f4384003a00 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.904+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439815f7f0 con 0x7f4384003a00 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.904+0000 7f438e7fc640 1 -- 172.21.15.28:0/3898864832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394003a20 con 0x7f4384003a00 2024-09-10T20:38:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.904+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f439815fdb0 con 0x7f4384003a00 2024-09-10T20:38:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.905+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43980b1870 con 0x7f4384003a00 2024-09-10T20:38:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.905+0000 7f438e7fc640 1 -- 172.21.15.28:0/3898864832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4394003bc0 con 0x7f4384003a00 2024-09-10T20:38:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.906+0000 7f438e7fc640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f437c077f70 0x7f437c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.906+0000 7f438e7fc640 1 -- 172.21.15.28:0/3898864832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f43940b9e20 con 0x7f4384003a00 2024-09-10T20:38:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.906+0000 7f439d0b2640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f437c077f70 0x7f437c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.907+0000 7f439d0b2640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f437c077f70 0x7f437c07a430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f439815f440 tx=0x7f4388005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:38.910+0000 7f438e7fc640 1 -- 172.21.15.28:0/3898864832 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f43940822d0 con 0x7f4384003a00 2024-09-10T20:38:39.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.100+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f4398074c40 con 0x7f4384003a00 2024-09-10T20:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.103+0000 7f438e7fc640 1 -- 172.21.15.28:0/3898864832 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f4394086d00 con 0x7f4384003a00 2024-09-10T20:38:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.105+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f437c077f70 msgr2=0x7f437c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.105+0000 7f439fb17640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f437c077f70 0x7f437c07a430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f439815f440 tx=0x7f4388005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.105+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 msgr2=0x7f43980ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.105+0000 7f439fb17640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 0x7f43980ca690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f439400c3b0 tx=0x7f4394001d20 comp rx=0 tx=0).stop 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.105+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 shutdown_connections 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.106+0000 7f439fb17640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f437c077f70 0x7f437c07a430 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.106+0000 7f439fb17640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4398160810 0x7f4398160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.106+0000 7f439fb17640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f439815efc0 0x7f4398150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.106+0000 7f439fb17640 1 --2- 172.21.15.28:0/3898864832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4384003a00 0x7f43980ca690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.106+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 >> 172.21.15.28:0/3898864832 conn(0x7f43980d4130 msgr2=0x7f43980b77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.106+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 shutdown_connections 2024-09-10T20:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.106+0000 7f439fb17640 1 -- 172.21.15.28:0/3898864832 wait complete. 2024-09-10T20:38:39.120 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:38:39.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:38:39.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:38:39.205 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:38:39.205 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:39.205 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:39.221 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:39.222 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:39.278 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:39.278 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:39.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:38:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.476+0000 7f0ec3710640 1 -- 172.21.15.28:0/1796227986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d4950 msgr2=0x7f0ebc0d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.476+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1796227986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d4950 0x7f0ebc0d2060 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0eac005fd0 tx=0x7f0eac02a540 comp rx=0 tx=0).stop 2024-09-10T20:38:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.477+0000 7f0ec3710640 1 -- 172.21.15.28:0/1796227986 shutdown_connections 2024-09-10T20:38:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.477+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1796227986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d4950 0x7f0ebc0d2060 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.477+0000 7f0ec3710640 1 -- 172.21.15.28:0/1796227986 >> 172.21.15.28:0/1796227986 conn(0x7f0ebc09d2d0 msgr2=0x7f0ebc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.477+0000 7f0ec3710640 1 -- 172.21.15.28:0/1796227986 shutdown_connections 2024-09-10T20:38:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.477+0000 7f0ec3710640 1 -- 172.21.15.28:0/1796227986 wait complete. 2024-09-10T20:38:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.478+0000 7f0ec3710640 1 Processor -- start 2024-09-10T20:38:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.478+0000 7f0ec3710640 1 -- start start 2024-09-10T20:38:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec3710640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ebc0d07e0 0x7f0ebc1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec3710640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d0bc0 0x7f0ebc15d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec3710640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ebc0d4950 0x7f0ebc0b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec3710640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc0d0030 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec3710640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc0cfeb0 con 0x7f0ebc0d4950 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec3710640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0ebc1696f0 con 0x7f0ebc0d07e0 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d0bc0 0x7f0ebc15d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d0bc0 0x7f0ebc15d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48824/0 (socket says 172.21.15.28:48824) 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 -- 172.21.15.28:0/1735227729 learned_addr learned my addr 172.21.15.28:0/1735227729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec14ac640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ebc0d07e0 0x7f0ebc1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec1cad640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ebc0d4950 0x7f0ebc0b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 -- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ebc0d07e0 msgr2=0x7f0ebc1545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ebc0d07e0 0x7f0ebc1545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 -- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ebc0d4950 msgr2=0x7f0ebc0b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ebc0d4950 0x7f0ebc0b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 -- 172.21.15.28:0/1735227729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc154ae0 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec0cab640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d0bc0 0x7f0ebc15d190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0eac034040 tx=0x7f0eac003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0eb1ffb640 1 -- 172.21.15.28:0/1735227729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac03e030 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0eb1ffb640 1 -- 172.21.15.28:0/1735227729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eac0047a0 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ebc15da30 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.479+0000 7f0eb1ffb640 1 -- 172.21.15.28:0/1735227729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac035040 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.480+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ebc15dff0 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.481+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ebc0d2060 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.481+0000 7f0eb1ffb640 1 -- 172.21.15.28:0/1735227729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0eac004bf0 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.482+0000 7f0eb1ffb640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ea0077f70 0x7f0ea007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.482+0000 7f0eb1ffb640 1 -- 172.21.15.28:0/1735227729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0eac0ba6c0 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.482+0000 7f0ec14ac640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ea0077f70 0x7f0ea007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.482+0000 7f0ec14ac640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ea0077f70 0x7f0ea007a430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8007fd0 tx=0x7f0eb8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.486+0000 7f0eb1ffb640 1 -- 172.21.15.28:0/1735227729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0ebc154ae0 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.682+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0ebc0b7280 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.683+0000 7f0eb1ffb640 1 -- 172.21.15.28:0/1735227729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f0eac082b60 con 0x7f0ebc0d0bc0 2024-09-10T20:38:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:38:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ea0077f70 msgr2=0x7f0ea007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ea0077f70 0x7f0ea007a430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8007fd0 tx=0x7f0eb8009040 comp rx=0 tx=0).stop 2024-09-10T20:38:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d0bc0 msgr2=0x7f0ebc15d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d0bc0 0x7f0ebc15d190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0eac034040 tx=0x7f0eac003970 comp rx=0 tx=0).stop 2024-09-10T20:38:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 shutdown_connections 2024-09-10T20:38:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ea0077f70 0x7f0ea007a430 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ebc0d4950 0x7f0ebc0b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0d0bc0 0x7f0ebc15d190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 --2- 172.21.15.28:0/1735227729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ebc0d07e0 0x7f0ebc1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 >> 172.21.15.28:0/1735227729 conn(0x7f0ebc09d2d0 msgr2=0x7f0ebc0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 shutdown_connections 2024-09-10T20:38:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:39.685+0000 7f0ec3710640 1 -- 172.21.15.28:0/1735227729 wait complete. 2024-09-10T20:38:39.699 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:38:39.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:39.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-10T20:38:39.727 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:38:39.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:39.727 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:38:39.795 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:39.795 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:39.815 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:39.816 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:39.875 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:39.875 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:39.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:38:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 -- 172.21.15.28:0/179411916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c0d0c80 msgr2=0x7f745c15bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179411916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c0d0c80 0x7f745c15bc40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f745c0b5e20 tx=0x7f744c02a4d0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 -- 172.21.15.28:0/179411916 shutdown_connections 2024-09-10T20:38:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179411916 >> v2:172.21.15.159:3300/0 conn(0x7f745c070280 0x7f745c0706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179411916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c0d0c80 0x7f745c15bc40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 -- 172.21.15.28:0/179411916 >> 172.21.15.28:0/179411916 conn(0x7f745c06dbf0 msgr2=0x7f745c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 -- 172.21.15.28:0/179411916 shutdown_connections 2024-09-10T20:38:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.035+0000 7f7461eb6640 1 -- 172.21.15.28:0/179411916 wait complete. 2024-09-10T20:38:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.036+0000 7f7461eb6640 1 Processor -- start 2024-09-10T20:38:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f7461eb6640 1 -- start start 2024-09-10T20:38:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f7461eb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c070280 0x7f745c1490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f7461eb6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745c0d0c80 0x7f745c14ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f7461eb6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745c0d1060 0x7f745c14b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f7461eb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f745c15edf0 con 0x7f745c070280 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f7461eb6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f745c0b54a0 con 0x7f745c0d1060 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f7461eb6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f745c15ef70 con 0x7f745c0d0c80 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f745b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c070280 0x7f745c1490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f745b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c070280 0x7f745c1490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48844/0 (socket says 172.21.15.28:48844) 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.037+0000 7f745b7fe640 1 -- 172.21.15.28:0/179524465 learned_addr learned my addr 172.21.15.28:0/179524465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745affd640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745c0d0c80 0x7f745c14ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745bfff640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745c0d1060 0x7f745c14b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745b7fe640 1 -- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745c0d0c80 msgr2=0x7f745c14ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745b7fe640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745c0d0c80 0x7f745c14ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745b7fe640 1 -- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745c0d1060 msgr2=0x7f745c14b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745b7fe640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745c0d1060 0x7f745c14b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745b7fe640 1 -- 172.21.15.28:0/179524465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745c1523b0 con 0x7f745c070280 2024-09-10T20:38:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745bfff640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745c0d1060 0x7f745c14b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f745b7fe640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c070280 0x7f745c1490e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f744c02aac0 tx=0x7f744c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f7443fff640 1 -- 172.21.15.28:0/179524465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c03f070 con 0x7f745c070280 2024-09-10T20:38:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f7443fff640 1 -- 172.21.15.28:0/179524465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f744c004890 con 0x7f745c070280 2024-09-10T20:38:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f745c152690 con 0x7f745c070280 2024-09-10T20:38:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f7443fff640 1 -- 172.21.15.28:0/179524465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c035030 con 0x7f745c070280 2024-09-10T20:38:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.038+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f745c152c50 con 0x7f745c070280 2024-09-10T20:38:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.039+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f745c070eb0 con 0x7f745c070280 2024-09-10T20:38:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.040+0000 7f7443fff640 1 -- 172.21.15.28:0/179524465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f744c004ce0 con 0x7f745c070280 2024-09-10T20:38:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.040+0000 7f7443fff640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7448077f70 0x7f744807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.041+0000 7f7443fff640 1 -- 172.21.15.28:0/179524465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f744c0baf10 con 0x7f745c070280 2024-09-10T20:38:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.041+0000 7f745affd640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7448077f70 0x7f744807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.041+0000 7f745affd640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7448077f70 0x7f744807a430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7450006fd0 tx=0x7f7450008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.045+0000 7f7443fff640 1 -- 172.21.15.28:0/179524465 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f744c083a10 con 0x7f745c070280 2024-09-10T20:38:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.241+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f745c153460 con 0x7f745c070280 2024-09-10T20:38:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.241+0000 7f7443fff640 1 -- 172.21.15.28:0/179524465 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f744c083310 con 0x7f745c070280 2024-09-10T20:38:40.246 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:38:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7448077f70 msgr2=0x7f744807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7448077f70 0x7f744807a430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7450006fd0 tx=0x7f7450008040 comp rx=0 tx=0).stop 2024-09-10T20:38:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c070280 msgr2=0x7f745c1490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c070280 0x7f745c1490e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f744c02aac0 tx=0x7f744c0039e0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 shutdown_connections 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7448077f70 0x7f744807a430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745c0d1060 0x7f745c14b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745c0d0c80 0x7f745c14ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 --2- 172.21.15.28:0/179524465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745c070280 0x7f745c1490e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 >> 172.21.15.28:0/179524465 conn(0x7f745c06dbf0 msgr2=0x7f745c06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 shutdown_connections 2024-09-10T20:38:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.243+0000 7f7461eb6640 1 -- 172.21.15.28:0/179524465 wait complete. 2024-09-10T20:38:40.258 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:38:40.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:40.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-09-10T20:38:40.290 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:38:40.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:40.291 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:38:40.355 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:40.356 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:40.372 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:40.373 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:40.429 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:38:40.429 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:40.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:38:40.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.629+0000 7fb071c9b640 1 -- 172.21.15.28:0/4110928264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c15df70 msgr2=0x7fb06c0b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.629+0000 7fb071c9b640 1 --2- 172.21.15.28:0/4110928264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c15df70 0x7fb06c0b2ba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005fd0 tx=0x7fb05c02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.629+0000 7fb071c9b640 1 -- 172.21.15.28:0/4110928264 shutdown_connections 2024-09-10T20:38:40.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.629+0000 7fb071c9b640 1 --2- 172.21.15.28:0/4110928264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c15df70 0x7fb06c0b2ba0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.629+0000 7fb071c9b640 1 -- 172.21.15.28:0/4110928264 >> 172.21.15.28:0/4110928264 conn(0x7fb06c06cf00 msgr2=0x7fb06c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:40.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.629+0000 7fb071c9b640 1 -- 172.21.15.28:0/4110928264 shutdown_connections 2024-09-10T20:38:40.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.629+0000 7fb071c9b640 1 -- 172.21.15.28:0/4110928264 wait complete. 2024-09-10T20:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.630+0000 7fb071c9b640 1 Processor -- start 2024-09-10T20:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.630+0000 7fb071c9b640 1 -- start start 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb071c9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0d00f0 0x7fb06c157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb071c9b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb06c0d0560 0x7fb06c149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb071c9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb06c15df70 0x7fb06c14c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb071c9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c0cead0 con 0x7fb06c0d00f0 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb071c9b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c0ce950 con 0x7fb06c15df70 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb071c9b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb06c166100 con 0x7fb06c0d0560 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb06b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0d00f0 0x7fb06c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb06b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0d00f0 0x7fb06c157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44154/0 (socket says 172.21.15.28:44154) 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.631+0000 7fb06b7fe640 1 -- 172.21.15.28:0/753129957 learned_addr learned my addr 172.21.15.28:0/753129957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06affd640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb06c0d0560 0x7fb06c149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06bfff640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb06c15df70 0x7fb06c14c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06b7fe640 1 -- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb06c0d0560 msgr2=0x7fb06c149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06b7fe640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb06c0d0560 0x7fb06c149690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06b7fe640 1 -- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb06c15df70 msgr2=0x7fb06c14c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06b7fe640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb06c15df70 0x7fb06c14c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06b7fe640 1 -- 172.21.15.28:0/753129957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06c1f5d00 con 0x7fb06c0d00f0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06bfff640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb06c15df70 0x7fb06c14c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb06b7fe640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0d00f0 0x7fb06c157010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb05c000c00 tx=0x7fb05c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb053fff640 1 -- 172.21.15.28:0/753129957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c03e030 con 0x7fb06c0d00f0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb053fff640 1 -- 172.21.15.28:0/753129957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb05c003ca0 con 0x7fb06c0d00f0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06c15cb30 con 0x7fb06c0d00f0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb053fff640 1 -- 172.21.15.28:0/753129957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c035040 con 0x7fb06c0d00f0 2024-09-10T20:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.632+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06c15cdd0 con 0x7fb06c0d00f0 2024-09-10T20:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.633+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb06c1f5850 con 0x7fb06c0d00f0 2024-09-10T20:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.634+0000 7fb053fff640 1 -- 172.21.15.28:0/753129957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb05c004a00 con 0x7fb06c0d00f0 2024-09-10T20:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.634+0000 7fb053fff640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb058077f70 0x7fb05807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.634+0000 7fb053fff640 1 -- 172.21.15.28:0/753129957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb05c0ba550 con 0x7fb06c0d00f0 2024-09-10T20:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.635+0000 7fb06affd640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb058077f70 0x7fb05807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.635+0000 7fb06affd640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb058077f70 0x7fb05807a430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb060004ea0 tx=0x7fb060009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.638+0000 7fb053fff640 1 -- 172.21.15.28:0/753129957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb05c0830e0 con 0x7fb06c0d00f0 2024-09-10T20:38:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.834+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb06c0ae0b0 con 0x7fb06c0d00f0 2024-09-10T20:38:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.834+0000 7fb053fff640 1 -- 172.21.15.28:0/753129957 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fb05c0829e0 con 0x7fb06c0d00f0 2024-09-10T20:38:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb058077f70 msgr2=0x7fb05807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb058077f70 0x7fb05807a430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb060004ea0 tx=0x7fb060009040 comp rx=0 tx=0).stop 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0d00f0 msgr2=0x7fb06c157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0d00f0 0x7fb06c157010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb05c000c00 tx=0x7fb05c0038a0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 shutdown_connections 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb058077f70 0x7fb05807a430 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb06c15df70 0x7fb06c14c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb06c0d0560 0x7fb06c149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 --2- 172.21.15.28:0/753129957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0d00f0 0x7fb06c157010 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 >> 172.21.15.28:0/753129957 conn(0x7fb06c06cf00 msgr2=0x7fb06c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 shutdown_connections 2024-09-10T20:38:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:40.837+0000 7fb071c9b640 1 -- 172.21.15.28:0/753129957 wait complete. 2024-09-10T20:38:40.851 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-09-10T20:38:40.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:40.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-09-10T20:38:40.878 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:38:40.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:40.878 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-10T20:38:40.935 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:40.935 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:40.954 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:40.955 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:41.014 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:38:41.014 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:38:41.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:38:41.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.175+0000 7f24c5816640 1 -- 172.21.15.28:0/2097863735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c015de70 msgr2=0x7f24c00d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.175+0000 7f24c5816640 1 --2- 172.21.15.28:0/2097863735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c015de70 0x7f24c00d2b90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009830 tx=0x7f24b002a510 comp rx=0 tx=0).stop 2024-09-10T20:38:41.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.175+0000 7f24c5816640 1 -- 172.21.15.28:0/2097863735 shutdown_connections 2024-09-10T20:38:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.175+0000 7f24c5816640 1 --2- 172.21.15.28:0/2097863735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c015de70 0x7f24c00d2b90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.175+0000 7f24c5816640 1 -- 172.21.15.28:0/2097863735 >> 172.21.15.28:0/2097863735 conn(0x7f24c00d3fb0 msgr2=0x7f24c00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.175+0000 7f24c5816640 1 -- 172.21.15.28:0/2097863735 shutdown_connections 2024-09-10T20:38:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.175+0000 7f24c5816640 1 -- 172.21.15.28:0/2097863735 wait complete. 2024-09-10T20:38:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.176+0000 7f24c5816640 1 Processor -- start 2024-09-10T20:38:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24c5816640 1 -- start start 2024-09-10T20:38:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24c5816640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24c00cf780 0x7f24c00a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24c5816640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24c015de70 0x7f24c006d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24c5816640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c0162260 0x7f24c006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24c5816640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24c00cff80 con 0x7f24c0162260 2024-09-10T20:38:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24c5816640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f24c00cfe00 con 0x7f24c00cf780 2024-09-10T20:38:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24c5816640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f24c015e470 con 0x7f24c015de70 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c0162260 0x7f24c006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c0162260 0x7f24c006da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44180/0 (socket says 172.21.15.28:44180) 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.177+0000 7f24bf7fe640 1 -- 172.21.15.28:0/2048809172 learned_addr learned my addr 172.21.15.28:0/2048809172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24beffd640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24c00cf780 0x7f24c00a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24bf7fe640 1 -- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24c015de70 msgr2=0x7f24c006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24bf7fe640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24c015de70 0x7f24c006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24bf7fe640 1 -- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24c00cf780 msgr2=0x7f24c00a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24bf7fe640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24c00cf780 0x7f24c00a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24bf7fe640 1 -- 172.21.15.28:0/2048809172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c01f4660 con 0x7f24c0162260 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24bf7fe640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c0162260 0x7f24c006da60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f24b800b990 tx=0x7f24b800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24a7fff640 1 -- 172.21.15.28:0/2048809172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8019070 con 0x7f24c0162260 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24a7fff640 1 -- 172.21.15.28:0/2048809172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24b800cc90 con 0x7f24c0162260 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c01f4940 con 0x7f24c0162260 2024-09-10T20:38:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24a7fff640 1 -- 172.21.15.28:0/2048809172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b80146a0 con 0x7f24c0162260 2024-09-10T20:38:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.178+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c01f4f00 con 0x7f24c0162260 2024-09-10T20:38:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.179+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c00b7830 con 0x7f24c0162260 2024-09-10T20:38:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.180+0000 7f24a7fff640 1 -- 172.21.15.28:0/2048809172 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f24b8014840 con 0x7f24c0162260 2024-09-10T20:38:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.180+0000 7f24a7fff640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f24ac077f70 0x7f24ac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.180+0000 7f24a7fff640 1 -- 172.21.15.28:0/2048809172 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f24b8099e30 con 0x7f24c0162260 2024-09-10T20:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.180+0000 7f24beffd640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f24ac077f70 0x7f24ac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.181+0000 7f24beffd640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f24ac077f70 0x7f24ac07a430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f24b0034000 tx=0x7f24b0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.184+0000 7f24a7fff640 1 -- 172.21.15.28:0/2048809172 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f24b8062930 con 0x7f24c0162260 2024-09-10T20:38:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.380+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f24c0076240 con 0x7f24c0162260 2024-09-10T20:38:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.380+0000 7f24a7fff640 1 -- 172.21.15.28:0/2048809172 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f24b8062230 con 0x7f24c0162260 2024-09-10T20:38:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:38:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.382+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f24ac077f70 msgr2=0x7f24ac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.382+0000 7f24c5816640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f24ac077f70 0x7f24ac07a430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f24b0034000 tx=0x7f24b0004730 comp rx=0 tx=0).stop 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.382+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c0162260 msgr2=0x7f24c006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.382+0000 7f24c5816640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c0162260 0x7f24c006da60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f24b800b990 tx=0x7f24b800be50 comp rx=0 tx=0).stop 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.382+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 shutdown_connections 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.383+0000 7f24c5816640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f24ac077f70 0x7f24ac07a430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.383+0000 7f24c5816640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c0162260 0x7f24c006da60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.383+0000 7f24c5816640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24c015de70 0x7f24c006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.383+0000 7f24c5816640 1 --2- 172.21.15.28:0/2048809172 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24c00cf780 0x7f24c00a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.383+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 >> 172.21.15.28:0/2048809172 conn(0x7f24c00d3fb0 msgr2=0x7f24c00d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.383+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 shutdown_connections 2024-09-10T20:38:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.383+0000 7f24c5816640 1 -- 172.21.15.28:0/2048809172 wait complete. 2024-09-10T20:38:41.397 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-09-10T20:38:41.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:41.398 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-09-10T20:38:41.429 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:38:41.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:38:41.430 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-10T20:38:41.489 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:38:41.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:41.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 -- 172.21.15.28:0/678695512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c15de70 msgr2=0x7f659c0d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 --2- 172.21.15.28:0/678695512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c15de70 0x7f659c0d2b90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f659c0698a0 tx=0x7f658c02aa30 comp rx=0 tx=0).stop 2024-09-10T20:38:41.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 -- 172.21.15.28:0/678695512 shutdown_connections 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 --2- 172.21.15.28:0/678695512 >> v2:172.21.15.159:3300/0 conn(0x7f659c161b50 0x7f659c161fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 --2- 172.21.15.28:0/678695512 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f659c0cf780 0x7f659c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 --2- 172.21.15.28:0/678695512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c15de70 0x7f659c0d2b90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 -- 172.21.15.28:0/678695512 >> 172.21.15.28:0/678695512 conn(0x7f659c0d3fb0 msgr2=0x7f659c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 -- 172.21.15.28:0/678695512 shutdown_connections 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.590+0000 7f65a192c640 1 -- 172.21.15.28:0/678695512 wait complete. 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.591+0000 7f65a192c640 1 Processor -- start 2024-09-10T20:38:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.592+0000 7f65a192c640 1 -- start start 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.592+0000 7f65a192c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c0cf780 0x7f659c152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.592+0000 7f65a192c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f659c15de70 0x7f659c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.592+0000 7f65a192c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f659c161b50 0x7f659c0b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.592+0000 7f65a192c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f659c0b6e00 con 0x7f659c0cf780 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.592+0000 7f65a192c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f659c15e460 con 0x7f659c15de70 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f65a192c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f659c0b6f80 con 0x7f659c161b50 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c0cf780 0x7f659c152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c0cf780 0x7f659c152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44194/0 (socket says 172.21.15.28:44194) 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 -- 172.21.15.28:0/3137537652 learned_addr learned my addr 172.21.15.28:0/3137537652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659a7fc640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f659c15de70 0x7f659c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659b7fe640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f659c161b50 0x7f659c0b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 -- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f659c161b50 msgr2=0x7f659c0b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f659c161b50 0x7f659c0b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 -- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f659c15de70 msgr2=0x7f659c0b1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f659c15de70 0x7f659c0b1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 -- 172.21.15.28:0/3137537652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659c152970 con 0x7f659c0cf780 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f659affd640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c0cf780 0x7f659c152430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f658c009d60 tx=0x7f658c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f6593fff640 1 -- 172.21.15.28:0/3137537652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c046070 con 0x7f659c0cf780 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f6593fff640 1 -- 172.21.15.28:0/3137537652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f658c004250 con 0x7f659c0cf780 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659c0b2820 con 0x7f659c0cf780 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f6593fff640 1 -- 172.21.15.28:0/3137537652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c033050 con 0x7f659c0cf780 2024-09-10T20:38:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.593+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f659c1f8850 con 0x7f659c0cf780 2024-09-10T20:38:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.595+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f659c0b2590 con 0x7f659c0cf780 2024-09-10T20:38:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.595+0000 7f6593fff640 1 -- 172.21.15.28:0/3137537652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f658c004660 con 0x7f659c0cf780 2024-09-10T20:38:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.596+0000 7f6593fff640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f657c077f70 0x7f657c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.596+0000 7f6593fff640 1 -- 172.21.15.28:0/3137537652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f658c0bb020 con 0x7f659c0cf780 2024-09-10T20:38:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.596+0000 7f659a7fc640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f657c077f70 0x7f657c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.596+0000 7f659a7fc640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f657c077f70 0x7f657c07a430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f659c1604c0 tx=0x7f6584008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.600+0000 7f6593fff640 1 -- 172.21.15.28:0/3137537652 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f659c152970 con 0x7f659c0cf780 2024-09-10T20:38:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.839+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f659c0d2b90 con 0x7f659c0cf780 2024-09-10T20:38:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.840+0000 7f6593fff640 1 -- 172.21.15.28:0/3137537652 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f658c083bb0 con 0x7f659c0cf780 2024-09-10T20:38:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:41.844 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f657c077f70 msgr2=0x7f657c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f657c077f70 0x7f657c07a430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f659c1604c0 tx=0x7f6584008040 comp rx=0 tx=0).stop 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c0cf780 msgr2=0x7f659c152430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c0cf780 0x7f659c152430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f658c009d60 tx=0x7f658c003f10 comp rx=0 tx=0).stop 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 shutdown_connections 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f657c077f70 0x7f657c07a430 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f659c161b50 0x7f659c0b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f659c15de70 0x7f659c0b1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 --2- 172.21.15.28:0/3137537652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659c0cf780 0x7f659c152430 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 >> 172.21.15.28:0/3137537652 conn(0x7f659c0d3fb0 msgr2=0x7f659c0b8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 shutdown_connections 2024-09-10T20:38:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.842+0000 7f65a192c640 1 -- 172.21.15.28:0/3137537652 wait complete. 2024-09-10T20:38:41.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.959+0000 7f64283a0640 1 -- 172.21.15.28:0/578650157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 msgr2=0x7f6408003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.959+0000 7f64283a0640 1 --2- 172.21.15.28:0/578650157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f6408003de0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6410009ed0 tx=0x7f6408005830 comp rx=0 tx=0).stop 2024-09-10T20:38:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.959+0000 7f64283a0640 1 -- 172.21.15.28:0/578650157 shutdown_connections 2024-09-10T20:38:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.959+0000 7f64283a0640 1 --2- 172.21.15.28:0/578650157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f6408003de0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.959+0000 7f64283a0640 1 -- 172.21.15.28:0/578650157 >> 172.21.15.28:0/578650157 conn(0x7f64200d0420 msgr2=0x7f64200d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.959+0000 7f64283a0640 1 -- 172.21.15.28:0/578650157 shutdown_connections 2024-09-10T20:38:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.959+0000 7f64283a0640 1 -- 172.21.15.28:0/578650157 wait complete. 2024-09-10T20:38:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.960+0000 7f64283a0640 1 Processor -- start 2024-09-10T20:38:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.960+0000 7f64283a0640 1 -- start start 2024-09-10T20:38:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f64283a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f64201f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f64283a0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200b0b10 0x7f64200b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f64283a0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64201ef6d0 0x7f64201efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f64283a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6420073cf0 con 0x7f6408003a00 2024-09-10T20:38:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f64283a0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f642015f640 con 0x7f64200b0b10 2024-09-10T20:38:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f64283a0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6420073e70 con 0x7f64201ef6d0 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f64201f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f64201f6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44212/0 (socket says 172.21.15.28:44212) 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 -- 172.21.15.28:0/3832312634 learned_addr learned my addr 172.21.15.28:0/3832312634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642693d640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64201ef6d0 0x7f64201efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642593b640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200b0b10 0x7f64200b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 -- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64201ef6d0 msgr2=0x7f64201efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64201ef6d0 0x7f64201efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 -- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200b0b10 msgr2=0x7f64200b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200b0b10 0x7f64200b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 -- 172.21.15.28:0/3832312634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64201599c0 con 0x7f6408003a00 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f642613c640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f64201f6030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6410009e20 tx=0x7f6410033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.961+0000 7f641effd640 1 -- 172.21.15.28:0/3832312634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410038040 con 0x7f6408003a00 2024-09-10T20:38:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.962+0000 7f641effd640 1 -- 172.21.15.28:0/3832312634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6410041070 con 0x7f6408003a00 2024-09-10T20:38:41.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.962+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6420159ca0 con 0x7f6408003a00 2024-09-10T20:38:41.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.962+0000 7f641effd640 1 -- 172.21.15.28:0/3832312634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641003c670 con 0x7f6408003a00 2024-09-10T20:38:41.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.962+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642015a260 con 0x7f6408003a00 2024-09-10T20:38:41.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.962+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64200cf300 con 0x7f6408003a00 2024-09-10T20:38:41.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.963+0000 7f641effd640 1 -- 172.21.15.28:0/3832312634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f641003b070 con 0x7f6408003a00 2024-09-10T20:38:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.964+0000 7f641effd640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f640c077fa0 0x7f640c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.964+0000 7f641effd640 1 -- 172.21.15.28:0/3832312634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f64100ba780 con 0x7f6408003a00 2024-09-10T20:38:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.964+0000 7f642593b640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f640c077fa0 0x7f640c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.964+0000 7f642593b640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f640c077fa0 0x7f640c07a460 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6414006990 tx=0x7f6414006d70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:41.968+0000 7f641effd640 1 -- 172.21.15.28:0/3832312634 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6410082ba0 con 0x7f6408003a00 2024-09-10T20:38:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.191+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6420068d10 con 0x7f6408003a00 2024-09-10T20:38:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.192+0000 7f641effd640 1 -- 172.21.15.28:0/3832312634 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f64100875d0 con 0x7f6408003a00 2024-09-10T20:38:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:42.196 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f640c077fa0 msgr2=0x7f640c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f640c077fa0 0x7f640c07a460 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6414006990 tx=0x7f6414006d70 comp rx=0 tx=0).stop 2024-09-10T20:38:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 msgr2=0x7f64201f6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f64201f6030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6410009e20 tx=0x7f6410033cb0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 shutdown_connections 2024-09-10T20:38:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f640c077fa0 0x7f640c07a460 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f64201ef6d0 0x7f64201efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f64200b0b10 0x7f64200b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 --2- 172.21.15.28:0/3832312634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6408003a00 0x7f64201f6030 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 >> 172.21.15.28:0/3832312634 conn(0x7f64200d0420 msgr2=0x7f64200ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 shutdown_connections 2024-09-10T20:38:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.194+0000 7f64283a0640 1 -- 172.21.15.28:0/3832312634 wait complete. 2024-09-10T20:38:42.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.311+0000 7f8c33a85640 1 -- 172.21.15.28:0/1558674596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c18004800 msgr2=0x7f8c18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.311+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1558674596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c18004800 0x7f8c18004c80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c009b50 tx=0x7f8c18005830 comp rx=0 tx=0).stop 2024-09-10T20:38:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.311+0000 7f8c33a85640 1 -- 172.21.15.28:0/1558674596 shutdown_connections 2024-09-10T20:38:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.311+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1558674596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c18004800 0x7f8c18004c80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.311+0000 7f8c33a85640 1 -- 172.21.15.28:0/1558674596 >> 172.21.15.28:0/1558674596 conn(0x7f8c2c0d1e40 msgr2=0x7f8c2c0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.311+0000 7f8c33a85640 1 -- 172.21.15.28:0/1558674596 shutdown_connections 2024-09-10T20:38:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.311+0000 7f8c33a85640 1 -- 172.21.15.28:0/1558674596 wait complete. 2024-09-10T20:38:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.312+0000 7f8c33a85640 1 Processor -- start 2024-09-10T20:38:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.312+0000 7f8c33a85640 1 -- start start 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c33a85640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c18004800 0x7f8c2c1f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c33a85640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c2c1515e0 0x7f8c2c1f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c33a85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c1f5e40 0x7f8c2c1f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c33a85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c0d2960 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c33a85640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c0d0d50 con 0x7f8c18004800 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c33a85640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8c2c0d2ae0 con 0x7f8c2c1515e0 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c32022640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c1f5e40 0x7f8c2c1f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c32022640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c1f5e40 0x7f8c2c1f62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44218/0 (socket says 172.21.15.28:44218) 2024-09-10T20:38:42.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c32022640 1 -- 172.21.15.28:0/1566200077 learned_addr learned my addr 172.21.15.28:0/1566200077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c31020640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c2c1515e0 0x7f8c2c1f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.313+0000 7f8c31821640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c18004800 0x7f8c2c1f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c32022640 1 -- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c2c1515e0 msgr2=0x7f8c2c1f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c32022640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c2c1515e0 0x7f8c2c1f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c32022640 1 -- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c18004800 msgr2=0x7f8c2c1f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c32022640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c18004800 0x7f8c2c1f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c32022640 1 -- 172.21.15.28:0/1566200077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c15e5a0 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c32022640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c1f5e40 0x7f8c2c1f62c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8c24000f80 tx=0x7f8c2400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c227fc640 1 -- 172.21.15.28:0/1566200077 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2400ca20 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c227fc640 1 -- 172.21.15.28:0/1566200077 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c24018070 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2c15e880 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c227fc640 1 -- 172.21.15.28:0/1566200077 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24011460 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.314+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c2c15ee40 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.315+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2c0cffa0 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.315+0000 7f8c227fc640 1 -- 172.21.15.28:0/1566200077 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8c24011600 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.316+0000 7f8c227fc640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8c10077fa0 0x7f8c1007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.316+0000 7f8c227fc640 1 -- 172.21.15.28:0/1566200077 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8c24099fa0 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.316+0000 7f8c31821640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8c10077fa0 0x7f8c1007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.317+0000 7f8c31821640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8c10077fa0 0x7f8c1007a460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000c00 tx=0x7f8c28006b70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:42.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.320+0000 7f8c227fc640 1 -- 172.21.15.28:0/1566200077 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8c24062450 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.505+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c2c0b65c0 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.506+0000 7f8c227fc640 1 -- 172.21.15.28:0/1566200077 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16669 (secure 0 0 0) 0x7f8c2406b1c0 con 0x7f8c2c1f5e40 2024-09-10T20:38:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:42.511 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:13.395359+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8c10077fa0 msgr2=0x7f8c1007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8c10077fa0 0x7f8c1007a460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000c00 tx=0x7f8c28006b70 comp rx=0 tx=0).stop 2024-09-10T20:38:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c1f5e40 msgr2=0x7f8c2c1f62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c1f5e40 0x7f8c2c1f62c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8c24000f80 tx=0x7f8c2400ed10 comp rx=0 tx=0).stop 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 shutdown_connections 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8c10077fa0 0x7f8c1007a460 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c1f5e40 0x7f8c2c1f62c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c2c1515e0 0x7f8c2c1f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 --2- 172.21.15.28:0/1566200077 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c18004800 0x7f8c2c1f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 >> 172.21.15.28:0/1566200077 conn(0x7f8c2c0d1e40 msgr2=0x7f8c2c0caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 shutdown_connections 2024-09-10T20:38:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.508+0000 7f8c33a85640 1 -- 172.21.15.28:0/1566200077 wait complete. 2024-09-10T20:38:42.524 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:38:42.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 -- 172.21.15.28:0/1995742808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3040162090 msgr2=0x7f30400b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 --2- 172.21.15.28:0/1995742808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3040162090 0x7f30400b6e00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f30400d4d80 tx=0x7f303002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 -- 172.21.15.28:0/1995742808 shutdown_connections 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 --2- 172.21.15.28:0/1995742808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3040162090 0x7f30400b6e00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 --2- 172.21.15.28:0/1995742808 >> v2:172.21.15.159:3300/0 conn(0x7f30400d2b90 0x7f3040161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 -- 172.21.15.28:0/1995742808 >> 172.21.15.28:0/1995742808 conn(0x7f30400d3fb0 msgr2=0x7f30400d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 -- 172.21.15.28:0/1995742808 shutdown_connections 2024-09-10T20:38:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.626+0000 7f30483ca640 1 -- 172.21.15.28:0/1995742808 wait complete. 2024-09-10T20:38:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 Processor -- start 2024-09-10T20:38:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 -- start start 2024-09-10T20:38:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30400d2b90 0x7f30401615a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f304015de70 0x7f30401f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3040162090 0x7f30400b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f304015e3d0 con 0x7f30400d2b90 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f304015e250 con 0x7f304015de70 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f30483ca640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f30400b7340 con 0x7f3040162090 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f3046166640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30400d2b90 0x7f30401615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.628+0000 7f3046166640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30400d2b90 0x7f30401615a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44230/0 (socket says 172.21.15.28:44230) 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046166640 1 -- 172.21.15.28:0/2412011770 learned_addr learned my addr 172.21.15.28:0/2412011770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046967640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3040162090 0x7f30400b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3045965640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f304015de70 0x7f30401f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046166640 1 -- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3040162090 msgr2=0x7f30400b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046166640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3040162090 0x7f30400b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046166640 1 -- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f304015de70 msgr2=0x7f30401f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046166640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f304015de70 0x7f30401f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046166640 1 -- 172.21.15.28:0/2412011770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30401f46f0 con 0x7f30400d2b90 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f3046166640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30400d2b90 0x7f30401615a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f303800c4c0 tx=0x7f303800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f303effd640 1 -- 172.21.15.28:0/2412011770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038016020 con 0x7f30400d2b90 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f303effd640 1 -- 172.21.15.28:0/2412011770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3038012940 con 0x7f30400d2b90 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30401f8220 con 0x7f30400d2b90 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f303effd640 1 -- 172.21.15.28:0/2412011770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038012d90 con 0x7f30400d2b90 2024-09-10T20:38:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.629+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30401f8a60 con 0x7f30400d2b90 2024-09-10T20:38:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.630+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30400b7810 con 0x7f30400d2b90 2024-09-10T20:38:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.631+0000 7f303effd640 1 -- 172.21.15.28:0/2412011770 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f303801a3b0 con 0x7f30400d2b90 2024-09-10T20:38:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.631+0000 7f303effd640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f302c077f70 0x7f302c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.632+0000 7f303effd640 1 -- 172.21.15.28:0/2412011770 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3038099df0 con 0x7f30400d2b90 2024-09-10T20:38:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.632+0000 7f3045965640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f302c077f70 0x7f302c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.632+0000 7f3045965640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f302c077f70 0x7f302c07a430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3040161480 tx=0x7f303400a4c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.636+0000 7f303effd640 1 -- 172.21.15.28:0/2412011770 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3038020030 con 0x7f30400d2b90 2024-09-10T20:38:42.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.857+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30400b9f30 con 0x7f30400d2b90 2024-09-10T20:38:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.858+0000 7f303effd640 1 -- 172.21.15.28:0/2412011770 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f30401f8220 con 0x7f30400d2b90 2024-09-10T20:38:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:42.863 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f302c077f70 msgr2=0x7f302c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f302c077f70 0x7f302c07a430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3040161480 tx=0x7f303400a4c0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30400d2b90 msgr2=0x7f30401615a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30400d2b90 0x7f30401615a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f303800c4c0 tx=0x7f303800c980 comp rx=0 tx=0).stop 2024-09-10T20:38:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 shutdown_connections 2024-09-10T20:38:42.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f302c077f70 0x7f302c07a430 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3040162090 0x7f30400b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f304015de70 0x7f30401f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 --2- 172.21.15.28:0/2412011770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30400d2b90 0x7f30401615a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 >> 172.21.15.28:0/2412011770 conn(0x7f30400d3fb0 msgr2=0x7f304015d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:42.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 shutdown_connections 2024-09-10T20:38:42.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.860+0000 7f30483ca640 1 -- 172.21.15.28:0/2412011770 wait complete. 2024-09-10T20:38:42.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.977+0000 7fb520e47640 1 -- 172.21.15.28:0/278330550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb508003a00 msgr2=0x7fb508003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.977+0000 7fb520e47640 1 --2- 172.21.15.28:0/278330550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb508003a00 0x7fb508003e20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb514006fd0 tx=0x7fb508005830 comp rx=0 tx=0).stop 2024-09-10T20:38:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.977+0000 7fb51ad9d640 1 -- 172.21.15.28:0/278330550 reap_dead start 2024-09-10T20:38:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.977+0000 7fb520e47640 1 -- 172.21.15.28:0/278330550 shutdown_connections 2024-09-10T20:38:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.977+0000 7fb520e47640 1 -- 172.21.15.28:0/278330550 >> 172.21.15.28:0/278330550 conn(0x7fb51c0d3fb0 msgr2=0x7fb51c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.977+0000 7fb520e47640 1 -- 172.21.15.28:0/278330550 shutdown_connections 2024-09-10T20:38:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.977+0000 7fb520e47640 1 -- 172.21.15.28:0/278330550 wait complete. 2024-09-10T20:38:42.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.978+0000 7fb520e47640 1 Processor -- start 2024-09-10T20:38:42.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.978+0000 7fb520e47640 1 -- start start 2024-09-10T20:38:42.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb520e47640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb508003a00 0x7fb51c0cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb520e47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51c15b670 0x7fb51c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb520e47640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb51c0b2cb0 0x7fb51c0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb520e47640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c15e3d0 con 0x7fb51c15b670 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb520e47640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c15e250 con 0x7fb508003a00 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb520e47640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb51c161b50 con 0x7fb51c0b2cb0 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb519d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51c15b670 0x7fb51c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb519d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51c15b670 0x7fb51c15ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44246/0 (socket says 172.21.15.28:44246) 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb519d9b640 1 -- 172.21.15.28:0/2333459218 learned_addr learned my addr 172.21.15.28:0/2333459218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb51ad9d640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb51c0b2cb0 0x7fb51c0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb51a59c640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb508003a00 0x7fb51c0cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb519d9b640 1 -- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb51c0b2cb0 msgr2=0x7fb51c0b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb519d9b640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb51c0b2cb0 0x7fb51c0b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb519d9b640 1 -- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb508003a00 msgr2=0x7fb51c0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.979+0000 7fb519d9b640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb508003a00 0x7fb51c0cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb519d9b640 1 -- 172.21.15.28:0/2333459218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c1f84e0 con 0x7fb51c15b670 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb51ad9d640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb51c0b2cb0 0x7fb51c0b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb51a59c640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb508003a00 0x7fb51c0cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb519d9b640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51c15b670 0x7fb51c15ae40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb50400f980 tx=0x7fb50400fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb512ffd640 1 -- 172.21.15.28:0/2333459218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50400bcb0 con 0x7fb51c15b670 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb512ffd640 1 -- 172.21.15.28:0/2333459218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb504011440 con 0x7fb51c15b670 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb512ffd640 1 -- 172.21.15.28:0/2333459218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5040118b0 con 0x7fb51c15b670 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51c0ce4d0 con 0x7fb51c15b670 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.980+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51c1f8a60 con 0x7fb51c15b670 2024-09-10T20:38:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.981+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51c0cf780 con 0x7fb51c15b670 2024-09-10T20:38:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.981+0000 7fb512ffd640 1 -- 172.21.15.28:0/2333459218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb504023030 con 0x7fb51c15b670 2024-09-10T20:38:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.982+0000 7fb512ffd640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb4fc077fa0 0x7fb4fc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.982+0000 7fb512ffd640 1 -- 172.21.15.28:0/2333459218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb50409a6a0 con 0x7fb51c15b670 2024-09-10T20:38:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.982+0000 7fb51a59c640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb4fc077fa0 0x7fb4fc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.983+0000 7fb51a59c640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb4fc077fa0 0x7fb4fc07a460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb514033000 tx=0x7fb514001db0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:42.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:42.986+0000 7fb512ffd640 1 -- 172.21.15.28:0/2333459218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb504063230 con 0x7fb51c15b670 2024-09-10T20:38:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.207+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb51c0b9f30 con 0x7fb51c15b670 2024-09-10T20:38:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.207+0000 7fb512ffd640 1 -- 172.21.15.28:0/2333459218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7fb504062b30 con 0x7fb51c15b670 2024-09-10T20:38:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-10T20:38:15:410160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3191523676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3191523676},{"type":"v1","addr":"172.21.15.28:6834","nonce":3191523676}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/239963078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":239963078},{"type":"v1","addr":"172.21.15.28:6835","nonce":239963078}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/2442588072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6836","nonce":2442588072},{"type":"v1","addr":"172.21.15.159:6837","nonce":2442588072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:11.751747+0000","modified":"2024-09-10T20:38:14.774063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4357":{"gid":4357,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/2925750660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2925750660},{"type":"v1","addr":"172.21.15.159:6835","nonce":2925750660}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4357,"qdb_cluster":[4357]},"id":1}]} 2024-09-10T20:38:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-10T20:38:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb4fc077fa0 msgr2=0x7fb4fc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb4fc077fa0 0x7fb4fc07a460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb514033000 tx=0x7fb514001db0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51c15b670 msgr2=0x7fb51c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51c15b670 0x7fb51c15ae40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb50400f980 tx=0x7fb50400fe40 comp rx=0 tx=0).stop 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 shutdown_connections 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb4fc077fa0 0x7fb4fc07a460 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb51c0b2cb0 0x7fb51c0b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51c15b670 0x7fb51c15ae40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 --2- 172.21.15.28:0/2333459218 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb508003a00 0x7fb51c0cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.209+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 >> 172.21.15.28:0/2333459218 conn(0x7fb51c0d3fb0 msgr2=0x7fb51c0b7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.210+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 shutdown_connections 2024-09-10T20:38:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.210+0000 7fb520e47640 1 -- 172.21.15.28:0/2333459218 wait complete. 2024-09-10T20:38:43.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:43.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 -- 172.21.15.28:0/3368403775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 msgr2=0x7f571c15e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 --2- 172.21.15.28:0/3368403775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 0x7f571c15e7e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5710009830 tx=0x7f571002a510 comp rx=0 tx=0).stop 2024-09-10T20:38:43.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 -- 172.21.15.28:0/3368403775 shutdown_connections 2024-09-10T20:38:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 --2- 172.21.15.28:0/3368403775 >> v2:172.21.15.159:3300/0 conn(0x7f571c15ed20 0x7f571c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 --2- 172.21.15.28:0/3368403775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 0x7f571c15e7e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 -- 172.21.15.28:0/3368403775 >> 172.21.15.28:0/3368403775 conn(0x7f571c0d0420 msgr2=0x7f571c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 -- 172.21.15.28:0/3368403775 shutdown_connections 2024-09-10T20:38:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.325+0000 7f5721862640 1 -- 172.21.15.28:0/3368403775 wait complete. 2024-09-10T20:38:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.326+0000 7f5721862640 1 Processor -- start 2024-09-10T20:38:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.327+0000 7f5721862640 1 -- start start 2024-09-10T20:38:43.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.327+0000 7f5721862640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 0x7f571c1efd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.327+0000 7f5721862640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571c151cc0 0x7f571c0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.327+0000 7f5721862640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c15ed20 0x7f571c1531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.327+0000 7f5721862640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f571c15f5b0 con 0x7f571c151cc0 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.327+0000 7f5721862640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f571c15f430 con 0x7f571c15ed20 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.327+0000 7f5721862640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f571c0cb750 con 0x7f571c0cf0f0 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571c151cc0 0x7f571c0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571c151cc0 0x7f571c0b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44248/0 (socket says 172.21.15.28:44248) 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 -- 172.21.15.28:0/3585051989 learned_addr learned my addr 172.21.15.28:0/3585051989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571affd640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 0x7f571c1efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571b7fe640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c15ed20 0x7f571c1531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 -- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 msgr2=0x7f571c1efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 0x7f571c1efd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 -- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c15ed20 msgr2=0x7f571c1531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c15ed20 0x7f571c1531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 -- 172.21.15.28:0/3585051989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c1523f0 con 0x7f571c151cc0 2024-09-10T20:38:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571b7fe640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c15ed20 0x7f571c1531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f571a7fc640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571c151cc0 0x7f571c0b09e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5710009950 tx=0x7f5710032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f5703fff640 1 -- 172.21.15.28:0/3585051989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5710040070 con 0x7f571c151cc0 2024-09-10T20:38:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f5703fff640 1 -- 172.21.15.28:0/3585051989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5710005310 con 0x7f571c151cc0 2024-09-10T20:38:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571c1526d0 con 0x7f571c151cc0 2024-09-10T20:38:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f5703fff640 1 -- 172.21.15.28:0/3585051989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5710005c90 con 0x7f571c151cc0 2024-09-10T20:38:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.328+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f571c152c90 con 0x7f571c151cc0 2024-09-10T20:38:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.329+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f571c0cbb70 con 0x7f571c151cc0 2024-09-10T20:38:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.330+0000 7f5703fff640 1 -- 172.21.15.28:0/3585051989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f571004c030 con 0x7f571c151cc0 2024-09-10T20:38:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.330+0000 7f5703fff640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5708077f70 0x7f570807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.330+0000 7f5703fff640 1 -- 172.21.15.28:0/3585051989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f57100baee0 con 0x7f571c151cc0 2024-09-10T20:38:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.330+0000 7f571affd640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5708077f70 0x7f570807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.331+0000 7f571affd640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5708077f70 0x7f570807a430 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f571c0cb4a0 tx=0x7f5714009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:43.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.335+0000 7f5703fff640 1 -- 172.21.15.28:0/3585051989 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5710083a70 con 0x7f571c151cc0 2024-09-10T20:38:43.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.524+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f571c068d10 con 0x7f571c151cc0 2024-09-10T20:38:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.525+0000 7f5703fff640 1 -- 172.21.15.28:0/3585051989 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16669 (secure 0 0 0) 0x7f5710083370 con 0x7f571c151cc0 2024-09-10T20:38:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:43.530 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:13.395359+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:09.596930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:10.711700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5708077f70 msgr2=0x7f570807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5708077f70 0x7f570807a430 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f571c0cb4a0 tx=0x7f5714009040 comp rx=0 tx=0).stop 2024-09-10T20:38:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571c151cc0 msgr2=0x7f571c0b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571c151cc0 0x7f571c0b09e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5710009950 tx=0x7f5710032c10 comp rx=0 tx=0).stop 2024-09-10T20:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 shutdown_connections 2024-09-10T20:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5708077f70 0x7f570807a430 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f571c15ed20 0x7f571c1531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571c151cc0 0x7f571c0b09e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 --2- 172.21.15.28:0/3585051989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f571c0cf0f0 0x7f571c1efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 >> 172.21.15.28:0/3585051989 conn(0x7f571c0d0420 msgr2=0x7f571c0cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 shutdown_connections 2024-09-10T20:38:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.527+0000 7f5721862640 1 -- 172.21.15.28:0/3585051989 wait complete. 2024-09-10T20:38:43.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:38:43.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.644+0000 7f98ed608640 1 -- 172.21.15.28:0/24036021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 msgr2=0x7f98d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.644+0000 7f98ed608640 1 --2- 172.21.15.28:0/24036021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 0x7f98d00051c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009b50 tx=0x7f98d0005830 comp rx=0 tx=0).stop 2024-09-10T20:38:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.644+0000 7f98e77fe640 1 -- 172.21.15.28:0/24036021 reap_dead start 2024-09-10T20:38:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.644+0000 7f98ed608640 1 -- 172.21.15.28:0/24036021 shutdown_connections 2024-09-10T20:38:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.644+0000 7f98ed608640 1 -- 172.21.15.28:0/24036021 >> 172.21.15.28:0/24036021 conn(0x7f98e80d4040 msgr2=0x7f98e80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.644+0000 7f98ed608640 1 -- 172.21.15.28:0/24036021 shutdown_connections 2024-09-10T20:38:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.644+0000 7f98ed608640 1 -- 172.21.15.28:0/24036021 wait complete. 2024-09-10T20:38:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.646+0000 7f98ed608640 1 Processor -- start 2024-09-10T20:38:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.646+0000 7f98ed608640 1 -- start start 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 0x7f98e81f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98e81f5800 0x7f98e8154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98e81f78d0 0x7f98e80cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98e80b7c40 con 0x7f98d0004d40 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f98e80d2e90 con 0x7f98e81f78d0 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f98e80b7dc0 con 0x7f98e81f5800 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 0x7f98e81f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 0x7f98e81f5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44262/0 (socket says 172.21.15.28:44262) 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 -- 172.21.15.28:0/3417814052 learned_addr learned my addr 172.21.15.28:0/3417814052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e67fc640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98e81f5800 0x7f98e8154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e77fe640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98e81f78d0 0x7f98e80cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 -- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98e81f5800 msgr2=0x7f98e8154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98e81f5800 0x7f98e8154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 -- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98e81f78d0 msgr2=0x7f98e80cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98e81f78d0 0x7f98e80cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 -- 172.21.15.28:0/3417814052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e80cb080 con 0x7f98d0004d40 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e67fc640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98e81f5800 0x7f98e8154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e77fe640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98e81f78d0 0x7f98e80cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98e6ffd640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 0x7f98e81f5d30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f98d8010c60 tx=0x7f98d800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98cf7fe640 1 -- 172.21.15.28:0/3417814052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8003b70 con 0x7f98d0004d40 2024-09-10T20:38:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98cf7fe640 1 -- 172.21.15.28:0/3417814052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98d8003e20 con 0x7f98d0004d40 2024-09-10T20:38:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98cf7fe640 1 -- 172.21.15.28:0/3417814052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8004290 con 0x7f98d0004d40 2024-09-10T20:38:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e80cb360 con 0x7f98d0004d40 2024-09-10T20:38:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.647+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e80cb920 con 0x7f98d0004d40 2024-09-10T20:38:43.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.648+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e80b8320 con 0x7f98d0004d40 2024-09-10T20:38:43.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.649+0000 7f98cf7fe640 1 -- 172.21.15.28:0/3417814052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f98d8024030 con 0x7f98d0004d40 2024-09-10T20:38:43.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.650+0000 7f98cf7fe640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98d4077fa0 0x7f98d407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:43.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.650+0000 7f98cf7fe640 1 -- 172.21.15.28:0/3417814052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f98d809ab60 con 0x7f98d0004d40 2024-09-10T20:38:43.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.650+0000 7f98e67fc640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98d4077fa0 0x7f98d407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:43.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.651+0000 7f98e67fc640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98d4077fa0 0x7f98d407a460 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f98dc001880 tx=0x7f98dc000f10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.654+0000 7f98cf7fe640 1 -- 172.21.15.28:0/3417814052 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f98d80636f0 con 0x7f98d0004d40 2024-09-10T20:38:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.875+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f98e80ce500 con 0x7f98d0004d40 2024-09-10T20:38:43.881 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:38:43.876+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:38:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.877+0000 7f98cf7fe640 1 -- 172.21.15.28:0/3417814052 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f98d8062ff0 con 0x7f98d0004d40 2024-09-10T20:38:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:38:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98d4077fa0 msgr2=0x7f98d407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98d4077fa0 0x7f98d407a460 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f98dc001880 tx=0x7f98dc000f10 comp rx=0 tx=0).stop 2024-09-10T20:38:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 msgr2=0x7f98e81f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 0x7f98e81f5d30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f98d8010c60 tx=0x7f98d800e720 comp rx=0 tx=0).stop 2024-09-10T20:38:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 shutdown_connections 2024-09-10T20:38:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98d4077fa0 0x7f98d407a460 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98e81f78d0 0x7f98e80cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98e81f5800 0x7f98e8154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 --2- 172.21.15.28:0/3417814052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d0004d40 0x7f98e81f5d30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 >> 172.21.15.28:0/3417814052 conn(0x7f98e80d4040 msgr2=0x7f98e80b94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 shutdown_connections 2024-09-10T20:38:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:43.880+0000 7f98ed608640 1 -- 172.21.15.28:0/3417814052 wait complete. 2024-09-10T20:38:43.890 INFO:tasks.ceph.mds.d.smithi159.stderr: -6882> 2024-09-10T20:38:17.342+0000 7f57378c9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:38:43.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:38:43.924 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:38:44.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.000+0000 7ff55750e640 1 -- 172.21.15.28:0/300878139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c0052e0 msgr2=0x7ff53c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:44.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.000+0000 7ff55750e640 1 --2- 172.21.15.28:0/300878139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c0052e0 0x7ff53c00b9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff548006fd0 tx=0x7ff53c005830 comp rx=0 tx=0).stop 2024-09-10T20:38:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.000+0000 7ff55750e640 1 -- 172.21.15.28:0/300878139 shutdown_connections 2024-09-10T20:38:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.000+0000 7ff55750e640 1 --2- 172.21.15.28:0/300878139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c0052e0 0x7ff53c00b9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.000+0000 7ff55750e640 1 -- 172.21.15.28:0/300878139 >> 172.21.15.28:0/300878139 conn(0x7ff5500d40f0 msgr2=0x7ff5500d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.000+0000 7ff55750e640 1 -- 172.21.15.28:0/300878139 shutdown_connections 2024-09-10T20:38:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.000+0000 7ff55750e640 1 -- 172.21.15.28:0/300878139 wait complete. 2024-09-10T20:38:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.001+0000 7ff55750e640 1 Processor -- start 2024-09-10T20:38:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.001+0000 7ff55750e640 1 -- start start 2024-09-10T20:38:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff55750e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff53c0052e0 0x7ff55015cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff55750e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5500c9350 0x7ff55015fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff55750e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff550160120 0x7ff5500b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff55750e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff550162150 con 0x7ff5500c9350 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff55750e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff550161fd0 con 0x7ff53c0052e0 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff55750e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff5501622d0 con 0x7ff550160120 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff554aa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5500c9350 0x7ff55015fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff554aa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5500c9350 0x7ff55015fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44278/0 (socket says 172.21.15.28:44278) 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff554aa9640 1 -- 172.21.15.28:0/534526862 learned_addr learned my addr 172.21.15.28:0/534526862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff5552aa640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff53c0052e0 0x7ff55015cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.002+0000 7ff555aab640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff550160120 0x7ff5500b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff554aa9640 1 -- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff550160120 msgr2=0x7ff5500b8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff554aa9640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff550160120 0x7ff5500b8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff554aa9640 1 -- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff53c0052e0 msgr2=0x7ff55015cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff554aa9640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff53c0052e0 0x7ff55015cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff554aa9640 1 -- 172.21.15.28:0/534526862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff550160610 con 0x7ff5500c9350 2024-09-10T20:38:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff554aa9640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5500c9350 0x7ff55015fbe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff540003f00 tx=0x7ff540001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff545ffb640 1 -- 172.21.15.28:0/534526862 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540011640 con 0x7ff5500c9350 2024-09-10T20:38:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff545ffb640 1 -- 172.21.15.28:0/534526862 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff540016020 con 0x7ff5500c9350 2024-09-10T20:38:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5501608f0 con 0x7ff5500c9350 2024-09-10T20:38:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff545ffb640 1 -- 172.21.15.28:0/534526862 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540011a80 con 0x7ff5500c9350 2024-09-10T20:38:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.003+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5501f49f0 con 0x7ff5500c9350 2024-09-10T20:38:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.004+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff550160e10 con 0x7ff5500c9350 2024-09-10T20:38:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.005+0000 7ff545ffb640 1 -- 172.21.15.28:0/534526862 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff540011c20 con 0x7ff5500c9350 2024-09-10T20:38:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.005+0000 7ff545ffb640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff534077f70 0x7ff53407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.006+0000 7ff545ffb640 1 -- 172.21.15.28:0/534526862 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7ff54009a860 con 0x7ff5500c9350 2024-09-10T20:38:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.006+0000 7ff5552aa640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff534077f70 0x7ff53407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.006+0000 7ff5552aa640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff534077f70 0x7ff53407a430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff54c003700 tx=0x7ff54c00b260 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.010+0000 7ff545ffb640 1 -- 172.21.15.28:0/534526862 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff550160e10 con 0x7ff5500c9350 2024-09-10T20:38:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.230+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff550162830 con 0x7ff5500c9350 2024-09-10T20:38:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.881+0000 7ff545ffb640 1 -- 172.21.15.28:0/534526862 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7ff540020020 con 0x7ff5500c9350 2024-09-10T20:38:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff534077f70 msgr2=0x7ff53407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff534077f70 0x7ff53407a430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff54c003700 tx=0x7ff54c00b260 comp rx=0 tx=0).stop 2024-09-10T20:38:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5500c9350 msgr2=0x7ff55015fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5500c9350 0x7ff55015fbe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff540003f00 tx=0x7ff540001630 comp rx=0 tx=0).stop 2024-09-10T20:38:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 shutdown_connections 2024-09-10T20:38:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff534077f70 0x7ff53407a430 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff550160120 0x7ff5500b8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5500c9350 0x7ff55015fbe0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 --2- 172.21.15.28:0/534526862 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff53c0052e0 0x7ff55015cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 >> 172.21.15.28:0/534526862 conn(0x7ff5500d40f0 msgr2=0x7ff55006e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 shutdown_connections 2024-09-10T20:38:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:44.884+0000 7ff55750e640 1 -- 172.21.15.28:0/534526862 wait complete. 2024-09-10T20:38:44.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:38:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 -- 172.21.15.28:0/1587586019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 msgr2=0x7f41280cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 --2- 172.21.15.28:0/1587586019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41280cdf00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4120005fd0 tx=0x7f412002b8f0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 -- 172.21.15.28:0/1587586019 shutdown_connections 2024-09-10T20:38:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 --2- 172.21.15.28:0/1587586019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41280cdf00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 --2- 172.21.15.28:0/1587586019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f412815a350 0x7f41280ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 -- 172.21.15.28:0/1587586019 >> 172.21.15.28:0/1587586019 conn(0x7f41280cfc80 msgr2=0x7f41280d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 -- 172.21.15.28:0/1587586019 shutdown_connections 2024-09-10T20:38:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.013+0000 7f412d670640 1 -- 172.21.15.28:0/1587586019 wait complete. 2024-09-10T20:38:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.014+0000 7f412d670640 1 Processor -- start 2024-09-10T20:38:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.014+0000 7f412d670640 1 -- start start 2024-09-10T20:38:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f412d670640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41280cabc0 0x7f4128158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f412d670640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41281f5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f412d670640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412815a350 0x7f4128156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f412d670640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f412806e390 con 0x7f41280cda70 2024-09-10T20:38:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f412d670640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f412815a940 con 0x7f412815a350 2024-09-10T20:38:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f412d670640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f412806e510 con 0x7f41280cabc0 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f41267fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41281f5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f41267fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41281f5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44294/0 (socket says 172.21.15.28:44294) 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f41267fc640 1 -- 172.21.15.28:0/4005471931 learned_addr learned my addr 172.21.15.28:0/4005471931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.015+0000 7f4126ffd640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41280cabc0 0x7f4128158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f41277fe640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412815a350 0x7f4128156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f41267fc640 1 -- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41280cabc0 msgr2=0x7f4128158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f41267fc640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41280cabc0 0x7f4128158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f41267fc640 1 -- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412815a350 msgr2=0x7f4128156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f41267fc640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412815a350 0x7f4128156050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f41267fc640 1 -- 172.21.15.28:0/4005471931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4128157940 con 0x7f41280cda70 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f4126ffd640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41280cabc0 0x7f4128158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f41267fc640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41281f5b60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c500 tx=0x7f411c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f410f7fe640 1 -- 172.21.15.28:0/4005471931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c0052d0 con 0x7f41280cda70 2024-09-10T20:38:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f410f7fe640 1 -- 172.21.15.28:0/4005471931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f411c005470 con 0x7f41280cda70 2024-09-10T20:38:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4128157be0 con 0x7f41280cda70 2024-09-10T20:38:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f410f7fe640 1 -- 172.21.15.28:0/4005471931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c012870 con 0x7f41280cda70 2024-09-10T20:38:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.016+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41281581a0 con 0x7f41280cda70 2024-09-10T20:38:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.017+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412806e9f0 con 0x7f41280cda70 2024-09-10T20:38:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.018+0000 7f410f7fe640 1 -- 172.21.15.28:0/4005471931 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f411c0035f0 con 0x7f41280cda70 2024-09-10T20:38:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.018+0000 7f410f7fe640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4114077f70 0x7f411407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.018+0000 7f410f7fe640 1 -- 172.21.15.28:0/4005471931 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f411c099970 con 0x7f41280cda70 2024-09-10T20:38:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.018+0000 7f4126ffd640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4114077f70 0x7f411407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.019+0000 7f4126ffd640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4114077f70 0x7f411407a430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f411800ab40 tx=0x7f411800a520 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:45.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.022+0000 7f410f7fe640 1 -- 172.21.15.28:0/4005471931 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f411c062420 con 0x7f41280cda70 2024-09-10T20:38:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.208+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f41280ce7d0 con 0x7f41280cda70 2024-09-10T20:38:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.883+0000 7f410f7fe640 1 -- 172.21.15.28:0/4005471931 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) ==== 181+0+0 (secure 0 0 0) 0x7f411c061d20 con 0x7f41280cda70 2024-09-10T20:38:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:38:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4114077f70 msgr2=0x7f411407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4114077f70 0x7f411407a430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f411800ab40 tx=0x7f411800a520 comp rx=0 tx=0).stop 2024-09-10T20:38:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 msgr2=0x7f41281f5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41281f5b60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c500 tx=0x7f411c00c9c0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 shutdown_connections 2024-09-10T20:38:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4114077f70 0x7f411407a430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412815a350 0x7f4128156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41280cda70 0x7f41281f5b60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 --2- 172.21.15.28:0/4005471931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41280cabc0 0x7f4128158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 >> 172.21.15.28:0/4005471931 conn(0x7f41280cfc80 msgr2=0x7f41280cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 shutdown_connections 2024-09-10T20:38:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:45.887+0000 7f412d670640 1 -- 172.21.15.28:0/4005471931 wait complete. 2024-09-10T20:38:45.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:38:46.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 -- 172.21.15.28:0/2948329306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 msgr2=0x7f5adc0b55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:46.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/2948329306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc0b55c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009ca0 tx=0x7f5ad402aa30 comp rx=0 tx=0).stop 2024-09-10T20:38:46.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 -- 172.21.15.28:0/2948329306 shutdown_connections 2024-09-10T20:38:46.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/2948329306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc0b55c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/2948329306 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5adc0cf590 0x7f5adc0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 -- 172.21.15.28:0/2948329306 >> 172.21.15.28:0/2948329306 conn(0x7f5adc0d1e40 msgr2=0x7f5adc0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 -- 172.21.15.28:0/2948329306 shutdown_connections 2024-09-10T20:38:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.019+0000 7f5ae1f17640 1 -- 172.21.15.28:0/2948329306 wait complete. 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.020+0000 7f5ae1f17640 1 Processor -- start 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.020+0000 7f5ae1f17640 1 -- start start 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5ae1f17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc1f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5ae1f17640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5adc0cf590 0x7f5adc0ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5ae1f17640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5adc15f540 0x7f5adc1f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5ae1f17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc0b5ec0 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5ae1f17640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc0b5d40 con 0x7f5adc0cf590 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5ae1f17640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5adc0b6040 con 0x7f5adc15f540 2024-09-10T20:38:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5adb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc1f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5adb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc1f5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44320/0 (socket says 172.21.15.28:44320) 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.021+0000 7f5adb7fe640 1 -- 172.21.15.28:0/1309079087 learned_addr learned my addr 172.21.15.28:0/1309079087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adaffd640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5adc0cf590 0x7f5adc0ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adbfff640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5adc15f540 0x7f5adc1f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adb7fe640 1 -- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5adc15f540 msgr2=0x7f5adc1f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adb7fe640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5adc15f540 0x7f5adc1f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adb7fe640 1 -- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5adc0cf590 msgr2=0x7f5adc0ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adb7fe640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5adc0cf590 0x7f5adc0ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adb7fe640 1 -- 172.21.15.28:0/1309079087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc1f2380 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adaffd640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5adc0cf590 0x7f5adc0ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5adb7fe640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc1f5490 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00b3f0 tx=0x7f5acc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5ac3fff640 1 -- 172.21.15.28:0/1309079087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc015020 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5ac3fff640 1 -- 172.21.15.28:0/1309079087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5acc011920 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc0ca940 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5ac3fff640 1 -- 172.21.15.28:0/1309079087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc011d70 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.022+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc0c8f10 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.023+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5adc15d5f0 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.024+0000 7f5ac3fff640 1 -- 172.21.15.28:0/1309079087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5acc01a400 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.024+0000 7f5ac3fff640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ac8077fa0 0x7f5ac807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.024+0000 7f5ac3fff640 1 -- 172.21.15.28:0/1309079087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5357+0+0 (secure 0 0 0) 0x7f5acc099a50 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.025+0000 7f5adaffd640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ac8077fa0 0x7f5ac807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.025+0000 7f5adaffd640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ac8077fa0 0x7f5ac807a460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5adc1f8fe0 tx=0x7f5ad000a5c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:46.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.028+0000 7f5ac3fff640 1 -- 172.21.15.28:0/1309079087 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5acc01f030 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.215+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5adc0d27d0 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.888+0000 7f5ac3fff640 1 -- 172.21.15.28:0/1309079087 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) ==== 169+0+0 (secure 0 0 0) 0x7f5acc062740 con 0x7f5adc0cd5a0 2024-09-10T20:38:46.893 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:38:46.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.891+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ac8077fa0 msgr2=0x7f5ac807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.891+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ac8077fa0 0x7f5ac807a460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5adc1f8fe0 tx=0x7f5ad000a5c0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.891+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 msgr2=0x7f5adc1f5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.891+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc1f5490 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00b3f0 tx=0x7f5acc00b8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 shutdown_connections 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5ac8077fa0 0x7f5ac807a460 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5adc15f540 0x7f5adc1f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5adc0cf590 0x7f5adc0ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 --2- 172.21.15.28:0/1309079087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5adc0cd5a0 0x7f5adc1f5490 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00b3f0 tx=0x7f5acc00b8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 >> 172.21.15.28:0/1309079087 conn(0x7f5adc0d1e40 msgr2=0x7f5adc15baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 shutdown_connections 2024-09-10T20:38:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:46.892+0000 7f5ae1f17640 1 -- 172.21.15.28:0/1309079087 wait complete. 2024-09-10T20:38:46.911 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:38:46.911 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:38:46.911 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:38:46.911 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:38:46.911 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:38:46.911 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:38:46.911 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:38:46.912 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:38:46.912 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:38:46.912 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:38:46.912 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:38:46.912 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:38:46.912 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:38:46.912 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:38:46.913 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:38:46.913 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:38:46.921 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:38:46.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:38:46.922 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:38:46.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:38:46.926 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:38:46.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:38:46.926 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:38:46.927 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:38:46.964 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:38:46.965 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:46.965 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:38:46.965 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:46.968 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:38:46.968 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:46.969 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:38:46.970 DEBUG:teuthology.parallel:result is None 2024-09-10T20:38:46.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:38:47.048 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:38:47.048 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:38:47.049 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:38:47.049 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:38:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 -- 172.21.15.28:0/2935513033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da00cf780 msgr2=0x7f4da00cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/2935513033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da00cf780 0x7f4da00cfbf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4da008c020 tx=0x7f4d9002a440 comp rx=0 tx=0).stop 2024-09-10T20:38:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 -- 172.21.15.28:0/2935513033 shutdown_connections 2024-09-10T20:38:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/2935513033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da00cf780 0x7f4da00cfbf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/2935513033 >> v2:172.21.15.159:3300/0 conn(0x7f4da015de70 0x7f4da00d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 -- 172.21.15.28:0/2935513033 >> 172.21.15.28:0/2935513033 conn(0x7f4da00d3fb0 msgr2=0x7f4da00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:47.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 -- 172.21.15.28:0/2935513033 shutdown_connections 2024-09-10T20:38:47.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.082+0000 7f4da7f2b640 1 -- 172.21.15.28:0/2935513033 wait complete. 2024-09-10T20:38:47.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.083+0000 7f4da7f2b640 1 Processor -- start 2024-09-10T20:38:47.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.084+0000 7f4da7f2b640 1 -- start start 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.084+0000 7f4da7f2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da00cf780 0x7f4da01f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.084+0000 7f4da7f2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da015de70 0x7f4da00a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da7f2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da0161b50 0x7f4da015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da7f2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4da00b7690 con 0x7f4da015de70 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da7f2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4da00b7510 con 0x7f4da0161b50 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da7f2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4da00b7810 con 0x7f4da00cf780 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da015de70 0x7f4da00a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da015de70 0x7f4da00a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44352/0 (socket says 172.21.15.28:44352) 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 -- 172.21.15.28:0/1757601458 learned_addr learned my addr 172.21.15.28:0/1757601458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da5cc7640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da00cf780 0x7f4da01f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da64c8640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da0161b50 0x7f4da015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 -- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da00cf780 msgr2=0x7f4da01f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da00cf780 0x7f4da01f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 -- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da0161b50 msgr2=0x7f4da015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da0161b50 0x7f4da015c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 -- 172.21.15.28:0/1757601458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da01f4730 con 0x7f4da015de70 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da5cc7640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da00cf780 0x7f4da01f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.085+0000 7f4da54c6640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da015de70 0x7f4da00a1830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4d90009920 tx=0x7f4d90033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.086+0000 7f4d967fc640 1 -- 172.21.15.28:0/1757601458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9003f020 con 0x7f4da015de70 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.086+0000 7f4d967fc640 1 -- 172.21.15.28:0/1757601458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d90033d90 con 0x7f4da015de70 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.086+0000 7f4d967fc640 1 -- 172.21.15.28:0/1757601458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90037040 con 0x7f4da015de70 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.086+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da00b3120 con 0x7f4da015de70 2024-09-10T20:38:47.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.086+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da00b3400 con 0x7f4da015de70 2024-09-10T20:38:47.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.087+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da00cfbf0 con 0x7f4da015de70 2024-09-10T20:38:47.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.087+0000 7f4d967fc640 1 -- 172.21.15.28:0/1757601458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4d9003a030 con 0x7f4da015de70 2024-09-10T20:38:47.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.088+0000 7f4d967fc640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d84077fa0 0x7f4d8407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:47.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.088+0000 7f4d967fc640 1 -- 172.21.15.28:0/1757601458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 4943+0+0 (secure 0 0 0) 0x7f4d900b9a30 con 0x7f4da015de70 2024-09-10T20:38:47.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.088+0000 7f4da5cc7640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d84077fa0 0x7f4d8407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:47.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.089+0000 7f4da5cc7640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d84077fa0 0x7f4d8407a460 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4da00b2760 tx=0x7f4d9c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.092+0000 7f4d967fc640 1 -- 172.21.15.28:0/1757601458 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4d90049030 con 0x7f4da015de70 2024-09-10T20:38:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.286+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f4da00b9f30 con 0x7f4da015de70 2024-09-10T20:38:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.904+0000 7f4d967fc640 1 -- 172.21.15.28:0/1757601458 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v26) ==== 97+0+0 (secure 0 0 0) 0x7f4d90082790 con 0x7f4da015de70 2024-09-10T20:38:47.909 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:38:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d84077fa0 msgr2=0x7f4d8407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d84077fa0 0x7f4d8407a460 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4da00b2760 tx=0x7f4d9c005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da015de70 msgr2=0x7f4da00a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da015de70 0x7f4da00a1830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4d90009920 tx=0x7f4d90033ba0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 shutdown_connections 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d84077fa0 0x7f4d8407a460 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4da0161b50 0x7f4da015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da015de70 0x7f4da00a1830 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 --2- 172.21.15.28:0/1757601458 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4da00cf780 0x7f4da01f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 >> 172.21.15.28:0/1757601458 conn(0x7f4da00d3fb0 msgr2=0x7f4da00b6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 shutdown_connections 2024-09-10T20:38:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:47.907+0000 7f4da7f2b640 1 -- 172.21.15.28:0/1757601458 wait complete. 2024-09-10T20:38:47.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:38:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.033+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/2930268696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 msgr2=0x7fa2b40d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.033+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/2930268696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40d2b90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40d01d0 tx=0x7fa2ac02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.033+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/2930268696 shutdown_connections 2024-09-10T20:38:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.033+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/2930268696 >> v2:172.21.15.159:3300/0 conn(0x7fa2b4162260 0x7fa2b40cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.033+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/2930268696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40d2b90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.033+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/2930268696 >> 172.21.15.28:0/2930268696 conn(0x7fa2b40d3fb0 msgr2=0x7fa2b40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.033+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/2930268696 shutdown_connections 2024-09-10T20:38:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.034+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/2930268696 wait complete. 2024-09-10T20:38:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.035+0000 7fa2b8ed6640 1 Processor -- start 2024-09-10T20:38:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.035+0000 7fa2b8ed6640 1 -- start start 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b8ed6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b40cf780 0x7fa2b40a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b8ed6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b8ed6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b4162260 0x7fa2b414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b8ed6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b40b6f80 con 0x7fa2b415de70 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b8ed6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b40b6e00 con 0x7fa2b4162260 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b8ed6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b40b7100 con 0x7fa2b40cf780 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44364/0 (socket says 172.21.15.28:44364) 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 -- 172.21.15.28:0/1868015755 learned_addr learned my addr 172.21.15.28:0/1868015755 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b2d9d640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b4162260 0x7fa2b414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b259c640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b40cf780 0x7fa2b40a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 -- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b40cf780 msgr2=0x7fa2b40a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b40cf780 0x7fa2b40a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 -- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b4162260 msgr2=0x7fa2b414f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b4162260 0x7fa2b414f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 -- 172.21.15.28:0/1868015755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b40cda10 con 0x7fa2b415de70 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b259c640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b40cf780 0x7fa2b40a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2b1d9b640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40a0de0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00c490 tx=0x7fa29c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2aaffd640 1 -- 172.21.15.28:0/1868015755 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c016020 con 0x7fa2b415de70 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2aaffd640 1 -- 172.21.15.28:0/1868015755 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa29c012940 con 0x7fa2b415de70 2024-09-10T20:38:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.036+0000 7fa2aaffd640 1 -- 172.21.15.28:0/1868015755 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c012d90 con 0x7fa2b415de70 2024-09-10T20:38:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.037+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b40cdcf0 con 0x7fa2b415de70 2024-09-10T20:38:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.037+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b414d730 con 0x7fa2b415de70 2024-09-10T20:38:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.038+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b40d2b90 con 0x7fa2b415de70 2024-09-10T20:38:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.038+0000 7fa2aaffd640 1 -- 172.21.15.28:0/1868015755 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa29c01a3b0 con 0x7fa2b415de70 2024-09-10T20:38:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.039+0000 7fa2aaffd640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa294077f70 0x7fa29407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.039+0000 7fa2aaffd640 1 -- 172.21.15.28:0/1868015755 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7fa29c099940 con 0x7fa2b415de70 2024-09-10T20:38:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.039+0000 7fa2b259c640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa294077f70 0x7fa29407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.039+0000 7fa2b259c640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa294077f70 0x7fa29407a430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac034000 tx=0x7fa2ac004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.043+0000 7fa2aaffd640 1 -- 172.21.15.28:0/1868015755 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa29c020030 con 0x7fa2b415de70 2024-09-10T20:38:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.298+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa2b40b9f30 con 0x7fa2b415de70 2024-09-10T20:38:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.299+0000 7fa2aaffd640 1 -- 172.21.15.28:0/1868015755 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+6645 (secure 0 0 0) 0x7fa29c062790 con 0x7fa2b415de70 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:38:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.307 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:38:48.308 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:38:48.309 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.309 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.309 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.309 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:38:48.312 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:38:48.314 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:38:48.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mds", 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAbruBmy0PGJBAANNmg5AxKjVW3ylFoNL2OTg==", 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mds" 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mgr", 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAbruBmJU/GJBAA6GenA3liZTJRjDLH15n93Q==", 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mgr" 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-osd", 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAbruBmmVbGJBAA/NMKLmN0IXYVICsUc1v3yw==", 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-osd" 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd", 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAbruBmk17GJBAAuumujt8I+UpifgL79wEN8A==", 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd" 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAbruBmx2vGJBAANPsttAGQp/KPJ4ZSLqLhaA==", 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rgw", 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAbruBmEHTGJBAAbaa4PsV7z6h8TMPyxOz4hA==", 2024-09-10T20:38:48.319 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rgw" 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:48.320 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:38:48.321 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa294077f70 msgr2=0x7fa29407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa294077f70 0x7fa29407a430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac034000 tx=0x7fa2ac004ab0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 msgr2=0x7fa2b40a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40a0de0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00c490 tx=0x7fa29c00c950 comp rx=0 tx=0).stop 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 shutdown_connections 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa294077f70 0x7fa29407a430 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b4162260 0x7fa2b414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b415de70 0x7fa2b40a0de0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 --2- 172.21.15.28:0/1868015755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b40cf780 0x7fa2b40a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 >> 172.21.15.28:0/1868015755 conn(0x7fa2b40d3fb0 msgr2=0x7fa2b4075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 shutdown_connections 2024-09-10T20:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.301+0000 7fa2b8ed6640 1 -- 172.21.15.28:0/1868015755 wait complete. 2024-09-10T20:38:48.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-09-10T20:38:48.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 -- 172.21.15.28:0/3518975438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff62015e0f0 msgr2=0x7ff6200d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 --2- 172.21.15.28:0/3518975438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff62015e0f0 0x7ff6200d2030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff6200698c0 tx=0x7ff61002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 -- 172.21.15.28:0/3518975438 shutdown_connections 2024-09-10T20:38:48.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 --2- 172.21.15.28:0/3518975438 >> v2:172.21.15.159:3300/0 conn(0x7ff6200b77f0 0x7ff6200b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 --2- 172.21.15.28:0/3518975438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff62015e0f0 0x7ff6200d2030 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 -- 172.21.15.28:0/3518975438 >> 172.21.15.28:0/3518975438 conn(0x7ff6200d4040 msgr2=0x7ff6200d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:48.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 -- 172.21.15.28:0/3518975438 shutdown_connections 2024-09-10T20:38:48.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.425+0000 7ff625608640 1 -- 172.21.15.28:0/3518975438 wait complete. 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.426+0000 7ff625608640 1 Processor -- start 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.426+0000 7ff625608640 1 -- start start 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff625608640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6200b77f0 0x7ff620152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff625608640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200d4c40 0x7ff62015b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff625608640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff62015e0f0 0x7ff6200c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff625608640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6200b8180 con 0x7ff6200d4c40 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff625608640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff6200d2e90 con 0x7ff62015e0f0 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff625608640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff6200b8300 con 0x7ff6200b77f0 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff61e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200d4c40 0x7ff62015b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff61e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200d4c40 0x7ff62015b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44378/0 (socket says 172.21.15.28:44378) 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.427+0000 7ff61e7fc640 1 -- 172.21.15.28:0/199213986 learned_addr learned my addr 172.21.15.28:0/199213986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61effd640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6200b77f0 0x7ff620152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61f7fe640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff62015e0f0 0x7ff6200c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61e7fc640 1 -- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6200b77f0 msgr2=0x7ff620152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61e7fc640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6200b77f0 0x7ff620152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61e7fc640 1 -- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff62015e0f0 msgr2=0x7ff6200c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61e7fc640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff62015e0f0 0x7ff6200c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61e7fc640 1 -- 172.21.15.28:0/199213986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff620153050 con 0x7ff6200d4c40 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61f7fe640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff62015e0f0 0x7ff6200c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff61e7fc640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200d4c40 0x7ff62015b9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff61400c420 tx=0x7ff61400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff6077fe640 1 -- 172.21.15.28:0/199213986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614011790 con 0x7ff6200d4c40 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff6077fe640 1 -- 172.21.15.28:0/199213986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff614017070 con 0x7ff6200d4c40 2024-09-10T20:38:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62015a920 con 0x7ff6200d4c40 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff6077fe640 1 -- 172.21.15.28:0/199213986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614011c80 con 0x7ff6200d4c40 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.428+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62015aee0 con 0x7ff6200d4c40 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.429+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6200b8fb0 con 0x7ff6200d4c40 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.430+0000 7ff6077fe640 1 -- 172.21.15.28:0/199213986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff61401c460 con 0x7ff6200d4c40 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.430+0000 7ff6077fe640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff60c077f70 0x7ff60c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.430+0000 7ff6077fe640 1 -- 172.21.15.28:0/199213986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7ff614099770 con 0x7ff6200d4c40 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.430+0000 7ff61effd640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff60c077f70 0x7ff60c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.431+0000 7ff61effd640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff60c077f70 0x7ff60c07a430 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff610034000 tx=0x7ff610004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.434+0000 7ff6077fe640 1 -- 172.21.15.28:0/199213986 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff614062650 con 0x7ff6200d4c40 2024-09-10T20:38:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.697+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7ff6200b9fc0 con 0x7ff6200d4c40 2024-09-10T20:38:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.700+0000 7ff6077fe640 1 -- 172.21.15.28:0/199213986 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v19) ==== 90+0+0 (secure 0 0 0) 0x7ff614061f50 con 0x7ff6200d4c40 2024-09-10T20:38:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff60c077f70 msgr2=0x7ff60c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff60c077f70 0x7ff60c07a430 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff610034000 tx=0x7ff610004ab0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200d4c40 msgr2=0x7ff62015b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200d4c40 0x7ff62015b9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff61400c420 tx=0x7ff61400c8e0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 shutdown_connections 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff60c077f70 0x7ff60c07a430 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff62015e0f0 0x7ff6200c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200d4c40 0x7ff62015b9c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 --2- 172.21.15.28:0/199213986 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6200b77f0 0x7ff620152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 >> 172.21.15.28:0/199213986 conn(0x7ff6200d4040 msgr2=0x7ff62006e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 shutdown_connections 2024-09-10T20:38:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.702+0000 7ff625608640 1 -- 172.21.15.28:0/199213986 wait complete. 2024-09-10T20:38:48.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-09-10T20:38:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.818+0000 7f2464f94640 1 -- 172.21.15.28:0/2005688872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244c003a30 msgr2=0x7f244c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.818+0000 7f2464f94640 1 --2- 172.21.15.28:0/2005688872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244c003a30 0x7f244c003e50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2450001d90 tx=0x7f244c005830 comp rx=0 tx=0).stop 2024-09-10T20:38:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.818+0000 7f245ed9d640 1 -- 172.21.15.28:0/2005688872 reap_dead start 2024-09-10T20:38:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.818+0000 7f2464f94640 1 -- 172.21.15.28:0/2005688872 shutdown_connections 2024-09-10T20:38:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.818+0000 7f2464f94640 1 -- 172.21.15.28:0/2005688872 >> 172.21.15.28:0/2005688872 conn(0x7f24600d0420 msgr2=0x7f24600d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.819+0000 7f2464f94640 1 -- 172.21.15.28:0/2005688872 shutdown_connections 2024-09-10T20:38:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.819+0000 7f2464f94640 1 -- 172.21.15.28:0/2005688872 wait complete. 2024-09-10T20:38:48.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.820+0000 7f2464f94640 1 Processor -- start 2024-09-10T20:38:48.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.820+0000 7f2464f94640 1 -- start start 2024-09-10T20:38:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2464f94640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f244c003a30 0x7f246014e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2464f94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24600b5130 0x7f24600b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2464f94640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24600b1540 0x7f24601f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2464f94640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24600cae00 con 0x7f24600b5130 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2464f94640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f24600cac80 con 0x7f244c003a30 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2464f94640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f246006ef30 con 0x7f24600b1540 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24600b5130 0x7f24600b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24600b5130 0x7f24600b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44404/0 (socket says 172.21.15.28:44404) 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 -- 172.21.15.28:0/3853658421 learned_addr learned my addr 172.21.15.28:0/3853658421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245e59c640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f244c003a30 0x7f246014e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245ed9d640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24600b1540 0x7f24601f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 -- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24600b1540 msgr2=0x7f24601f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24600b1540 0x7f24601f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 -- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f244c003a30 msgr2=0x7f246014e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f244c003a30 0x7f246014e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 -- 172.21.15.28:0/3853658421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24601f3510 con 0x7f24600b5130 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245e59c640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f244c003a30 0x7f246014e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245ed9d640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24600b1540 0x7f24601f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f245dd9b640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24600b5130 0x7f24600b1000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f244800fa60 tx=0x7f244800ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2456ffd640 1 -- 172.21.15.28:0/3853658421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244800bcb0 con 0x7f24600b5130 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2456ffd640 1 -- 172.21.15.28:0/3853658421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2448011440 con 0x7f24600b5130 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24600b4650 con 0x7f24600b5130 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.821+0000 7f2456ffd640 1 -- 172.21.15.28:0/3853658421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448011890 con 0x7f24600b5130 2024-09-10T20:38:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.822+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24600b4c10 con 0x7f24600b5130 2024-09-10T20:38:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.822+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24600b1950 con 0x7f24600b5130 2024-09-10T20:38:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.823+0000 7f2456ffd640 1 -- 172.21.15.28:0/3853658421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2448023030 con 0x7f24600b5130 2024-09-10T20:38:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.824+0000 7f2456ffd640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2440077fa0 0x7f244007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.824+0000 7f2456ffd640 1 -- 172.21.15.28:0/3853658421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f244809a320 con 0x7f24600b5130 2024-09-10T20:38:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.824+0000 7f245e59c640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2440077fa0 0x7f244007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.824+0000 7f245e59c640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2440077fa0 0x7f244007a460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2450032000 tx=0x7f2450001510 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:48.828+0000 7f2456ffd640 1 -- 172.21.15.28:0/3853658421 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f24600b1950 con 0x7f24600b5130 2024-09-10T20:38:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.085+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f24600b4440 con 0x7f24600b5130 2024-09-10T20:38:49.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.087+0000 7f2456ffd640 1 -- 172.21.15.28:0/3853658421 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v20) ==== 90+0+0 (secure 0 0 0) 0x7f2448063160 con 0x7f24600b5130 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2440077fa0 msgr2=0x7f244007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2440077fa0 0x7f244007a460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2450032000 tx=0x7f2450001510 comp rx=0 tx=0).stop 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24600b5130 msgr2=0x7f24600b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24600b5130 0x7f24600b1000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f244800fa60 tx=0x7f244800ff20 comp rx=0 tx=0).stop 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 shutdown_connections 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2440077fa0 0x7f244007a460 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f24600b1540 0x7f24601f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24600b5130 0x7f24600b1000 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 --2- 172.21.15.28:0/3853658421 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f244c003a30 0x7f246014e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 >> 172.21.15.28:0/3853658421 conn(0x7f24600d0420 msgr2=0x7f24600706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 shutdown_connections 2024-09-10T20:38:49.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.090+0000 7f2464f94640 1 -- 172.21.15.28:0/3853658421 wait complete. 2024-09-10T20:38:49.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-09-10T20:38:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.206+0000 7fc7ad32c640 1 -- 172.21.15.28:0/4193053292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a815de70 msgr2=0x7fc7a80d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.206+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/4193053292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a815de70 0x7fc7a80d2b90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0005fd0 tx=0x7fc7a002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.206+0000 7fc7ad32c640 1 -- 172.21.15.28:0/4193053292 shutdown_connections 2024-09-10T20:38:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.206+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/4193053292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a815de70 0x7fc7a80d2b90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.206+0000 7fc7ad32c640 1 -- 172.21.15.28:0/4193053292 >> 172.21.15.28:0/4193053292 conn(0x7fc7a80d3fb0 msgr2=0x7fc7a80d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.206+0000 7fc7ad32c640 1 -- 172.21.15.28:0/4193053292 shutdown_connections 2024-09-10T20:38:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.206+0000 7fc7ad32c640 1 -- 172.21.15.28:0/4193053292 wait complete. 2024-09-10T20:38:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.207+0000 7fc7ad32c640 1 Processor -- start 2024-09-10T20:38:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.207+0000 7fc7ad32c640 1 -- start start 2024-09-10T20:38:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7ad32c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a80cf780 0x7fc7a80cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7ad32c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a815de70 0x7fc7a81f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7ad32c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8162260 0x7fc7a81f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7ad32c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a80cff80 con 0x7fc7a8162260 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7ad32c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a80cfe00 con 0x7fc7a80cf780 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7ad32c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a815e470 con 0x7fc7a815de70 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8162260 0x7fc7a81f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8162260 0x7fc7a81f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44412/0 (socket says 172.21.15.28:44412) 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.208+0000 7fc7a77fe640 1 -- 172.21.15.28:0/3420768761 learned_addr learned my addr 172.21.15.28:0/3420768761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a6ffd640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a80cf780 0x7fc7a80cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a67fc640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a815de70 0x7fc7a81f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a77fe640 1 -- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a815de70 msgr2=0x7fc7a81f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a77fe640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a815de70 0x7fc7a81f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a77fe640 1 -- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a80cf780 msgr2=0x7fc7a80cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a77fe640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a80cf780 0x7fc7a80cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a77fe640 1 -- 172.21.15.28:0/3420768761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a81f6340 con 0x7fc7a8162260 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7a77fe640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8162260 0x7fc7a81f73e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc79800b410 tx=0x7fc79800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc79f7fe640 1 -- 172.21.15.28:0/3420768761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798015030 con 0x7fc7a8162260 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc79f7fe640 1 -- 172.21.15.28:0/3420768761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc798004020 con 0x7fc7a8162260 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a81f6900 con 0x7fc7a8162260 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc79f7fe640 1 -- 172.21.15.28:0/3420768761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798004470 con 0x7fc7a8162260 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.209+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a80cd840 con 0x7fc7a8162260 2024-09-10T20:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.210+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a80b7850 con 0x7fc7a8162260 2024-09-10T20:38:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.211+0000 7fc79f7fe640 1 -- 172.21.15.28:0/3420768761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc798022060 con 0x7fc7a8162260 2024-09-10T20:38:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.211+0000 7fc79f7fe640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc788077f70 0x7fc78807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.211+0000 7fc79f7fe640 1 -- 172.21.15.28:0/3420768761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7fc7980994e0 con 0x7fc7a8162260 2024-09-10T20:38:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.211+0000 7fc7a6ffd640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc788077f70 0x7fc78807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.212+0000 7fc7a6ffd640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc788077f70 0x7fc78807a430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0034000 tx=0x7fc7a0004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.215+0000 7fc79f7fe640 1 -- 172.21.15.28:0/3420768761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc798062330 con 0x7fc7a8162260 2024-09-10T20:38:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.472+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7fc7a80b6720 con 0x7fc7a8162260 2024-09-10T20:38:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.474+0000 7fc79f7fe640 1 -- 172.21.15.28:0/3420768761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v21) ==== 90+0+0 (secure 0 0 0) 0x7fc798061c30 con 0x7fc7a8162260 2024-09-10T20:38:49.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc788077f70 msgr2=0x7fc78807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc788077f70 0x7fc78807a430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0034000 tx=0x7fc7a0004ab0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8162260 msgr2=0x7fc7a81f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8162260 0x7fc7a81f73e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc79800b410 tx=0x7fc79800b8d0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 shutdown_connections 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc788077f70 0x7fc78807a430 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8162260 0x7fc7a81f73e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7a815de70 0x7fc7a81f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 --2- 172.21.15.28:0/3420768761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7a80cf780 0x7fc7a80cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 >> 172.21.15.28:0/3420768761 conn(0x7fc7a80d3fb0 msgr2=0x7fc7a80b8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 shutdown_connections 2024-09-10T20:38:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.476+0000 7fc7ad32c640 1 -- 172.21.15.28:0/3420768761 wait complete. 2024-09-10T20:38:49.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-09-10T20:38:49.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 -- 172.21.15.28:0/1086285091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400b4340 msgr2=0x7fcd400d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1086285091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400b4340 0x7fcd400d0090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcd400d0c80 tx=0x7fcd2802a530 comp rx=0 tx=0).stop 2024-09-10T20:38:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 -- 172.21.15.28:0/1086285091 shutdown_connections 2024-09-10T20:38:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1086285091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400b4340 0x7fcd400d0090 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1086285091 >> v2:172.21.15.159:3300/0 conn(0x7fcd400cd3d0 0x7fcd400cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 -- 172.21.15.28:0/1086285091 >> 172.21.15.28:0/1086285091 conn(0x7fcd4006cf00 msgr2=0x7fcd4006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 -- 172.21.15.28:0/1086285091 shutdown_connections 2024-09-10T20:38:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.594+0000 7fcd462e6640 1 -- 172.21.15.28:0/1086285091 wait complete. 2024-09-10T20:38:49.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.596+0000 7fcd462e6640 1 Processor -- start 2024-09-10T20:38:49.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.596+0000 7fcd462e6640 1 -- start start 2024-09-10T20:38:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd462e6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd400b4340 0x7fcd400ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd462e6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd400b47d0 0x7fcd4014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd462e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400cd3d0 0x7fcd400739e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd462e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd400d0960 con 0x7fcd400cd3d0 2024-09-10T20:38:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd462e6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcd400d07e0 con 0x7fcd400b47d0 2024-09-10T20:38:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd462e6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcd400b3590 con 0x7fcd400b4340 2024-09-10T20:38:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400cd3d0 0x7fcd400739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400cd3d0 0x7fcd400739e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44434/0 (socket says 172.21.15.28:44434) 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 -- 172.21.15.28:0/1293303733 learned_addr learned my addr 172.21.15.28:0/1293303733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd3f7fe640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd400b47d0 0x7fcd4014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd3ffff640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd400b4340 0x7fcd400ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 -- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd400b4340 msgr2=0x7fcd400ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd400b4340 0x7fcd400ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 -- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd400b47d0 msgr2=0x7fcd4014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd400b47d0 0x7fcd4014dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 -- 172.21.15.28:0/1293303733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd400ca750 con 0x7fcd400cd3d0 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd3ffff640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd400b4340 0x7fcd400ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd44883640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400cd3d0 0x7fcd400739e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd3800cbf0 tx=0x7fcd3800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd3cff9640 1 -- 172.21.15.28:0/1293303733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38016020 con 0x7fcd400cd3d0 2024-09-10T20:38:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd3cff9640 1 -- 172.21.15.28:0/1293303733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd3800acf0 con 0x7fcd400cd3d0 2024-09-10T20:38:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd40072940 con 0x7fcd400cd3d0 2024-09-10T20:38:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.597+0000 7fcd3cff9640 1 -- 172.21.15.28:0/1293303733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38012680 con 0x7fcd400cd3d0 2024-09-10T20:38:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.598+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd40072f00 con 0x7fcd400cd3d0 2024-09-10T20:38:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.598+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd400b3ab0 con 0x7fcd400cd3d0 2024-09-10T20:38:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.599+0000 7fcd3cff9640 1 -- 172.21.15.28:0/1293303733 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fcd38012820 con 0x7fcd400cd3d0 2024-09-10T20:38:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.600+0000 7fcd3cff9640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcd20077f70 0x7fcd2007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.600+0000 7fcd3cff9640 1 -- 172.21.15.28:0/1293303733 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7fcd38099ff0 con 0x7fcd400cd3d0 2024-09-10T20:38:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.600+0000 7fcd3ffff640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcd20077f70 0x7fcd2007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.601+0000 7fcd3ffff640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcd20077f70 0x7fcd2007a430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000ab40 tx=0x7fcd3000a520 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.604+0000 7fcd3cff9640 1 -- 172.21.15.28:0/1293303733 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcd38020030 con 0x7fcd400cd3d0 2024-09-10T20:38:49.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.858+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fcd4015bc20 con 0x7fcd400cd3d0 2024-09-10T20:38:49.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.860+0000 7fcd3cff9640 1 -- 172.21.15.28:0/1293303733 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v22) ==== 90+0+0 (secure 0 0 0) 0x7fcd38062e70 con 0x7fcd400cd3d0 2024-09-10T20:38:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcd20077f70 msgr2=0x7fcd2007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcd20077f70 0x7fcd2007a430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000ab40 tx=0x7fcd3000a520 comp rx=0 tx=0).stop 2024-09-10T20:38:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400cd3d0 msgr2=0x7fcd400739e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400cd3d0 0x7fcd400739e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd3800cbf0 tx=0x7fcd3800a430 comp rx=0 tx=0).stop 2024-09-10T20:38:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 shutdown_connections 2024-09-10T20:38:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcd20077f70 0x7fcd2007a430 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd400cd3d0 0x7fcd400739e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd400b47d0 0x7fcd4014dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 --2- 172.21.15.28:0/1293303733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd400b4340 0x7fcd400ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 >> 172.21.15.28:0/1293303733 conn(0x7fcd4006cf00 msgr2=0x7fcd4009b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:49.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 shutdown_connections 2024-09-10T20:38:49.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.862+0000 7fcd462e6640 1 -- 172.21.15.28:0/1293303733 wait complete. 2024-09-10T20:38:49.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-09-10T20:38:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.979+0000 7f7e10d00640 1 -- 172.21.15.28:0/214001263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c162260 msgr2=0x7f7e0c0b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.979+0000 7f7e10d00640 1 --2- 172.21.15.28:0/214001263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c162260 0x7f7e0c0b6e00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0d1d00 tx=0x7f7e0402a780 comp rx=0 tx=0).stop 2024-09-10T20:38:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.979+0000 7f7e10d00640 1 -- 172.21.15.28:0/214001263 shutdown_connections 2024-09-10T20:38:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.979+0000 7f7e10d00640 1 --2- 172.21.15.28:0/214001263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c162260 0x7f7e0c0b6e00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.979+0000 7f7e10d00640 1 -- 172.21.15.28:0/214001263 >> 172.21.15.28:0/214001263 conn(0x7f7e0c0d3fb0 msgr2=0x7f7e0c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.979+0000 7f7e10d00640 1 -- 172.21.15.28:0/214001263 shutdown_connections 2024-09-10T20:38:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.980+0000 7f7e10d00640 1 -- 172.21.15.28:0/214001263 wait complete. 2024-09-10T20:38:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.981+0000 7f7e10d00640 1 Processor -- start 2024-09-10T20:38:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.981+0000 7f7e10d00640 1 -- start start 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e10d00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0d2b90 0x7f7e0c14cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e10d00640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e0c15de70 0x7f7e0c15f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e10d00640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e0c162260 0x7f7e0c15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e10d00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c0b7340 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e10d00640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c15e460 con 0x7f7e0c15de70 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e10d00640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0c0b74c0 con 0x7f7e0c162260 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0d2b90 0x7f7e0c14cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0d2b90 0x7f7e0c14cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44442/0 (socket says 172.21.15.28:44442) 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 -- 172.21.15.28:0/1553072685 learned_addr learned my addr 172.21.15.28:0/1553072685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e03fff640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e0c15de70 0x7f7e0c15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0ad9d640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e0c162260 0x7f7e0c15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 -- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e0c162260 msgr2=0x7f7e0c15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e0c162260 0x7f7e0c15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 -- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e0c15de70 msgr2=0x7f7e0c15f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e0c15de70 0x7f7e0c15f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 -- 172.21.15.28:0/1553072685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c1f8eb0 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.982+0000 7f7e0a59c640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0d2b90 0x7f7e0c14cd60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c460 tx=0x7f7dfc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.983+0000 7f7e02ffd640 1 -- 172.21.15.28:0/1553072685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc011790 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.983+0000 7f7e02ffd640 1 -- 172.21.15.28:0/1553072685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7dfc017070 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.983+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0c160190 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.983+0000 7f7e02ffd640 1 -- 172.21.15.28:0/1553072685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc011d00 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.983+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e0c160a30 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.984+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e0c160580 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.984+0000 7f7e02ffd640 1 -- 172.21.15.28:0/1553072685 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7dfc01c460 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.985+0000 7f7e02ffd640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7dec077f70 0x7f7dec07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.985+0000 7f7e02ffd640 1 -- 172.21.15.28:0/1553072685 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f7dfc099710 con 0x7f7e0c0d2b90 2024-09-10T20:38:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.985+0000 7f7e03fff640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7dec077f70 0x7f7dec07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.985+0000 7f7e03fff640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7dec077f70 0x7f7dec07a430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0ce390 tx=0x7f7df4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:49.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:49.989+0000 7f7e02ffd640 1 -- 172.21.15.28:0/1553072685 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7dfc061f10 con 0x7f7e0c0d2b90 2024-09-10T20:38:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.242+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f7e0c0d38e0 con 0x7f7e0c0d2b90 2024-09-10T20:38:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.244+0000 7f7e02ffd640 1 -- 172.21.15.28:0/1553072685 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v23) ==== 97+0+0 (secure 0 0 0) 0x7f7e0c0d38e0 con 0x7f7e0c0d2b90 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7dec077f70 msgr2=0x7f7dec07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7dec077f70 0x7f7dec07a430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0ce390 tx=0x7f7df4009040 comp rx=0 tx=0).stop 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0d2b90 msgr2=0x7f7e0c14cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0d2b90 0x7f7e0c14cd60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c460 tx=0x7f7dfc00c920 comp rx=0 tx=0).stop 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 shutdown_connections 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7dec077f70 0x7f7dec07a430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e0c162260 0x7f7e0c15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e0c15de70 0x7f7e0c15f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 --2- 172.21.15.28:0/1553072685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0d2b90 0x7f7e0c14cd60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 >> 172.21.15.28:0/1553072685 conn(0x7f7e0c0d3fb0 msgr2=0x7f7e0c0b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 shutdown_connections 2024-09-10T20:38:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.247+0000 7f7e10d00640 1 -- 172.21.15.28:0/1553072685 wait complete. 2024-09-10T20:38:50.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-09-10T20:38:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.364+0000 7f566953d640 1 -- 172.21.15.28:0/2574438624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 msgr2=0x7f56640b5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.364+0000 7f566953d640 1 --2- 172.21.15.28:0/2574438624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640b5d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f565c005fd0 tx=0x7f565c02a940 comp rx=0 tx=0).stop 2024-09-10T20:38:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.364+0000 7f566953d640 1 -- 172.21.15.28:0/2574438624 shutdown_connections 2024-09-10T20:38:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.364+0000 7f566953d640 1 --2- 172.21.15.28:0/2574438624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640b5d20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.364+0000 7f566953d640 1 -- 172.21.15.28:0/2574438624 >> 172.21.15.28:0/2574438624 conn(0x7f56640d1e40 msgr2=0x7f56640d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.364+0000 7f566953d640 1 -- 172.21.15.28:0/2574438624 shutdown_connections 2024-09-10T20:38:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.364+0000 7f566953d640 1 -- 172.21.15.28:0/2574438624 wait complete. 2024-09-10T20:38:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.365+0000 7f566953d640 1 Processor -- start 2024-09-10T20:38:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.365+0000 7f566953d640 1 -- start start 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f566953d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56640d05d0 0x7f566406f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f566953d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f566953d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566415bf30 0x7f56640cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f566953d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f56640cd600 con 0x7f56640d2960 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f566953d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f56640d0d50 con 0x7f566415bf30 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f566953d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f56640b6260 con 0x7f56640d05d0 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640c8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56502/0 (socket says 172.21.15.28:56502) 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 -- 172.21.15.28:0/4158466188 learned_addr learned my addr 172.21.15.28:0/4158466188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56637fe640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566415bf30 0x7f56640cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f5662ffd640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56640d05d0 0x7f566406f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 -- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56640d05d0 msgr2=0x7f566406f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56640d05d0 0x7f566406f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 -- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566415bf30 msgr2=0x7f56640cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566415bf30 0x7f56640cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 -- 172.21.15.28:0/4158466188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566406e6a0 con 0x7f56640d2960 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56637fe640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566415bf30 0x7f56640cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.366+0000 7f56627fc640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640c8820 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f565800c500 tx=0x7f565800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.367+0000 7f564b7fe640 1 -- 172.21.15.28:0/4158466188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580052d0 con 0x7f56640d2960 2024-09-10T20:38:50.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.367+0000 7f564b7fe640 1 -- 172.21.15.28:0/4158466188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5658005470 con 0x7f56640d2960 2024-09-10T20:38:50.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.367+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56641590f0 con 0x7f56640d2960 2024-09-10T20:38:50.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.367+0000 7f564b7fe640 1 -- 172.21.15.28:0/4158466188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5658012870 con 0x7f56640d2960 2024-09-10T20:38:50.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.367+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641593d0 con 0x7f56640d2960 2024-09-10T20:38:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.368+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56640b6710 con 0x7f56640d2960 2024-09-10T20:38:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.368+0000 7f564b7fe640 1 -- 172.21.15.28:0/4158466188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f56580035f0 con 0x7f56640d2960 2024-09-10T20:38:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.369+0000 7f564b7fe640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5650077f70 0x7f565007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.369+0000 7f564b7fe640 1 -- 172.21.15.28:0/4158466188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f56580995f0 con 0x7f56640d2960 2024-09-10T20:38:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.369+0000 7f5662ffd640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5650077f70 0x7f565007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.370+0000 7f5662ffd640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5650077f70 0x7f565007a430 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5654000f60 tx=0x7f5654009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:50.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.373+0000 7f564b7fe640 1 -- 172.21.15.28:0/4158466188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5658062450 con 0x7f56640d2960 2024-09-10T20:38:50.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.630+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f56640b7640 con 0x7f56640d2960 2024-09-10T20:38:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.633+0000 7f564b7fe640 1 -- 172.21.15.28:0/4158466188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v24) ==== 90+0+0 (secure 0 0 0) 0x7f5658061d50 con 0x7f56640d2960 2024-09-10T20:38:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.634+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5650077f70 msgr2=0x7f565007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.634+0000 7f566953d640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5650077f70 0x7f565007a430 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5654000f60 tx=0x7f5654009040 comp rx=0 tx=0).stop 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 msgr2=0x7f56640c8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640c8820 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f565800c500 tx=0x7f565800c9c0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 shutdown_connections 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5650077f70 0x7f565007a430 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566415bf30 0x7f56640cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56640d2960 0x7f56640c8820 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 --2- 172.21.15.28:0/4158466188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f56640d05d0 0x7f566406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 >> 172.21.15.28:0/4158466188 conn(0x7f56640d1e40 msgr2=0x7f56641e84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 shutdown_connections 2024-09-10T20:38:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.635+0000 7f566953d640 1 -- 172.21.15.28:0/4158466188 wait complete. 2024-09-10T20:38:50.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:38:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 -- 172.21.15.28:0/1847110898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d015bf30 msgr2=0x7f29d00d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 --2- 172.21.15.28:0/1847110898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d015bf30 0x7f29d00d2960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f29d00d1a20 tx=0x7f29cc02a530 comp rx=0 tx=0).stop 2024-09-10T20:38:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 -- 172.21.15.28:0/1847110898 shutdown_connections 2024-09-10T20:38:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 --2- 172.21.15.28:0/1847110898 >> v2:172.21.15.159:3300/0 conn(0x7f29d00cfec0 0x7f29d00d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 --2- 172.21.15.28:0/1847110898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d015bf30 0x7f29d00d2960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 -- 172.21.15.28:0/1847110898 >> 172.21.15.28:0/1847110898 conn(0x7f29d00d1e40 msgr2=0x7f29d00d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 -- 172.21.15.28:0/1847110898 shutdown_connections 2024-09-10T20:38:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.751+0000 7f29d7623640 1 -- 172.21.15.28:0/1847110898 wait complete. 2024-09-10T20:38:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.752+0000 7f29d7623640 1 Processor -- start 2024-09-10T20:38:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.752+0000 7f29d7623640 1 -- start start 2024-09-10T20:38:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d7623640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d00ce5f0 0x7f29d01eecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d7623640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00cfec0 0x7f29d015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d7623640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d015bf30 0x7f29d0153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d7623640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29d00d0cc0 con 0x7f29d00cfec0 2024-09-10T20:38:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d7623640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f29d00d0b40 con 0x7f29d015bf30 2024-09-10T20:38:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d7623640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f29d00b6620 con 0x7f29d00ce5f0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d4bbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00cfec0 0x7f29d015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d4bbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00cfec0 0x7f29d015dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56524/0 (socket says 172.21.15.28:56524) 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d4bbe640 1 -- 172.21.15.28:0/3655095589 learned_addr learned my addr 172.21.15.28:0/3655095589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d53bf640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d00ce5f0 0x7f29d01eecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d5bc0640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d015bf30 0x7f29d0153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d4bbe640 1 -- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d00ce5f0 msgr2=0x7f29d01eecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d4bbe640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d00ce5f0 0x7f29d01eecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d4bbe640 1 -- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d015bf30 msgr2=0x7f29d0153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.753+0000 7f29d4bbe640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d015bf30 0x7f29d0153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.754+0000 7f29d4bbe640 1 -- 172.21.15.28:0/3655095589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d006dd50 con 0x7f29d00cfec0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.754+0000 7f29d4bbe640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00cfec0 0x7f29d015dc60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f29c000d950 tx=0x7f29c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.754+0000 7f29c5ffb640 1 -- 172.21.15.28:0/3655095589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0017070 con 0x7f29d00cfec0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.754+0000 7f29c5ffb640 1 -- 172.21.15.28:0/3655095589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29c0009d20 con 0x7f29d00cfec0 2024-09-10T20:38:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.754+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d006e030 con 0x7f29d00cfec0 2024-09-10T20:38:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.754+0000 7f29c5ffb640 1 -- 172.21.15.28:0/3655095589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c00106c0 con 0x7f29d00cfec0 2024-09-10T20:38:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.754+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d006e5f0 con 0x7f29d00cfec0 2024-09-10T20:38:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.755+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d00d0330 con 0x7f29d00cfec0 2024-09-10T20:38:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.756+0000 7f29c5ffb640 1 -- 172.21.15.28:0/3655095589 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f29c0010860 con 0x7f29d00cfec0 2024-09-10T20:38:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.756+0000 7f29c5ffb640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4077f70 0x7f29b407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:50.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.756+0000 7f29c5ffb640 1 -- 172.21.15.28:0/3655095589 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f29c0099e60 con 0x7f29d00cfec0 2024-09-10T20:38:50.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.757+0000 7f29d53bf640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4077f70 0x7f29b407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:50.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.757+0000 7f29d53bf640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4077f70 0x7f29b407a430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f29cc034000 tx=0x7f29cc004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:50.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.761+0000 7f29c5ffb640 1 -- 172.21.15.28:0/3655095589 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f29c0062ca0 con 0x7f29d00cfec0 2024-09-10T20:38:50.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.947+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f29d015a520 con 0x7f29d00cfec0 2024-09-10T20:38:50.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.947+0000 7f29c5ffb640 1 -- 172.21.15.28:0/3655095589 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) ==== 84+0+65 (secure 0 0 0) 0x7f29c00625a0 con 0x7f29d00cfec0 2024-09-10T20:38:50.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:50.953 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:38:50.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:38:50.953 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:38:50.953 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:38:50.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:38:50.953 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.949+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4077f70 msgr2=0x7f29b407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.949+0000 7f29d7623640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4077f70 0x7f29b407a430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f29cc034000 tx=0x7f29cc004730 comp rx=0 tx=0).stop 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.949+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00cfec0 msgr2=0x7f29d015dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.949+0000 7f29d7623640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00cfec0 0x7f29d015dc60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f29c000d950 tx=0x7f29c000de10 comp rx=0 tx=0).stop 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 shutdown_connections 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4077f70 0x7f29b407a430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d015bf30 0x7f29d0153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00cfec0 0x7f29d015dc60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 --2- 172.21.15.28:0/3655095589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d00ce5f0 0x7f29d01eecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 >> 172.21.15.28:0/3655095589 conn(0x7f29d00d1e40 msgr2=0x7f29d00d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 shutdown_connections 2024-09-10T20:38:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:50.950+0000 7f29d7623640 1 -- 172.21.15.28:0/3655095589 wait complete. 2024-09-10T20:38:50.966 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:38:50.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:38:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.071+0000 7f61a2ef7640 1 -- 172.21.15.28:0/2501804512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61880051c0 msgr2=0x7f618800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.071+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/2501804512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61880051c0 0x7f618800b9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6194009b50 tx=0x7f6188005830 comp rx=0 tx=0).stop 2024-09-10T20:38:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.071+0000 7f61a2ef7640 1 -- 172.21.15.28:0/2501804512 shutdown_connections 2024-09-10T20:38:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.071+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/2501804512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61880051c0 0x7f618800b9c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.071+0000 7f61a2ef7640 1 -- 172.21.15.28:0/2501804512 >> 172.21.15.28:0/2501804512 conn(0x7f619c0d1da0 msgr2=0x7f619c0d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.071+0000 7f61a2ef7640 1 -- 172.21.15.28:0/2501804512 shutdown_connections 2024-09-10T20:38:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.071+0000 7f61a2ef7640 1 -- 172.21.15.28:0/2501804512 wait complete. 2024-09-10T20:38:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.072+0000 7f61a2ef7640 1 Processor -- start 2024-09-10T20:38:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.072+0000 7f61a2ef7640 1 -- start start 2024-09-10T20:38:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a2ef7640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61880051c0 0x7f619c14cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a2ef7640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f619c150660 0x7f619c0b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a2ef7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c14e000 0x7f619c14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a2ef7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f619c0ced10 con 0x7f619c14e000 2024-09-10T20:38:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a2ef7640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f619c0ceb90 con 0x7f61880051c0 2024-09-10T20:38:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a2ef7640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f619c0cee90 con 0x7f619c150660 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a1494640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c14e000 0x7f619c14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a1494640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c14e000 0x7f619c14e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56546/0 (socket says 172.21.15.28:56546) 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a1494640 1 -- 172.21.15.28:0/3876720053 learned_addr learned my addr 172.21.15.28:0/3876720053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f61a0c93640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61880051c0 0x7f619c14cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.073+0000 7f619bfff640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f619c150660 0x7f619c0b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a1494640 1 -- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f619c150660 msgr2=0x7f619c0b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a1494640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f619c150660 0x7f619c0b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a1494640 1 -- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61880051c0 msgr2=0x7f619c14cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a1494640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61880051c0 0x7f619c14cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a1494640 1 -- 172.21.15.28:0/3876720053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c150e50 con 0x7f619c14e000 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a0c93640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61880051c0 0x7f619c14cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a1494640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c14e000 0x7f619c14e4b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6194009c70 tx=0x7f619402de60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61997fa640 1 -- 172.21.15.28:0/3876720053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619402d070 con 0x7f619c14e000 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61997fa640 1 -- 172.21.15.28:0/3876720053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6194041070 con 0x7f619c14e000 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c14d230 con 0x7f619c14e000 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61997fa640 1 -- 172.21.15.28:0/3876720053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619402d450 con 0x7f619c14e000 2024-09-10T20:38:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.074+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c14d7f0 con 0x7f619c14e000 2024-09-10T20:38:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.075+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f619c06d750 con 0x7f619c14e000 2024-09-10T20:38:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.076+0000 7f61997fa640 1 -- 172.21.15.28:0/3876720053 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f619402b800 con 0x7f619c14e000 2024-09-10T20:38:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.076+0000 7f61997fa640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f617c077fa0 0x7f617c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.076+0000 7f61997fa640 1 -- 172.21.15.28:0/3876720053 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f61940b9ef0 con 0x7f619c14e000 2024-09-10T20:38:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.076+0000 7f61a0c93640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f617c077fa0 0x7f617c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.077+0000 7f61a0c93640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f617c077fa0 0x7f617c07a460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f618c0018a0 tx=0x7f618c001cb0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.080+0000 7f61997fa640 1 -- 172.21.15.28:0/3876720053 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6194082dd0 con 0x7f619c14e000 2024-09-10T20:38:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.266+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f619c0ce590 con 0x7f619c14e000 2024-09-10T20:38:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.917+0000 7f61997fa640 1 -- 172.21.15.28:0/3876720053 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v29) ==== 138+0+0 (secure 0 0 0) 0x7f61940826d0 con 0x7f619c14e000 2024-09-10T20:38:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f617c077fa0 msgr2=0x7f617c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f617c077fa0 0x7f617c07a460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f618c0018a0 tx=0x7f618c001cb0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c14e000 msgr2=0x7f619c14e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c14e000 0x7f619c14e4b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6194009c70 tx=0x7f619402de60 comp rx=0 tx=0).stop 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 shutdown_connections 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f617c077fa0 0x7f617c07a460 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c14e000 0x7f619c14e4b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f619c150660 0x7f619c0b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 --2- 172.21.15.28:0/3876720053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61880051c0 0x7f619c14cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 >> 172.21.15.28:0/3876720053 conn(0x7f619c0d1da0 msgr2=0x7f619c0b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 shutdown_connections 2024-09-10T20:38:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:51.921+0000 7f61a2ef7640 1 -- 172.21.15.28:0/3876720053 wait complete. 2024-09-10T20:38:51.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:38:52.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 -- 172.21.15.28:0/3448911737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2f40 msgr2=0x7f21640b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:52.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 --2- 172.21.15.28:0/3448911737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2f40 0x7f21640b1ab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2158009830 tx=0x7f215802a510 comp rx=0 tx=0).stop 2024-09-10T20:38:52.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 -- 172.21.15.28:0/3448911737 shutdown_connections 2024-09-10T20:38:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 --2- 172.21.15.28:0/3448911737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2f40 0x7f21640b1ab0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 --2- 172.21.15.28:0/3448911737 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f216415f4d0 0x7f21640b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 -- 172.21.15.28:0/3448911737 >> 172.21.15.28:0/3448911737 conn(0x7f21640d1ef0 msgr2=0x7f21640d2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 -- 172.21.15.28:0/3448911737 shutdown_connections 2024-09-10T20:38:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.057+0000 7f216a90e640 1 -- 172.21.15.28:0/3448911737 wait complete. 2024-09-10T20:38:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.059+0000 7f216a90e640 1 Processor -- start 2024-09-10T20:38:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.059+0000 7f216a90e640 1 -- start start 2024-09-10T20:38:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2ab0 0x7f216415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21640b2f40 0x7f21641f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f216415f4d0 0x7f21641f8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21640d0ae0 con 0x7f21640b2ab0 2024-09-10T20:38:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f21640d0960 con 0x7f216415f4d0 2024-09-10T20:38:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f21641f93b0 con 0x7f21640b2f40 2024-09-10T20:38:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2ab0 0x7f216415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2ab0 0x7f216415b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56576/0 (socket says 172.21.15.28:56576) 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 -- 172.21.15.28:0/1732351619 learned_addr learned my addr 172.21.15.28:0/1732351619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f21637fe640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21640b2f40 0x7f21641f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2168eab640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f216415f4d0 0x7f21641f8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 -- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21640b2f40 msgr2=0x7f21641f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21640b2f40 0x7f21641f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 -- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f216415f4d0 msgr2=0x7f21641f8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f216415f4d0 0x7f21641f8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 -- 172.21.15.28:0/1732351619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2164071bc0 con 0x7f21640b2ab0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2163fff640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2ab0 0x7f216415b1f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f215400d740 tx=0x7f215400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2160ff9640 1 -- 172.21.15.28:0/1732351619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154017070 con 0x7f21640b2ab0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2160ff9640 1 -- 172.21.15.28:0/1732351619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2154009cc0 con 0x7f21640b2ab0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2164071ea0 con 0x7f21640b2ab0 2024-09-10T20:38:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f2160ff9640 1 -- 172.21.15.28:0/1732351619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21540106c0 con 0x7f21640b2ab0 2024-09-10T20:38:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.060+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21641f54a0 con 0x7f21640b2ab0 2024-09-10T20:38:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.062+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21640b1570 con 0x7f21640b2ab0 2024-09-10T20:38:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.063+0000 7f2160ff9640 1 -- 172.21.15.28:0/1732351619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2154010860 con 0x7f21640b2ab0 2024-09-10T20:38:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.063+0000 7f2160ff9640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2150077f70 0x7f215007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.063+0000 7f2160ff9640 1 -- 172.21.15.28:0/1732351619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5227+0+0 (secure 0 0 0) 0x7f2154099ff0 con 0x7f21640b2ab0 2024-09-10T20:38:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.064+0000 7f21637fe640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2150077f70 0x7f215007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.064+0000 7f21637fe640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2150077f70 0x7f215007a430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2158034000 tx=0x7f2158004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.068+0000 7f2160ff9640 1 -- 172.21.15.28:0/1732351619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f21540625e0 con 0x7f21640b2ab0 2024-09-10T20:38:52.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.254+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f21640cf6b0 con 0x7f21640b2ab0 2024-09-10T20:38:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.921+0000 7f2160ff9640 1 -- 172.21.15.28:0/1732351619 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v30) ==== 170+0+0 (secure 0 0 0) 0x7f2154067010 con 0x7f21640b2ab0 2024-09-10T20:38:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.924+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2150077f70 msgr2=0x7f215007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.924+0000 7f216a90e640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2150077f70 0x7f215007a430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2158034000 tx=0x7f2158004730 comp rx=0 tx=0).stop 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.924+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2ab0 msgr2=0x7f216415b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.924+0000 7f216a90e640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2ab0 0x7f216415b1f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f215400d740 tx=0x7f215400dc00 comp rx=0 tx=0).stop 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.925+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 shutdown_connections 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.925+0000 7f216a90e640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2150077f70 0x7f215007a430 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.925+0000 7f216a90e640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f216415f4d0 0x7f21641f8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.925+0000 7f216a90e640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f21640b2f40 0x7f21641f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.925+0000 7f216a90e640 1 --2- 172.21.15.28:0/1732351619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21640b2ab0 0x7f216415b1f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.925+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 >> 172.21.15.28:0/1732351619 conn(0x7f21640d1ef0 msgr2=0x7f21640cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.926+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 shutdown_connections 2024-09-10T20:38:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:52.926+0000 7f216a90e640 1 -- 172.21.15.28:0/1732351619 wait complete. 2024-09-10T20:38:52.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:38:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.069+0000 7fa1d7060640 1 -- 172.21.15.28:0/3803115739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1bc004800 msgr2=0x7fa1bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.069+0000 7fa1d7060640 1 --2- 172.21.15.28:0/3803115739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1bc004800 0x7fa1bc004c80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa1d015b1f0 tx=0x7fa1bc005830 comp rx=0 tx=0).stop 2024-09-10T20:38:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.069+0000 7fa1d55fd640 1 -- 172.21.15.28:0/3803115739 reap_dead start 2024-09-10T20:38:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.069+0000 7fa1d7060640 1 -- 172.21.15.28:0/3803115739 shutdown_connections 2024-09-10T20:38:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.069+0000 7fa1d7060640 1 -- 172.21.15.28:0/3803115739 >> 172.21.15.28:0/3803115739 conn(0x7fa1d006cf00 msgr2=0x7fa1d006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.069+0000 7fa1d7060640 1 -- 172.21.15.28:0/3803115739 shutdown_connections 2024-09-10T20:38:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.069+0000 7fa1d7060640 1 -- 172.21.15.28:0/3803115739 wait complete. 2024-09-10T20:38:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.070+0000 7fa1d7060640 1 Processor -- start 2024-09-10T20:38:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.070+0000 7fa1d7060640 1 -- start start 2024-09-10T20:38:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d7060640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1bc004800 0x7fa1d01551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d7060640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0154c90 0x7fa1d014cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d7060640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1d00c7000 0x7fa1d00c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d7060640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d006fde0 con 0x7fa1d0154c90 2024-09-10T20:38:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d7060640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d00cb4f0 con 0x7fa1bc004800 2024-09-10T20:38:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d7060640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa1d006ff60 con 0x7fa1d00c7000 2024-09-10T20:38:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0154c90 0x7fa1d014cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0154c90 0x7fa1d014cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56586/0 (socket says 172.21.15.28:56586) 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1cffff640 1 -- 172.21.15.28:0/2635968934 learned_addr learned my addr 172.21.15.28:0/2635968934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d4dfc640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1bc004800 0x7fa1d01551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.071+0000 7fa1d55fd640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1d00c7000 0x7fa1d00c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cffff640 1 -- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1d00c7000 msgr2=0x7fa1d00c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cffff640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1d00c7000 0x7fa1d00c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cffff640 1 -- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1bc004800 msgr2=0x7fa1d01551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cffff640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1bc004800 0x7fa1d01551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cffff640 1 -- 172.21.15.28:0/2635968934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d00c5d70 con 0x7fa1d0154c90 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cffff640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0154c90 0x7fa1d014cb50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8036040 tx=0x7fa1b8033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cd7fa640 1 -- 172.21.15.28:0/2635968934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8043070 con 0x7fa1d0154c90 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cd7fa640 1 -- 172.21.15.28:0/2635968934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1b8004140 con 0x7fa1d0154c90 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d00c6050 con 0x7fa1d0154c90 2024-09-10T20:38:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1cd7fa640 1 -- 172.21.15.28:0/2635968934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8038040 con 0x7fa1d0154c90 2024-09-10T20:38:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.072+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d0157ff0 con 0x7fa1d0154c90 2024-09-10T20:38:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.073+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d00b4350 con 0x7fa1d0154c90 2024-09-10T20:38:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.074+0000 7fa1cd7fa640 1 -- 172.21.15.28:0/2635968934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa1b803b030 con 0x7fa1d0154c90 2024-09-10T20:38:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.074+0000 7fa1cd7fa640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa1b0077f70 0x7fa1b007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.075+0000 7fa1cd7fa640 1 -- 172.21.15.28:0/2635968934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5609+0+0 (secure 0 0 0) 0x7fa1b80ba140 con 0x7fa1d0154c90 2024-09-10T20:38:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.075+0000 7fa1d4dfc640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa1b0077f70 0x7fa1b007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.075+0000 7fa1d4dfc640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa1b0077f70 0x7fa1b007a430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c0009500 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:53.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.079+0000 7fa1cd7fa640 1 -- 172.21.15.28:0/2635968934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa1d00b4350 con 0x7fa1d0154c90 2024-09-10T20:38:53.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.301+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa1d015e7a0 con 0x7fa1d0154c90 2024-09-10T20:38:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.933+0000 7fa1cd7fa640 1 -- 172.21.15.28:0/2635968934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7fa1b80826e0 con 0x7fa1d0154c90 2024-09-10T20:38:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 4 and data pool 5 2024-09-10T20:38:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.948+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa1b0077f70 msgr2=0x7fa1b007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.948+0000 7fa1d7060640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa1b0077f70 0x7fa1b007a430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c0009500 comp rx=0 tx=0).stop 2024-09-10T20:38:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.948+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0154c90 msgr2=0x7fa1d014cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.948+0000 7fa1d7060640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0154c90 0x7fa1d014cb50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8036040 tx=0x7fa1b8033670 comp rx=0 tx=0).stop 2024-09-10T20:38:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.949+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 shutdown_connections 2024-09-10T20:38:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.949+0000 7fa1d7060640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa1b0077f70 0x7fa1b007a430 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.949+0000 7fa1d7060640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1d00c7000 0x7fa1d00c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.949+0000 7fa1d7060640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0154c90 0x7fa1d014cb50 secure :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8036040 tx=0x7fa1b8033670 comp rx=0 tx=0).stop 2024-09-10T20:38:53.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.949+0000 7fa1d7060640 1 --2- 172.21.15.28:0/2635968934 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1bc004800 0x7fa1d01551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:53.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.949+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 >> 172.21.15.28:0/2635968934 conn(0x7fa1d006cf00 msgr2=0x7fa1d00b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.955+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 shutdown_connections 2024-09-10T20:38:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:53.955+0000 7fa1d7060640 1 -- 172.21.15.28:0/2635968934 wait complete. 2024-09-10T20:38:53.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.084+0000 7f2f3b35d640 1 -- 172.21.15.28:0/2637531376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3415de70 msgr2=0x7f2f340d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.084+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/2637531376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3415de70 0x7f2f340d2b90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2f24005fd0 tx=0x7f2f2402a8b0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.084+0000 7f2f3b35d640 1 -- 172.21.15.28:0/2637531376 shutdown_connections 2024-09-10T20:38:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.084+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/2637531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f340cf780 0x7f2f340cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.084+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/2637531376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3415de70 0x7f2f340d2b90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.084+0000 7f2f3b35d640 1 -- 172.21.15.28:0/2637531376 >> 172.21.15.28:0/2637531376 conn(0x7f2f340d3fb0 msgr2=0x7f2f340d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.085+0000 7f2f3b35d640 1 -- 172.21.15.28:0/2637531376 shutdown_connections 2024-09-10T20:38:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.085+0000 7f2f3b35d640 1 -- 172.21.15.28:0/2637531376 wait complete. 2024-09-10T20:38:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 Processor -- start 2024-09-10T20:38:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 -- start start 2024-09-10T20:38:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f340cf780 0x7f2f3415fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f3415de70 0x7f2f3414caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f34161b50 0x7f2f341f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f34162350 con 0x7f2f340cf780 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2f341621d0 con 0x7f2f3415de70 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.086+0000 7f2f3b35d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2f341624d0 con 0x7f2f34161b50 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f340cf780 0x7f2f3415fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f340cf780 0x7f2f3415fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56606/0 (socket says 172.21.15.28:56606) 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 -- 172.21.15.28:0/1251626364 learned_addr learned my addr 172.21.15.28:0/1251626364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f398fa640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f34161b50 0x7f2f341f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f388f8640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f3415de70 0x7f2f3414caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 -- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f34161b50 msgr2=0x7f2f341f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f34161b50 0x7f2f341f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 -- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f3415de70 msgr2=0x7f2f3414caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f3415de70 0x7f2f3414caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 -- 172.21.15.28:0/1251626364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f34160400 con 0x7f2f340cf780 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f398fa640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f34161b50 0x7f2f341f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f388f8640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f3415de70 0x7f2f3414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f390f9640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f340cf780 0x7f2f3415fec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2f24000f80 tx=0x7f2f240039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f31ffb640 1 -- 172.21.15.28:0/1251626364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24040070 con 0x7f2f340cf780 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f31ffb640 1 -- 172.21.15.28:0/1251626364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f24004130 con 0x7f2f340cf780 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f341606c0 con 0x7f2f340cf780 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.087+0000 7f2f31ffb640 1 -- 172.21.15.28:0/1251626364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24004580 con 0x7f2f340cf780 2024-09-10T20:38:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.088+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f34160c80 con 0x7f2f340cf780 2024-09-10T20:38:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.089+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f341537e0 con 0x7f2f340cf780 2024-09-10T20:38:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.089+0000 7f2f31ffb640 1 -- 172.21.15.28:0/1251626364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2f2404d030 con 0x7f2f340cf780 2024-09-10T20:38:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.090+0000 7f2f31ffb640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f20077f70 0x7f2f2007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.090+0000 7f2f31ffb640 1 -- 172.21.15.28:0/1251626364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f2f240bb4d0 con 0x7f2f340cf780 2024-09-10T20:38:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.090+0000 7f2f388f8640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f20077f70 0x7f2f2007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.091+0000 7f2f388f8640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f20077f70 0x7f2f2007a430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2f340b0fa0 tx=0x7f2f28005e70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.094+0000 7f2f31ffb640 1 -- 172.21.15.28:0/1251626364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2f24083f30 con 0x7f2f340cf780 2024-09-10T20:38:54.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.279+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f34074670 con 0x7f2f340cf780 2024-09-10T20:38:54.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.280+0000 7f2f31ffb640 1 -- 172.21.15.28:0/1251626364 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+16682 (secure 0 0 0) 0x7f2f24083830 con 0x7f2f340cf780 2024-09-10T20:38:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:54.284 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":31,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:53.302365+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:51.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:52.258677+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f20077f70 msgr2=0x7f2f2007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f20077f70 0x7f2f2007a430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2f340b0fa0 tx=0x7f2f28005e70 comp rx=0 tx=0).stop 2024-09-10T20:38:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f340cf780 msgr2=0x7f2f3415fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f340cf780 0x7f2f3415fec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2f24000f80 tx=0x7f2f240039f0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 shutdown_connections 2024-09-10T20:38:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f20077f70 0x7f2f2007a430 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f34161b50 0x7f2f341f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f3415de70 0x7f2f3414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 --2- 172.21.15.28:0/1251626364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f340cf780 0x7f2f3415fec0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 >> 172.21.15.28:0/1251626364 conn(0x7f2f340d3fb0 msgr2=0x7f2f340b81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 shutdown_connections 2024-09-10T20:38:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.282+0000 7f2f3b35d640 1 -- 172.21.15.28:0/1251626364 wait complete. 2024-09-10T20:38:54.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:54.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 -- 172.21.15.28:0/3177211606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc06d1f0 msgr2=0x7f3efc06ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 --2- 172.21.15.28:0/3177211606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc06d1f0 0x7f3efc06ef30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0b3120 tx=0x7f3eec02a4d0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 -- 172.21.15.28:0/3177211606 shutdown_connections 2024-09-10T20:38:54.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 --2- 172.21.15.28:0/3177211606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc06d1f0 0x7f3efc06ef30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 --2- 172.21.15.28:0/3177211606 >> v2:172.21.15.159:3300/0 conn(0x7f3efc156130 0x7f3efc0caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 -- 172.21.15.28:0/3177211606 >> 172.21.15.28:0/3177211606 conn(0x7f3efc0d0420 msgr2=0x7f3efc0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 -- 172.21.15.28:0/3177211606 shutdown_connections 2024-09-10T20:38:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.398+0000 7f3f01841640 1 -- 172.21.15.28:0/3177211606 wait complete. 2024-09-10T20:38:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.399+0000 7f3f01841640 1 Processor -- start 2024-09-10T20:38:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.400+0000 7f3f01841640 1 -- start start 2024-09-10T20:38:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.400+0000 7f3f01841640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3efc06d1f0 0x7f3efc0c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.400+0000 7f3f01841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc156130 0x7f3efc15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.400+0000 7f3f01841640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3efc156510 0x7f3efc0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.400+0000 7f3f01841640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc073580 con 0x7f3efc156130 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.400+0000 7f3f01841640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc073400 con 0x7f3efc156510 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.400+0000 7f3f01841640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc073700 con 0x7f3efc06d1f0 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc156130 0x7f3efc15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc156130 0x7f3efc15f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56626/0 (socket says 172.21.15.28:56626) 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 -- 172.21.15.28:0/1502802697 learned_addr learned my addr 172.21.15.28:0/1502802697 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efb7fe640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3efc156510 0x7f3efc0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efaffd640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3efc06d1f0 0x7f3efc0c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 -- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3efc06d1f0 msgr2=0x7f3efc0c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3efc06d1f0 0x7f3efc0c72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 -- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3efc156510 msgr2=0x7f3efc0c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3efc156510 0x7f3efc0c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 -- 172.21.15.28:0/1502802697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc0c5490 con 0x7f3efc156130 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efaffd640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3efc06d1f0 0x7f3efc0c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3efa7fc640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc156130 0x7f3efc15f9f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400b3c0 tx=0x7f3ee400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3ef3fff640 1 -- 172.21.15.28:0/1502802697 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4011830 con 0x7f3efc156130 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3ef3fff640 1 -- 172.21.15.28:0/1502802697 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ee4016070 con 0x7f3efc156130 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc0c5770 con 0x7f3efc156130 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3ef3fff640 1 -- 172.21.15.28:0/1502802697 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4011d80 con 0x7f3efc156130 2024-09-10T20:38:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.401+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc0c5d30 con 0x7f3efc156130 2024-09-10T20:38:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.402+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3efc073c10 con 0x7f3efc156130 2024-09-10T20:38:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.403+0000 7f3ef3fff640 1 -- 172.21.15.28:0/1502802697 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3ee401b460 con 0x7f3efc156130 2024-09-10T20:38:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.403+0000 7f3ef3fff640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3edc077f70 0x7f3edc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.403+0000 7f3ef3fff640 1 -- 172.21.15.28:0/1502802697 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f3ee4099a60 con 0x7f3efc156130 2024-09-10T20:38:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.404+0000 7f3efaffd640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3edc077f70 0x7f3edc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.404+0000 7f3efaffd640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3edc077f70 0x7f3edc07a430 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4007eb0 tx=0x7f3ef40073c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:54.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.407+0000 7f3ef3fff640 1 -- 172.21.15.28:0/1502802697 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3ee4062550 con 0x7f3efc156130 2024-09-10T20:38:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.595+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3efc0d11c0 con 0x7f3efc156130 2024-09-10T20:38:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.596+0000 7f3ef3fff640 1 -- 172.21.15.28:0/1502802697 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+16682 (secure 0 0 0) 0x7f3ee4061e50 con 0x7f3efc156130 2024-09-10T20:38:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:54.601 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":31,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:53.302365+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:51.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:52.258677+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3edc077f70 msgr2=0x7f3edc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3edc077f70 0x7f3edc07a430 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4007eb0 tx=0x7f3ef40073c0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc156130 msgr2=0x7f3efc15f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc156130 0x7f3efc15f9f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400b3c0 tx=0x7f3ee400b880 comp rx=0 tx=0).stop 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 shutdown_connections 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3edc077f70 0x7f3edc07a430 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3efc156510 0x7f3efc0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3efc156130 0x7f3efc15f9f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 --2- 172.21.15.28:0/1502802697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3efc06d1f0 0x7f3efc0c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 >> 172.21.15.28:0/1502802697 conn(0x7f3efc0d0420 msgr2=0x7f3efc06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 shutdown_connections 2024-09-10T20:38:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.599+0000 7f3f01841640 1 -- 172.21.15.28:0/1502802697 wait complete. 2024-09-10T20:38:54.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:38:54.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 -- 172.21.15.28:0/1696590306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e300b3580 msgr2=0x7f9e3006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 --2- 172.21.15.28:0/1696590306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e300b3580 0x7f9e3006e400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9e3015d260 tx=0x7f9e2402a560 comp rx=0 tx=0).stop 2024-09-10T20:38:54.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 -- 172.21.15.28:0/1696590306 shutdown_connections 2024-09-10T20:38:54.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 --2- 172.21.15.28:0/1696590306 >> v2:172.21.15.159:3300/0 conn(0x7f9e3006e940 0x7f9e300ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 --2- 172.21.15.28:0/1696590306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e300b3580 0x7f9e3006e400 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 -- 172.21.15.28:0/1696590306 >> 172.21.15.28:0/1696590306 conn(0x7f9e3006d110 msgr2=0x7f9e300b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:54.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 -- 172.21.15.28:0/1696590306 shutdown_connections 2024-09-10T20:38:54.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.718+0000 7f9e35fad640 1 -- 172.21.15.28:0/1696590306 wait complete. 2024-09-10T20:38:54.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.719+0000 7f9e35fad640 1 Processor -- start 2024-09-10T20:38:54.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e35fad640 1 -- start start 2024-09-10T20:38:54.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e35fad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e3006e940 0x7f9e30155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e35fad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e30152e00 0x7f9e301568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e35fad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e30156e30 0x7f9e3015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e35fad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3015f820 con 0x7f9e3006e940 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e35fad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9e300cf5b0 con 0x7f9e30156e30 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e35fad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3015f9a0 con 0x7f9e30152e00 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e3006e940 0x7f9e30155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e3006e940 0x7f9e30155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56648/0 (socket says 172.21.15.28:56648) 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.720+0000 7f9e2f7fe640 1 -- 172.21.15.28:0/996475766 learned_addr learned my addr 172.21.15.28:0/996475766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2ffff640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e30156e30 0x7f9e3015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2effd640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e30152e00 0x7f9e301568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2f7fe640 1 -- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e30152e00 msgr2=0x7f9e301568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2f7fe640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e30152e00 0x7f9e301568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2f7fe640 1 -- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e30156e30 msgr2=0x7f9e3015d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2f7fe640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e30156e30 0x7f9e3015d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2f7fe640 1 -- 172.21.15.28:0/996475766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e301f4ff0 con 0x7f9e3006e940 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e2f7fe640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e3006e940 0x7f9e30155d80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c3d0 tx=0x7f9e2000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e17fff640 1 -- 172.21.15.28:0/996475766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20017070 con 0x7f9e3006e940 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e17fff640 1 -- 172.21.15.28:0/996475766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e200053c0 con 0x7f9e3006e940 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e301f52d0 con 0x7f9e3006e940 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e17fff640 1 -- 172.21.15.28:0/996475766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20003850 con 0x7f9e3006e940 2024-09-10T20:38:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.721+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3015c650 con 0x7f9e3006e940 2024-09-10T20:38:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.722+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3015fec0 con 0x7f9e3006e940 2024-09-10T20:38:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.723+0000 7f9e17fff640 1 -- 172.21.15.28:0/996475766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9e200039f0 con 0x7f9e3006e940 2024-09-10T20:38:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.723+0000 7f9e17fff640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e1c077fa0 0x7f9e1c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.724+0000 7f9e17fff640 1 -- 172.21.15.28:0/996475766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f9e200992c0 con 0x7f9e3006e940 2024-09-10T20:38:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.724+0000 7f9e2effd640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e1c077fa0 0x7f9e1c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.724+0000 7f9e2effd640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e1c077fa0 0x7f9e1c07a460 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9e24033000 tx=0x7f9e24004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:54.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.728+0000 7f9e17fff640 1 -- 172.21.15.28:0/996475766 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9e20061770 con 0x7f9e3006e940 2024-09-10T20:38:54.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:54.951+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9e300b3340 con 0x7f9e3006e940 2024-09-10T20:38:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.954+0000 7f9e17fff640 1 -- 172.21.15.28:0/996475766 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f9e200661a0 con 0x7f9e3006e940 2024-09-10T20:38:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e1c077fa0 msgr2=0x7f9e1c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e1c077fa0 0x7f9e1c07a460 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9e24033000 tx=0x7f9e24004730 comp rx=0 tx=0).stop 2024-09-10T20:38:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e3006e940 msgr2=0x7f9e30155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e3006e940 0x7f9e30155d80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c3d0 tx=0x7f9e2000c890 comp rx=0 tx=0).stop 2024-09-10T20:38:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 shutdown_connections 2024-09-10T20:38:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e1c077fa0 0x7f9e1c07a460 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e30156e30 0x7f9e3015d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e30152e00 0x7f9e301568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 --2- 172.21.15.28:0/996475766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e3006e940 0x7f9e30155d80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 >> 172.21.15.28:0/996475766 conn(0x7f9e3006d110 msgr2=0x7f9e300d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 shutdown_connections 2024-09-10T20:38:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:55.957+0000 7f9e35fad640 1 -- 172.21.15.28:0/996475766 wait complete. 2024-09-10T20:38:55.975 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:38:55.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:38:56.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.087+0000 7f132813d640 1 -- 172.21.15.28:0/2665694474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1308003a50 msgr2=0x7f1308003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:56.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.087+0000 7f132813d640 1 --2- 172.21.15.28:0/2665694474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1308003a50 0x7f1308003e70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1310002230 tx=0x7f1308005830 comp rx=0 tx=0).stop 2024-09-10T20:38:56.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.087+0000 7f132813d640 1 -- 172.21.15.28:0/2665694474 shutdown_connections 2024-09-10T20:38:56.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.087+0000 7f132813d640 1 --2- 172.21.15.28:0/2665694474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1308003a50 0x7f1308003e70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:56.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.087+0000 7f132813d640 1 -- 172.21.15.28:0/2665694474 >> 172.21.15.28:0/2665694474 conn(0x7f13200d1f30 msgr2=0x7f13200d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:56.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.087+0000 7f132813d640 1 -- 172.21.15.28:0/2665694474 shutdown_connections 2024-09-10T20:38:56.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.087+0000 7f132813d640 1 -- 172.21.15.28:0/2665694474 wait complete. 2024-09-10T20:38:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.088+0000 7f132813d640 1 Processor -- start 2024-09-10T20:38:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.088+0000 7f132813d640 1 -- start start 2024-09-10T20:38:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f132813d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1308003a50 0x7f13200b7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f132813d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13200c8080 0x7f13200cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f132813d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1320070a90 0x7f1320070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f132813d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13200ce930 con 0x7f13200c8080 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f132813d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f13200ce7b0 con 0x7f1320070a90 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f132813d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f13200ceab0 con 0x7f1308003a50 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f13256d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13200c8080 0x7f13200cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f13256d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13200c8080 0x7f13200cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56666/0 (socket says 172.21.15.28:56666) 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f13256d8640 1 -- 172.21.15.28:0/3411559276 learned_addr learned my addr 172.21.15.28:0/3411559276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f1325ed9640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1308003a50 0x7f13200b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f13266da640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1320070a90 0x7f1320070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f13256d8640 1 -- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1308003a50 msgr2=0x7f13200b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f13256d8640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1308003a50 0x7f13200b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.089+0000 7f13256d8640 1 -- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1320070a90 msgr2=0x7f1320070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f13256d8640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1320070a90 0x7f1320070f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f13256d8640 1 -- 172.21.15.28:0/3411559276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13200c85f0 con 0x7f13200c8080 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f1325ed9640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1308003a50 0x7f13200b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f13256d8640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13200c8080 0x7f13200cbcc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1314003fb0 tx=0x7f1314001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f131e7fc640 1 -- 172.21.15.28:0/3411559276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1314015070 con 0x7f13200c8080 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f131e7fc640 1 -- 172.21.15.28:0/3411559276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1314011840 con 0x7f13200c8080 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13200c88d0 con 0x7f13200c8080 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f131e7fc640 1 -- 172.21.15.28:0/3411559276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1314011c90 con 0x7f13200c8080 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.090+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13200c9530 con 0x7f13200c8080 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.091+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1320161aa0 con 0x7f13200c8080 2024-09-10T20:38:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.092+0000 7f131e7fc640 1 -- 172.21.15.28:0/3411559276 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f13140103c0 con 0x7f13200c8080 2024-09-10T20:38:56.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.092+0000 7f131e7fc640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f130c077f70 0x7f130c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:56.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.092+0000 7f131e7fc640 1 -- 172.21.15.28:0/3411559276 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f131409abd0 con 0x7f13200c8080 2024-09-10T20:38:56.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.093+0000 7f1325ed9640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f130c077f70 0x7f130c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:56.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.093+0000 7f1325ed9640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f130c077f70 0x7f130c07a430 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1310033000 tx=0x7f1310036040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:56.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.096+0000 7f131e7fc640 1 -- 172.21.15.28:0/3411559276 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1320161aa0 con 0x7f13200c8080 2024-09-10T20:38:56.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.322+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f132006a360 con 0x7f13200c8080 2024-09-10T20:38:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.962+0000 7f131e7fc640 1 -- 172.21.15.28:0/3411559276 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f132006a360 con 0x7f13200c8080 2024-09-10T20:38:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.965+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f130c077f70 msgr2=0x7f130c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.965+0000 7f132813d640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f130c077f70 0x7f130c07a430 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1310033000 tx=0x7f1310036040 comp rx=0 tx=0).stop 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.965+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13200c8080 msgr2=0x7f13200cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.965+0000 7f132813d640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13200c8080 0x7f13200cbcc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1314003fb0 tx=0x7f1314001630 comp rx=0 tx=0).stop 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.965+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 shutdown_connections 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.965+0000 7f132813d640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f130c077f70 0x7f130c07a430 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.966+0000 7f132813d640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1320070a90 0x7f1320070f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.966+0000 7f132813d640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13200c8080 0x7f13200cbcc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.966+0000 7f132813d640 1 --2- 172.21.15.28:0/3411559276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1308003a50 0x7f13200b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.966+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 >> 172.21.15.28:0/3411559276 conn(0x7f13200d1f30 msgr2=0x7f132006dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.966+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 shutdown_connections 2024-09-10T20:38:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:56.966+0000 7f132813d640 1 -- 172.21.15.28:0/3411559276 wait complete. 2024-09-10T20:38:56.983 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:38:56.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.091+0000 7feeab333640 1 -- 172.21.15.28:0/2797262572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 msgr2=0x7fee90003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.091+0000 7feeab333640 1 --2- 172.21.15.28:0/2797262572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7fee90003e70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feea0006fd0 tx=0x7fee90005830 comp rx=0 tx=0).stop 2024-09-10T20:38:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.092+0000 7feeab333640 1 -- 172.21.15.28:0/2797262572 shutdown_connections 2024-09-10T20:38:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.092+0000 7feeab333640 1 --2- 172.21.15.28:0/2797262572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7fee90003e70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.092+0000 7feeab333640 1 -- 172.21.15.28:0/2797262572 >> 172.21.15.28:0/2797262572 conn(0x7feea406dbf0 msgr2=0x7feea406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.092+0000 7feeab333640 1 -- 172.21.15.28:0/2797262572 shutdown_connections 2024-09-10T20:38:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.092+0000 7feeab333640 1 -- 172.21.15.28:0/2797262572 wait complete. 2024-09-10T20:38:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.093+0000 7feeab333640 1 Processor -- start 2024-09-10T20:38:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.093+0000 7feeab333640 1 -- start start 2024-09-10T20:38:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feeab333640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7feea415d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feeab333640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea41392f0 0x7feea41f7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feeab333640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea4139820 0x7feea40706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feeab333640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feea40d1ac0 con 0x7fee90003a50 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feeab333640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7feea40d1940 con 0x7feea41392f0 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feeab333640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7feea4161950 con 0x7feea4139820 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7feea415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7feea415d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56694/0 (socket says 172.21.15.28:56694) 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 -- 172.21.15.28:0/1387462350 learned_addr learned my addr 172.21.15.28:0/1387462350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea88ce640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea41392f0 0x7feea41f7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea98d0640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea4139820 0x7feea40706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 -- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea4139820 msgr2=0x7feea40706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea4139820 0x7feea40706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 -- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea41392f0 msgr2=0x7feea41f7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea41392f0 0x7feea41f7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 -- 172.21.15.28:0/1387462350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea415dcc0 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feea90cf640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7feea415d780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7feea0000c00 tx=0x7feea0001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7fee99ffb640 1 -- 172.21.15.28:0/1387462350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0003a00 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7fee99ffb640 1 -- 172.21.15.28:0/1387462350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feea0003cb0 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.094+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea41f87e0 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.095+0000 7fee99ffb640 1 -- 172.21.15.28:0/1387462350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea002e6e0 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.095+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea41f8da0 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.096+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea406cae0 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.096+0000 7fee99ffb640 1 -- 172.21.15.28:0/1387462350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7feea002e880 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.097+0000 7fee99ffb640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fee88077f70 0x7fee8807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.097+0000 7fee99ffb640 1 -- 172.21.15.28:0/1387462350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7feea00ba100 con 0x7fee90003a50 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.097+0000 7feea88ce640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fee88077f70 0x7fee8807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.098+0000 7feea88ce640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fee88077f70 0x7fee8807a430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feea40b3f20 tx=0x7fee94006ca0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.102+0000 7fee99ffb640 1 -- 172.21.15.28:0/1387462350 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feea0047030 con 0x7fee90003a50 2024-09-10T20:38:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.322+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feea40b5d50 con 0x7fee90003a50 2024-09-10T20:38:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.322+0000 7fee99ffb640 1 -- 172.21.15.28:0/1387462350 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5226 (secure 0 0 0) 0x7feea415dcc0 con 0x7fee90003a50 2024-09-10T20:38:57.326 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:57.327 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"btime":"2024-09-10T20:38:56:959095+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1687267914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1687267914},{"type":"v1","addr":"172.21.15.159:6836","nonce":1687267914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/649109556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":649109556},{"type":"v1","addr":"172.21.15.159:6837","nonce":649109556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4851,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2646744984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2646744984},{"type":"v1","addr":"172.21.15.28:6834","nonce":2646744984}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:53.302329+0000","modified":"2024-09-10T20:38:56.323633+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4854":{"gid":4854,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1884641453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1884641453},{"type":"v1","addr":"172.21.15.28:6835","nonce":1884641453}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4854,"qdb_cluster":[4854]},"id":2}]} 2024-09-10T20:38:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 16 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fee88077f70 msgr2=0x7fee8807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fee88077f70 0x7fee8807a430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feea40b3f20 tx=0x7fee94006ca0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 msgr2=0x7feea415d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7feea415d780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7feea0000c00 tx=0x7feea0001cf0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 shutdown_connections 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fee88077f70 0x7fee8807a430 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feea4139820 0x7feea40706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feea41392f0 0x7feea41f7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 --2- 172.21.15.28:0/1387462350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee90003a50 0x7feea415d780 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 >> 172.21.15.28:0/1387462350 conn(0x7feea406dbf0 msgr2=0x7feea40ae590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:57.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 shutdown_connections 2024-09-10T20:38:57.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.325+0000 7feeab333640 1 -- 172.21.15.28:0/1387462350 wait complete. 2024-09-10T20:38:57.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:38:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.440+0000 7fb54d3a1640 1 -- 172.21.15.28:0/659507507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54815acf0 msgr2=0x7fb548074390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.440+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/659507507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54815acf0 0x7fb548074390 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb540005fd0 tx=0x7fb54002a820 comp rx=0 tx=0).stop 2024-09-10T20:38:57.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.440+0000 7fb54d3a1640 1 -- 172.21.15.28:0/659507507 shutdown_connections 2024-09-10T20:38:57.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.441+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/659507507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54815acf0 0x7fb548074390 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.441+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/659507507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb5480d0670 0x7fb5480732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.441+0000 7fb54d3a1640 1 -- 172.21.15.28:0/659507507 >> 172.21.15.28:0/659507507 conn(0x7fb5480cf990 msgr2=0x7fb5480cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:57.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.441+0000 7fb54d3a1640 1 -- 172.21.15.28:0/659507507 shutdown_connections 2024-09-10T20:38:57.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.441+0000 7fb54d3a1640 1 -- 172.21.15.28:0/659507507 wait complete. 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.442+0000 7fb54d3a1640 1 Processor -- start 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.442+0000 7fb54d3a1640 1 -- start start 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54d3a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5480d0670 0x7fb5481563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54d3a1640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb54815a910 0x7fb548157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54d3a1640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb54815acf0 0x7fb5480b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54d3a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5480ceaf0 con 0x7fb5480d0670 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54d3a1640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb5480ce970 con 0x7fb54815acf0 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54d3a1640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb5480cec70 con 0x7fb54815a910 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5480d0670 0x7fb5481563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5480d0670 0x7fb5481563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56710/0 (socket says 172.21.15.28:56710) 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 -- 172.21.15.28:0/1060661389 learned_addr learned my addr 172.21.15.28:0/1060661389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb5477fe640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb54815a910 0x7fb548157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54cba0640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb54815acf0 0x7fb5480b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 -- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb54815a910 msgr2=0x7fb548157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb54815a910 0x7fb548157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 -- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb54815acf0 msgr2=0x7fb5480b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb54815acf0 0x7fb5480b4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 -- 172.21.15.28:0/1060661389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5481f3f40 con 0x7fb5480d0670 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb54cba0640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb54815acf0 0x7fb5480b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.443+0000 7fb547fff640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5480d0670 0x7fb5481563f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb53800c530 tx=0x7fb53800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.444+0000 7fb544ff9640 1 -- 172.21.15.28:0/1060661389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538017070 con 0x7fb5480d0670 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.444+0000 7fb544ff9640 1 -- 172.21.15.28:0/1060661389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5380035f0 con 0x7fb5480d0670 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.444+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5480aa110 con 0x7fb5480d0670 2024-09-10T20:38:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.444+0000 7fb544ff9640 1 -- 172.21.15.28:0/1060661389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538003a40 con 0x7fb5480d0670 2024-09-10T20:38:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.444+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5480aa6d0 con 0x7fb5480d0670 2024-09-10T20:38:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.445+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb548074390 con 0x7fb5480d0670 2024-09-10T20:38:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.445+0000 7fb544ff9640 1 -- 172.21.15.28:0/1060661389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb538003be0 con 0x7fb5480d0670 2024-09-10T20:38:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.446+0000 7fb544ff9640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb534077f70 0x7fb53407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.446+0000 7fb544ff9640 1 -- 172.21.15.28:0/1060661389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fb538099380 con 0x7fb5480d0670 2024-09-10T20:38:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.446+0000 7fb5477fe640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb534077f70 0x7fb53407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.447+0000 7fb5477fe640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb534077f70 0x7fb53407a430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb5480ae600 tx=0x7fb53c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:57.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.450+0000 7fb544ff9640 1 -- 172.21.15.28:0/1060661389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb5380618c0 con 0x7fb5480d0670 2024-09-10T20:38:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.635+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb548069810 con 0x7fb5480d0670 2024-09-10T20:38:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.636+0000 7fb544ff9640 1 -- 172.21.15.28:0/1060661389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16669 (secure 0 0 0) 0x7fb5380662f0 con 0x7fb5480d0670 2024-09-10T20:38:57.640 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:57.640 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:54.948562+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:51.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:52.258677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb534077f70 msgr2=0x7fb53407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb534077f70 0x7fb53407a430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb5480ae600 tx=0x7fb53c005f90 comp rx=0 tx=0).stop 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5480d0670 msgr2=0x7fb5481563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5480d0670 0x7fb5481563f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb53800c530 tx=0x7fb53800c9f0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 shutdown_connections 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb534077f70 0x7fb53407a430 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb54815acf0 0x7fb5480b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb54815a910 0x7fb548157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 --2- 172.21.15.28:0/1060661389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5480d0670 0x7fb5481563f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 >> 172.21.15.28:0/1060661389 conn(0x7fb5480cf990 msgr2=0x7fb548072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 shutdown_connections 2024-09-10T20:38:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.638+0000 7fb54d3a1640 1 -- 172.21.15.28:0/1060661389 wait complete. 2024-09-10T20:38:57.653 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:38:57.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.755+0000 7f82f58e6640 1 -- 172.21.15.28:0/3332160975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82d80040b0 msgr2=0x7f82d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.755+0000 7f82f58e6640 1 --2- 172.21.15.28:0/3332160975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82d80040b0 0x7f82d8004800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f82f0074830 tx=0x7f82d8005830 comp rx=0 tx=0).stop 2024-09-10T20:38:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.755+0000 7f82ef7fe640 1 -- 172.21.15.28:0/3332160975 reap_dead start 2024-09-10T20:38:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.755+0000 7f82f58e6640 1 -- 172.21.15.28:0/3332160975 shutdown_connections 2024-09-10T20:38:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.755+0000 7f82f58e6640 1 -- 172.21.15.28:0/3332160975 >> 172.21.15.28:0/3332160975 conn(0x7f82f00d0420 msgr2=0x7f82f00d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.755+0000 7f82f58e6640 1 -- 172.21.15.28:0/3332160975 shutdown_connections 2024-09-10T20:38:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.755+0000 7f82f58e6640 1 -- 172.21.15.28:0/3332160975 wait complete. 2024-09-10T20:38:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.756+0000 7f82f58e6640 1 Processor -- start 2024-09-10T20:38:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82f58e6640 1 -- start start 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82f58e6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82d80040b0 0x7f82f00c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82f58e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82f006cd50 0x7f82f00b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82f58e6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f006d1b0 0x7f82f00b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82f58e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82f0073e70 con 0x7f82f006cd50 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82f58e6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f82f0073cf0 con 0x7f82d80040b0 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82f58e6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f82f0073ff0 con 0x7f82f006d1b0 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82ee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82f006cd50 0x7f82f00b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82ee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82f006cd50 0x7f82f00b33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56734/0 (socket says 172.21.15.28:56734) 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.757+0000 7f82ee7fc640 1 -- 172.21.15.28:0/3294282032 learned_addr learned my addr 172.21.15.28:0/3294282032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ef7fe640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f006d1b0 0x7f82f00b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82eeffd640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82d80040b0 0x7f82f00c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ee7fc640 1 -- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f006d1b0 msgr2=0x7f82f00b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ee7fc640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f006d1b0 0x7f82f00b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ee7fc640 1 -- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82d80040b0 msgr2=0x7f82f00c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ee7fc640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82d80040b0 0x7f82f00c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ee7fc640 1 -- 172.21.15.28:0/3294282032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f00b4320 con 0x7f82f006cd50 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ef7fe640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f006d1b0 0x7f82f00b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82ee7fc640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82f006cd50 0x7f82f00b33f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f82e4001780 tx=0x7f82e4006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82d7fff640 1 -- 172.21.15.28:0/3294282032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4038070 con 0x7f82f006cd50 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82d7fff640 1 -- 172.21.15.28:0/3294282032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82e4003af0 con 0x7f82f006cd50 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f00b4040 con 0x7f82f006cd50 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82d7fff640 1 -- 172.21.15.28:0/3294282032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4033420 con 0x7f82f006cd50 2024-09-10T20:38:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.758+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f00c5cd0 con 0x7f82f006cd50 2024-09-10T20:38:57.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.759+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f00cf400 con 0x7f82f006cd50 2024-09-10T20:38:57.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.760+0000 7f82d7fff640 1 -- 172.21.15.28:0/3294282032 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f82e40335c0 con 0x7f82f006cd50 2024-09-10T20:38:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.760+0000 7f82d7fff640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f82dc077f70 0x7f82dc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.760+0000 7f82d7fff640 1 -- 172.21.15.28:0/3294282032 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f82e40b94b0 con 0x7f82f006cd50 2024-09-10T20:38:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.761+0000 7f82eeffd640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f82dc077f70 0x7f82dc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.761+0000 7f82eeffd640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f82dc077f70 0x7f82dc07a430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82e0003700 tx=0x7f82e000b260 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.764+0000 7f82d7fff640 1 -- 172.21.15.28:0/3294282032 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f82e4082040 con 0x7f82f006cd50 2024-09-10T20:38:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.944+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82f0068d10 con 0x7f82dc077f70 2024-09-10T20:38:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.949+0000 7f82d7fff640 1 -- 172.21.15.28:0/3294282032 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+244553 (secure 0 0 0) 0x7f82f0068d10 con 0x7f82dc077f70 2024-09-10T20:38:57.956 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:57.957 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:57.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f82dc077f70 msgr2=0x7f82dc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f82dc077f70 0x7f82dc07a430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82e0003700 tx=0x7f82e000b260 comp rx=0 tx=0).stop 2024-09-10T20:38:57.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82f006cd50 msgr2=0x7f82f00b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:57.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82f006cd50 0x7f82f00b33f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f82e4001780 tx=0x7f82e4006360 comp rx=0 tx=0).stop 2024-09-10T20:38:57.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 shutdown_connections 2024-09-10T20:38:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f82dc077f70 0x7f82dc07a430 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f006d1b0 0x7f82f00b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82f006cd50 0x7f82f00b33f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 --2- 172.21.15.28:0/3294282032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82d80040b0 0x7f82f00c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 >> 172.21.15.28:0/3294282032 conn(0x7f82f00d0420 msgr2=0x7f82f015d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 shutdown_connections 2024-09-10T20:38:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:57.952+0000 7f82f58e6640 1 -- 172.21.15.28:0/3294282032 wait complete. 2024-09-10T20:38:57.967 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-10T20:38:55.985076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592002,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":130,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":236600,"kb_used_data":3708,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738120,"statfs":{"total":773094113280,"available":772851834880,"internally_reserved":0,"allocated":3796992,"data_stored":2144534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1634,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.063277"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959137+0000","last_change":"2024-09-10T20:38:53.948628+0000","last_active":"2024-09-10T20:38:54.959137+0000","last_peered":"2024-09-10T20:38:54.959137+0000","last_clean":"2024-09-10T20:38:54.959137+0000","last_became_active":"2024-09-10T20:38:53.948363+0000","last_became_peered":"2024-09-10T20:38:53.948363+0000","last_unstale":"2024-09-10T20:38:54.959137+0000","last_undegraded":"2024-09-10T20:38:54.959137+0000","last_fullsized":"2024-09-10T20:38:54.959137+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:39.816989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959852+0000","last_change":"2024-09-10T20:38:53.968306+0000","last_active":"2024-09-10T20:38:54.959852+0000","last_peered":"2024-09-10T20:38:54.959852+0000","last_clean":"2024-09-10T20:38:54.959852+0000","last_became_active":"2024-09-10T20:38:53.967930+0000","last_became_peered":"2024-09-10T20:38:53.967930+0000","last_unstale":"2024-09-10T20:38:54.959852+0000","last_undegraded":"2024-09-10T20:38:54.959852+0000","last_fullsized":"2024-09-10T20:38:54.959852+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:07.430068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958626+0000","last_change":"2024-09-10T20:38:53.949808+0000","last_active":"2024-09-10T20:38:54.958626+0000","last_peered":"2024-09-10T20:38:54.958626+0000","last_clean":"2024-09-10T20:38:54.958626+0000","last_became_active":"2024-09-10T20:38:53.949487+0000","last_became_peered":"2024-09-10T20:38:53.949487+0000","last_unstale":"2024-09-10T20:38:54.958626+0000","last_undegraded":"2024-09-10T20:38:54.958626+0000","last_fullsized":"2024-09-10T20:38:54.958626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:27:40.923439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.970291+0000","last_change":"2024-09-10T20:38:53.968185+0000","last_active":"2024-09-10T20:38:53.970291+0000","last_peered":"2024-09-10T20:38:53.970291+0000","last_clean":"2024-09-10T20:38:53.970291+0000","last_became_active":"2024-09-10T20:38:53.967916+0000","last_became_peered":"2024-09-10T20:38:53.967916+0000","last_unstale":"2024-09-10T20:38:53.970291+0000","last_undegraded":"2024-09-10T20:38:53.970291+0000","last_fullsized":"2024-09-10T20:38:53.970291+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:42.503646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959537+0000","last_change":"2024-09-10T20:38:53.943826+0000","last_active":"2024-09-10T20:38:54.959537+0000","last_peered":"2024-09-10T20:38:54.959537+0000","last_clean":"2024-09-10T20:38:54.959537+0000","last_became_active":"2024-09-10T20:38:53.943586+0000","last_became_peered":"2024-09-10T20:38:53.943586+0000","last_unstale":"2024-09-10T20:38:54.959537+0000","last_undegraded":"2024-09-10T20:38:54.959537+0000","last_fullsized":"2024-09-10T20:38:54.959537+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:58:28.628213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961750+0000","last_change":"2024-09-10T20:38:53.959657+0000","last_active":"2024-09-10T20:38:53.961750+0000","last_peered":"2024-09-10T20:38:53.961750+0000","last_clean":"2024-09-10T20:38:53.961750+0000","last_became_active":"2024-09-10T20:38:53.959387+0000","last_became_peered":"2024-09-10T20:38:53.959387+0000","last_unstale":"2024-09-10T20:38:53.961750+0000","last_undegraded":"2024-09-10T20:38:53.961750+0000","last_fullsized":"2024-09-10T20:38:53.961750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:23.725358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959250+0000","last_change":"2024-09-10T20:38:53.944660+0000","last_active":"2024-09-10T20:38:54.959250+0000","last_peered":"2024-09-10T20:38:54.959250+0000","last_clean":"2024-09-10T20:38:54.959250+0000","last_became_active":"2024-09-10T20:38:53.944344+0000","last_became_peered":"2024-09-10T20:38:53.944344+0000","last_unstale":"2024-09-10T20:38:54.959250+0000","last_undegraded":"2024-09-10T20:38:54.959250+0000","last_fullsized":"2024-09-10T20:38:54.959250+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:13:24.439493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957440+0000","last_change":"2024-09-10T20:38:53.948646+0000","last_active":"2024-09-10T20:38:53.957440+0000","last_peered":"2024-09-10T20:38:53.957440+0000","last_clean":"2024-09-10T20:38:53.957440+0000","last_became_active":"2024-09-10T20:38:53.948390+0000","last_became_peered":"2024-09-10T20:38:53.948390+0000","last_unstale":"2024-09-10T20:38:53.957440+0000","last_undegraded":"2024-09-10T20:38:53.957440+0000","last_fullsized":"2024-09-10T20:38:53.957440+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:37.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957114+0000","last_change":"2024-09-10T20:38:53.947931+0000","last_active":"2024-09-10T20:38:53.957114+0000","last_peered":"2024-09-10T20:38:53.957114+0000","last_clean":"2024-09-10T20:38:53.957114+0000","last_became_active":"2024-09-10T20:38:53.947664+0000","last_became_peered":"2024-09-10T20:38:53.947664+0000","last_unstale":"2024-09-10T20:38:53.957114+0000","last_undegraded":"2024-09-10T20:38:53.957114+0000","last_fullsized":"2024-09-10T20:38:53.957114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:31.935869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961051+0000","last_change":"2024-09-10T20:38:53.957959+0000","last_active":"2024-09-10T20:38:53.961051+0000","last_peered":"2024-09-10T20:38:53.961051+0000","last_clean":"2024-09-10T20:38:53.961051+0000","last_became_active":"2024-09-10T20:38:53.957680+0000","last_became_peered":"2024-09-10T20:38:53.957680+0000","last_unstale":"2024-09-10T20:38:53.961051+0000","last_undegraded":"2024-09-10T20:38:53.961051+0000","last_fullsized":"2024-09-10T20:38:53.961051+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:42:12.458273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.952649+0000","last_change":"2024-09-10T20:38:53.947018+0000","last_active":"2024-09-10T20:38:53.952649+0000","last_peered":"2024-09-10T20:38:53.952649+0000","last_clean":"2024-09-10T20:38:53.952649+0000","last_became_active":"2024-09-10T20:38:53.946745+0000","last_became_peered":"2024-09-10T20:38:53.946745+0000","last_unstale":"2024-09-10T20:38:53.952649+0000","last_undegraded":"2024-09-10T20:38:53.952649+0000","last_fullsized":"2024-09-10T20:38:53.952649+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:15.368370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959787+0000","last_change":"2024-09-10T20:38:53.948524+0000","last_active":"2024-09-10T20:38:54.959787+0000","last_peered":"2024-09-10T20:38:54.959787+0000","last_clean":"2024-09-10T20:38:54.959787+0000","last_became_active":"2024-09-10T20:38:53.947361+0000","last_became_peered":"2024-09-10T20:38:53.947361+0000","last_unstale":"2024-09-10T20:38:54.959787+0000","last_undegraded":"2024-09-10T20:38:54.959787+0000","last_fullsized":"2024-09-10T20:38:54.959787+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:58.590534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.960053+0000","last_change":"2024-09-10T20:38:53.950830+0000","last_active":"2024-09-10T20:38:54.960053+0000","last_peered":"2024-09-10T20:38:54.960053+0000","last_clean":"2024-09-10T20:38:54.960053+0000","last_became_active":"2024-09-10T20:38:53.950569+0000","last_became_peered":"2024-09-10T20:38:53.950569+0000","last_unstale":"2024-09-10T20:38:54.960053+0000","last_undegraded":"2024-09-10T20:38:54.960053+0000","last_fullsized":"2024-09-10T20:38:54.960053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:53.427390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958599+0000","last_change":"2024-09-10T20:38:53.954909+0000","last_active":"2024-09-10T20:38:54.958599+0000","last_peered":"2024-09-10T20:38:54.958599+0000","last_clean":"2024-09-10T20:38:54.958599+0000","last_became_active":"2024-09-10T20:38:53.954398+0000","last_became_peered":"2024-09-10T20:38:53.954398+0000","last_unstale":"2024-09-10T20:38:54.958599+0000","last_undegraded":"2024-09-10T20:38:54.958599+0000","last_fullsized":"2024-09-10T20:38:54.958599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:48:53.378497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957242+0000","last_change":"2024-09-10T20:38:53.948778+0000","last_active":"2024-09-10T20:38:53.957242+0000","last_peered":"2024-09-10T20:38:53.957242+0000","last_clean":"2024-09-10T20:38:53.957242+0000","last_became_active":"2024-09-10T20:38:53.948545+0000","last_became_peered":"2024-09-10T20:38:53.948545+0000","last_unstale":"2024-09-10T20:38:53.957242+0000","last_undegraded":"2024-09-10T20:38:53.957242+0000","last_fullsized":"2024-09-10T20:38:53.957242+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:16.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959795+0000","last_change":"2024-09-10T20:38:52.940550+0000","last_active":"2024-09-10T20:38:54.959795+0000","last_peered":"2024-09-10T20:38:54.959795+0000","last_clean":"2024-09-10T20:38:54.959795+0000","last_became_active":"2024-09-10T20:38:52.940254+0000","last_became_peered":"2024-09-10T20:38:52.940254+0000","last_unstale":"2024-09-10T20:38:54.959795+0000","last_undegraded":"2024-09-10T20:38:54.959795+0000","last_fullsized":"2024-09-10T20:38:54.959795+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:50:56.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"31'2","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.959123+0000","last_change":"2024-09-10T20:38:52.940726+0000","last_active":"2024-09-10T20:38:53.959123+0000","last_peered":"2024-09-10T20:38:53.959123+0000","last_clean":"2024-09-10T20:38:53.959123+0000","last_became_active":"2024-09-10T20:38:52.940218+0000","last_became_peered":"2024-09-10T20:38:52.940218+0000","last_unstale":"2024-09-10T20:38:53.959123+0000","last_undegraded":"2024-09-10T20:38:53.959123+0000","last_fullsized":"2024-09-10T20:38:53.959123+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:32.485722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959676+0000","last_change":"2024-09-10T20:38:53.951827+0000","last_active":"2024-09-10T20:38:54.959676+0000","last_peered":"2024-09-10T20:38:54.959676+0000","last_clean":"2024-09-10T20:38:54.959676+0000","last_became_active":"2024-09-10T20:38:53.951569+0000","last_became_peered":"2024-09-10T20:38:53.951569+0000","last_unstale":"2024-09-10T20:38:54.959676+0000","last_undegraded":"2024-09-10T20:38:54.959676+0000","last_fullsized":"2024-09-10T20:38:54.959676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:55:29.069213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959942+0000","last_change":"2024-09-10T20:38:52.932378+0000","last_active":"2024-09-10T20:38:54.959942+0000","last_peered":"2024-09-10T20:38:54.959942+0000","last_clean":"2024-09-10T20:38:54.959942+0000","last_became_active":"2024-09-10T20:38:52.932104+0000","last_became_peered":"2024-09-10T20:38:52.932104+0000","last_unstale":"2024-09-10T20:38:54.959942+0000","last_undegraded":"2024-09-10T20:38:54.959942+0000","last_fullsized":"2024-09-10T20:38:54.959942+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:51:25.951863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.942427+0000","last_change":"2024-09-10T20:38:52.936643+0000","last_active":"2024-09-10T20:38:53.942427+0000","last_peered":"2024-09-10T20:38:53.942427+0000","last_clean":"2024-09-10T20:38:53.942427+0000","last_became_active":"2024-09-10T20:38:52.935792+0000","last_became_peered":"2024-09-10T20:38:52.935792+0000","last_unstale":"2024-09-10T20:38:53.942427+0000","last_undegraded":"2024-09-10T20:38:53.942427+0000","last_fullsized":"2024-09-10T20:38:53.942427+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:57.044959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958696+0000","last_change":"2024-09-10T20:38:53.946792+0000","last_active":"2024-09-10T20:38:54.958696+0000","last_peered":"2024-09-10T20:38:54.958696+0000","last_clean":"2024-09-10T20:38:54.958696+0000","last_became_active":"2024-09-10T20:38:53.946486+0000","last_became_peered":"2024-09-10T20:38:53.946486+0000","last_unstale":"2024-09-10T20:38:54.958696+0000","last_undegraded":"2024-09-10T20:38:54.958696+0000","last_fullsized":"2024-09-10T20:38:54.958696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:41:36.164988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958697+0000","last_change":"2024-09-10T20:38:52.942691+0000","last_active":"2024-09-10T20:38:54.958697+0000","last_peered":"2024-09-10T20:38:54.958697+0000","last_clean":"2024-09-10T20:38:54.958697+0000","last_became_active":"2024-09-10T20:38:52.942436+0000","last_became_peered":"2024-09-10T20:38:52.942436+0000","last_unstale":"2024-09-10T20:38:54.958697+0000","last_undegraded":"2024-09-10T20:38:54.958697+0000","last_fullsized":"2024-09-10T20:38:54.958697+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:19.514284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959929+0000","last_change":"2024-09-10T20:38:53.943792+0000","last_active":"2024-09-10T20:38:54.959929+0000","last_peered":"2024-09-10T20:38:54.959929+0000","last_clean":"2024-09-10T20:38:54.959929+0000","last_became_active":"2024-09-10T20:38:53.943522+0000","last_became_peered":"2024-09-10T20:38:53.943522+0000","last_unstale":"2024-09-10T20:38:54.959929+0000","last_undegraded":"2024-09-10T20:38:54.959929+0000","last_fullsized":"2024-09-10T20:38:54.959929+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:54.872129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959239+0000","last_change":"2024-09-10T20:38:52.938493+0000","last_active":"2024-09-10T20:38:54.959239+0000","last_peered":"2024-09-10T20:38:54.959239+0000","last_clean":"2024-09-10T20:38:54.959239+0000","last_became_active":"2024-09-10T20:38:52.938210+0000","last_became_peered":"2024-09-10T20:38:52.938210+0000","last_unstale":"2024-09-10T20:38:54.959239+0000","last_undegraded":"2024-09-10T20:38:54.959239+0000","last_fullsized":"2024-09-10T20:38:54.959239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:44:18.774129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.960087+0000","last_change":"2024-09-10T20:38:53.968268+0000","last_active":"2024-09-10T20:38:54.960087+0000","last_peered":"2024-09-10T20:38:54.960087+0000","last_clean":"2024-09-10T20:38:54.960087+0000","last_became_active":"2024-09-10T20:38:53.967903+0000","last_became_peered":"2024-09-10T20:38:53.967903+0000","last_unstale":"2024-09-10T20:38:54.960087+0000","last_undegraded":"2024-09-10T20:38:54.960087+0000","last_fullsized":"2024-09-10T20:38:54.960087+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:13:18.224569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.971432+0000","last_change":"2024-09-10T20:38:52.936161+0000","last_active":"2024-09-10T20:38:53.971432+0000","last_peered":"2024-09-10T20:38:53.971432+0000","last_clean":"2024-09-10T20:38:53.971432+0000","last_became_active":"2024-09-10T20:38:52.935643+0000","last_became_peered":"2024-09-10T20:38:52.935643+0000","last_unstale":"2024-09-10T20:38:53.971432+0000","last_undegraded":"2024-09-10T20:38:53.971432+0000","last_fullsized":"2024-09-10T20:38:53.971432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:34.761867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.956815+0000","last_change":"2024-09-10T20:38:53.948474+0000","last_active":"2024-09-10T20:38:53.956815+0000","last_peered":"2024-09-10T20:38:53.956815+0000","last_clean":"2024-09-10T20:38:53.956815+0000","last_became_active":"2024-09-10T20:38:53.948184+0000","last_became_peered":"2024-09-10T20:38:53.948184+0000","last_unstale":"2024-09-10T20:38:53.956815+0000","last_undegraded":"2024-09-10T20:38:53.956815+0000","last_fullsized":"2024-09-10T20:38:53.956815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:46.112293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959225+0000","last_change":"2024-09-10T20:38:53.957247+0000","last_active":"2024-09-10T20:38:54.959225+0000","last_peered":"2024-09-10T20:38:54.959225+0000","last_clean":"2024-09-10T20:38:54.959225+0000","last_became_active":"2024-09-10T20:38:53.956475+0000","last_became_peered":"2024-09-10T20:38:53.956475+0000","last_unstale":"2024-09-10T20:38:54.959225+0000","last_undegraded":"2024-09-10T20:38:54.959225+0000","last_fullsized":"2024-09-10T20:38:54.959225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:00:36.701444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959281+0000","last_change":"2024-09-10T20:38:52.941073+0000","last_active":"2024-09-10T20:38:54.959281+0000","last_peered":"2024-09-10T20:38:54.959281+0000","last_clean":"2024-09-10T20:38:54.959281+0000","last_became_active":"2024-09-10T20:38:52.940767+0000","last_became_peered":"2024-09-10T20:38:52.940767+0000","last_unstale":"2024-09-10T20:38:54.959281+0000","last_undegraded":"2024-09-10T20:38:54.959281+0000","last_fullsized":"2024-09-10T20:38:54.959281+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:41.632356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958826+0000","last_change":"2024-09-10T20:38:52.938620+0000","last_active":"2024-09-10T20:38:54.958826+0000","last_peered":"2024-09-10T20:38:54.958826+0000","last_clean":"2024-09-10T20:38:54.958826+0000","last_became_active":"2024-09-10T20:38:52.938261+0000","last_became_peered":"2024-09-10T20:38:52.938261+0000","last_unstale":"2024-09-10T20:38:54.958826+0000","last_undegraded":"2024-09-10T20:38:54.958826+0000","last_fullsized":"2024-09-10T20:38:54.958826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:57:27.195064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"31'8","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.965749+0000","last_change":"2024-09-10T20:38:52.943742+0000","last_active":"2024-09-10T20:38:53.965749+0000","last_peered":"2024-09-10T20:38:53.965749+0000","last_clean":"2024-09-10T20:38:53.965749+0000","last_became_active":"2024-09-10T20:38:52.943394+0000","last_became_peered":"2024-09-10T20:38:52.943394+0000","last_unstale":"2024-09-10T20:38:53.965749+0000","last_undegraded":"2024-09-10T20:38:53.965749+0000","last_fullsized":"2024-09-10T20:38:53.965749+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:12:49.651268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"31'1","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961502+0000","last_change":"2024-09-10T20:38:52.942078+0000","last_active":"2024-09-10T20:38:53.961502+0000","last_peered":"2024-09-10T20:38:53.961502+0000","last_clean":"2024-09-10T20:38:53.961502+0000","last_became_active":"2024-09-10T20:38:52.941617+0000","last_became_peered":"2024-09-10T20:38:52.941617+0000","last_unstale":"2024-09-10T20:38:53.961502+0000","last_undegraded":"2024-09-10T20:38:53.961502+0000","last_fullsized":"2024-09-10T20:38:53.961502+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:58.849546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958478+0000","last_change":"2024-09-10T20:38:53.945772+0000","last_active":"2024-09-10T20:38:54.958478+0000","last_peered":"2024-09-10T20:38:54.958478+0000","last_clean":"2024-09-10T20:38:54.958478+0000","last_became_active":"2024-09-10T20:38:53.945499+0000","last_became_peered":"2024-09-10T20:38:53.945499+0000","last_unstale":"2024-09-10T20:38:54.958478+0000","last_undegraded":"2024-09-10T20:38:54.958478+0000","last_fullsized":"2024-09-10T20:38:54.958478+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:44.793307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961602+0000","last_change":"2024-09-10T20:38:53.960281+0000","last_active":"2024-09-10T20:38:53.961602+0000","last_peered":"2024-09-10T20:38:53.961602+0000","last_clean":"2024-09-10T20:38:53.961602+0000","last_became_active":"2024-09-10T20:38:53.959983+0000","last_became_peered":"2024-09-10T20:38:53.959983+0000","last_unstale":"2024-09-10T20:38:53.961602+0000","last_undegraded":"2024-09-10T20:38:53.961602+0000","last_fullsized":"2024-09-10T20:38:53.961602+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:52.590068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.958287+0000","last_change":"2024-09-10T20:38:53.956597+0000","last_active":"2024-09-10T20:38:53.958287+0000","last_peered":"2024-09-10T20:38:53.958287+0000","last_clean":"2024-09-10T20:38:53.958287+0000","last_became_active":"2024-09-10T20:38:53.956337+0000","last_became_peered":"2024-09-10T20:38:53.956337+0000","last_unstale":"2024-09-10T20:38:53.958287+0000","last_undegraded":"2024-09-10T20:38:53.958287+0000","last_fullsized":"2024-09-10T20:38:53.958287+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:03.034001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958826+0000","last_change":"2024-09-10T20:38:52.936958+0000","last_active":"2024-09-10T20:38:54.958826+0000","last_peered":"2024-09-10T20:38:54.958826+0000","last_clean":"2024-09-10T20:38:54.958826+0000","last_became_active":"2024-09-10T20:38:52.936672+0000","last_became_peered":"2024-09-10T20:38:52.936672+0000","last_unstale":"2024-09-10T20:38:54.958826+0000","last_undegraded":"2024-09-10T20:38:54.958826+0000","last_fullsized":"2024-09-10T20:38:54.958826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:39.604422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958811+0000","last_change":"2024-09-10T20:38:53.948631+0000","last_active":"2024-09-10T20:38:54.958811+0000","last_peered":"2024-09-10T20:38:54.958811+0000","last_clean":"2024-09-10T20:38:54.958811+0000","last_became_active":"2024-09-10T20:38:53.948344+0000","last_became_peered":"2024-09-10T20:38:53.948344+0000","last_unstale":"2024-09-10T20:38:54.958811+0000","last_undegraded":"2024-09-10T20:38:54.958811+0000","last_fullsized":"2024-09-10T20:38:54.958811+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:14:49.187533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957116+0000","last_change":"2024-09-10T20:38:53.952723+0000","last_active":"2024-09-10T20:38:53.957116+0000","last_peered":"2024-09-10T20:38:53.957116+0000","last_clean":"2024-09-10T20:38:53.957116+0000","last_became_active":"2024-09-10T20:38:53.952399+0000","last_became_peered":"2024-09-10T20:38:53.952399+0000","last_unstale":"2024-09-10T20:38:53.957116+0000","last_undegraded":"2024-09-10T20:38:53.957116+0000","last_fullsized":"2024-09-10T20:38:53.957116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:13:27.524745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961568+0000","last_change":"2024-09-10T20:38:53.959514+0000","last_active":"2024-09-10T20:38:53.961568+0000","last_peered":"2024-09-10T20:38:53.961568+0000","last_clean":"2024-09-10T20:38:53.961568+0000","last_became_active":"2024-09-10T20:38:53.959256+0000","last_became_peered":"2024-09-10T20:38:53.959256+0000","last_unstale":"2024-09-10T20:38:53.961568+0000","last_undegraded":"2024-09-10T20:38:53.961568+0000","last_fullsized":"2024-09-10T20:38:53.961568+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:52:16.377413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'2","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.959736+0000","last_change":"2024-09-10T20:38:52.940948+0000","last_active":"2024-09-10T20:38:53.959736+0000","last_peered":"2024-09-10T20:38:53.959736+0000","last_clean":"2024-09-10T20:38:53.959736+0000","last_became_active":"2024-09-10T20:38:52.939610+0000","last_became_peered":"2024-09-10T20:38:52.939610+0000","last_unstale":"2024-09-10T20:38:53.959736+0000","last_undegraded":"2024-09-10T20:38:53.959736+0000","last_fullsized":"2024-09-10T20:38:53.959736+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:06:54.772555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959322+0000","last_change":"2024-09-10T20:38:52.938788+0000","last_active":"2024-09-10T20:38:54.959322+0000","last_peered":"2024-09-10T20:38:54.959322+0000","last_clean":"2024-09-10T20:38:54.959322+0000","last_became_active":"2024-09-10T20:38:52.937799+0000","last_became_peered":"2024-09-10T20:38:52.937799+0000","last_unstale":"2024-09-10T20:38:54.959322+0000","last_undegraded":"2024-09-10T20:38:54.959322+0000","last_fullsized":"2024-09-10T20:38:54.959322+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:08.661653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957386+0000","last_change":"2024-09-10T20:38:53.955780+0000","last_active":"2024-09-10T20:38:53.957386+0000","last_peered":"2024-09-10T20:38:53.957386+0000","last_clean":"2024-09-10T20:38:53.957386+0000","last_became_active":"2024-09-10T20:38:53.955494+0000","last_became_peered":"2024-09-10T20:38:53.955494+0000","last_unstale":"2024-09-10T20:38:53.957386+0000","last_undegraded":"2024-09-10T20:38:53.957386+0000","last_fullsized":"2024-09-10T20:38:53.957386+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:45.472013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.944217+0000","last_change":"2024-09-10T20:38:52.938501+0000","last_active":"2024-09-10T20:38:53.944217+0000","last_peered":"2024-09-10T20:38:53.944217+0000","last_clean":"2024-09-10T20:38:53.944217+0000","last_became_active":"2024-09-10T20:38:52.938201+0000","last_became_peered":"2024-09-10T20:38:52.938201+0000","last_unstale":"2024-09-10T20:38:53.944217+0000","last_undegraded":"2024-09-10T20:38:53.944217+0000","last_fullsized":"2024-09-10T20:38:53.944217+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:50.069288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.946490+0000","last_change":"2024-09-10T20:38:52.937078+0000","last_active":"2024-09-10T20:38:53.946490+0000","last_peered":"2024-09-10T20:38:53.946490+0000","last_clean":"2024-09-10T20:38:53.946490+0000","last_became_active":"2024-09-10T20:38:52.936803+0000","last_became_peered":"2024-09-10T20:38:52.936803+0000","last_unstale":"2024-09-10T20:38:53.946490+0000","last_undegraded":"2024-09-10T20:38:53.946490+0000","last_fullsized":"2024-09-10T20:38:53.946490+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:44:29.761081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959573+0000","last_change":"2024-09-10T20:38:53.955680+0000","last_active":"2024-09-10T20:38:54.959573+0000","last_peered":"2024-09-10T20:38:54.959573+0000","last_clean":"2024-09-10T20:38:54.959573+0000","last_became_active":"2024-09-10T20:38:53.952088+0000","last_became_peered":"2024-09-10T20:38:53.952088+0000","last_unstale":"2024-09-10T20:38:54.959573+0000","last_undegraded":"2024-09-10T20:38:54.959573+0000","last_fullsized":"2024-09-10T20:38:54.959573+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:25:51.845077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.943486+0000","last_change":"2024-09-10T20:38:52.942225+0000","last_active":"2024-09-10T20:38:53.943486+0000","last_peered":"2024-09-10T20:38:53.943486+0000","last_clean":"2024-09-10T20:38:53.943486+0000","last_became_active":"2024-09-10T20:38:52.941887+0000","last_became_peered":"2024-09-10T20:38:52.941887+0000","last_unstale":"2024-09-10T20:38:53.943486+0000","last_undegraded":"2024-09-10T20:38:53.943486+0000","last_fullsized":"2024-09-10T20:38:53.943486+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:47:38.085507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958736+0000","last_change":"2024-09-10T20:38:53.964770+0000","last_active":"2024-09-10T20:38:54.958736+0000","last_peered":"2024-09-10T20:38:54.958736+0000","last_clean":"2024-09-10T20:38:54.958736+0000","last_became_active":"2024-09-10T20:38:53.964456+0000","last_became_peered":"2024-09-10T20:38:53.964456+0000","last_unstale":"2024-09-10T20:38:54.958736+0000","last_undegraded":"2024-09-10T20:38:54.958736+0000","last_fullsized":"2024-09-10T20:38:54.958736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:35.620408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959476+0000","last_change":"2024-09-10T20:38:52.942992+0000","last_active":"2024-09-10T20:38:54.959476+0000","last_peered":"2024-09-10T20:38:54.959476+0000","last_clean":"2024-09-10T20:38:54.959476+0000","last_became_active":"2024-09-10T20:38:52.942733+0000","last_became_peered":"2024-09-10T20:38:52.942733+0000","last_unstale":"2024-09-10T20:38:54.959476+0000","last_undegraded":"2024-09-10T20:38:54.959476+0000","last_fullsized":"2024-09-10T20:38:54.959476+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:07:36.996717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:52.919556+0000","last_change":"2024-09-10T20:38:52.919556+0000","last_active":"2024-09-10T20:38:52.919556+0000","last_peered":"2024-09-10T20:38:52.919556+0000","last_clean":"2024-09-10T20:38:52.919556+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:52.919556+0000","last_undegraded":"2024-09-10T20:38:52.919556+0000","last_fullsized":"2024-09-10T20:38:52.919556+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:38:51.913751+0000","last_change":"2024-09-10T20:38:51.913751+0000","last_active":"2024-09-10T20:38:51.913751+0000","last_peered":"2024-09-10T20:38:51.913751+0000","last_clean":"2024-09-10T20:38:51.913751+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:38:51.913751+0000","last_undegraded":"2024-09-10T20:38:51.913751+0000","last_fullsized":"2024-09-10T20:38:51.913751+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":121,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-10T20:38:48.593420+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:48.593420+0000","last_peered":"2024-09-10T20:38:48.593420+0000","last_clean":"2024-09-10T20:38:48.593420+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:48.593420+0000","last_undegraded":"2024-09-10T20:38:48.593420+0000","last_fullsized":"2024-09-10T20:38:48.593420+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":60,"acting":60,"num_store_stats":4},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1634,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":36,"acting":36,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509454,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31492,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":303104,"data_stored":110560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31488,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340352,"statfs":{"total":96636764160,"available":96604520448,"internally_reserved":0,"allocated":303104,"data_stored":110560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":897024,"data_stored":700928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31484,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":303104,"data_stored":110560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542158,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":344064,"data_stored":129449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542159,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":352256,"data_stored":131039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542157,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":937984,"data_stored":720377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":356352,"data_stored":131061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:57.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:38:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 -- 172.21.15.28:0/3166166772 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15a190 msgr2=0x7f588c0b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 --2- 172.21.15.28:0/3166166772 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15a190 0x7f588c0b2ba0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f587c005fd0 tx=0x7f587c02aa10 comp rx=0 tx=0).stop 2024-09-10T20:38:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 -- 172.21.15.28:0/3166166772 shutdown_connections 2024-09-10T20:38:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 --2- 172.21.15.28:0/3166166772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f588c0b4d10 0x7f588c0ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 --2- 172.21.15.28:0/3166166772 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15a190 0x7f588c0b2ba0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 -- 172.21.15.28:0/3166166772 >> 172.21.15.28:0/3166166772 conn(0x7f588c06cf00 msgr2=0x7f588c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 -- 172.21.15.28:0/3166166772 shutdown_connections 2024-09-10T20:38:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.108+0000 7f5891aed640 1 -- 172.21.15.28:0/3166166772 wait complete. 2024-09-10T20:38:58.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.109+0000 7f5891aed640 1 Processor -- start 2024-09-10T20:38:58.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f5891aed640 1 -- start start 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f5891aed640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f588c0b4d10 0x7f588c1f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f5891aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c15a190 0x7f588c151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f5891aed640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15df70 0x7f588c157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f5891aed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f588c0d0270 con 0x7f588c15a190 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f5891aed640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f588c0d00f0 con 0x7f588c15df70 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f5891aed640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f588c0d03f0 con 0x7f588c0b4d10 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f588affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c15a190 0x7f588c151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.110+0000 7f588affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c15a190 0x7f588c151d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56738/0 (socket says 172.21.15.28:56738) 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588affd640 1 -- 172.21.15.28:0/792545302 learned_addr learned my addr 172.21.15.28:0/792545302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588bfff640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15df70 0x7f588c157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588b7fe640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f588c0b4d10 0x7f588c1f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588affd640 1 -- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f588c0b4d10 msgr2=0x7f588c1f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588affd640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f588c0b4d10 0x7f588c1f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588affd640 1 -- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15df70 msgr2=0x7f588c157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588affd640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15df70 0x7f588c157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588affd640 1 -- 172.21.15.28:0/792545302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588c1f6c20 con 0x7f588c15a190 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f588affd640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c15a190 0x7f588c151d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f588000c370 tx=0x7f588000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f5873fff640 1 -- 172.21.15.28:0/792545302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880017070 con 0x7f588c15a190 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f5873fff640 1 -- 172.21.15.28:0/792545302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58800053c0 con 0x7f588c15a190 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588c152250 con 0x7f588c15a190 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f5873fff640 1 -- 172.21.15.28:0/792545302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880003850 con 0x7f588c15a190 2024-09-10T20:38:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.111+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f588c1f6660 con 0x7f588c15a190 2024-09-10T20:38:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.112+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588c0addf0 con 0x7f588c15a190 2024-09-10T20:38:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.113+0000 7f5873fff640 1 -- 172.21.15.28:0/792545302 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f58800039f0 con 0x7f588c15a190 2024-09-10T20:38:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.113+0000 7f5873fff640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5878077f70 0x7f587807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.114+0000 7f5873fff640 1 -- 172.21.15.28:0/792545302 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f5880099540 con 0x7f588c15a190 2024-09-10T20:38:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.114+0000 7f588b7fe640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5878077f70 0x7f587807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.114+0000 7f588b7fe640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5878077f70 0x7f587807a430 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f587c035000 tx=0x7f587c004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:58.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.118+0000 7f5873fff640 1 -- 172.21.15.28:0/792545302 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f588c0addf0 con 0x7f588c15a190 2024-09-10T20:38:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.298+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f588c0ce030 con 0x7f5878077f70 2024-09-10T20:38:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.303+0000 7f5873fff640 1 -- 172.21.15.28:0/792545302 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251938 (secure 0 0 0) 0x7f588c0ce030 con 0x7f5878077f70 2024-09-10T20:38:58.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:38:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5878077f70 msgr2=0x7f587807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5878077f70 0x7f587807a430 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f587c035000 tx=0x7f587c004ab0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c15a190 msgr2=0x7f588c151d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c15a190 0x7f588c151d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f588000c370 tx=0x7f588000c830 comp rx=0 tx=0).stop 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 shutdown_connections 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5878077f70 0x7f587807a430 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f588c15df70 0x7f588c157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c15a190 0x7f588c151d10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 --2- 172.21.15.28:0/792545302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f588c0b4d10 0x7f588c1f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 >> 172.21.15.28:0/792545302 conn(0x7f588c06cf00 msgr2=0x7f588c0b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 shutdown_connections 2024-09-10T20:38:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.306+0000 7f5891aed640 1 -- 172.21.15.28:0/792545302 wait complete. 2024-09-10T20:38:58.321 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-10T20:38:57.985469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236828,"kb_used_data":3928,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737892,"statfs":{"total":773094113280,"available":772851601408,"internally_reserved":0,"allocated":4022272,"data_stored":2246172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000993"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211315+0000","last_change":"2024-09-10T20:38:53.955417+0000","last_active":"2024-09-10T20:38:55.211315+0000","last_peered":"2024-09-10T20:38:55.211315+0000","last_clean":"2024-09-10T20:38:55.211315+0000","last_became_active":"2024-09-10T20:38:53.955157+0000","last_became_peered":"2024-09-10T20:38:53.955157+0000","last_unstale":"2024-09-10T20:38:55.211315+0000","last_undegraded":"2024-09-10T20:38:55.211315+0000","last_fullsized":"2024-09-10T20:38:55.211315+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:22:21.078139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959137+0000","last_change":"2024-09-10T20:38:53.948628+0000","last_active":"2024-09-10T20:38:54.959137+0000","last_peered":"2024-09-10T20:38:54.959137+0000","last_clean":"2024-09-10T20:38:54.959137+0000","last_became_active":"2024-09-10T20:38:53.948363+0000","last_became_peered":"2024-09-10T20:38:53.948363+0000","last_unstale":"2024-09-10T20:38:54.959137+0000","last_undegraded":"2024-09-10T20:38:54.959137+0000","last_fullsized":"2024-09-10T20:38:54.959137+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:39.816989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.212470+0000","last_change":"2024-09-10T20:38:53.967880+0000","last_active":"2024-09-10T20:38:55.212470+0000","last_peered":"2024-09-10T20:38:55.212470+0000","last_clean":"2024-09-10T20:38:55.212470+0000","last_became_active":"2024-09-10T20:38:53.967582+0000","last_became_peered":"2024-09-10T20:38:53.967582+0000","last_unstale":"2024-09-10T20:38:55.212470+0000","last_undegraded":"2024-09-10T20:38:55.212470+0000","last_fullsized":"2024-09-10T20:38:55.212470+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:15.368370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959852+0000","last_change":"2024-09-10T20:38:53.968306+0000","last_active":"2024-09-10T20:38:54.959852+0000","last_peered":"2024-09-10T20:38:54.959852+0000","last_clean":"2024-09-10T20:38:54.959852+0000","last_became_active":"2024-09-10T20:38:53.967930+0000","last_became_peered":"2024-09-10T20:38:53.967930+0000","last_unstale":"2024-09-10T20:38:54.959852+0000","last_undegraded":"2024-09-10T20:38:54.959852+0000","last_fullsized":"2024-09-10T20:38:54.959852+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:07.430068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958626+0000","last_change":"2024-09-10T20:38:53.949808+0000","last_active":"2024-09-10T20:38:54.958626+0000","last_peered":"2024-09-10T20:38:54.958626+0000","last_clean":"2024-09-10T20:38:54.958626+0000","last_became_active":"2024-09-10T20:38:53.949487+0000","last_became_peered":"2024-09-10T20:38:53.949487+0000","last_unstale":"2024-09-10T20:38:54.958626+0000","last_undegraded":"2024-09-10T20:38:54.958626+0000","last_fullsized":"2024-09-10T20:38:54.958626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:27:40.923439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.970291+0000","last_change":"2024-09-10T20:38:53.968185+0000","last_active":"2024-09-10T20:38:53.970291+0000","last_peered":"2024-09-10T20:38:53.970291+0000","last_clean":"2024-09-10T20:38:53.970291+0000","last_became_active":"2024-09-10T20:38:53.967916+0000","last_became_peered":"2024-09-10T20:38:53.967916+0000","last_unstale":"2024-09-10T20:38:53.970291+0000","last_undegraded":"2024-09-10T20:38:53.970291+0000","last_fullsized":"2024-09-10T20:38:53.970291+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:42.503646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211732+0000","last_change":"2024-09-10T20:38:53.953713+0000","last_active":"2024-09-10T20:38:55.211732+0000","last_peered":"2024-09-10T20:38:55.211732+0000","last_clean":"2024-09-10T20:38:55.211732+0000","last_became_active":"2024-09-10T20:38:53.951258+0000","last_became_peered":"2024-09-10T20:38:53.951258+0000","last_unstale":"2024-09-10T20:38:55.211732+0000","last_undegraded":"2024-09-10T20:38:55.211732+0000","last_fullsized":"2024-09-10T20:38:55.211732+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:48:53.378497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.210862+0000","last_change":"2024-09-10T20:38:53.963864+0000","last_active":"2024-09-10T20:38:55.210862+0000","last_peered":"2024-09-10T20:38:55.210862+0000","last_clean":"2024-09-10T20:38:55.210862+0000","last_became_active":"2024-09-10T20:38:53.963559+0000","last_became_peered":"2024-09-10T20:38:53.963559+0000","last_unstale":"2024-09-10T20:38:55.210862+0000","last_undegraded":"2024-09-10T20:38:55.210862+0000","last_fullsized":"2024-09-10T20:38:55.210862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:02:38.117424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211390+0000","last_change":"2024-09-10T20:38:53.957891+0000","last_active":"2024-09-10T20:38:55.211390+0000","last_peered":"2024-09-10T20:38:55.211390+0000","last_clean":"2024-09-10T20:38:55.211390+0000","last_became_active":"2024-09-10T20:38:53.957561+0000","last_became_peered":"2024-09-10T20:38:53.957561+0000","last_unstale":"2024-09-10T20:38:55.211390+0000","last_undegraded":"2024-09-10T20:38:55.211390+0000","last_fullsized":"2024-09-10T20:38:55.211390+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:47:39.091312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959537+0000","last_change":"2024-09-10T20:38:53.943826+0000","last_active":"2024-09-10T20:38:54.959537+0000","last_peered":"2024-09-10T20:38:54.959537+0000","last_clean":"2024-09-10T20:38:54.959537+0000","last_became_active":"2024-09-10T20:38:53.943586+0000","last_became_peered":"2024-09-10T20:38:53.943586+0000","last_unstale":"2024-09-10T20:38:54.959537+0000","last_undegraded":"2024-09-10T20:38:54.959537+0000","last_fullsized":"2024-09-10T20:38:54.959537+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:58:28.628213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211396+0000","last_change":"2024-09-10T20:38:53.962581+0000","last_active":"2024-09-10T20:38:55.211396+0000","last_peered":"2024-09-10T20:38:55.211396+0000","last_clean":"2024-09-10T20:38:55.211396+0000","last_became_active":"2024-09-10T20:38:53.961952+0000","last_became_peered":"2024-09-10T20:38:53.961952+0000","last_unstale":"2024-09-10T20:38:55.211396+0000","last_undegraded":"2024-09-10T20:38:55.211396+0000","last_fullsized":"2024-09-10T20:38:55.211396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:21.487501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.212273+0000","last_change":"2024-09-10T20:38:53.949468+0000","last_active":"2024-09-10T20:38:55.212273+0000","last_peered":"2024-09-10T20:38:55.212273+0000","last_clean":"2024-09-10T20:38:55.212273+0000","last_became_active":"2024-09-10T20:38:53.949187+0000","last_became_peered":"2024-09-10T20:38:53.949187+0000","last_unstale":"2024-09-10T20:38:55.212273+0000","last_undegraded":"2024-09-10T20:38:55.212273+0000","last_fullsized":"2024-09-10T20:38:55.212273+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:44:30.766885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961750+0000","last_change":"2024-09-10T20:38:53.959657+0000","last_active":"2024-09-10T20:38:53.961750+0000","last_peered":"2024-09-10T20:38:53.961750+0000","last_clean":"2024-09-10T20:38:53.961750+0000","last_became_active":"2024-09-10T20:38:53.959387+0000","last_became_peered":"2024-09-10T20:38:53.959387+0000","last_unstale":"2024-09-10T20:38:53.961750+0000","last_undegraded":"2024-09-10T20:38:53.961750+0000","last_fullsized":"2024-09-10T20:38:53.961750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:23.725358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959250+0000","last_change":"2024-09-10T20:38:53.944660+0000","last_active":"2024-09-10T20:38:54.959250+0000","last_peered":"2024-09-10T20:38:54.959250+0000","last_clean":"2024-09-10T20:38:54.959250+0000","last_became_active":"2024-09-10T20:38:53.944344+0000","last_became_peered":"2024-09-10T20:38:53.944344+0000","last_unstale":"2024-09-10T20:38:54.959250+0000","last_undegraded":"2024-09-10T20:38:54.959250+0000","last_fullsized":"2024-09-10T20:38:54.959250+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:13:24.439493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955123+0000","last_change":"2024-09-10T20:38:53.951923+0000","last_active":"2024-09-10T20:38:54.955123+0000","last_peered":"2024-09-10T20:38:54.955123+0000","last_clean":"2024-09-10T20:38:54.955123+0000","last_became_active":"2024-09-10T20:38:53.951634+0000","last_became_peered":"2024-09-10T20:38:53.951634+0000","last_unstale":"2024-09-10T20:38:54.955123+0000","last_undegraded":"2024-09-10T20:38:54.955123+0000","last_fullsized":"2024-09-10T20:38:54.955123+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:32.296363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955124+0000","last_change":"2024-09-10T20:38:53.954291+0000","last_active":"2024-09-10T20:38:54.955124+0000","last_peered":"2024-09-10T20:38:54.955124+0000","last_clean":"2024-09-10T20:38:54.955124+0000","last_became_active":"2024-09-10T20:38:53.954009+0000","last_became_peered":"2024-09-10T20:38:53.954009+0000","last_unstale":"2024-09-10T20:38:54.955124+0000","last_undegraded":"2024-09-10T20:38:54.955124+0000","last_fullsized":"2024-09-10T20:38:54.955124+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:49:04.889488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.210979+0000","last_change":"2024-09-10T20:38:53.951817+0000","last_active":"2024-09-10T20:38:55.210979+0000","last_peered":"2024-09-10T20:38:55.210979+0000","last_clean":"2024-09-10T20:38:55.210979+0000","last_became_active":"2024-09-10T20:38:53.951481+0000","last_became_peered":"2024-09-10T20:38:53.951481+0000","last_unstale":"2024-09-10T20:38:55.210979+0000","last_undegraded":"2024-09-10T20:38:55.210979+0000","last_fullsized":"2024-09-10T20:38:55.210979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:03.653219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957440+0000","last_change":"2024-09-10T20:38:53.948646+0000","last_active":"2024-09-10T20:38:53.957440+0000","last_peered":"2024-09-10T20:38:53.957440+0000","last_clean":"2024-09-10T20:38:53.957440+0000","last_became_active":"2024-09-10T20:38:53.948390+0000","last_became_peered":"2024-09-10T20:38:53.948390+0000","last_unstale":"2024-09-10T20:38:53.957440+0000","last_undegraded":"2024-09-10T20:38:53.957440+0000","last_fullsized":"2024-09-10T20:38:53.957440+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:37.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957114+0000","last_change":"2024-09-10T20:38:53.947931+0000","last_active":"2024-09-10T20:38:53.957114+0000","last_peered":"2024-09-10T20:38:53.957114+0000","last_clean":"2024-09-10T20:38:53.957114+0000","last_became_active":"2024-09-10T20:38:53.947664+0000","last_became_peered":"2024-09-10T20:38:53.947664+0000","last_unstale":"2024-09-10T20:38:53.957114+0000","last_undegraded":"2024-09-10T20:38:53.957114+0000","last_fullsized":"2024-09-10T20:38:53.957114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:31.935869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961051+0000","last_change":"2024-09-10T20:38:53.957959+0000","last_active":"2024-09-10T20:38:53.961051+0000","last_peered":"2024-09-10T20:38:53.961051+0000","last_clean":"2024-09-10T20:38:53.961051+0000","last_became_active":"2024-09-10T20:38:53.957680+0000","last_became_peered":"2024-09-10T20:38:53.957680+0000","last_unstale":"2024-09-10T20:38:53.961051+0000","last_undegraded":"2024-09-10T20:38:53.961051+0000","last_fullsized":"2024-09-10T20:38:53.961051+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:42:12.458273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955425+0000","last_change":"2024-09-10T20:38:53.950699+0000","last_active":"2024-09-10T20:38:54.955425+0000","last_peered":"2024-09-10T20:38:54.955425+0000","last_clean":"2024-09-10T20:38:54.955425+0000","last_became_active":"2024-09-10T20:38:53.950387+0000","last_became_peered":"2024-09-10T20:38:53.950387+0000","last_unstale":"2024-09-10T20:38:54.955425+0000","last_undegraded":"2024-09-10T20:38:54.955425+0000","last_fullsized":"2024-09-10T20:38:54.955425+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:09.011207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.210934+0000","last_change":"2024-09-10T20:38:53.953560+0000","last_active":"2024-09-10T20:38:55.210934+0000","last_peered":"2024-09-10T20:38:55.210934+0000","last_clean":"2024-09-10T20:38:55.210934+0000","last_became_active":"2024-09-10T20:38:53.951229+0000","last_became_peered":"2024-09-10T20:38:53.951229+0000","last_unstale":"2024-09-10T20:38:55.210934+0000","last_undegraded":"2024-09-10T20:38:55.210934+0000","last_fullsized":"2024-09-10T20:38:55.210934+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:55.884655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.952649+0000","last_change":"2024-09-10T20:38:53.947018+0000","last_active":"2024-09-10T20:38:53.952649+0000","last_peered":"2024-09-10T20:38:53.952649+0000","last_clean":"2024-09-10T20:38:53.952649+0000","last_became_active":"2024-09-10T20:38:53.946745+0000","last_became_peered":"2024-09-10T20:38:53.946745+0000","last_unstale":"2024-09-10T20:38:53.952649+0000","last_undegraded":"2024-09-10T20:38:53.952649+0000","last_fullsized":"2024-09-10T20:38:53.952649+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:15.368370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211065+0000","last_change":"2024-09-10T20:38:53.966352+0000","last_active":"2024-09-10T20:38:55.211065+0000","last_peered":"2024-09-10T20:38:55.211065+0000","last_clean":"2024-09-10T20:38:55.211065+0000","last_became_active":"2024-09-10T20:38:53.963758+0000","last_became_peered":"2024-09-10T20:38:53.963758+0000","last_unstale":"2024-09-10T20:38:55.211065+0000","last_undegraded":"2024-09-10T20:38:55.211065+0000","last_fullsized":"2024-09-10T20:38:55.211065+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:32.296363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959787+0000","last_change":"2024-09-10T20:38:53.948524+0000","last_active":"2024-09-10T20:38:54.959787+0000","last_peered":"2024-09-10T20:38:54.959787+0000","last_clean":"2024-09-10T20:38:54.959787+0000","last_became_active":"2024-09-10T20:38:53.947361+0000","last_became_peered":"2024-09-10T20:38:53.947361+0000","last_unstale":"2024-09-10T20:38:54.959787+0000","last_undegraded":"2024-09-10T20:38:54.959787+0000","last_fullsized":"2024-09-10T20:38:54.959787+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:58.590534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.960053+0000","last_change":"2024-09-10T20:38:53.950830+0000","last_active":"2024-09-10T20:38:54.960053+0000","last_peered":"2024-09-10T20:38:54.960053+0000","last_clean":"2024-09-10T20:38:54.960053+0000","last_became_active":"2024-09-10T20:38:53.950569+0000","last_became_peered":"2024-09-10T20:38:53.950569+0000","last_unstale":"2024-09-10T20:38:54.960053+0000","last_undegraded":"2024-09-10T20:38:54.960053+0000","last_fullsized":"2024-09-10T20:38:54.960053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:53.427390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958599+0000","last_change":"2024-09-10T20:38:53.954909+0000","last_active":"2024-09-10T20:38:54.958599+0000","last_peered":"2024-09-10T20:38:54.958599+0000","last_clean":"2024-09-10T20:38:54.958599+0000","last_became_active":"2024-09-10T20:38:53.954398+0000","last_became_peered":"2024-09-10T20:38:53.954398+0000","last_unstale":"2024-09-10T20:38:54.958599+0000","last_undegraded":"2024-09-10T20:38:54.958599+0000","last_fullsized":"2024-09-10T20:38:54.958599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:48:53.378497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211715+0000","last_change":"2024-09-10T20:38:53.967567+0000","last_active":"2024-09-10T20:38:55.211715+0000","last_peered":"2024-09-10T20:38:55.211715+0000","last_clean":"2024-09-10T20:38:55.211715+0000","last_became_active":"2024-09-10T20:38:53.967304+0000","last_became_peered":"2024-09-10T20:38:53.967304+0000","last_unstale":"2024-09-10T20:38:55.211715+0000","last_undegraded":"2024-09-10T20:38:55.211715+0000","last_fullsized":"2024-09-10T20:38:55.211715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:30.808994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955204+0000","last_change":"2024-09-10T20:38:53.946752+0000","last_active":"2024-09-10T20:38:54.955204+0000","last_peered":"2024-09-10T20:38:54.955204+0000","last_clean":"2024-09-10T20:38:54.955204+0000","last_became_active":"2024-09-10T20:38:53.946394+0000","last_became_peered":"2024-09-10T20:38:53.946394+0000","last_unstale":"2024-09-10T20:38:54.955204+0000","last_undegraded":"2024-09-10T20:38:54.955204+0000","last_fullsized":"2024-09-10T20:38:54.955204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:14:18.773328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955145+0000","last_change":"2024-09-10T20:38:53.947152+0000","last_active":"2024-09-10T20:38:54.955145+0000","last_peered":"2024-09-10T20:38:54.955145+0000","last_clean":"2024-09-10T20:38:54.955145+0000","last_became_active":"2024-09-10T20:38:53.946677+0000","last_became_peered":"2024-09-10T20:38:53.946677+0000","last_unstale":"2024-09-10T20:38:54.955145+0000","last_undegraded":"2024-09-10T20:38:54.955145+0000","last_fullsized":"2024-09-10T20:38:54.955145+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:22:21.078139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211758+0000","last_change":"2024-09-10T20:38:53.962148+0000","last_active":"2024-09-10T20:38:55.211758+0000","last_peered":"2024-09-10T20:38:55.211758+0000","last_clean":"2024-09-10T20:38:55.211758+0000","last_became_active":"2024-09-10T20:38:53.961846+0000","last_became_peered":"2024-09-10T20:38:53.961846+0000","last_unstale":"2024-09-10T20:38:55.211758+0000","last_undegraded":"2024-09-10T20:38:55.211758+0000","last_fullsized":"2024-09-10T20:38:55.211758+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:09.011207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.210878+0000","last_change":"2024-09-10T20:38:53.969154+0000","last_active":"2024-09-10T20:38:55.210878+0000","last_peered":"2024-09-10T20:38:55.210878+0000","last_clean":"2024-09-10T20:38:55.210878+0000","last_became_active":"2024-09-10T20:38:53.968870+0000","last_became_peered":"2024-09-10T20:38:53.968870+0000","last_unstale":"2024-09-10T20:38:55.210878+0000","last_undegraded":"2024-09-10T20:38:55.210878+0000","last_fullsized":"2024-09-10T20:38:55.210878+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:41:40.532854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957242+0000","last_change":"2024-09-10T20:38:53.948778+0000","last_active":"2024-09-10T20:38:53.957242+0000","last_peered":"2024-09-10T20:38:53.957242+0000","last_clean":"2024-09-10T20:38:53.957242+0000","last_became_active":"2024-09-10T20:38:53.948545+0000","last_became_peered":"2024-09-10T20:38:53.948545+0000","last_unstale":"2024-09-10T20:38:53.957242+0000","last_undegraded":"2024-09-10T20:38:53.957242+0000","last_fullsized":"2024-09-10T20:38:53.957242+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:16.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"31'4","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.212459+0000","last_change":"2024-09-10T20:38:52.940161+0000","last_active":"2024-09-10T20:38:55.212459+0000","last_peered":"2024-09-10T20:38:55.212459+0000","last_clean":"2024-09-10T20:38:55.212459+0000","last_became_active":"2024-09-10T20:38:52.939877+0000","last_became_peered":"2024-09-10T20:38:52.939877+0000","last_unstale":"2024-09-10T20:38:55.212459+0000","last_undegraded":"2024-09-10T20:38:55.212459+0000","last_fullsized":"2024-09-10T20:38:55.212459+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:48.341074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211519+0000","last_change":"2024-09-10T20:38:53.966588+0000","last_active":"2024-09-10T20:38:55.211519+0000","last_peered":"2024-09-10T20:38:55.211519+0000","last_clean":"2024-09-10T20:38:55.211519+0000","last_became_active":"2024-09-10T20:38:53.963931+0000","last_became_peered":"2024-09-10T20:38:53.963931+0000","last_unstale":"2024-09-10T20:38:55.211519+0000","last_undegraded":"2024-09-10T20:38:55.211519+0000","last_fullsized":"2024-09-10T20:38:55.211519+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:14:18.773328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959795+0000","last_change":"2024-09-10T20:38:52.940550+0000","last_active":"2024-09-10T20:38:54.959795+0000","last_peered":"2024-09-10T20:38:54.959795+0000","last_clean":"2024-09-10T20:38:54.959795+0000","last_became_active":"2024-09-10T20:38:52.940254+0000","last_became_peered":"2024-09-10T20:38:52.940254+0000","last_unstale":"2024-09-10T20:38:54.959795+0000","last_undegraded":"2024-09-10T20:38:54.959795+0000","last_fullsized":"2024-09-10T20:38:54.959795+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:50:56.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211914+0000","last_change":"2024-09-10T20:38:53.967511+0000","last_active":"2024-09-10T20:38:55.211914+0000","last_peered":"2024-09-10T20:38:55.211914+0000","last_clean":"2024-09-10T20:38:55.211914+0000","last_became_active":"2024-09-10T20:38:53.967203+0000","last_became_peered":"2024-09-10T20:38:53.967203+0000","last_unstale":"2024-09-10T20:38:55.211914+0000","last_undegraded":"2024-09-10T20:38:55.211914+0000","last_fullsized":"2024-09-10T20:38:55.211914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:45.085156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"31'2","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.959123+0000","last_change":"2024-09-10T20:38:52.940726+0000","last_active":"2024-09-10T20:38:53.959123+0000","last_peered":"2024-09-10T20:38:53.959123+0000","last_clean":"2024-09-10T20:38:53.959123+0000","last_became_active":"2024-09-10T20:38:52.940218+0000","last_became_peered":"2024-09-10T20:38:52.940218+0000","last_unstale":"2024-09-10T20:38:53.959123+0000","last_undegraded":"2024-09-10T20:38:53.959123+0000","last_fullsized":"2024-09-10T20:38:53.959123+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:32.485722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959676+0000","last_change":"2024-09-10T20:38:53.951827+0000","last_active":"2024-09-10T20:38:54.959676+0000","last_peered":"2024-09-10T20:38:54.959676+0000","last_clean":"2024-09-10T20:38:54.959676+0000","last_became_active":"2024-09-10T20:38:53.951569+0000","last_became_peered":"2024-09-10T20:38:53.951569+0000","last_unstale":"2024-09-10T20:38:54.959676+0000","last_undegraded":"2024-09-10T20:38:54.959676+0000","last_fullsized":"2024-09-10T20:38:54.959676+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:55:29.069213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959942+0000","last_change":"2024-09-10T20:38:52.932378+0000","last_active":"2024-09-10T20:38:54.959942+0000","last_peered":"2024-09-10T20:38:54.959942+0000","last_clean":"2024-09-10T20:38:54.959942+0000","last_became_active":"2024-09-10T20:38:52.932104+0000","last_became_peered":"2024-09-10T20:38:52.932104+0000","last_unstale":"2024-09-10T20:38:54.959942+0000","last_undegraded":"2024-09-10T20:38:54.959942+0000","last_fullsized":"2024-09-10T20:38:54.959942+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:51:25.951863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211120+0000","last_change":"2024-09-10T20:38:53.960002+0000","last_active":"2024-09-10T20:38:55.211120+0000","last_peered":"2024-09-10T20:38:55.211120+0000","last_clean":"2024-09-10T20:38:55.211120+0000","last_became_active":"2024-09-10T20:38:53.959734+0000","last_became_peered":"2024-09-10T20:38:53.959734+0000","last_unstale":"2024-09-10T20:38:55.211120+0000","last_undegraded":"2024-09-10T20:38:55.211120+0000","last_fullsized":"2024-09-10T20:38:55.211120+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:44.394017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211227+0000","last_change":"2024-09-10T20:38:52.930707+0000","last_active":"2024-09-10T20:38:55.211227+0000","last_peered":"2024-09-10T20:38:55.211227+0000","last_clean":"2024-09-10T20:38:55.211227+0000","last_became_active":"2024-09-10T20:38:52.930397+0000","last_became_peered":"2024-09-10T20:38:52.930397+0000","last_unstale":"2024-09-10T20:38:55.211227+0000","last_undegraded":"2024-09-10T20:38:55.211227+0000","last_fullsized":"2024-09-10T20:38:55.211227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:44:29.761081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.210941+0000","last_change":"2024-09-10T20:38:53.963610+0000","last_active":"2024-09-10T20:38:55.210941+0000","last_peered":"2024-09-10T20:38:55.210941+0000","last_clean":"2024-09-10T20:38:55.210941+0000","last_became_active":"2024-09-10T20:38:53.963305+0000","last_became_peered":"2024-09-10T20:38:53.963305+0000","last_unstale":"2024-09-10T20:38:55.210941+0000","last_undegraded":"2024-09-10T20:38:55.210941+0000","last_fullsized":"2024-09-10T20:38:55.210941+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:48:20.965520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211055+0000","last_change":"2024-09-10T20:38:52.940300+0000","last_active":"2024-09-10T20:38:55.211055+0000","last_peered":"2024-09-10T20:38:55.211055+0000","last_clean":"2024-09-10T20:38:55.211055+0000","last_became_active":"2024-09-10T20:38:52.939983+0000","last_became_peered":"2024-09-10T20:38:52.939983+0000","last_unstale":"2024-09-10T20:38:55.211055+0000","last_undegraded":"2024-09-10T20:38:55.211055+0000","last_fullsized":"2024-09-10T20:38:55.211055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:20:43.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211904+0000","last_change":"2024-09-10T20:38:53.962286+0000","last_active":"2024-09-10T20:38:55.211904+0000","last_peered":"2024-09-10T20:38:55.211904+0000","last_clean":"2024-09-10T20:38:55.211904+0000","last_became_active":"2024-09-10T20:38:53.961996+0000","last_became_peered":"2024-09-10T20:38:53.961996+0000","last_unstale":"2024-09-10T20:38:55.211904+0000","last_undegraded":"2024-09-10T20:38:55.211904+0000","last_fullsized":"2024-09-10T20:38:55.211904+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:51.075093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.942427+0000","last_change":"2024-09-10T20:38:52.936643+0000","last_active":"2024-09-10T20:38:53.942427+0000","last_peered":"2024-09-10T20:38:53.942427+0000","last_clean":"2024-09-10T20:38:53.942427+0000","last_became_active":"2024-09-10T20:38:52.935792+0000","last_became_peered":"2024-09-10T20:38:52.935792+0000","last_unstale":"2024-09-10T20:38:53.942427+0000","last_undegraded":"2024-09-10T20:38:53.942427+0000","last_fullsized":"2024-09-10T20:38:53.942427+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:57.044959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958696+0000","last_change":"2024-09-10T20:38:53.946792+0000","last_active":"2024-09-10T20:38:54.958696+0000","last_peered":"2024-09-10T20:38:54.958696+0000","last_clean":"2024-09-10T20:38:54.958696+0000","last_became_active":"2024-09-10T20:38:53.946486+0000","last_became_peered":"2024-09-10T20:38:53.946486+0000","last_unstale":"2024-09-10T20:38:54.958696+0000","last_undegraded":"2024-09-10T20:38:54.958696+0000","last_fullsized":"2024-09-10T20:38:54.958696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:41:36.164988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211366+0000","last_change":"2024-09-10T20:38:52.936697+0000","last_active":"2024-09-10T20:38:55.211366+0000","last_peered":"2024-09-10T20:38:55.211366+0000","last_clean":"2024-09-10T20:38:55.211366+0000","last_became_active":"2024-09-10T20:38:52.936253+0000","last_became_peered":"2024-09-10T20:38:52.936253+0000","last_unstale":"2024-09-10T20:38:55.211366+0000","last_undegraded":"2024-09-10T20:38:55.211366+0000","last_fullsized":"2024-09-10T20:38:55.211366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:34.761867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211936+0000","last_change":"2024-09-10T20:38:53.950710+0000","last_active":"2024-09-10T20:38:55.211936+0000","last_peered":"2024-09-10T20:38:55.211936+0000","last_clean":"2024-09-10T20:38:55.211936+0000","last_became_active":"2024-09-10T20:38:53.949413+0000","last_became_peered":"2024-09-10T20:38:53.949413+0000","last_unstale":"2024-09-10T20:38:55.211936+0000","last_undegraded":"2024-09-10T20:38:55.211936+0000","last_fullsized":"2024-09-10T20:38:55.211936+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:59:44.238906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958697+0000","last_change":"2024-09-10T20:38:52.942691+0000","last_active":"2024-09-10T20:38:54.958697+0000","last_peered":"2024-09-10T20:38:54.958697+0000","last_clean":"2024-09-10T20:38:54.958697+0000","last_became_active":"2024-09-10T20:38:52.942436+0000","last_became_peered":"2024-09-10T20:38:52.942436+0000","last_unstale":"2024-09-10T20:38:54.958697+0000","last_undegraded":"2024-09-10T20:38:54.958697+0000","last_fullsized":"2024-09-10T20:38:54.958697+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:19.514284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959929+0000","last_change":"2024-09-10T20:38:53.943792+0000","last_active":"2024-09-10T20:38:54.959929+0000","last_peered":"2024-09-10T20:38:54.959929+0000","last_clean":"2024-09-10T20:38:54.959929+0000","last_became_active":"2024-09-10T20:38:53.943522+0000","last_became_peered":"2024-09-10T20:38:53.943522+0000","last_unstale":"2024-09-10T20:38:54.959929+0000","last_undegraded":"2024-09-10T20:38:54.959929+0000","last_fullsized":"2024-09-10T20:38:54.959929+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:54.872129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959239+0000","last_change":"2024-09-10T20:38:52.938493+0000","last_active":"2024-09-10T20:38:54.959239+0000","last_peered":"2024-09-10T20:38:54.959239+0000","last_clean":"2024-09-10T20:38:54.959239+0000","last_became_active":"2024-09-10T20:38:52.938210+0000","last_became_peered":"2024-09-10T20:38:52.938210+0000","last_unstale":"2024-09-10T20:38:54.959239+0000","last_undegraded":"2024-09-10T20:38:54.959239+0000","last_fullsized":"2024-09-10T20:38:54.959239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:44:18.774129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.960087+0000","last_change":"2024-09-10T20:38:53.968268+0000","last_active":"2024-09-10T20:38:54.960087+0000","last_peered":"2024-09-10T20:38:54.960087+0000","last_clean":"2024-09-10T20:38:54.960087+0000","last_became_active":"2024-09-10T20:38:53.967903+0000","last_became_peered":"2024-09-10T20:38:53.967903+0000","last_unstale":"2024-09-10T20:38:54.960087+0000","last_undegraded":"2024-09-10T20:38:54.960087+0000","last_fullsized":"2024-09-10T20:38:54.960087+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:13:18.224569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211297+0000","last_change":"2024-09-10T20:38:52.938618+0000","last_active":"2024-09-10T20:38:55.211297+0000","last_peered":"2024-09-10T20:38:55.211297+0000","last_clean":"2024-09-10T20:38:55.211297+0000","last_became_active":"2024-09-10T20:38:52.938331+0000","last_became_peered":"2024-09-10T20:38:52.938331+0000","last_unstale":"2024-09-10T20:38:55.211297+0000","last_undegraded":"2024-09-10T20:38:55.211297+0000","last_fullsized":"2024-09-10T20:38:55.211297+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:44.079351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211239+0000","last_change":"2024-09-10T20:38:53.962267+0000","last_active":"2024-09-10T20:38:55.211239+0000","last_peered":"2024-09-10T20:38:55.211239+0000","last_clean":"2024-09-10T20:38:55.211239+0000","last_became_active":"2024-09-10T20:38:53.961587+0000","last_became_peered":"2024-09-10T20:38:53.961587+0000","last_unstale":"2024-09-10T20:38:55.211239+0000","last_undegraded":"2024-09-10T20:38:55.211239+0000","last_fullsized":"2024-09-10T20:38:55.211239+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:43.521918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.971432+0000","last_change":"2024-09-10T20:38:52.936161+0000","last_active":"2024-09-10T20:38:53.971432+0000","last_peered":"2024-09-10T20:38:53.971432+0000","last_clean":"2024-09-10T20:38:53.971432+0000","last_became_active":"2024-09-10T20:38:52.935643+0000","last_became_peered":"2024-09-10T20:38:52.935643+0000","last_unstale":"2024-09-10T20:38:53.971432+0000","last_undegraded":"2024-09-10T20:38:53.971432+0000","last_fullsized":"2024-09-10T20:38:53.971432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:34.761867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.956815+0000","last_change":"2024-09-10T20:38:53.948474+0000","last_active":"2024-09-10T20:38:53.956815+0000","last_peered":"2024-09-10T20:38:53.956815+0000","last_clean":"2024-09-10T20:38:53.956815+0000","last_became_active":"2024-09-10T20:38:53.948184+0000","last_became_peered":"2024-09-10T20:38:53.948184+0000","last_unstale":"2024-09-10T20:38:53.956815+0000","last_undegraded":"2024-09-10T20:38:53.956815+0000","last_fullsized":"2024-09-10T20:38:53.956815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:46.112293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211328+0000","last_change":"2024-09-10T20:38:52.934856+0000","last_active":"2024-09-10T20:38:55.211328+0000","last_peered":"2024-09-10T20:38:55.211328+0000","last_clean":"2024-09-10T20:38:55.211328+0000","last_became_active":"2024-09-10T20:38:52.934489+0000","last_became_peered":"2024-09-10T20:38:52.934489+0000","last_unstale":"2024-09-10T20:38:55.211328+0000","last_undegraded":"2024-09-10T20:38:55.211328+0000","last_fullsized":"2024-09-10T20:38:55.211328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:50.069288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959225+0000","last_change":"2024-09-10T20:38:53.957247+0000","last_active":"2024-09-10T20:38:54.959225+0000","last_peered":"2024-09-10T20:38:54.959225+0000","last_clean":"2024-09-10T20:38:54.959225+0000","last_became_active":"2024-09-10T20:38:53.956475+0000","last_became_peered":"2024-09-10T20:38:53.956475+0000","last_unstale":"2024-09-10T20:38:54.959225+0000","last_undegraded":"2024-09-10T20:38:54.959225+0000","last_fullsized":"2024-09-10T20:38:54.959225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:00:36.701444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959281+0000","last_change":"2024-09-10T20:38:52.941073+0000","last_active":"2024-09-10T20:38:54.959281+0000","last_peered":"2024-09-10T20:38:54.959281+0000","last_clean":"2024-09-10T20:38:54.959281+0000","last_became_active":"2024-09-10T20:38:52.940767+0000","last_became_peered":"2024-09-10T20:38:52.940767+0000","last_unstale":"2024-09-10T20:38:54.959281+0000","last_undegraded":"2024-09-10T20:38:54.959281+0000","last_fullsized":"2024-09-10T20:38:54.959281+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:41.632356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211265+0000","last_change":"2024-09-10T20:38:53.954329+0000","last_active":"2024-09-10T20:38:55.211265+0000","last_peered":"2024-09-10T20:38:55.211265+0000","last_clean":"2024-09-10T20:38:55.211265+0000","last_became_active":"2024-09-10T20:38:53.954037+0000","last_became_peered":"2024-09-10T20:38:53.954037+0000","last_unstale":"2024-09-10T20:38:55.211265+0000","last_undegraded":"2024-09-10T20:38:55.211265+0000","last_fullsized":"2024-09-10T20:38:55.211265+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:52.918936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958826+0000","last_change":"2024-09-10T20:38:52.938620+0000","last_active":"2024-09-10T20:38:54.958826+0000","last_peered":"2024-09-10T20:38:54.958826+0000","last_clean":"2024-09-10T20:38:54.958826+0000","last_became_active":"2024-09-10T20:38:52.938261+0000","last_became_peered":"2024-09-10T20:38:52.938261+0000","last_unstale":"2024-09-10T20:38:54.958826+0000","last_undegraded":"2024-09-10T20:38:54.958826+0000","last_fullsized":"2024-09-10T20:38:54.958826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:57:27.195064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211183+0000","last_change":"2024-09-10T20:38:53.968134+0000","last_active":"2024-09-10T20:38:55.211183+0000","last_peered":"2024-09-10T20:38:55.211183+0000","last_clean":"2024-09-10T20:38:55.211183+0000","last_became_active":"2024-09-10T20:38:53.967874+0000","last_became_peered":"2024-09-10T20:38:53.967874+0000","last_unstale":"2024-09-10T20:38:55.211183+0000","last_undegraded":"2024-09-10T20:38:55.211183+0000","last_fullsized":"2024-09-10T20:38:55.211183+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:42:59.105906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'8","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.965749+0000","last_change":"2024-09-10T20:38:52.943742+0000","last_active":"2024-09-10T20:38:53.965749+0000","last_peered":"2024-09-10T20:38:53.965749+0000","last_clean":"2024-09-10T20:38:53.965749+0000","last_became_active":"2024-09-10T20:38:52.943394+0000","last_became_peered":"2024-09-10T20:38:52.943394+0000","last_unstale":"2024-09-10T20:38:53.965749+0000","last_undegraded":"2024-09-10T20:38:53.965749+0000","last_fullsized":"2024-09-10T20:38:53.965749+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:12:49.651268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.212152+0000","last_change":"2024-09-10T20:38:53.961857+0000","last_active":"2024-09-10T20:38:55.212152+0000","last_peered":"2024-09-10T20:38:55.212152+0000","last_clean":"2024-09-10T20:38:55.212152+0000","last_became_active":"2024-09-10T20:38:53.961538+0000","last_became_peered":"2024-09-10T20:38:53.961538+0000","last_unstale":"2024-09-10T20:38:55.212152+0000","last_undegraded":"2024-09-10T20:38:55.212152+0000","last_fullsized":"2024-09-10T20:38:55.212152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:11.299175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"31'1","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961502+0000","last_change":"2024-09-10T20:38:52.942078+0000","last_active":"2024-09-10T20:38:53.961502+0000","last_peered":"2024-09-10T20:38:53.961502+0000","last_clean":"2024-09-10T20:38:53.961502+0000","last_became_active":"2024-09-10T20:38:52.941617+0000","last_became_peered":"2024-09-10T20:38:52.941617+0000","last_unstale":"2024-09-10T20:38:53.961502+0000","last_undegraded":"2024-09-10T20:38:53.961502+0000","last_fullsized":"2024-09-10T20:38:53.961502+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:58.849546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211586+0000","last_change":"2024-09-10T20:38:53.958839+0000","last_active":"2024-09-10T20:38:55.211586+0000","last_peered":"2024-09-10T20:38:55.211586+0000","last_clean":"2024-09-10T20:38:55.211586+0000","last_became_active":"2024-09-10T20:38:53.958547+0000","last_became_peered":"2024-09-10T20:38:53.958547+0000","last_unstale":"2024-09-10T20:38:55.211586+0000","last_undegraded":"2024-09-10T20:38:55.211586+0000","last_fullsized":"2024-09-10T20:38:55.211586+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:06:55.778360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211698+0000","last_change":"2024-09-10T20:38:52.932802+0000","last_active":"2024-09-10T20:38:55.211698+0000","last_peered":"2024-09-10T20:38:55.211698+0000","last_clean":"2024-09-10T20:38:55.211698+0000","last_became_active":"2024-09-10T20:38:52.932421+0000","last_became_peered":"2024-09-10T20:38:52.932421+0000","last_unstale":"2024-09-10T20:38:55.211698+0000","last_undegraded":"2024-09-10T20:38:55.211698+0000","last_fullsized":"2024-09-10T20:38:55.211698+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:56:23.033330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958478+0000","last_change":"2024-09-10T20:38:53.945772+0000","last_active":"2024-09-10T20:38:54.958478+0000","last_peered":"2024-09-10T20:38:54.958478+0000","last_clean":"2024-09-10T20:38:54.958478+0000","last_became_active":"2024-09-10T20:38:53.945499+0000","last_became_peered":"2024-09-10T20:38:53.945499+0000","last_unstale":"2024-09-10T20:38:54.958478+0000","last_undegraded":"2024-09-10T20:38:54.958478+0000","last_fullsized":"2024-09-10T20:38:54.958478+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:44.793307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211452+0000","last_change":"2024-09-10T20:38:52.943354+0000","last_active":"2024-09-10T20:38:55.211452+0000","last_peered":"2024-09-10T20:38:55.211452+0000","last_clean":"2024-09-10T20:38:55.211452+0000","last_became_active":"2024-09-10T20:38:52.942442+0000","last_became_peered":"2024-09-10T20:38:52.942442+0000","last_unstale":"2024-09-10T20:38:55.211452+0000","last_undegraded":"2024-09-10T20:38:55.211452+0000","last_fullsized":"2024-09-10T20:38:55.211452+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:15.041513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961602+0000","last_change":"2024-09-10T20:38:53.960281+0000","last_active":"2024-09-10T20:38:53.961602+0000","last_peered":"2024-09-10T20:38:53.961602+0000","last_clean":"2024-09-10T20:38:53.961602+0000","last_became_active":"2024-09-10T20:38:53.959983+0000","last_became_peered":"2024-09-10T20:38:53.959983+0000","last_unstale":"2024-09-10T20:38:53.961602+0000","last_undegraded":"2024-09-10T20:38:53.961602+0000","last_fullsized":"2024-09-10T20:38:53.961602+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:52.590068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955333+0000","last_change":"2024-09-10T20:38:52.942032+0000","last_active":"2024-09-10T20:38:54.955333+0000","last_peered":"2024-09-10T20:38:54.955333+0000","last_clean":"2024-09-10T20:38:54.955333+0000","last_became_active":"2024-09-10T20:38:52.941755+0000","last_became_peered":"2024-09-10T20:38:52.941755+0000","last_unstale":"2024-09-10T20:38:54.955333+0000","last_undegraded":"2024-09-10T20:38:54.955333+0000","last_fullsized":"2024-09-10T20:38:54.955333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:22.719553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.958287+0000","last_change":"2024-09-10T20:38:53.956597+0000","last_active":"2024-09-10T20:38:53.958287+0000","last_peered":"2024-09-10T20:38:53.958287+0000","last_clean":"2024-09-10T20:38:53.958287+0000","last_became_active":"2024-09-10T20:38:53.956337+0000","last_became_peered":"2024-09-10T20:38:53.956337+0000","last_unstale":"2024-09-10T20:38:53.958287+0000","last_undegraded":"2024-09-10T20:38:53.958287+0000","last_fullsized":"2024-09-10T20:38:53.958287+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:03.034001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958826+0000","last_change":"2024-09-10T20:38:52.936958+0000","last_active":"2024-09-10T20:38:54.958826+0000","last_peered":"2024-09-10T20:38:54.958826+0000","last_clean":"2024-09-10T20:38:54.958826+0000","last_became_active":"2024-09-10T20:38:52.936672+0000","last_became_peered":"2024-09-10T20:38:52.936672+0000","last_unstale":"2024-09-10T20:38:54.958826+0000","last_undegraded":"2024-09-10T20:38:54.958826+0000","last_fullsized":"2024-09-10T20:38:54.958826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:39.604422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958811+0000","last_change":"2024-09-10T20:38:53.948631+0000","last_active":"2024-09-10T20:38:54.958811+0000","last_peered":"2024-09-10T20:38:54.958811+0000","last_clean":"2024-09-10T20:38:54.958811+0000","last_became_active":"2024-09-10T20:38:53.948344+0000","last_became_peered":"2024-09-10T20:38:53.948344+0000","last_unstale":"2024-09-10T20:38:54.958811+0000","last_undegraded":"2024-09-10T20:38:54.958811+0000","last_fullsized":"2024-09-10T20:38:54.958811+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:14:49.187533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211138+0000","last_change":"2024-09-10T20:38:52.940795+0000","last_active":"2024-09-10T20:38:55.211138+0000","last_peered":"2024-09-10T20:38:55.211138+0000","last_clean":"2024-09-10T20:38:55.211138+0000","last_became_active":"2024-09-10T20:38:52.940448+0000","last_became_peered":"2024-09-10T20:38:52.940448+0000","last_unstale":"2024-09-10T20:38:55.211138+0000","last_undegraded":"2024-09-10T20:38:55.211138+0000","last_fullsized":"2024-09-10T20:38:55.211138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:47:38.085507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957116+0000","last_change":"2024-09-10T20:38:53.952723+0000","last_active":"2024-09-10T20:38:53.957116+0000","last_peered":"2024-09-10T20:38:53.957116+0000","last_clean":"2024-09-10T20:38:53.957116+0000","last_became_active":"2024-09-10T20:38:53.952399+0000","last_became_peered":"2024-09-10T20:38:53.952399+0000","last_unstale":"2024-09-10T20:38:53.957116+0000","last_undegraded":"2024-09-10T20:38:53.957116+0000","last_fullsized":"2024-09-10T20:38:53.957116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:13:27.524745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211916+0000","last_change":"2024-09-10T20:38:52.938916+0000","last_active":"2024-09-10T20:38:55.211916+0000","last_peered":"2024-09-10T20:38:55.211916+0000","last_clean":"2024-09-10T20:38:55.211916+0000","last_became_active":"2024-09-10T20:38:52.938627+0000","last_became_peered":"2024-09-10T20:38:52.938627+0000","last_unstale":"2024-09-10T20:38:55.211916+0000","last_undegraded":"2024-09-10T20:38:55.211916+0000","last_fullsized":"2024-09-10T20:38:55.211916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:53.493089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211804+0000","last_change":"2024-09-10T20:38:53.959523+0000","last_active":"2024-09-10T20:38:55.211804+0000","last_peered":"2024-09-10T20:38:55.211804+0000","last_clean":"2024-09-10T20:38:55.211804+0000","last_became_active":"2024-09-10T20:38:53.959228+0000","last_became_peered":"2024-09-10T20:38:53.959228+0000","last_unstale":"2024-09-10T20:38:55.211804+0000","last_undegraded":"2024-09-10T20:38:55.211804+0000","last_fullsized":"2024-09-10T20:38:55.211804+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:35.767672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.210980+0000","last_change":"2024-09-10T20:38:52.944389+0000","last_active":"2024-09-10T20:38:55.210980+0000","last_peered":"2024-09-10T20:38:55.210980+0000","last_clean":"2024-09-10T20:38:55.210980+0000","last_became_active":"2024-09-10T20:38:52.944082+0000","last_became_peered":"2024-09-10T20:38:52.944082+0000","last_unstale":"2024-09-10T20:38:55.210980+0000","last_undegraded":"2024-09-10T20:38:55.210980+0000","last_fullsized":"2024-09-10T20:38:55.210980+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:08.005402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.961568+0000","last_change":"2024-09-10T20:38:53.959514+0000","last_active":"2024-09-10T20:38:53.961568+0000","last_peered":"2024-09-10T20:38:53.961568+0000","last_clean":"2024-09-10T20:38:53.961568+0000","last_became_active":"2024-09-10T20:38:53.959256+0000","last_became_peered":"2024-09-10T20:38:53.959256+0000","last_unstale":"2024-09-10T20:38:53.961568+0000","last_undegraded":"2024-09-10T20:38:53.961568+0000","last_fullsized":"2024-09-10T20:38:53.961568+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:52:16.377413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'2","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.959736+0000","last_change":"2024-09-10T20:38:52.940948+0000","last_active":"2024-09-10T20:38:53.959736+0000","last_peered":"2024-09-10T20:38:53.959736+0000","last_clean":"2024-09-10T20:38:53.959736+0000","last_became_active":"2024-09-10T20:38:52.939610+0000","last_became_peered":"2024-09-10T20:38:52.939610+0000","last_unstale":"2024-09-10T20:38:53.959736+0000","last_undegraded":"2024-09-10T20:38:53.959736+0000","last_fullsized":"2024-09-10T20:38:53.959736+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:06:54.772555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.212546+0000","last_change":"2024-09-10T20:38:53.965477+0000","last_active":"2024-09-10T20:38:55.212546+0000","last_peered":"2024-09-10T20:38:55.212546+0000","last_clean":"2024-09-10T20:38:55.212546+0000","last_became_active":"2024-09-10T20:38:53.965178+0000","last_became_peered":"2024-09-10T20:38:53.965178+0000","last_unstale":"2024-09-10T20:38:55.212546+0000","last_undegraded":"2024-09-10T20:38:55.212546+0000","last_fullsized":"2024-09-10T20:38:55.212546+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:59:00.394081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959322+0000","last_change":"2024-09-10T20:38:52.938788+0000","last_active":"2024-09-10T20:38:54.959322+0000","last_peered":"2024-09-10T20:38:54.959322+0000","last_clean":"2024-09-10T20:38:54.959322+0000","last_became_active":"2024-09-10T20:38:52.937799+0000","last_became_peered":"2024-09-10T20:38:52.937799+0000","last_unstale":"2024-09-10T20:38:54.959322+0000","last_undegraded":"2024-09-10T20:38:54.959322+0000","last_fullsized":"2024-09-10T20:38:54.959322+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:08.661653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.957386+0000","last_change":"2024-09-10T20:38:53.955780+0000","last_active":"2024-09-10T20:38:53.957386+0000","last_peered":"2024-09-10T20:38:53.957386+0000","last_clean":"2024-09-10T20:38:53.957386+0000","last_became_active":"2024-09-10T20:38:53.955494+0000","last_became_peered":"2024-09-10T20:38:53.955494+0000","last_unstale":"2024-09-10T20:38:53.957386+0000","last_undegraded":"2024-09-10T20:38:53.957386+0000","last_fullsized":"2024-09-10T20:38:53.957386+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:45.472013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.944217+0000","last_change":"2024-09-10T20:38:52.938501+0000","last_active":"2024-09-10T20:38:53.944217+0000","last_peered":"2024-09-10T20:38:53.944217+0000","last_clean":"2024-09-10T20:38:53.944217+0000","last_became_active":"2024-09-10T20:38:52.938201+0000","last_became_peered":"2024-09-10T20:38:52.938201+0000","last_unstale":"2024-09-10T20:38:53.944217+0000","last_undegraded":"2024-09-10T20:38:53.944217+0000","last_fullsized":"2024-09-10T20:38:53.944217+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:50.069288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955341+0000","last_change":"2024-09-10T20:38:53.956867+0000","last_active":"2024-09-10T20:38:54.955341+0000","last_peered":"2024-09-10T20:38:54.955341+0000","last_clean":"2024-09-10T20:38:54.955341+0000","last_became_active":"2024-09-10T20:38:53.956504+0000","last_became_peered":"2024-09-10T20:38:53.956504+0000","last_unstale":"2024-09-10T20:38:54.955341+0000","last_undegraded":"2024-09-10T20:38:54.955341+0000","last_fullsized":"2024-09-10T20:38:54.955341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:30.808994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.946490+0000","last_change":"2024-09-10T20:38:52.937078+0000","last_active":"2024-09-10T20:38:53.946490+0000","last_peered":"2024-09-10T20:38:53.946490+0000","last_clean":"2024-09-10T20:38:53.946490+0000","last_became_active":"2024-09-10T20:38:52.936803+0000","last_became_peered":"2024-09-10T20:38:52.936803+0000","last_unstale":"2024-09-10T20:38:53.946490+0000","last_undegraded":"2024-09-10T20:38:53.946490+0000","last_fullsized":"2024-09-10T20:38:53.946490+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:44:29.761081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959573+0000","last_change":"2024-09-10T20:38:53.955680+0000","last_active":"2024-09-10T20:38:54.959573+0000","last_peered":"2024-09-10T20:38:54.959573+0000","last_clean":"2024-09-10T20:38:54.959573+0000","last_became_active":"2024-09-10T20:38:53.952088+0000","last_became_peered":"2024-09-10T20:38:53.952088+0000","last_unstale":"2024-09-10T20:38:54.959573+0000","last_undegraded":"2024-09-10T20:38:54.959573+0000","last_fullsized":"2024-09-10T20:38:54.959573+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:25:51.845077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-10T20:38:53.943486+0000","last_change":"2024-09-10T20:38:52.942225+0000","last_active":"2024-09-10T20:38:53.943486+0000","last_peered":"2024-09-10T20:38:53.943486+0000","last_clean":"2024-09-10T20:38:53.943486+0000","last_became_active":"2024-09-10T20:38:52.941887+0000","last_became_peered":"2024-09-10T20:38:52.941887+0000","last_unstale":"2024-09-10T20:38:53.943486+0000","last_undegraded":"2024-09-10T20:38:53.943486+0000","last_fullsized":"2024-09-10T20:38:53.943486+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:47:38.085507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.958736+0000","last_change":"2024-09-10T20:38:53.964770+0000","last_active":"2024-09-10T20:38:54.958736+0000","last_peered":"2024-09-10T20:38:54.958736+0000","last_clean":"2024-09-10T20:38:54.958736+0000","last_became_active":"2024-09-10T20:38:53.964456+0000","last_became_peered":"2024-09-10T20:38:53.964456+0000","last_unstale":"2024-09-10T20:38:54.958736+0000","last_undegraded":"2024-09-10T20:38:54.958736+0000","last_fullsized":"2024-09-10T20:38:54.958736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:35.620408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211858+0000","last_change":"2024-09-10T20:38:52.937518+0000","last_active":"2024-09-10T20:38:55.211858+0000","last_peered":"2024-09-10T20:38:55.211858+0000","last_clean":"2024-09-10T20:38:55.211858+0000","last_became_active":"2024-09-10T20:38:52.937288+0000","last_became_peered":"2024-09-10T20:38:52.937288+0000","last_unstale":"2024-09-10T20:38:55.211858+0000","last_undegraded":"2024-09-10T20:38:55.211858+0000","last_fullsized":"2024-09-10T20:38:55.211858+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:06:54.772555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211394+0000","last_change":"2024-09-10T20:38:53.958513+0000","last_active":"2024-09-10T20:38:55.211394+0000","last_peered":"2024-09-10T20:38:55.211394+0000","last_clean":"2024-09-10T20:38:55.211394+0000","last_became_active":"2024-09-10T20:38:53.958247+0000","last_became_peered":"2024-09-10T20:38:53.958247+0000","last_unstale":"2024-09-10T20:38:55.211394+0000","last_undegraded":"2024-09-10T20:38:55.211394+0000","last_fullsized":"2024-09-10T20:38:55.211394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:04.555451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.959476+0000","last_change":"2024-09-10T20:38:52.942992+0000","last_active":"2024-09-10T20:38:54.959476+0000","last_peered":"2024-09-10T20:38:54.959476+0000","last_clean":"2024-09-10T20:38:54.959476+0000","last_became_active":"2024-09-10T20:38:52.942733+0000","last_became_peered":"2024-09-10T20:38:52.942733+0000","last_unstale":"2024-09-10T20:38:54.959476+0000","last_undegraded":"2024-09-10T20:38:54.959476+0000","last_fullsized":"2024-09-10T20:38:54.959476+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:07:36.996717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:54.955407+0000","last_change":"2024-09-10T20:38:53.954985+0000","last_active":"2024-09-10T20:38:54.955407+0000","last_peered":"2024-09-10T20:38:54.955407+0000","last_clean":"2024-09-10T20:38:54.955407+0000","last_became_active":"2024-09-10T20:38:53.954703+0000","last_became_peered":"2024-09-10T20:38:53.954703+0000","last_unstale":"2024-09-10T20:38:54.955407+0000","last_undegraded":"2024-09-10T20:38:54.955407+0000","last_fullsized":"2024-09-10T20:38:54.955407+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:52.919556+0000","last_clean_scrub_stamp":"2024-09-10T20:38:52.919556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:11:29.648816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211500+0000","last_change":"2024-09-10T20:38:52.942556+0000","last_active":"2024-09-10T20:38:55.211500+0000","last_peered":"2024-09-10T20:38:55.211500+0000","last_clean":"2024-09-10T20:38:55.211500+0000","last_became_active":"2024-09-10T20:38:52.941998+0000","last_became_peered":"2024-09-10T20:38:52.941998+0000","last_unstale":"2024-09-10T20:38:55.211500+0000","last_undegraded":"2024-09-10T20:38:55.211500+0000","last_fullsized":"2024-09-10T20:38:55.211500+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:51.913751+0000","last_clean_scrub_stamp":"2024-09-10T20:38:51.913751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:07:31.927944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":129,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-10T20:38:55.211938+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:38:55.211938+0000","last_peered":"2024-09-10T20:38:55.211938+0000","last_clean":"2024-09-10T20:38:55.211938+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:38:55.211938+0000","last_undegraded":"2024-09-10T20:38:55.211938+0000","last_fullsized":"2024-09-10T20:38:55.211938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31552,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":368640,"data_stored":136503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509454,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":356352,"data_stored":135761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509454,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":950272,"data_stored":726165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":356352,"data_stored":135817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542158,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":344064,"data_stored":129449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542159,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":352256,"data_stored":131039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542157,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":937984,"data_stored":720377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":356352,"data_stored":131061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:38:58.325 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:38:58.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:38:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.470+0000 7f154ed33640 1 -- 172.21.15.28:0/2360522998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1530004d40 msgr2=0x7f15300051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.470+0000 7f154ed33640 1 --2- 172.21.15.28:0/2360522998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1530004d40 0x7f15300051a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f154000c640 tx=0x7f1530005830 comp rx=0 tx=0).stop 2024-09-10T20:38:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.470+0000 7f154d2d0640 1 -- 172.21.15.28:0/2360522998 reap_dead start 2024-09-10T20:38:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.470+0000 7f154ed33640 1 -- 172.21.15.28:0/2360522998 shutdown_connections 2024-09-10T20:38:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.470+0000 7f154ed33640 1 -- 172.21.15.28:0/2360522998 >> 172.21.15.28:0/2360522998 conn(0x7f154806cf00 msgr2=0x7f154806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.470+0000 7f154ed33640 1 -- 172.21.15.28:0/2360522998 shutdown_connections 2024-09-10T20:38:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.470+0000 7f154ed33640 1 -- 172.21.15.28:0/2360522998 wait complete. 2024-09-10T20:38:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.471+0000 7f154ed33640 1 Processor -- start 2024-09-10T20:38:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.471+0000 7f154ed33640 1 -- start start 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154ed33640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1530004d40 0x7f15480b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154ed33640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15481403b0 0x7f1548142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154ed33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f154815b440 0x7f15481f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154ed33640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1548070430 con 0x7f154815b440 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154ed33640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f15480aef70 con 0x7f15481403b0 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154ed33640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f15480705b0 con 0x7f1530004d40 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f154815b440 0x7f15481f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f154815b440 0x7f15481f5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56760/0 (socket says 172.21.15.28:56760) 2024-09-10T20:38:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 -- 172.21.15.28:0/1877118171 learned_addr learned my addr 172.21.15.28:0/1877118171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154cacf640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1530004d40 0x7f15480b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f1547fff640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15481403b0 0x7f1548142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 -- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1530004d40 msgr2=0x7f15480b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1530004d40 0x7f15480b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 -- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15481403b0 msgr2=0x7f1548142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15481403b0 0x7f1548142df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.472+0000 7f154d2d0640 1 -- 172.21.15.28:0/1877118171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15480b2530 con 0x7f154815b440 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.473+0000 7f154d2d0640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f154815b440 0x7f15481f5cd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f154000c760 tx=0x7f1540001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.473+0000 7f15457fa640 1 -- 172.21.15.28:0/1877118171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1540001f80 con 0x7f154815b440 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.473+0000 7f15457fa640 1 -- 172.21.15.28:0/1877118171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f154002b940 con 0x7f154815b440 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.473+0000 7f15457fa640 1 -- 172.21.15.28:0/1877118171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154002bd70 con 0x7f154815b440 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.473+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15481541f0 con 0x7f154815b440 2024-09-10T20:38:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.473+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15481547b0 con 0x7f154815b440 2024-09-10T20:38:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.474+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1548154010 con 0x7f154815b440 2024-09-10T20:38:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.474+0000 7f15457fa640 1 -- 172.21.15.28:0/1877118171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1540046060 con 0x7f154815b440 2024-09-10T20:38:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.475+0000 7f15457fa640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1534077fa0 0x7f153407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.475+0000 7f15457fa640 1 -- 172.21.15.28:0/1877118171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f15400b92c0 con 0x7f154815b440 2024-09-10T20:38:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.475+0000 7f154cacf640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1534077fa0 0x7f153407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.476+0000 7f154cacf640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1534077fa0 0x7f153407a460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1538006590 tx=0x7f1538006ba0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.479+0000 7f15457fa640 1 -- 172.21.15.28:0/1877118171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1548154010 con 0x7f154815b440 2024-09-10T20:38:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.736+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f15480aeb40 con 0x7f154815b440 2024-09-10T20:38:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.738+0000 7f15457fa640 1 -- 172.21.15.28:0/1877118171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) ==== 190+0+147 (secure 0 0 0) 0x7f15480aeb40 con 0x7f154815b440 2024-09-10T20:38:58.742 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:38:58.742 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:38:58.742 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:38:58.742 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:38:58.742 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:38:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:38:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1534077fa0 msgr2=0x7f153407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1534077fa0 0x7f153407a460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1538006590 tx=0x7f1538006ba0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f154815b440 msgr2=0x7f15481f5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f154815b440 0x7f15481f5cd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f154000c760 tx=0x7f1540001aa0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 shutdown_connections 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1534077fa0 0x7f153407a460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f154815b440 0x7f15481f5cd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15481403b0 0x7f1548142df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 --2- 172.21.15.28:0/1877118171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1530004d40 0x7f15480b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 >> 172.21.15.28:0/1877118171 conn(0x7f154806cf00 msgr2=0x7f1548075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 shutdown_connections 2024-09-10T20:38:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.741+0000 7f154ed33640 1 -- 172.21.15.28:0/1877118171 wait complete. 2024-09-10T20:38:58.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:38:58.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 -- 172.21.15.28:0/1447525659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd540cf780 msgr2=0x7fbd540cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/1447525659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd540cf780 0x7fbd540cfbf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbd5408c020 tx=0x7fbd4402a530 comp rx=0 tx=0).stop 2024-09-10T20:38:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 -- 172.21.15.28:0/1447525659 shutdown_connections 2024-09-10T20:38:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/1447525659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd540cf780 0x7fbd540cfbf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/1447525659 >> v2:172.21.15.159:3300/0 conn(0x7fbd5415de70 0x7fbd540d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 -- 172.21.15.28:0/1447525659 >> 172.21.15.28:0/1447525659 conn(0x7fbd540d3fb0 msgr2=0x7fbd540d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 -- 172.21.15.28:0/1447525659 shutdown_connections 2024-09-10T20:38:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.856+0000 7fbd5c5df640 1 -- 172.21.15.28:0/1447525659 wait complete. 2024-09-10T20:38:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 Processor -- start 2024-09-10T20:38:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 -- start start 2024-09-10T20:38:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbd540cf780 0x7fbd540cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd5415de70 0x7fbd5415fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd54161b50 0x7fbd54159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd540b7690 con 0x7fbd5415de70 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbd540b7510 con 0x7fbd54161b50 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.858+0000 7fbd5c5df640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbd540b7810 con 0x7fbd540cf780 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd5415de70 0x7fbd5415fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd5415de70 0x7fbd5415fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56774/0 (socket says 172.21.15.28:56774) 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 -- 172.21.15.28:0/216868656 learned_addr learned my addr 172.21.15.28:0/216868656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd5ab7c640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd54161b50 0x7fbd54159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd5a37b640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbd540cf780 0x7fbd540cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 -- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbd540cf780 msgr2=0x7fbd540cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbd540cf780 0x7fbd540cc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 -- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd54161b50 msgr2=0x7fbd54159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd54161b50 0x7fbd54159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 -- 172.21.15.28:0/216868656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd541f7fa0 con 0x7fbd5415de70 2024-09-10T20:38:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd5ab7c640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd54161b50 0x7fbd54159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd59b7a640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd5415de70 0x7fbd5415fec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009920 tx=0x7fbd44032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd4affd640 1 -- 172.21.15.28:0/216868656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44040070 con 0x7fbd5415de70 2024-09-10T20:38:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd4affd640 1 -- 172.21.15.28:0/216868656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd44005310 con 0x7fbd5415de70 2024-09-10T20:38:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd541f8280 con 0x7fbd5415de70 2024-09-10T20:38:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd4affd640 1 -- 172.21.15.28:0/216868656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44003800 con 0x7fbd5415de70 2024-09-10T20:38:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.859+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd541f8840 con 0x7fbd5415de70 2024-09-10T20:38:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.860+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd540d2b00 con 0x7fbd5415de70 2024-09-10T20:38:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.861+0000 7fbd4affd640 1 -- 172.21.15.28:0/216868656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbd4404c030 con 0x7fbd5415de70 2024-09-10T20:38:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.862+0000 7fbd4affd640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbd38077f70 0x7fbd3807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.862+0000 7fbd4affd640 1 -- 172.21.15.28:0/216868656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fbd440bafa0 con 0x7fbd5415de70 2024-09-10T20:38:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.862+0000 7fbd5a37b640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbd38077f70 0x7fbd3807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.862+0000 7fbd5a37b640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbd38077f70 0x7fbd3807a430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000a810 tx=0x7fbd50008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:58.866+0000 7fbd4affd640 1 -- 172.21.15.28:0/216868656 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbd540d2b00 con 0x7fbd5415de70 2024-09-10T20:38:58.916 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-10T20:38:58.912+0000 7f9713104640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:38:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.091+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd540cfbf0 con 0x7fbd5415de70 2024-09-10T20:38:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.091+0000 7fbd4affd640 1 -- 172.21.15.28:0/216868656 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5226 (secure 0 0 0) 0x7fbd44083aa0 con 0x7fbd5415de70 2024-09-10T20:38:59.096 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:38:59.096 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"btime":"2024-09-10T20:38:56:959095+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1687267914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1687267914},{"type":"v1","addr":"172.21.15.159:6836","nonce":1687267914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/649109556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":649109556},{"type":"v1","addr":"172.21.15.159:6837","nonce":649109556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4851,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2646744984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2646744984},{"type":"v1","addr":"172.21.15.28:6834","nonce":2646744984}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:53.302329+0000","modified":"2024-09-10T20:38:56.323633+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4854":{"gid":4854,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1884641453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1884641453},{"type":"v1","addr":"172.21.15.28:6835","nonce":1884641453}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4854,"qdb_cluster":[4854]},"id":2}]} 2024-09-10T20:38:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 16 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbd38077f70 msgr2=0x7fbd3807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbd38077f70 0x7fbd3807a430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000a810 tx=0x7fbd50008040 comp rx=0 tx=0).stop 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd5415de70 msgr2=0x7fbd5415fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd5415de70 0x7fbd5415fec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009920 tx=0x7fbd44032d50 comp rx=0 tx=0).stop 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 shutdown_connections 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbd38077f70 0x7fbd3807a430 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd54161b50 0x7fbd54159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd5415de70 0x7fbd5415fec0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 --2- 172.21.15.28:0/216868656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbd540cf780 0x7fbd540cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 >> 172.21.15.28:0/216868656 conn(0x7fbd540d3fb0 msgr2=0x7fbd540b7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 shutdown_connections 2024-09-10T20:38:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.094+0000 7fbd5c5df640 1 -- 172.21.15.28:0/216868656 wait complete. 2024-09-10T20:38:59.511 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:38:53.302329+0000', 'modified': '2024-09-10T20:38:56.323633+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4854}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4854': {'gid': 4854, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1884641453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1884641453}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1884641453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4854, 'qdb_cluster': [4854]} 2024-09-10T20:38:59.511 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:38:59.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-10T20:38:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 -- 172.21.15.28:0/4041190936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc0d2b90 msgr2=0x7ff3bc161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/4041190936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc0d2b90 0x7ff3bc161b50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0d01d0 tx=0x7ff3ac02a560 comp rx=0 tx=0).stop 2024-09-10T20:38:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 -- 172.21.15.28:0/4041190936 shutdown_connections 2024-09-10T20:38:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/4041190936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc0d2b90 0x7ff3bc161b50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/4041190936 >> v2:172.21.15.159:3300/0 conn(0x7ff3bc162090 0x7ff3bc0b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 -- 172.21.15.28:0/4041190936 >> 172.21.15.28:0/4041190936 conn(0x7ff3bc0d3fb0 msgr2=0x7ff3bc0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 -- 172.21.15.28:0/4041190936 shutdown_connections 2024-09-10T20:38:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.619+0000 7ff3c3e08640 1 -- 172.21.15.28:0/4041190936 wait complete. 2024-09-10T20:38:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 Processor -- start 2024-09-10T20:38:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 -- start start 2024-09-10T20:38:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3bc0d2b90 0x7ff3bc14e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc15de70 0x7ff3bc161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3bc162090 0x7ff3bc1f8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc15e3d0 con 0x7ff3bc15de70 2024-09-10T20:38:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc15e250 con 0x7ff3bc0d2b90 2024-09-10T20:38:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c3e08640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff3bc0b7340 con 0x7ff3bc162090 2024-09-10T20:38:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c13a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc15de70 0x7ff3bc161230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.621+0000 7ff3c13a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc15de70 0x7ff3bc161230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56790/0 (socket says 172.21.15.28:56790) 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2535456715 learned_addr learned my addr 172.21.15.28:0/2535456715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3bc0d2b90 0x7ff3bc14e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c23a5640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3bc162090 0x7ff3bc1f8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3bc162090 msgr2=0x7ff3bc1f8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3bc162090 0x7ff3bc1f8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3bc0d2b90 msgr2=0x7ff3bc14e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3bc0d2b90 0x7ff3bc14e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2535456715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3bc1f4a10 con 0x7ff3bc15de70 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc15de70 0x7ff3bc161230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0099c0 tx=0x7ff3ac0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3b27fc640 1 -- 172.21.15.28:0/2535456715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac03f070 con 0x7ff3bc15de70 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3b27fc640 1 -- 172.21.15.28:0/2535456715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3ac004890 con 0x7ff3bc15de70 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3bc14e730 con 0x7ff3bc15de70 2024-09-10T20:38:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3b27fc640 1 -- 172.21.15.28:0/2535456715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac035030 con 0x7ff3bc15de70 2024-09-10T20:38:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.622+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3bc14ecf0 con 0x7ff3bc15de70 2024-09-10T20:38:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.624+0000 7ff3b27fc640 1 -- 172.21.15.28:0/2535456715 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff3ac004ce0 con 0x7ff3bc15de70 2024-09-10T20:38:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.624+0000 7ff3b27fc640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff3a0077f70 0x7ff3a007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.624+0000 7ff3b27fc640 1 -- 172.21.15.28:0/2535456715 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7ff3ac0bafa0 con 0x7ff3bc15de70 2024-09-10T20:38:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.625+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff3a0077f70 0x7ff3a007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.625+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff3a0077f70 0x7ff3a007a430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0a1db0 tx=0x7ff3b8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 Processor -- start 2024-09-10T20:38:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 -- start start 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee23b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee6c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee215f0 0x56470ee21a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56470ed98820 con 0x56470ee4c1b0 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56470ed87c50 con 0x56470ee215f0 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56470ed8ba60 con 0x56470ee6bdf0 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c1ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee23b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c1ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee23b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56798/0 (socket says 172.21.15.28:56798) 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.626+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/3976957392 learned_addr learned my addr 172.21.15.28:0/3976957392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 msgr2=0x56470ee6c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee6c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee6c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c23a5640 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee215f0 0x56470ee21a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee215f0 msgr2=0x56470ee21a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee215f0 0x56470ee21a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/3976957392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56470eae9850 con 0x56470ee4c1b0 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee23b20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800fb70 tx=0x7ff3b8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6a71e0990c84df server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff39f7fe640 1 -- 172.21.15.28:0/3976957392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8025070 con 0x56470ee4c1b0 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff39f7fe640 1 -- 172.21.15.28:0/3976957392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3b801b050 con 0x56470ee4c1b0 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.627+0000 7ff39f7fe640 1 -- 172.21.15.28:0/3976957392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8020410 con 0x56470ee4c1b0 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/3976957392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 msgr2=0x56470ee23b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee23b20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800fb70 tx=0x7ff3b8012000 comp rx=0 tx=0).stop 2024-09-10T20:38:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/3976957392 shutdown_connections 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee215f0 0x56470ee21a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee6c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/3976957392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee23b20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/3976957392 >> 172.21.15.28:0/3976957392 conn(0x56470ee226a0 msgr2=0x56470ee260c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/3976957392 shutdown_connections 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/3976957392 wait complete. 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 -- start start 2024-09-10T20:38:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.628+0000 7ff3c6bdb740 1 Processor -- start 2024-09-10T20:38:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee7e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee46800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee4a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56470eaf8620 con 0x56470ee4c1b0 2024-09-10T20:38:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56470ed2d5f0 con 0x56470ee7ba50 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56470ee20b30 con 0x56470ee6bdf0 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c6bdb740 10 client.0 _ll_register_callbacks cb 0x56470ed310a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c1ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee7e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c1ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee7e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56810/0 (socket says 172.21.15.28:56810) 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.629+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/438085004 learned_addr learned my addr 172.21.15.28:0/438085004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c23a5640 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee4a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 msgr2=0x56470ee46800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee46800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee46800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 msgr2=0x56470ee4a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee4a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c1ba4640 1 -- 172.21.15.28:0/438085004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56470eb606b0 con 0x56470ee4c1b0 2024-09-10T20:38:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee7e890 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007fa0 tx=0x7ff3b8020a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f447b1c150a1fb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff39cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff39cff9640 1 -- 172.21.15.28:0/438085004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b802f070 con 0x56470ee4c1b0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff39cff9640 1 -- 172.21.15.28:0/438085004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7ff3b8007020 con 0x56470ee4c1b0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3b801b050 con 0x56470ee4c1b0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff39cff9640 1 -- 172.21.15.28:0/438085004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ff3b8037050 con 0x56470ee4c1b0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c6bdb740 10 client.5040 fetch_fsmap learned FSMap version 16 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x56470ee46d40 con 0x56470ee4c1b0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff39cff9640 1 -- 172.21.15.28:0/438085004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3b801b050 con 0x56470ee4c1b0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.630+0000 7ff39cff9640 1 -- 172.21.15.28:0/438085004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 16) ==== 3502+0+0 (secure 0 0 0) 0x7ff3b803a020 con 0x56470ee4c1b0 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c6bdb740 10 client.5040 fetch_fsmap finished waiting for FSMap version 16 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c6bdb740 10 client.5040 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c6bdb740 20 client.5040 populate_metadata read hostname 'smithi028' 2024-09-10T20:38:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee7a910 0x56470ee7acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c6bdb740 4 client.5040 mds_command: new command op to 4854 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 --> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56470ed9a770 con 0x56470ee7a910 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee7a910 0x56470ee7acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff39cff9640 10 client.5040 ms_handle_connect on v2:172.21.15.28:6833/1884641453 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.631+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee7a910 0x56470ee7acf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1725e22d13de253 server_cookie=33c22f19543c6243 in_seq=0 out_seq=0 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff39cff9640 1 -- 172.21.15.28:0/438085004 <== mds.0 v2:172.21.15.28:6833/1884641453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56470ed9a770 con 0x56470ee7a910 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff39cff9640 10 client.5040 handle_command_reply: tid=0 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 client.5040 shutdown 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 msgr2=0x56470ee7e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee7e890 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007fa0 tx=0x7ff3b8020a00 comp rx=0 tx=0).stop 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 shutdown_connections 2024-09-10T20:38:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee7a910 0x56470ee7acf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee4a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee6bdf0 0x56470ee46800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff39cff9640 0 client.5040 ms_handle_reset on v2:172.21.15.28:6833/1884641453 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/438085004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee4c1b0 0x56470ee7e890 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 >> 172.21.15.28:0/438085004 conn(0x56470ee215f0 msgr2=0x56470ee6cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 shutdown_connections 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/438085004 wait complete. 2024-09-10T20:38:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.632+0000 7ff3c6bdb740 20 client.5040 trim_cache size 0 max 16384 2024-09-10T20:38:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 Processor -- start 2024-09-10T20:38:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 -- start start 2024-09-10T20:38:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470ee7e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee6bdf0 0x56470ee7e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee7ba50 0x56470ee6ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc2003c0 con 0x56470ee7ba50 2024-09-10T20:38:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc15f310 con 0x56470ee6bdf0 2024-09-10T20:38:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.651+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff3bc1f55f0 con 0x56470ee4c1b0 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee7ba50 0x56470ee6ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee7ba50 0x56470ee6ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56814/0 (socket says 172.21.15.28:56814) 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 -- 172.21.15.28:0/1577278188 learned_addr learned my addr 172.21.15.28:0/1577278188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee6bdf0 0x56470ee7e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470ee7e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 -- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 msgr2=0x56470ee7e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470ee7e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 -- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee6bdf0 msgr2=0x56470ee7e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee6bdf0 0x56470ee7e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 -- 172.21.15.28:0/1577278188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56470ee1c5d0 con 0x56470ee7ba50 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee6bdf0 0x56470ee7e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff3c23a5640 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee7ba50 0x56470ee6ab30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009fa0 tx=0x7ff3b4009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913cfdbcb17b2e82 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff39effd640 1 -- 172.21.15.28:0/1577278188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b40157d0 con 0x56470ee7ba50 2024-09-10T20:38:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff39effd640 1 -- 172.21.15.28:0/1577278188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3b401b070 con 0x56470ee7ba50 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.652+0000 7ff39effd640 1 -- 172.21.15.28:0/1577278188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4015bf0 con 0x56470ee7ba50 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/1577278188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee7ba50 msgr2=0x56470ee6ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee7ba50 0x56470ee6ab30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009fa0 tx=0x7ff3b4009650 comp rx=0 tx=0).stop 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/1577278188 shutdown_connections 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee7ba50 0x56470ee6ab30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee6bdf0 0x56470ee7e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/1577278188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470ee7e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/1577278188 >> 172.21.15.28:0/1577278188 conn(0x56470ee215f0 msgr2=0x56470ee4b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/1577278188 shutdown_connections 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/1577278188 wait complete. 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 -- start start 2024-09-10T20:38:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.653+0000 7ff3c6bdb740 1 Processor -- start 2024-09-10T20:38:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470eeeb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee6bdf0 0x56470ee48fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c6bdb740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee494f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56470ee44550 con 0x56470ee6bdf0 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56470ee7c330 con 0x56470ee7ba50 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c6bdb740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56470ee7bec0 con 0x56470ee4c1b0 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c6bdb740 10 client.0 _ll_register_callbacks cb 0x7ff3b8023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c13a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee6bdf0 0x56470ee48fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c13a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee6bdf0 0x56470ee48fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56828/0 (socket says 172.21.15.28:56828) 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.654+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2508523386 learned_addr learned my addr 172.21.15.28:0/2508523386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c23a5640 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470eeeb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 msgr2=0x56470eeeb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470eeeb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 msgr2=0x56470ee494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c13a3640 1 -- 172.21.15.28:0/2508523386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56470ee84c20 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c13a3640 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee6bdf0 0x56470ee48fb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac02ab50 tx=0x7ff3ac0a5280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54b49b44e7d20c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff39cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff39cff9640 1 -- 172.21.15.28:0/2508523386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac049030 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff39cff9640 1 -- 172.21.15.28:0/2508523386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7ff3ac08a2b0 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3b401b070 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff39cff9640 1 -- 172.21.15.28:0/2508523386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ff3ac08c140 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c6bdb740 10 client.5046 fetch_fsmap learned FSMap version 16 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x56470ee6d160 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff39cff9640 1 -- 172.21.15.28:0/2508523386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3b401b070 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff39cff9640 1 -- 172.21.15.28:0/2508523386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 16) ==== 3502+0+0 (secure 0 0 0) 0x7ff3ac08c370 con 0x56470ee6bdf0 2024-09-10T20:38:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c6bdb740 10 client.5046 fetch_fsmap finished waiting for FSMap version 16 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c6bdb740 10 client.5046 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.655+0000 7ff3c6bdb740 20 client.5046 populate_metadata read hostname 'smithi028' 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee6a8d0 0x56470ee6acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 4 client.5046 mds_command: new command op to 4854 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 --> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] -- command(tid 0: {"prefix": "status"}) -- 0x56470ed96220 con 0x56470ee6a8d0 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee6a8d0 0x56470ee6acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c1ba4640 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee6a8d0 0x56470ee6acb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca9af39f5b786aeb server_cookie=1a4d244e13e0c52d in_seq=0 out_seq=0 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff39cff9640 10 client.5046 ms_handle_connect on v2:172.21.15.28:6833/1884641453 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff39cff9640 1 -- 172.21.15.28:0/2508523386 <== mds.0 v2:172.21.15.28:6833/1884641453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x56470ed96220 con 0x56470ee6a8d0 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff39cff9640 10 client.5046 handle_command_reply: tid=0 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 client.5046 shutdown 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee6bdf0 msgr2=0x56470ee48fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee6bdf0 0x56470ee48fb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac02ab50 tx=0x7ff3ac0a5280 comp rx=0 tx=0).stop 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 shutdown_connections 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:6833/1884641453,v1:172.21.15.28:6835/1884641453] conn(0x56470ee6a8d0 0x56470ee6acb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56470ee7ba50 0x56470ee494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56470ee6bdf0 0x56470ee48fb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 --2- 172.21.15.28:0/2508523386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56470ee4c1b0 0x56470eeeb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff39cff9640 0 client.5046 ms_handle_reset on v2:172.21.15.28:6833/1884641453 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 >> 172.21.15.28:0/2508523386 conn(0x56470ee215f0 msgr2=0x56470ee7acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 shutdown_connections 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.656+0000 7ff3c6bdb740 1 -- 172.21.15.28:0/2508523386 wait complete. 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c6bdb740 20 client.5046 trim_cache size 0 max 16384 2024-09-10T20:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4854, 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7223222710000003, 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 16, 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 32, 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 32, 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.609823036 2024-09-10T20:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:38:59.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff3a0077f70 msgr2=0x7ff3a007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff3a0077f70 0x7ff3a007a430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0a1db0 tx=0x7ff3b8009040 comp rx=0 tx=0).stop 2024-09-10T20:38:59.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc15de70 msgr2=0x7ff3bc161230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc15de70 0x7ff3bc161230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0099c0 tx=0x7ff3ac0039e0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 shutdown_connections 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff3a0077f70 0x7ff3a007a430 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3bc162090 0x7ff3bc1f8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3bc15de70 0x7ff3bc161230 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 --2- 172.21.15.28:0/2535456715 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3bc0d2b90 0x7ff3bc14e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 >> 172.21.15.28:0/2535456715 conn(0x7ff3bc0d3fb0 msgr2=0x7ff3bc06ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 shutdown_connections 2024-09-10T20:38:59.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.657+0000 7ff3c3e08640 1 -- 172.21.15.28:0/2535456715 wait complete. 2024-09-10T20:38:59.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:38:59.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.772+0000 7f341c6f7640 1 -- 172.21.15.28:0/2381169456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3414162260 msgr2=0x7f34140b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.772+0000 7f341c6f7640 1 --2- 172.21.15.28:0/2381169456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3414162260 0x7f34140b6e00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f34140d1d00 tx=0x7f340402a890 comp rx=0 tx=0).stop 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.772+0000 7f341c6f7640 1 -- 172.21.15.28:0/2381169456 shutdown_connections 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.772+0000 7f341c6f7640 1 --2- 172.21.15.28:0/2381169456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3414162260 0x7f34140b6e00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.772+0000 7f341c6f7640 1 -- 172.21.15.28:0/2381169456 >> 172.21.15.28:0/2381169456 conn(0x7f34140d3fb0 msgr2=0x7f34140d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.772+0000 7f341c6f7640 1 -- 172.21.15.28:0/2381169456 shutdown_connections 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.772+0000 7f341c6f7640 1 -- 172.21.15.28:0/2381169456 wait complete. 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.773+0000 7f341c6f7640 1 Processor -- start 2024-09-10T20:38:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341c6f7640 1 -- start start 2024-09-10T20:38:59.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341c6f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34140d2b90 0x7f341415b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341c6f7640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f341415de70 0x7f3414160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341c6f7640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3414162260 0x7f3414161080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341c6f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34140b7340 con 0x7f34140d2b90 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341c6f7640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f341415e460 con 0x7f341415de70 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341c6f7640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f34140b74c0 con 0x7f3414162260 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341a493640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34140d2b90 0x7f341415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341a493640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34140d2b90 0x7f341415b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56838/0 (socket says 172.21.15.28:56838) 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.774+0000 7f341a493640 1 -- 172.21.15.28:0/3056679878 learned_addr learned my addr 172.21.15.28:0/3056679878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f3419c92640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f341415de70 0x7f3414160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341ac94640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3414162260 0x7f3414161080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341a493640 1 -- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3414162260 msgr2=0x7f3414161080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341a493640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3414162260 0x7f3414161080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341a493640 1 -- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f341415de70 msgr2=0x7f3414160b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341a493640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f341415de70 0x7f3414160b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341a493640 1 -- 172.21.15.28:0/3056679878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34141f9050 con 0x7f34140d2b90 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f3419c92640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f341415de70 0x7f3414160b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:38:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341a493640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34140d2b90 0x7f341415b080 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f340c00c430 tx=0x7f340c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f3412ffd640 1 -- 172.21.15.28:0/3056679878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c011790 con 0x7f34140d2b90 2024-09-10T20:38:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f3412ffd640 1 -- 172.21.15.28:0/3056679878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f340c017070 con 0x7f34140d2b90 2024-09-10T20:38:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34141f4c80 con 0x7f34140d2b90 2024-09-10T20:38:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f3412ffd640 1 -- 172.21.15.28:0/3056679878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c011d00 con 0x7f34140d2b90 2024-09-10T20:38:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.775+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34141f5240 con 0x7f34140d2b90 2024-09-10T20:38:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.776+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34140b7960 con 0x7f34140d2b90 2024-09-10T20:38:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.777+0000 7f3412ffd640 1 -- 172.21.15.28:0/3056679878 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f340c01c460 con 0x7f34140d2b90 2024-09-10T20:38:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.777+0000 7f3412ffd640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3400077f70 0x7f340007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:38:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.778+0000 7f3412ffd640 1 -- 172.21.15.28:0/3056679878 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f340c099bf0 con 0x7f34140d2b90 2024-09-10T20:38:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.778+0000 7f3419c92640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3400077f70 0x7f340007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:38:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.778+0000 7f3419c92640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3400077f70 0x7f340007a430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34141603d0 tx=0x7f3408009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:38:59.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:38:59.782+0000 7f3412ffd640 1 -- 172.21.15.28:0/3056679878 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f340c062780 con 0x7f34140d2b90 2024-09-10T20:39:00.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.008+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f34140b9f30 con 0x7f34140d2b90 2024-09-10T20:39:00.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.009+0000 7f3412ffd640 1 -- 172.21.15.28:0/3056679878 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f340c062080 con 0x7f34140d2b90 2024-09-10T20:39:00.013 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3400077f70 msgr2=0x7f340007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3400077f70 0x7f340007a430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34141603d0 tx=0x7f3408009040 comp rx=0 tx=0).stop 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34140d2b90 msgr2=0x7f341415b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34140d2b90 0x7f341415b080 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f340c00c430 tx=0x7f340c00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 shutdown_connections 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3400077f70 0x7f340007a430 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3414162260 0x7f3414161080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f341415de70 0x7f3414160b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 --2- 172.21.15.28:0/3056679878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34140d2b90 0x7f341415b080 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 >> 172.21.15.28:0/3056679878 conn(0x7f34140d3fb0 msgr2=0x7f34140b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 shutdown_connections 2024-09-10T20:39:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.011+0000 7f341c6f7640 1 -- 172.21.15.28:0/3056679878 wait complete. 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:39:00.026 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:39:00.027 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:39:00.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:00.027 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:39:00.044 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:39:00.044 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:39:00.044 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:39:00.044 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:00.044 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:39:00.045 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42691sec preferred_lft 42691sec 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:00.046 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:00.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:00.048 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:39:00.109 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:39:00.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:00.110 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:39:00.168 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:39:00.168 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:39:00.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:39:00.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:39:00.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:39:00.168 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:39:00.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:39:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:00.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:00.297 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:39:00.313 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:39:00.313 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:39:00.313 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:39:00.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:00.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:39:00.380 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:00.380 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:00.435 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:39:00.435 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:00.435 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:00.491 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:39:00.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:00.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:00.576 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:00.576 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:39:00.576 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:39:00.585 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:00.585 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:39:00.585 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:39:00.585 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:00.585 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:00.590 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:00.590 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:39:00.590 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:39:00.590 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:00.590 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:39:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:39:00.593 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:39:00.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:00.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:00.619 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:39:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:39:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout:client5058 2024-09-10T20:39:00.722 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:00.722 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:00.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:39:00.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:39:00.825 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:39:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.883+0000 7fe06e008640 1 -- 172.21.15.28:0/3318266729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 msgr2=0x7fe050004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.883+0000 7fe06e008640 1 --2- 172.21.15.28:0/3318266729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 0x7fe050004c80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009b50 tx=0x7fe050005830 comp rx=0 tx=0).stop 2024-09-10T20:39:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.883+0000 7fe067fff640 1 -- 172.21.15.28:0/3318266729 reap_dead start 2024-09-10T20:39:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.883+0000 7fe06e008640 1 -- 172.21.15.28:0/3318266729 shutdown_connections 2024-09-10T20:39:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.883+0000 7fe06e008640 1 -- 172.21.15.28:0/3318266729 >> 172.21.15.28:0/3318266729 conn(0x7fe06806cf00 msgr2=0x7fe06806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.883+0000 7fe06e008640 1 -- 172.21.15.28:0/3318266729 shutdown_connections 2024-09-10T20:39:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.883+0000 7fe06e008640 1 -- 172.21.15.28:0/3318266729 wait complete. 2024-09-10T20:39:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.884+0000 7fe06e008640 1 Processor -- start 2024-09-10T20:39:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.884+0000 7fe06e008640 1 -- start start 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe06e008640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 0x7fe068153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe06e008640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe068157690 0x7fe0681555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe06e008640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0680b23e0 0x7fe0681523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe06e008640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0680d0500 con 0x7fe050004800 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe06e008640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe0680cc830 con 0x7fe068157690 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe06e008640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe0680d0680 con 0x7fe0680b23e0 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 0x7fe068153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 0x7fe068153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49924/0 (socket says 172.21.15.28:49924) 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 -- 172.21.15.28:0/1252939592 learned_addr learned my addr 172.21.15.28:0/1252939592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe067fff640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0680b23e0 0x7fe0681523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe066ffd640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe068157690 0x7fe0681555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 -- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0680b23e0 msgr2=0x7fe0681523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0680b23e0 0x7fe0681523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 -- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe068157690 msgr2=0x7fe0681555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe068157690 0x7fe0681555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 -- 172.21.15.28:0/1252939592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe068154760 con 0x7fe050004800 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.885+0000 7fe0677fe640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 0x7fe068153f30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe058011a50 tx=0x7fe058011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.886+0000 7fe04ffff640 1 -- 172.21.15.28:0/1252939592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05801f070 con 0x7fe050004800 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.886+0000 7fe04ffff640 1 -- 172.21.15.28:0/1252939592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe05800fdc0 con 0x7fe050004800 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.886+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0680af810 con 0x7fe050004800 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.886+0000 7fe04ffff640 1 -- 172.21.15.28:0/1252939592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0580126b0 con 0x7fe050004800 2024-09-10T20:39:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.886+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0680afdd0 con 0x7fe050004800 2024-09-10T20:39:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.887+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0680b47a0 con 0x7fe050004800 2024-09-10T20:39:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.887+0000 7fe04ffff640 1 -- 172.21.15.28:0/1252939592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe058012850 con 0x7fe050004800 2024-09-10T20:39:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.888+0000 7fe04ffff640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe054077f70 0x7fe05407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.888+0000 7fe04ffff640 1 -- 172.21.15.28:0/1252939592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fe05809a950 con 0x7fe050004800 2024-09-10T20:39:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.888+0000 7fe066ffd640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe054077f70 0x7fe05407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.888+0000 7fe066ffd640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe054077f70 0x7fe05407a430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe05c035000 tx=0x7fe05c02d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:00.892+0000 7fe04ffff640 1 -- 172.21.15.28:0/1252939592 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe058062e10 con 0x7fe050004800 2024-09-10T20:39:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.117+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe06806ff00 con 0x7fe050004800 2024-09-10T20:39:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.117+0000 7fe04ffff640 1 -- 172.21.15.28:0/1252939592 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7fe058067840 con 0x7fe050004800 2024-09-10T20:39:01.121 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:39:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe054077f70 msgr2=0x7fe05407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe054077f70 0x7fe05407a430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe05c035000 tx=0x7fe05c02d040 comp rx=0 tx=0).stop 2024-09-10T20:39:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 msgr2=0x7fe068153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 0x7fe068153f30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe058011a50 tx=0x7fe058011f10 comp rx=0 tx=0).stop 2024-09-10T20:39:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 shutdown_connections 2024-09-10T20:39:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe054077f70 0x7fe05407a430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0680b23e0 0x7fe0681523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe068157690 0x7fe0681555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 --2- 172.21.15.28:0/1252939592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe050004800 0x7fe068153f30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.119+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 >> 172.21.15.28:0/1252939592 conn(0x7fe06806cf00 msgr2=0x7fe06815ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.120+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 shutdown_connections 2024-09-10T20:39:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.120+0000 7fe06e008640 1 -- 172.21.15.28:0/1252939592 wait complete. 2024-09-10T20:39:01.134 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:39:01.134 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:39:01.134 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:39:01.134 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:39:01.134 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:01.135 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:39:01.135 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:39:01.135 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:39:01.135 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:39:01.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:01.135 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:39:01.156 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:39:01.156 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:39:01.156 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:39:01.157 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:01.157 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:39:01.157 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:01.157 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:01.157 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:01.157 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:39:01.158 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:01.158 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:01.158 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:39:01.158 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:01.158 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:01.158 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42708sec preferred_lft 42708sec 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:39:01.159 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:39:01.160 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:39:01.161 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:01.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:01.161 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:39:01.226 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:39:01.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:01.228 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:39:01.286 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:39:01.286 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:39:01.286 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:39:01.286 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:39:01.286 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:39:01.287 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:39:01.287 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:39:01.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:39:01.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:39:01.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:39:01.289 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:39:01.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:39:01.408 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:01.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:01.414 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:39:01.430 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:39:01.430 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:39:01.430 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:39:01.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:01.430 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:39:01.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:01.496 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:01.553 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:39:01.554 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:01.554 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:01.609 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:39:01.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:01.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:01.695 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:01.695 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:39:01.695 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:39:01.704 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:01.704 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:39:01.704 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:39:01.705 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:01.705 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:01.715 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:01.715 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:39:01.715 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:39:01.715 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:01.715 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:01.715 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:39:01.716 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:39:01.716 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:39:01.719 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:39:01.719 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:39:01.719 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:39:01.720 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:39:01.720 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:39:01.720 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:39:01.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:01.720 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:01.746 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:39:01.836 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:39:01.838 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:01.844 INFO:teuthology.orchestra.run.smithi159.stdout:client4757 2024-09-10T20:39:01.845 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:01.845 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:01.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:39:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 -- 172.21.15.28:0/4284570907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 msgr2=0x7f29d00d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 --2- 172.21.15.28:0/4284570907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d00d1c60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f29c8005fd0 tx=0x7f29c802a940 comp rx=0 tx=0).stop 2024-09-10T20:39:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 -- 172.21.15.28:0/4284570907 shutdown_connections 2024-09-10T20:39:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 --2- 172.21.15.28:0/4284570907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d00d1c60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 --2- 172.21.15.28:0/4284570907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d0161850 0x7f29d00d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 -- 172.21.15.28:0/4284570907 >> 172.21.15.28:0/4284570907 conn(0x7f29d00d3f90 msgr2=0x7f29d00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 -- 172.21.15.28:0/4284570907 shutdown_connections 2024-09-10T20:39:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.979+0000 7f29d6c52640 1 -- 172.21.15.28:0/4284570907 wait complete. 2024-09-10T20:39:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.980+0000 7f29d6c52640 1 Processor -- start 2024-09-10T20:39:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.980+0000 7f29d6c52640 1 -- start start 2024-09-10T20:39:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d6c52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d0147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d6c52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d0161850 0x7f29d01450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d6c52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d0162000 0x7f29d00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d6c52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29d00d2320 con 0x7f29d00b4c20 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d6c52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f29d00d21a0 con 0x7f29d0161850 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d6c52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f29d00d24a0 con 0x7f29d0162000 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d49ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d0147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d49ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d0147120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49954/0 (socket says 172.21.15.28:49954) 2024-09-10T20:39:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d49ee640 1 -- 172.21.15.28:0/777310689 learned_addr learned my addr 172.21.15.28:0/777310689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29d51ef640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d0162000 0x7f29d00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.981+0000 7f29cffff640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d0161850 0x7f29d01450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d49ee640 1 -- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d0162000 msgr2=0x7f29d00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d49ee640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d0162000 0x7f29d00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d49ee640 1 -- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d0161850 msgr2=0x7f29d01450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d49ee640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d0161850 0x7f29d01450c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d49ee640 1 -- 172.21.15.28:0/777310689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d0147710 con 0x7f29d00b4c20 2024-09-10T20:39:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d49ee640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d0147120 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f29c000c430 tx=0x7f29c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29cd7fa640 1 -- 172.21.15.28:0/777310689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0011790 con 0x7f29d00b4c20 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29cd7fa640 1 -- 172.21.15.28:0/777310689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29c0017070 con 0x7f29d00b4c20 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d01479f0 con 0x7f29d00b4c20 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29cd7fa640 1 -- 172.21.15.28:0/777310689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0011ce0 con 0x7f29d00b4c20 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.982+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d01f4cd0 con 0x7f29d00b4c20 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.983+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d00d1c60 con 0x7f29d00b4c20 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.984+0000 7f29cd7fa640 1 -- 172.21.15.28:0/777310689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f29c001c460 con 0x7f29d00b4c20 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.984+0000 7f29cd7fa640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b0077f70 0x7f29b007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.984+0000 7f29cd7fa640 1 -- 172.21.15.28:0/777310689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f29c0099970 con 0x7f29d00b4c20 2024-09-10T20:39:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.984+0000 7f29cffff640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b0077f70 0x7f29b007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.985+0000 7f29cffff640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b0077f70 0x7f29b007a430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f29d0160cc0 tx=0x7f29b8005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:01.989+0000 7f29cd7fa640 1 -- 172.21.15.28:0/777310689 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f29c0061e20 con 0x7f29d00b4c20 2024-09-10T20:39:02.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.213+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f29d00d0ba0 con 0x7f29d00b4c20 2024-09-10T20:39:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.979+0000 7f29cd7fa640 1 -- 172.21.15.28:0/777310689 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v17) ==== 136+0+0 (secure 0 0 0) 0x7f29c0066850 con 0x7f29d00b4c20 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.982+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b0077f70 msgr2=0x7f29b007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.982+0000 7f29d6c52640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b0077f70 0x7f29b007a430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f29d0160cc0 tx=0x7f29b8005f90 comp rx=0 tx=0).stop 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.982+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 msgr2=0x7f29d0147120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.982+0000 7f29d6c52640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d0147120 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f29c000c430 tx=0x7f29c000c8f0 comp rx=0 tx=0).stop 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 shutdown_connections 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b0077f70 0x7f29b007a430 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d0162000 0x7f29d00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d0161850 0x7f29d01450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 --2- 172.21.15.28:0/777310689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d00b4c20 0x7f29d0147120 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 >> 172.21.15.28:0/777310689 conn(0x7f29d00d3f90 msgr2=0x7f29d0160f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 shutdown_connections 2024-09-10T20:39:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:02.983+0000 7f29d6c52640 1 -- 172.21.15.28:0/777310689 wait complete. 2024-09-10T20:39:03.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:39:03.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 -- 172.21.15.28:0/2487271805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bb40 msgr2=0x7fc0400af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 --2- 172.21.15.28:0/2487271805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bb40 0x7fc0400af660 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009900 tx=0x7fc03c02a610 comp rx=0 tx=0).stop 2024-09-10T20:39:03.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 -- 172.21.15.28:0/2487271805 shutdown_connections 2024-09-10T20:39:03.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 --2- 172.21.15.28:0/2487271805 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc04015bf20 0x7fc0400afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 --2- 172.21.15.28:0/2487271805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bb40 0x7fc0400af660 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 -- 172.21.15.28:0/2487271805 >> 172.21.15.28:0/2487271805 conn(0x7fc04006dbf0 msgr2=0x7fc04006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 -- 172.21.15.28:0/2487271805 shutdown_connections 2024-09-10T20:39:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.109+0000 7fc047910640 1 -- 172.21.15.28:0/2487271805 wait complete. 2024-09-10T20:39:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.110+0000 7fc047910640 1 Processor -- start 2024-09-10T20:39:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.110+0000 7fc047910640 1 -- start start 2024-09-10T20:39:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc047910640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc04006cae0 0x7fc0400d1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc047910640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc04015bb40 0x7fc0401451b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc047910640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bf20 0x7fc0401456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc047910640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc04006d3a0 con 0x7fc04015bf20 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc047910640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc04006d220 con 0x7fc04015bb40 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc047910640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc0400b4330 con 0x7fc04006cae0 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bf20 0x7fc0401456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bf20 0x7fc0401456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49964/0 (socket says 172.21.15.28:49964) 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 -- 172.21.15.28:0/627040132 learned_addr learned my addr 172.21.15.28:0/627040132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc0456ac640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc04006cae0 0x7fc0400d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc044eab640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc04015bb40 0x7fc0401451b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 -- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc04006cae0 msgr2=0x7fc0400d1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc04006cae0 0x7fc0400d1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 -- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc04015bb40 msgr2=0x7fc0401451b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc04015bb40 0x7fc0401451b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.111+0000 7fc045ead640 1 -- 172.21.15.28:0/627040132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc040156a60 con 0x7fc04015bf20 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.112+0000 7fc045ead640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bf20 0x7fc0401456f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc03800c7d0 tx=0x7fc03800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.112+0000 7fc035ffb640 1 -- 172.21.15.28:0/627040132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc038003b70 con 0x7fc04015bf20 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.112+0000 7fc035ffb640 1 -- 172.21.15.28:0/627040132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc038003d30 con 0x7fc04015bf20 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.112+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc040145d70 con 0x7fc04015bf20 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.112+0000 7fc035ffb640 1 -- 172.21.15.28:0/627040132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03800aab0 con 0x7fc04015bf20 2024-09-10T20:39:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.112+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc040143170 con 0x7fc04015bf20 2024-09-10T20:39:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.113+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0400b47b0 con 0x7fc04015bf20 2024-09-10T20:39:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.113+0000 7fc035ffb640 1 -- 172.21.15.28:0/627040132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc03800ac50 con 0x7fc04015bf20 2024-09-10T20:39:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.114+0000 7fc035ffb640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc024077f70 0x7fc02407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.114+0000 7fc035ffb640 1 -- 172.21.15.28:0/627040132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fc038099ad0 con 0x7fc04015bf20 2024-09-10T20:39:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.114+0000 7fc0456ac640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc024077f70 0x7fc02407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.115+0000 7fc0456ac640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc024077f70 0x7fc02407a430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc03c033000 tx=0x7fc03c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.119+0000 7fc035ffb640 1 -- 172.21.15.28:0/627040132 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc0380625d0 con 0x7fc04015bf20 2024-09-10T20:39:03.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.328+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fc040074500 con 0x7fc024077f70 2024-09-10T20:39:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.330+0000 7fc035ffb640 1 -- 172.21.15.28:0/627040132 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fc040074500 con 0x7fc024077f70 2024-09-10T20:39:03.333 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:39:03.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:03.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:39:03.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:03.334 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:39:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc024077f70 msgr2=0x7fc02407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc024077f70 0x7fc02407a430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc03c033000 tx=0x7fc03c004730 comp rx=0 tx=0).stop 2024-09-10T20:39:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bf20 msgr2=0x7fc0401456f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bf20 0x7fc0401456f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc03800c7d0 tx=0x7fc03800cc90 comp rx=0 tx=0).stop 2024-09-10T20:39:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 shutdown_connections 2024-09-10T20:39:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc024077f70 0x7fc02407a430 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04015bf20 0x7fc0401456f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc04015bb40 0x7fc0401451b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 --2- 172.21.15.28:0/627040132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc04006cae0 0x7fc0400d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 >> 172.21.15.28:0/627040132 conn(0x7fc04006dbf0 msgr2=0x7fc04006dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 shutdown_connections 2024-09-10T20:39:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.332+0000 7fc047910640 1 -- 172.21.15.28:0/627040132 wait complete. 2024-09-10T20:39:03.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 -- 172.21.15.28:0/2832072491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 msgr2=0x7f782815cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 --2- 172.21.15.28:0/2832072491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f782815cbd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f782806db50 tx=0x7f782002a820 comp rx=0 tx=0).stop 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 -- 172.21.15.28:0/2832072491 shutdown_connections 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 --2- 172.21.15.28:0/2832072491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f782815cbd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 --2- 172.21.15.28:0/2832072491 >> v2:172.21.15.159:3300/0 conn(0x7f78281587a0 0x7f782815b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 -- 172.21.15.28:0/2832072491 >> 172.21.15.28:0/2832072491 conn(0x7f782806d110 msgr2=0x7f78280b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 -- 172.21.15.28:0/2832072491 shutdown_connections 2024-09-10T20:39:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.452+0000 7f782ea16640 1 -- 172.21.15.28:0/2832072491 wait complete. 2024-09-10T20:39:03.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.453+0000 7f782ea16640 1 Processor -- start 2024-09-10T20:39:03.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.453+0000 7f782ea16640 1 -- start start 2024-09-10T20:39:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782ea16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f78280af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782ea16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78281587a0 0x7f7828143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782ea16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f782815e7d0 0x7f78281436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782ea16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f782815d4e0 con 0x7f782806fc20 2024-09-10T20:39:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782ea16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f782815d360 con 0x7f78281587a0 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782ea16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f782815d660 con 0x7f782815e7d0 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f78280af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f78280af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49982/0 (socket says 172.21.15.28:49982) 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 -- 172.21.15.28:0/1615998820 learned_addr learned my addr 172.21.15.28:0/1615998820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782cfb3640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f782815e7d0 0x7f78281436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f78277fe640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78281587a0 0x7f7828143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 -- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f782815e7d0 msgr2=0x7f78281436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f782815e7d0 0x7f78281436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 -- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78281587a0 msgr2=0x7f7828143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78281587a0 0x7f7828143170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 -- 172.21.15.28:0/1615998820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78280b1270 con 0x7f782806fc20 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f782cfb3640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f782815e7d0 0x7f78281436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f78277fe640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78281587a0 0x7f7828143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.454+0000 7f7827fff640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f78280af0b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7818009970 tx=0x7f7818009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.455+0000 7f7824ff9640 1 -- 172.21.15.28:0/1615998820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781800d020 con 0x7f782806fc20 2024-09-10T20:39:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.455+0000 7f7824ff9640 1 -- 172.21.15.28:0/1615998820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f781800d1c0 con 0x7f782806fc20 2024-09-10T20:39:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.455+0000 7f7824ff9640 1 -- 172.21.15.28:0/1615998820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818007ac0 con 0x7f782806fc20 2024-09-10T20:39:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.455+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78280b06f0 con 0x7f782806fc20 2024-09-10T20:39:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.455+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78280b0cb0 con 0x7f782806fc20 2024-09-10T20:39:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.456+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f782815db80 con 0x7f782806fc20 2024-09-10T20:39:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.456+0000 7f7824ff9640 1 -- 172.21.15.28:0/1615998820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7818007c60 con 0x7f782806fc20 2024-09-10T20:39:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.457+0000 7f7824ff9640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7814077f70 0x7f781407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.457+0000 7f7824ff9640 1 -- 172.21.15.28:0/1615998820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f781809b810 con 0x7f782806fc20 2024-09-10T20:39:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.457+0000 7f78277fe640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7814077f70 0x7f781407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.458+0000 7f78277fe640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7814077f70 0x7f781407a430 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f78281f4d10 tx=0x7f781c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.461+0000 7f7824ff9640 1 -- 172.21.15.28:0/1615998820 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f78180643a0 con 0x7f782806fc20 2024-09-10T20:39:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.646+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f78280b21f0 con 0x7f782806fc20 2024-09-10T20:39:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.647+0000 7f7824ff9640 1 -- 172.21.15.28:0/1615998820 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f7818063ca0 con 0x7f782806fc20 2024-09-10T20:39:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7814077f70 msgr2=0x7f781407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7814077f70 0x7f781407a430 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f78281f4d10 tx=0x7f781c009040 comp rx=0 tx=0).stop 2024-09-10T20:39:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 msgr2=0x7f78280af0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f78280af0b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7818009970 tx=0x7f7818009e30 comp rx=0 tx=0).stop 2024-09-10T20:39:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 shutdown_connections 2024-09-10T20:39:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7814077f70 0x7f781407a430 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f782815e7d0 0x7f78281436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78281587a0 0x7f7828143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 --2- 172.21.15.28:0/1615998820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f782806fc20 0x7f78280af0b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:03.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 >> 172.21.15.28:0/1615998820 conn(0x7f782806d110 msgr2=0x7f7828073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:03.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 shutdown_connections 2024-09-10T20:39:03.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:03.649+0000 7f782ea16640 1 -- 172.21.15.28:0/1615998820 wait complete. 2024-09-10T20:39:03.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:03.665 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp61360/subvol51900) 2024-09-10T20:39:03.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:03.696 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:39:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:39:03.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:03.754 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp61360/subvol51900) 2024-09-10T20:39:03.831 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:03.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp61360/subvol51900/.meta 2024-09-10T20:39:03.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol51900 subvol_grp61360 2024-09-10T20:39:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.047+0000 7fbaae29b640 1 -- 172.21.15.28:0/1070428411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 msgr2=0x7fbaa80cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.047+0000 7fbaae29b640 1 --2- 172.21.15.28:0/1070428411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa80cb040 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fba90009830 tx=0x7fba9002a5a0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.047+0000 7fbaae29b640 1 -- 172.21.15.28:0/1070428411 shutdown_connections 2024-09-10T20:39:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.047+0000 7fbaae29b640 1 --2- 172.21.15.28:0/1070428411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa80cb040 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.047+0000 7fbaae29b640 1 -- 172.21.15.28:0/1070428411 >> 172.21.15.28:0/1070428411 conn(0x7fbaa80d0410 msgr2=0x7fbaa80d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.047+0000 7fbaae29b640 1 -- 172.21.15.28:0/1070428411 shutdown_connections 2024-09-10T20:39:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.047+0000 7fbaae29b640 1 -- 172.21.15.28:0/1070428411 wait complete. 2024-09-10T20:39:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.048+0000 7fbaae29b640 1 Processor -- start 2024-09-10T20:39:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.048+0000 7fbaae29b640 1 -- start start 2024-09-10T20:39:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaae29b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa80cb580 0x7fbaa81f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaae29b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa8155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaae29b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa815e4a0 0x7fbaa80b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaae29b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa806d850 con 0x7fbaa80cef00 2024-09-10T20:39:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaae29b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa8073190 con 0x7fbaa80cb580 2024-09-10T20:39:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaae29b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa806d9d0 con 0x7fbaa815e4a0 2024-09-10T20:39:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa8155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa8155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49992/0 (socket says 172.21.15.28:49992) 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 -- 172.21.15.28:0/3965845071 learned_addr learned my addr 172.21.15.28:0/3965845071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaac838640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa815e4a0 0x7fbaa80b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa7fff640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa80cb580 0x7fbaa81f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 -- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa815e4a0 msgr2=0x7fbaa80b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa815e4a0 0x7fbaa80b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 -- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa80cb580 msgr2=0x7fbaa81f6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa80cb580 0x7fbaa81f6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.049+0000 7fbaa77fe640 1 -- 172.21.15.28:0/3965845071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa8159b60 con 0x7fbaa80cef00 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.050+0000 7fbaa77fe640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa8155960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fba90009950 tx=0x7fba90032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.050+0000 7fbaa4ff9640 1 -- 172.21.15.28:0/3965845071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90040070 con 0x7fbaa80cef00 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.050+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa8159e40 con 0x7fbaa80cef00 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.050+0000 7fbaa4ff9640 1 -- 172.21.15.28:0/3965845071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba90005310 con 0x7fbaa80cef00 2024-09-10T20:39:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.050+0000 7fbaa4ff9640 1 -- 172.21.15.28:0/3965845071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900047a0 con 0x7fbaa80cef00 2024-09-10T20:39:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.050+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa80ca0d0 con 0x7fbaa80cef00 2024-09-10T20:39:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.051+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa80cb040 con 0x7fbaa80cef00 2024-09-10T20:39:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.052+0000 7fbaa4ff9640 1 -- 172.21.15.28:0/3965845071 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fba90004940 con 0x7fbaa80cef00 2024-09-10T20:39:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.052+0000 7fbaa4ff9640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fba88077f70 0x7fba8807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.052+0000 7fbaa4ff9640 1 -- 172.21.15.28:0/3965845071 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fba900bb030 con 0x7fbaa80cef00 2024-09-10T20:39:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.052+0000 7fbaa7fff640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fba88077f70 0x7fba8807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.053+0000 7fbaa7fff640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fba88077f70 0x7fba8807a430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba98007fd0 tx=0x7fba98009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:04.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.056+0000 7fbaa4ff9640 1 -- 172.21.15.28:0/3965845071 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fba9004b030 con 0x7fbaa80cef00 2024-09-10T20:39:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.268+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol51900", "group_name": "subvol_grp61360", "target": ["mon-mgr", ""]}) -- 0x7fbaa8155ea0 con 0x7fba88077f70 2024-09-10T20:39:04.277 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:04.273+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:04.277 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:04.273+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:04.277 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:04.273+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:04.277 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:04.273+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:04.277 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:04.273+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.310+0000 7fbaa4ff9640 1 -- 172.21.15.28:0/3965845071 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fba980012a0 con 0x7fba88077f70 2024-09-10T20:39:04.314 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp61360/subvol51900 2024-09-10T20:39:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fba88077f70 msgr2=0x7fba8807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fba88077f70 0x7fba8807a430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba98007fd0 tx=0x7fba98009040 comp rx=0 tx=0).stop 2024-09-10T20:39:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 msgr2=0x7fbaa8155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa8155960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fba90009950 tx=0x7fba90032c10 comp rx=0 tx=0).stop 2024-09-10T20:39:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 shutdown_connections 2024-09-10T20:39:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fba88077f70 0x7fba8807a430 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa815e4a0 0x7fbaa80b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa80cef00 0x7fbaa8155960 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 --2- 172.21.15.28:0/3965845071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa80cb580 0x7fbaa81f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 >> 172.21.15.28:0/3965845071 conn(0x7fbaa80d0410 msgr2=0x7fbaa80afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:04.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 shutdown_connections 2024-09-10T20:39:04.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.312+0000 7fbaae29b640 1 -- 172.21.15.28:0/3965845071 wait complete. 2024-09-10T20:39:04.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol51900 subvol_grp61360 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.432+0000 7fd46642a640 1 -- 172.21.15.28:0/209072190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 msgr2=0x7fd4600d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.432+0000 7fd46642a640 1 --2- 172.21.15.28:0/209072190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 0x7fd4600d1c60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd458005fd0 tx=0x7fd45802ab20 comp rx=0 tx=0).stop 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.432+0000 7fd46642a640 1 -- 172.21.15.28:0/209072190 shutdown_connections 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.432+0000 7fd46642a640 1 --2- 172.21.15.28:0/209072190 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 0x7fd4600d1c60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.432+0000 7fd46642a640 1 --2- 172.21.15.28:0/209072190 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd460162000 0x7fd4600d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.432+0000 7fd46642a640 1 -- 172.21.15.28:0/209072190 >> 172.21.15.28:0/209072190 conn(0x7fd4600d3f90 msgr2=0x7fd4600d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.433+0000 7fd46642a640 1 -- 172.21.15.28:0/209072190 shutdown_connections 2024-09-10T20:39:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.433+0000 7fd46642a640 1 -- 172.21.15.28:0/209072190 wait complete. 2024-09-10T20:39:04.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.434+0000 7fd46642a640 1 Processor -- start 2024-09-10T20:39:04.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.434+0000 7fd46642a640 1 -- start start 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd46642a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4600b4c20 0x7fd46015a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd46642a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 0x7fd4600b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd46642a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd460162000 0x7fd460158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd46642a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4600d2320 con 0x7fd4600b4c20 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd46642a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd4600d21a0 con 0x7fd460161850 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd46642a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd4600d24a0 con 0x7fd460162000 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4600b4c20 0x7fd46015a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4600b4c20 0x7fd46015a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50004/0 (socket says 172.21.15.28:50004) 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 -- 172.21.15.28:0/3718079922 learned_addr learned my addr 172.21.15.28:0/3718079922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd4649c7640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd460162000 0x7fd460158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45f7fe640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 0x7fd4600b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 -- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd460162000 msgr2=0x7fd460158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd460162000 0x7fd460158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 -- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 msgr2=0x7fd4600b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 0x7fd4600b7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 -- 172.21.15.28:0/3718079922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46015ac20 con 0x7fd4600b4c20 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45f7fe640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 0x7fd4600b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd4649c7640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd460162000 0x7fd460158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45ffff640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4600b4c20 0x7fd46015a640 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd45000e9f0 tx=0x7fd45000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45cff9640 1 -- 172.21.15.28:0/3718079922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45000cd80 con 0x7fd4600b4c20 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45cff9640 1 -- 172.21.15.28:0/3718079922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd450004020 con 0x7fd4600b4c20 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.435+0000 7fd45cff9640 1 -- 172.21.15.28:0/3718079922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450004470 con 0x7fd4600b4c20 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.436+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4600cc6e0 con 0x7fd4600b4c20 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.436+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4600ccca0 con 0x7fd4600b4c20 2024-09-10T20:39:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.437+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4600d1c60 con 0x7fd4600b4c20 2024-09-10T20:39:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.437+0000 7fd45cff9640 1 -- 172.21.15.28:0/3718079922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd4500035f0 con 0x7fd4600b4c20 2024-09-10T20:39:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.438+0000 7fd45cff9640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd44c077f70 0x7fd44c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.438+0000 7fd45cff9640 1 -- 172.21.15.28:0/3718079922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fd45009a840 con 0x7fd4600b4c20 2024-09-10T20:39:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.438+0000 7fd45f7fe640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd44c077f70 0x7fd44c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.439+0000 7fd45f7fe640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd44c077f70 0x7fd44c07a430 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd4601f9040 tx=0x7fd454009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:04.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.442+0000 7fd45cff9640 1 -- 172.21.15.28:0/3718079922 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd450063340 con 0x7fd4600b4c20 2024-09-10T20:39:04.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.654+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol51900", "group_name": "subvol_grp61360", "target": ["mon-mgr", ""]}) -- 0x7fd4600b9f10 con 0x7fd44c077f70 2024-09-10T20:39:04.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.661+0000 7fd45cff9640 1 -- 172.21.15.28:0/3718079922 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd4600b9f10 con 0x7fd44c077f70 2024-09-10T20:39:04.665 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp61360/subvol51900 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd44c077f70 msgr2=0x7fd44c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd44c077f70 0x7fd44c07a430 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd4601f9040 tx=0x7fd454009040 comp rx=0 tx=0).stop 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4600b4c20 msgr2=0x7fd46015a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4600b4c20 0x7fd46015a640 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd45000e9f0 tx=0x7fd45000eeb0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 shutdown_connections 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd44c077f70 0x7fd44c07a430 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd460162000 0x7fd460158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd460161850 0x7fd4600b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 --2- 172.21.15.28:0/3718079922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4600b4c20 0x7fd46015a640 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 >> 172.21.15.28:0/3718079922 conn(0x7fd4600d3f90 msgr2=0x7fd46015cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 shutdown_connections 2024-09-10T20:39:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:04.663+0000 7fd46642a640 1 -- 172.21.15.28:0/3718079922 wait complete. 2024-09-10T20:39:04.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:04.678 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/f333d7f19210e5c1d147080fcf73dff2.meta) 2024-09-10T20:39:07.057 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:39:07.058 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:39:07.058 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:39:07.058 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/subvol_grp61360/subvol51900 2024-09-10T20:39:07.058 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:39:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:07.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol51900 subvol_grp61360 2024-09-10T20:39:07.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 -- 172.21.15.28:0/1818866561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92180b5030 msgr2=0x7f921815f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1818866561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92180b5030 0x7f921815f850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9214005fd0 tx=0x7f921402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 -- 172.21.15.28:0/1818866561 shutdown_connections 2024-09-10T20:39:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1818866561 >> v2:172.21.15.159:3300/0 conn(0x7f9218073d40 0x7f92180741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1818866561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92180b5030 0x7f921815f850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 -- 172.21.15.28:0/1818866561 >> 172.21.15.28:0/1818866561 conn(0x7f92180cfc80 msgr2=0x7f92180d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 -- 172.21.15.28:0/1818866561 shutdown_connections 2024-09-10T20:39:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.176+0000 7f921ed5d640 1 -- 172.21.15.28:0/1818866561 wait complete. 2024-09-10T20:39:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.177+0000 7f921ed5d640 1 Processor -- start 2024-09-10T20:39:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.178+0000 7f921ed5d640 1 -- start start 2024-09-10T20:39:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.178+0000 7f921ed5d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921806e390 0x7f92180aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.178+0000 7f921ed5d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9218073d40 0x7f92180c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.178+0000 7f921ed5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92180b5030 0x7f92180aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.178+0000 7f921ed5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92180ca450 con 0x7f92180b5030 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.178+0000 7f921ed5d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f92180ca2d0 con 0x7f9218073d40 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.178+0000 7f921ed5d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92180ca5d0 con 0x7f921806e390 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92180b5030 0x7f92180aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92180b5030 0x7f92180aaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50008/0 (socket says 172.21.15.28:50008) 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 -- 172.21.15.28:0/1261348346 learned_addr learned my addr 172.21.15.28:0/1261348346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921caf9640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921806e390 0x7f92180aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f920ffff640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9218073d40 0x7f92180c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 -- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921806e390 msgr2=0x7f92180aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921806e390 0x7f92180aa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 -- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9218073d40 msgr2=0x7f92180c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9218073d40 0x7f92180c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 -- 172.21.15.28:0/1261348346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92180a9540 con 0x7f92180b5030 2024-09-10T20:39:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921d2fa640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92180b5030 0x7f92180aaa00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f921000eb10 tx=0x7f921000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f920d7fa640 1 -- 172.21.15.28:0/1261348346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210004b20 con 0x7f92180b5030 2024-09-10T20:39:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f920d7fa640 1 -- 172.21.15.28:0/1261348346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9210004ce0 con 0x7f92180b5030 2024-09-10T20:39:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92180a97e0 con 0x7f92180b5030 2024-09-10T20:39:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f920d7fa640 1 -- 172.21.15.28:0/1261348346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210005280 con 0x7f92180b5030 2024-09-10T20:39:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.179+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92180ab4a0 con 0x7f92180b5030 2024-09-10T20:39:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.180+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92180a9bd0 con 0x7f92180b5030 2024-09-10T20:39:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.181+0000 7f920d7fa640 1 -- 172.21.15.28:0/1261348346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 6) ==== 100652+0+0 (secure 0 0 0) 0x7f9210005420 con 0x7f92180b5030 2024-09-10T20:39:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.182+0000 7f920d7fa640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9204078000 0x7f920407a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.182+0000 7f920d7fa640 1 -- 172.21.15.28:0/1261348346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f921009aa20 con 0x7f92180b5030 2024-09-10T20:39:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.182+0000 7f921caf9640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9204078000 0x7f920407a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.182+0000 7f921caf9640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9204078000 0x7f920407a4c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9214005fd0 tx=0x7f9214004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.186+0000 7f920d7fa640 1 -- 172.21.15.28:0/1261348346 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9210062e60 con 0x7f92180b5030 2024-09-10T20:39:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.399+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol51900", "group_name": "subvol_grp61360", "target": ["mon-mgr", ""]}) -- 0x7f92180a99b0 con 0x7f9204078000 2024-09-10T20:39:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.421+0000 7f920d7fa640 1 -- 172.21.15.28:0/1261348346 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f92180a99b0 con 0x7f9204078000 2024-09-10T20:39:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9204078000 msgr2=0x7f920407a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9204078000 0x7f920407a4c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9214005fd0 tx=0x7f9214004730 comp rx=0 tx=0).stop 2024-09-10T20:39:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92180b5030 msgr2=0x7f92180aaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92180b5030 0x7f92180aaa00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f921000eb10 tx=0x7f921000c2d0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 shutdown_connections 2024-09-10T20:39:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9204078000 0x7f920407a4c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92180b5030 0x7f92180aaa00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9218073d40 0x7f92180c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 --2- 172.21.15.28:0/1261348346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f921806e390 0x7f92180aa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 >> 172.21.15.28:0/1261348346 conn(0x7f92180cfc80 msgr2=0x7f92180d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 shutdown_connections 2024-09-10T20:39:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.424+0000 7f921ed5d640 1 -- 172.21.15.28:0/1261348346 wait complete. 2024-09-10T20:39:07.429 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.425+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.429 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.425+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.429 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.425+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.429 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.425+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.430 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.425+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.439 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-10T20:39:07.444 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.440+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.445 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.440+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.445 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.440+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.445 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.440+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.445 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:07.441+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:39:07.492 INFO:teuthology.orchestra.run.smithi028.stdout:2 2024-09-10T20:39:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:07.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp61360 2024-09-10T20:39:07.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.602+0000 7f985cbc6640 1 -- 172.21.15.28:0/3096776291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9858151db0 msgr2=0x7f98580706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.602+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3096776291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9858151db0 0x7f98580706e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9848009c10 tx=0x7f984802a870 comp rx=0 tx=0).stop 2024-09-10T20:39:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.602+0000 7f985cbc6640 1 -- 172.21.15.28:0/3096776291 shutdown_connections 2024-09-10T20:39:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.602+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3096776291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9858151db0 0x7f98580706e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.602+0000 7f985cbc6640 1 -- 172.21.15.28:0/3096776291 >> 172.21.15.28:0/3096776291 conn(0x7f98580d0d50 msgr2=0x7f98580d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.602+0000 7f985cbc6640 1 -- 172.21.15.28:0/3096776291 shutdown_connections 2024-09-10T20:39:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.602+0000 7f985cbc6640 1 -- 172.21.15.28:0/3096776291 wait complete. 2024-09-10T20:39:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 Processor -- start 2024-09-10T20:39:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 -- start start 2024-09-10T20:39:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9858073ea0 0x7f98581f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9858151db0 0x7f98580cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f985815bd80 0x7f98581511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98580ccc40 con 0x7f985815bd80 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f98580ccac0 con 0x7f9858073ea0 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.604+0000 7f985cbc6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f98580ccdc0 con 0x7f9858151db0 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f985815bd80 0x7f98581511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f985815bd80 0x7f98581511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50018/0 (socket says 172.21.15.28:50018) 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 -- 172.21.15.28:0/3421721090 learned_addr learned my addr 172.21.15.28:0/3421721090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f985659c640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9858073ea0 0x7f98581f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9855d9b640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9858151db0 0x7f98580cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 -- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9858151db0 msgr2=0x7f98580cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9858151db0 0x7f98580cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 -- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9858073ea0 msgr2=0x7f98581f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9858073ea0 0x7f98581f2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 -- 172.21.15.28:0/3421721090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98581f34f0 con 0x7f985815bd80 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f985659c640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9858073ea0 0x7f98581f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f9856d9d640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f985815bd80 0x7f98581511d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f985000c390 tx=0x7f985000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f984effd640 1 -- 172.21.15.28:0/3421721090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850006890 con 0x7f985815bd80 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f984effd640 1 -- 172.21.15.28:0/3421721090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9850006b20 con 0x7f985815bd80 2024-09-10T20:39:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98581f37d0 con 0x7f985815bd80 2024-09-10T20:39:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f984effd640 1 -- 172.21.15.28:0/3421721090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850007020 con 0x7f985815bd80 2024-09-10T20:39:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.605+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9858151710 con 0x7f985815bd80 2024-09-10T20:39:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.606+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98580706e0 con 0x7f985815bd80 2024-09-10T20:39:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.607+0000 7f984effd640 1 -- 172.21.15.28:0/3421721090 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 6) ==== 100652+0+0 (secure 0 0 0) 0x7f98500071c0 con 0x7f985815bd80 2024-09-10T20:39:07.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.607+0000 7f984effd640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9838078000 0x7f983807a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:07.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.607+0000 7f984effd640 1 -- 172.21.15.28:0/3421721090 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f9850099800 con 0x7f985815bd80 2024-09-10T20:39:07.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.608+0000 7f985659c640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9838078000 0x7f983807a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:07.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.608+0000 7f985659c640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9838078000 0x7f983807a4c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9848034000 tx=0x7f9848004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:07.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.612+0000 7f984effd640 1 -- 172.21.15.28:0/3421721090 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9850062290 con 0x7f985815bd80 2024-09-10T20:39:07.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.824+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp61360", "target": ["mon-mgr", ""]}) -- 0x7f98580749a0 con 0x7f9838078000 2024-09-10T20:39:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.829+0000 7f984effd640 1 -- 172.21.15.28:0/3421721090 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f98580749a0 con 0x7f9838078000 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9838078000 msgr2=0x7f983807a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9838078000 0x7f983807a4c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9848034000 tx=0x7f9848004ab0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f985815bd80 msgr2=0x7f98581511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f985815bd80 0x7f98581511d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f985000c390 tx=0x7f985000c850 comp rx=0 tx=0).stop 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 shutdown_connections 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9838078000 0x7f983807a4c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f985815bd80 0x7f98581511d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9858151db0 0x7f98580cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 --2- 172.21.15.28:0/3421721090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9858073ea0 0x7f98581f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 >> 172.21.15.28:0/3421721090 conn(0x7f98580d0d50 msgr2=0x7f98581543a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 shutdown_connections 2024-09-10T20:39:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:07.831+0000 7f985cbc6640 1 -- 172.21.15.28:0/3421721090 wait complete. 2024-09-10T20:39:07.845 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:39:07.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:39:07.884 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:39:07.945 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:39:07.945 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:39:07.945 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:07.945 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:07.962 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:07.962 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:08.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:39:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 -- 172.21.15.28:0/1457181740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 msgr2=0x7f66680cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 --2- 172.21.15.28:0/1457181740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f66680cfbf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f666808c020 tx=0x7f665c02a560 comp rx=0 tx=0).stop 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 -- 172.21.15.28:0/1457181740 shutdown_connections 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 --2- 172.21.15.28:0/1457181740 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6668161b50 0x7f6668161f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 --2- 172.21.15.28:0/1457181740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f66680cfbf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 --2- 172.21.15.28:0/1457181740 >> v2:172.21.15.159:3300/0 conn(0x7f666815de70 0x7f66680d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 -- 172.21.15.28:0/1457181740 >> 172.21.15.28:0/1457181740 conn(0x7f66680d3fb0 msgr2=0x7f66680d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 -- 172.21.15.28:0/1457181740 shutdown_connections 2024-09-10T20:39:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.160+0000 7f666f7ad640 1 -- 172.21.15.28:0/1457181740 wait complete. 2024-09-10T20:39:08.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 Processor -- start 2024-09-10T20:39:08.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 -- start start 2024-09-10T20:39:08.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f666814d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f666815de70 0x7f66681f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6668161b50 0x7f66681f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66680b6e00 con 0x7f66680cf780 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f666815e460 con 0x7f666815de70 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666f7ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f66680b6f80 con 0x7f6668161b50 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666d549640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f666814d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.162+0000 7f666d549640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f666814d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50026/0 (socket says 172.21.15.28:50026) 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666d549640 1 -- 172.21.15.28:0/2945043027 learned_addr learned my addr 172.21.15.28:0/2945043027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666d549640 1 -- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6668161b50 msgr2=0x7f66681f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666dd4a640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6668161b50 0x7f66681f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666cd48640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f666815de70 0x7f66681f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666d549640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6668161b50 0x7f66681f1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666d549640 1 -- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f666815de70 msgr2=0x7f66681f52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666d549640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f666815de70 0x7f66681f52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666d549640 1 -- 172.21.15.28:0/2945043027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66680cc170 con 0x7f66680cf780 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666d549640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f666814d200 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f666000b4a0 tx=0x7f666000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f6665ffb640 1 -- 172.21.15.28:0/2945043027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660014070 con 0x7f66680cf780 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f6665ffb640 1 -- 172.21.15.28:0/2945043027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6660010700 con 0x7f66680cf780 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66680cc6c0 con 0x7f66680cf780 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f6665ffb640 1 -- 172.21.15.28:0/2945043027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660010b50 con 0x7f66680cf780 2024-09-10T20:39:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.163+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66680ccc80 con 0x7f66680cf780 2024-09-10T20:39:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.164+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66680d2b90 con 0x7f66680cf780 2024-09-10T20:39:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.165+0000 7f6665ffb640 1 -- 172.21.15.28:0/2945043027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f6660017360 con 0x7f66680cf780 2024-09-10T20:39:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.166+0000 7f6665ffb640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6654078120 0x7f665407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.166+0000 7f6665ffb640 1 -- 172.21.15.28:0/2945043027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f6660099f60 con 0x7f66680cf780 2024-09-10T20:39:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.166+0000 7f666cd48640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6654078120 0x7f665407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.167+0000 7f666cd48640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6654078120 0x7f665407a5e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f665c033000 tx=0x7f665c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:08.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.170+0000 7f6665ffb640 1 -- 172.21.15.28:0/2945043027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6660062640 con 0x7f66680cf780 2024-09-10T20:39:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.366+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f66680b9f30 con 0x7f66680cf780 2024-09-10T20:39:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.366+0000 7f6665ffb640 1 -- 172.21.15.28:0/2945043027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v32) ==== 80+0+0 (secure 0 0 0) 0x7f6660061f40 con 0x7f66680cf780 2024-09-10T20:39:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:39:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6654078120 msgr2=0x7f665407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6654078120 0x7f665407a5e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f665c033000 tx=0x7f665c004730 comp rx=0 tx=0).stop 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 msgr2=0x7f666814d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f666814d200 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f666000b4a0 tx=0x7f666000b960 comp rx=0 tx=0).stop 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 shutdown_connections 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6654078120 0x7f665407a5e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6668161b50 0x7f66681f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f666815de70 0x7f66681f52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 --2- 172.21.15.28:0/2945043027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66680cf780 0x7f666814d200 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 >> 172.21.15.28:0/2945043027 conn(0x7f66680d3fb0 msgr2=0x7f66680b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 shutdown_connections 2024-09-10T20:39:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.368+0000 7f666f7ad640 1 -- 172.21.15.28:0/2945043027 wait complete. 2024-09-10T20:39:08.383 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:39:08.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:08.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:08.430 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:39:08.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:08.430 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:39:08.462 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:39:08.462 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:08.462 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:08.479 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:08.479 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:08.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:39:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.640+0000 7fa73ae8c640 1 -- 172.21.15.28:0/994771200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 msgr2=0x7fa720004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.640+0000 7fa73ae8c640 1 --2- 172.21.15.28:0/994771200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 0x7fa720004c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009ed0 tx=0x7fa720005830 comp rx=0 tx=0).stop 2024-09-10T20:39:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.640+0000 7fa739429640 1 -- 172.21.15.28:0/994771200 reap_dead start 2024-09-10T20:39:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.640+0000 7fa73ae8c640 1 -- 172.21.15.28:0/994771200 shutdown_connections 2024-09-10T20:39:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.640+0000 7fa73ae8c640 1 -- 172.21.15.28:0/994771200 >> 172.21.15.28:0/994771200 conn(0x7fa7340d1e40 msgr2=0x7fa7340d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.640+0000 7fa73ae8c640 1 -- 172.21.15.28:0/994771200 shutdown_connections 2024-09-10T20:39:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.640+0000 7fa73ae8c640 1 -- 172.21.15.28:0/994771200 wait complete. 2024-09-10T20:39:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.641+0000 7fa73ae8c640 1 Processor -- start 2024-09-10T20:39:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.641+0000 7fa73ae8c640 1 -- start start 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa73ae8c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 0x7fa7341540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa73ae8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa734159130 0x7fa7340aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa73ae8c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7340cbb10 0x7fa7340af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa73ae8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7340b13e0 con 0x7fa734159130 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa73ae8c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa7340cf170 con 0x7fa7340cbb10 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa73ae8c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa7340b1560 con 0x7fa720004800 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa734159130 0x7fa7340aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa734159130 0x7fa7340aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50038/0 (socket says 172.21.15.28:50038) 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 -- 172.21.15.28:0/1618163500 learned_addr learned my addr 172.21.15.28:0/1618163500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa739429640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7340cbb10 0x7fa7340af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa738c28640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 0x7fa7341540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 -- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 msgr2=0x7fa7341540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 0x7fa7341540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 -- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7340cbb10 msgr2=0x7fa7340af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7340cbb10 0x7fa7340af0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.642+0000 7fa733fff640 1 -- 172.21.15.28:0/1618163500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7341599f0 con 0x7fa734159130 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.643+0000 7fa738c28640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 0x7fa7341540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.643+0000 7fa733fff640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa734159130 0x7fa7340aeb90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009ed0 tx=0x7fa71c033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.643+0000 7fa7317fa640 1 -- 172.21.15.28:0/1618163500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c043070 con 0x7fa734159130 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.643+0000 7fa7317fa640 1 -- 172.21.15.28:0/1618163500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa71c003c60 con 0x7fa734159130 2024-09-10T20:39:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.643+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7340ca090 con 0x7fa734159130 2024-09-10T20:39:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.643+0000 7fa7317fa640 1 -- 172.21.15.28:0/1618163500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c038040 con 0x7fa734159130 2024-09-10T20:39:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.643+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7340ca610 con 0x7fa734159130 2024-09-10T20:39:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.644+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa73415a8b0 con 0x7fa734159130 2024-09-10T20:39:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.645+0000 7fa7317fa640 1 -- 172.21.15.28:0/1618163500 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fa71c03b030 con 0x7fa734159130 2024-09-10T20:39:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.645+0000 7fa7317fa640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa714078150 0x7fa71407a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.645+0000 7fa7317fa640 1 -- 172.21.15.28:0/1618163500 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fa71c0bad10 con 0x7fa734159130 2024-09-10T20:39:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.645+0000 7fa738c28640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa714078150 0x7fa71407a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.646+0000 7fa738c28640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa714078150 0x7fa71407a610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa724001650 tx=0x7fa7240067a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.649+0000 7fa7317fa640 1 -- 172.21.15.28:0/1618163500 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa71c082ec0 con 0x7fa734159130 2024-09-10T20:39:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.847+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa734159680 con 0x7fa734159130 2024-09-10T20:39:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.847+0000 7fa7317fa640 1 -- 172.21.15.28:0/1618163500 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v32) ==== 80+0+0 (secure 0 0 0) 0x7fa71c0878f0 con 0x7fa734159130 2024-09-10T20:39:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:39:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa714078150 msgr2=0x7fa71407a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa714078150 0x7fa71407a610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa724001650 tx=0x7fa7240067a0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa734159130 msgr2=0x7fa7340aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa734159130 0x7fa7340aeb90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009ed0 tx=0x7fa71c033df0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 shutdown_connections 2024-09-10T20:39:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa714078150 0x7fa71407a610 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7340cbb10 0x7fa7340af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa734159130 0x7fa7340aeb90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 --2- 172.21.15.28:0/1618163500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa720004800 0x7fa7341540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 >> 172.21.15.28:0/1618163500 conn(0x7fa7340d1e40 msgr2=0x7fa734072160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 shutdown_connections 2024-09-10T20:39:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:08.849+0000 7fa73ae8c640 1 -- 172.21.15.28:0/1618163500 wait complete. 2024-09-10T20:39:08.864 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:39:08.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:08.864 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:08.900 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:39:08.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:08.901 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:39:08.958 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:39:08.958 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:08.958 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:08.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:39:08.975 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:39:08.975 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:08.975 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:09.014 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:39:09.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.117+0000 7f29f548c640 1 -- 172.21.15.28:0/1359992975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00cd3d0 msgr2=0x7f29f00cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.117+0000 7f29f548c640 1 --2- 172.21.15.28:0/1359992975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00cd3d0 0x7f29f00cc570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29e0005fd0 tx=0x7f29e002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.118+0000 7f29f548c640 1 -- 172.21.15.28:0/1359992975 shutdown_connections 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.118+0000 7f29f548c640 1 --2- 172.21.15.28:0/1359992975 >> v2:172.21.15.159:3300/0 conn(0x7f29f00b4340 0x7f29f00b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.118+0000 7f29f548c640 1 --2- 172.21.15.28:0/1359992975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00cd3d0 0x7f29f00cc570 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.118+0000 7f29f548c640 1 -- 172.21.15.28:0/1359992975 >> 172.21.15.28:0/1359992975 conn(0x7f29f006cf00 msgr2=0x7f29f006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.118+0000 7f29f548c640 1 -- 172.21.15.28:0/1359992975 shutdown_connections 2024-09-10T20:39:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.118+0000 7f29f548c640 1 -- 172.21.15.28:0/1359992975 wait complete. 2024-09-10T20:39:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.119+0000 7f29f548c640 1 Processor -- start 2024-09-10T20:39:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.119+0000 7f29f548c640 1 -- start start 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29f548c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00b4340 0x7f29f0155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29f548c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f00cd3d0 0x7f29f0072870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29f548c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29f00d0090 0x7f29f00b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29f548c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29f00d09a0 con 0x7f29f00cd3d0 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29f548c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f29f00d0820 con 0x7f29f00d0090 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29f548c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f29f00b3590 con 0x7f29f00b4340 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f00cd3d0 0x7f29f0072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f00cd3d0 0x7f29f0072870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50052/0 (socket says 172.21.15.28:50052) 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 -- 172.21.15.28:0/3273247911 learned_addr learned my addr 172.21.15.28:0/3273247911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ef7fe640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29f00d0090 0x7f29f00b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29eeffd640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00b4340 0x7f29f0155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 -- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00b4340 msgr2=0x7f29f0155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00b4340 0x7f29f0155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 -- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29f00d0090 msgr2=0x7f29f00b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29f00d0090 0x7f29f00b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 -- 172.21.15.28:0/3273247911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f0072db0 con 0x7f29f00cd3d0 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29eeffd640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00b4340 0x7f29f0155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.120+0000 7f29ee7fc640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f00cd3d0 0x7f29f0072870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f29d800e9f0 tx=0x7f29d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.121+0000 7f29e77fe640 1 -- 172.21.15.28:0/3273247911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d800cde0 con 0x7f29f00cd3d0 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.121+0000 7f29e77fe640 1 -- 172.21.15.28:0/3273247911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29d8010520 con 0x7f29f00cd3d0 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.121+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f0073050 con 0x7f29f00cd3d0 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.121+0000 7f29e77fe640 1 -- 172.21.15.28:0/3273247911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8010970 con 0x7f29f00cd3d0 2024-09-10T20:39:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.121+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f0073610 con 0x7f29f00cd3d0 2024-09-10T20:39:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.122+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f00b3ab0 con 0x7f29f00cd3d0 2024-09-10T20:39:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.122+0000 7f29e77fe640 1 -- 172.21.15.28:0/3273247911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f29d8010b10 con 0x7f29f00cd3d0 2024-09-10T20:39:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.123+0000 7f29e77fe640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29d0078120 0x7f29d007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.123+0000 7f29e77fe640 1 -- 172.21.15.28:0/3273247911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f29d809abe0 con 0x7f29f00cd3d0 2024-09-10T20:39:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.123+0000 7f29eeffd640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29d0078120 0x7f29d007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.124+0000 7f29eeffd640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29d0078120 0x7f29d007a5e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f29e0033000 tx=0x7f29e0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.127+0000 7f29e77fe640 1 -- 172.21.15.28:0/3273247911 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f29d8062d00 con 0x7f29f00cd3d0 2024-09-10T20:39:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.350+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29f006d450 con 0x7f29f00cd3d0 2024-09-10T20:39:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.351+0000 7f29e77fe640 1 -- 172.21.15.28:0/3273247911 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5226 (secure 0 0 0) 0x7f29d8067730 con 0x7f29f00cd3d0 2024-09-10T20:39:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:09.355 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"btime":"2024-09-10T20:39:02:975887+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1687267914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1687267914},{"type":"v1","addr":"172.21.15.159:6836","nonce":1687267914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/649109556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":649109556},{"type":"v1","addr":"172.21.15.159:6837","nonce":649109556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4851,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2646744984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2646744984},{"type":"v1","addr":"172.21.15.28:6834","nonce":2646744984}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:53.302329+0000","modified":"2024-09-10T20:38:56.323633+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4854":{"gid":4854,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1884641453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1884641453},{"type":"v1","addr":"172.21.15.28:6835","nonce":1884641453}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4854,"qdb_cluster":[4854]},"id":2}]} 2024-09-10T20:39:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-10T20:39:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29d0078120 msgr2=0x7f29d007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29d0078120 0x7f29d007a5e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f29e0033000 tx=0x7f29e0004730 comp rx=0 tx=0).stop 2024-09-10T20:39:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f00cd3d0 msgr2=0x7f29f0072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f00cd3d0 0x7f29f0072870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f29d800e9f0 tx=0x7f29d800eeb0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 shutdown_connections 2024-09-10T20:39:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29d0078120 0x7f29d007a5e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29f00d0090 0x7f29f00b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f00cd3d0 0x7f29f0072870 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 --2- 172.21.15.28:0/3273247911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29f00b4340 0x7f29f0155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 >> 172.21.15.28:0/3273247911 conn(0x7f29f006cf00 msgr2=0x7f29f009bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 shutdown_connections 2024-09-10T20:39:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.353+0000 7f29f548c640 1 -- 172.21.15.28:0/3273247911 wait complete. 2024-09-10T20:39:09.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.470+0000 7fc871433640 1 -- 172.21.15.28:0/3273298157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858003a00 msgr2=0x7fc858003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.470+0000 7fc871433640 1 --2- 172.21.15.28:0/3273298157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858003a00 0x7fc858003e20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc864009f00 tx=0x7fc858005830 comp rx=0 tx=0).stop 2024-09-10T20:39:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.470+0000 7fc86b7fe640 1 -- 172.21.15.28:0/3273298157 reap_dead start 2024-09-10T20:39:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.470+0000 7fc871433640 1 -- 172.21.15.28:0/3273298157 shutdown_connections 2024-09-10T20:39:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.470+0000 7fc871433640 1 -- 172.21.15.28:0/3273298157 >> 172.21.15.28:0/3273298157 conn(0x7fc86c0d4130 msgr2=0x7fc86c0d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.470+0000 7fc871433640 1 -- 172.21.15.28:0/3273298157 shutdown_connections 2024-09-10T20:39:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.470+0000 7fc871433640 1 -- 172.21.15.28:0/3273298157 wait complete. 2024-09-10T20:39:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.471+0000 7fc871433640 1 Processor -- start 2024-09-10T20:39:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.471+0000 7fc871433640 1 -- start start 2024-09-10T20:39:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc871433640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc858003a00 0x7fc86c0b9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc871433640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86c151ae0 0x7fc86c0c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc871433640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86c1621b0 0x7fc86c0c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc871433640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c15f350 con 0x7fc86c151ae0 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc871433640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c15f1d0 con 0x7fc858003a00 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc871433640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc86c15f4d0 con 0x7fc86c1621b0 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc86a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86c151ae0 0x7fc86c0c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc86a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86c151ae0 0x7fc86c0c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50072/0 (socket says 172.21.15.28:50072) 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc86a7fc640 1 -- 172.21.15.28:0/2295868455 learned_addr learned my addr 172.21.15.28:0/2295868455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc86affd640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc858003a00 0x7fc86c0b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.472+0000 7fc86b7fe640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86c1621b0 0x7fc86c0c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc86a7fc640 1 -- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86c1621b0 msgr2=0x7fc86c0c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc86a7fc640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86c1621b0 0x7fc86c0c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc86a7fc640 1 -- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc858003a00 msgr2=0x7fc86c0b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc86a7fc640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc858003a00 0x7fc86c0b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc86a7fc640 1 -- 172.21.15.28:0/2295868455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c1f8130 con 0x7fc86c151ae0 2024-09-10T20:39:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc86b7fe640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86c1621b0 0x7fc86c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc86a7fc640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86c151ae0 0x7fc86c0c7c20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc8540118d0 tx=0x7fc854011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc8637fe640 1 -- 172.21.15.28:0/2295868455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85401e070 con 0x7fc86c151ae0 2024-09-10T20:39:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc8637fe640 1 -- 172.21.15.28:0/2295868455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc85400fbb0 con 0x7fc86c151ae0 2024-09-10T20:39:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86c1f8410 con 0x7fc86c151ae0 2024-09-10T20:39:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc8637fe640 1 -- 172.21.15.28:0/2295868455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc854019410 con 0x7fc86c151ae0 2024-09-10T20:39:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.473+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86c1f89d0 con 0x7fc86c151ae0 2024-09-10T20:39:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.474+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86c0d1d00 con 0x7fc86c151ae0 2024-09-10T20:39:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.475+0000 7fc8637fe640 1 -- 172.21.15.28:0/2295868455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fc8540195b0 con 0x7fc86c151ae0 2024-09-10T20:39:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.475+0000 7fc8637fe640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc84c078120 0x7fc84c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.475+0000 7fc8637fe640 1 -- 172.21.15.28:0/2295868455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fc85409ac80 con 0x7fc86c151ae0 2024-09-10T20:39:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.476+0000 7fc86affd640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc84c078120 0x7fc84c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.476+0000 7fc86affd640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc84c078120 0x7fc84c07a5e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc864035000 tx=0x7fc864004a40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.479+0000 7fc8637fe640 1 -- 172.21.15.28:0/2295868455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc854062dc0 con 0x7fc86c151ae0 2024-09-10T20:39:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.701+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc86c069680 con 0x7fc86c151ae0 2024-09-10T20:39:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.701+0000 7fc8637fe640 1 -- 172.21.15.28:0/2295868455 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5226 (secure 0 0 0) 0x7fc8540677f0 con 0x7fc86c151ae0 2024-09-10T20:39:09.705 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:09.705 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"btime":"2024-09-10T20:39:02:975887+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1687267914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1687267914},{"type":"v1","addr":"172.21.15.159:6836","nonce":1687267914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/649109556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":649109556},{"type":"v1","addr":"172.21.15.159:6837","nonce":649109556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4851,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2646744984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2646744984},{"type":"v1","addr":"172.21.15.28:6834","nonce":2646744984}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:53.302329+0000","modified":"2024-09-10T20:38:56.323633+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4854":{"gid":4854,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1884641453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1884641453},{"type":"v1","addr":"172.21.15.28:6835","nonce":1884641453}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4854,"qdb_cluster":[4854]},"id":2}]} 2024-09-10T20:39:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-10T20:39:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc84c078120 msgr2=0x7fc84c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc84c078120 0x7fc84c07a5e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc864035000 tx=0x7fc864004a40 comp rx=0 tx=0).stop 2024-09-10T20:39:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86c151ae0 msgr2=0x7fc86c0c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86c151ae0 0x7fc86c0c7c20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc8540118d0 tx=0x7fc854011d90 comp rx=0 tx=0).stop 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 shutdown_connections 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc84c078120 0x7fc84c07a5e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86c1621b0 0x7fc86c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86c151ae0 0x7fc86c0c7c20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 --2- 172.21.15.28:0/2295868455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc858003a00 0x7fc86c0b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 >> 172.21.15.28:0/2295868455 conn(0x7fc86c0d4130 msgr2=0x7fc86c06e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 shutdown_connections 2024-09-10T20:39:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.703+0000 7fc871433640 1 -- 172.21.15.28:0/2295868455 wait complete. 2024-09-10T20:39:09.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:39:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 -- 172.21.15.28:0/1001600444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 msgr2=0x7f5df00cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1001600444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00cc570 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5de0005fd0 tx=0x7f5de002a530 comp rx=0 tx=0).stop 2024-09-10T20:39:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 -- 172.21.15.28:0/1001600444 shutdown_connections 2024-09-10T20:39:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1001600444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5df00d07a0 0x7f5df00b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1001600444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00cc570 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 -- 172.21.15.28:0/1001600444 >> 172.21.15.28:0/1001600444 conn(0x7f5df006cf00 msgr2=0x7f5df006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 -- 172.21.15.28:0/1001600444 shutdown_connections 2024-09-10T20:39:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.820+0000 7f5df58c6640 1 -- 172.21.15.28:0/1001600444 wait complete. 2024-09-10T20:39:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.821+0000 7f5df58c6640 1 Processor -- start 2024-09-10T20:39:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.821+0000 7f5df58c6640 1 -- start start 2024-09-10T20:39:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5df58c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5df00b4340 0x7f5df00b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5df58c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5df58c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5df00d07a0 0x7f5df00aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5df58c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5df00b33c0 con 0x7f5df00cd3d0 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5df58c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5df00b4af0 con 0x7f5df00b4340 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5df58c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5df00b3540 con 0x7f5df00d07a0 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5de7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5de7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00acd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50098/0 (socket says 172.21.15.28:50098) 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5de7fff640 1 -- 172.21.15.28:0/1333230495 learned_addr learned my addr 172.21.15.28:0/1333230495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5deeffd640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5df00b4340 0x7f5df00b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.822+0000 7f5def7fe640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5df00d07a0 0x7f5df00aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5de7fff640 1 -- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5df00d07a0 msgr2=0x7f5df00aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5de7fff640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5df00d07a0 0x7f5df00aec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5de7fff640 1 -- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5df00b4340 msgr2=0x7f5df00b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5de7fff640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5df00b4340 0x7f5df00b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5de7fff640 1 -- 172.21.15.28:0/1333230495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df00af7c0 con 0x7f5df00cd3d0 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5deeffd640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5df00b4340 0x7f5df00b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5de7fff640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00acd60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800c430 tx=0x7f5dd800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5decff9640 1 -- 172.21.15.28:0/1333230495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8011790 con 0x7f5df00cd3d0 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5decff9640 1 -- 172.21.15.28:0/1333230495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dd8017070 con 0x7f5df00cd3d0 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5decff9640 1 -- 172.21.15.28:0/1333230495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8011d00 con 0x7f5df00cd3d0 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df00ad3f0 con 0x7f5df00cd3d0 2024-09-10T20:39:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.823+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df00adc90 con 0x7f5df00cd3d0 2024-09-10T20:39:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.824+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df00b3a50 con 0x7f5df00cd3d0 2024-09-10T20:39:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.825+0000 7f5decff9640 1 -- 172.21.15.28:0/1333230495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f5dd801c460 con 0x7f5df00cd3d0 2024-09-10T20:39:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.825+0000 7f5decff9640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5dd0078120 0x7f5dd007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.826+0000 7f5decff9640 1 -- 172.21.15.28:0/1333230495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f5dd8099e30 con 0x7f5df00cd3d0 2024-09-10T20:39:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.826+0000 7f5deeffd640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5dd0078120 0x7f5dd007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.826+0000 7f5deeffd640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5dd0078120 0x7f5dd007a5e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5de0033000 tx=0x7f5de0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:09.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:09.830+0000 7f5decff9640 1 -- 172.21.15.28:0/1333230495 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5dd8061fe0 con 0x7f5df00cd3d0 2024-09-10T20:39:10.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.018+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5df00cef90 con 0x7f5df00cd3d0 2024-09-10T20:39:10.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.019+0000 7f5decff9640 1 -- 172.21.15.28:0/1333230495 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16669 (secure 0 0 0) 0x7f5dd8066a10 con 0x7f5df00cd3d0 2024-09-10T20:39:10.023 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:10.023 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:54.948562+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:51.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:52.258677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:39:10.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5dd0078120 msgr2=0x7f5dd007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5dd0078120 0x7f5dd007a5e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5de0033000 tx=0x7f5de0004730 comp rx=0 tx=0).stop 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 msgr2=0x7f5df00acd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00acd60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800c430 tx=0x7f5dd800c8f0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 shutdown_connections 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5dd0078120 0x7f5dd007a5e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5df00d07a0 0x7f5df00aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df00cd3d0 0x7f5df00acd60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 --2- 172.21.15.28:0/1333230495 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5df00b4340 0x7f5df00b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.021+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 >> 172.21.15.28:0/1333230495 conn(0x7f5df006cf00 msgr2=0x7f5df009b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.022+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 shutdown_connections 2024-09-10T20:39:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.022+0000 7f5df58c6640 1 -- 172.21.15.28:0/1333230495 wait complete. 2024-09-10T20:39:10.037 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:39:10.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.139+0000 7f864573d640 1 -- 172.21.15.28:0/3343719157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8628004d40 msgr2=0x7f86280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.139+0000 7f864573d640 1 --2- 172.21.15.28:0/3343719157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8628004d40 0x7f86280051c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8638009ed0 tx=0x7f8628005830 comp rx=0 tx=0).stop 2024-09-10T20:39:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.139+0000 7f863f7fe640 1 -- 172.21.15.28:0/3343719157 reap_dead start 2024-09-10T20:39:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.139+0000 7f864573d640 1 -- 172.21.15.28:0/3343719157 shutdown_connections 2024-09-10T20:39:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.139+0000 7f864573d640 1 -- 172.21.15.28:0/3343719157 >> 172.21.15.28:0/3343719157 conn(0x7f864006cf00 msgr2=0x7f864006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.139+0000 7f864573d640 1 -- 172.21.15.28:0/3343719157 shutdown_connections 2024-09-10T20:39:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.139+0000 7f864573d640 1 -- 172.21.15.28:0/3343719157 wait complete. 2024-09-10T20:39:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.141+0000 7f864573d640 1 Processor -- start 2024-09-10T20:39:10.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.141+0000 7f864573d640 1 -- start start 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f864573d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8628004d40 0x7f86400740b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f864573d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8640073b30 0x7f864014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f864573d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400b0ce0 0x7f864014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f864573d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86400b2e80 con 0x7f8640073b30 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f864573d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f86400b4b00 con 0x7f8628004d40 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f864573d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f86400b3000 con 0x7f86400b0ce0 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8640073b30 0x7f864014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8640073b30 0x7f864014e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50108/0 (socket says 172.21.15.28:50108) 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 -- 172.21.15.28:0/3099848093 learned_addr learned my addr 172.21.15.28:0/3099848093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863effd640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8628004d40 0x7f86400740b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863f7fe640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400b0ce0 0x7f864014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 -- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400b0ce0 msgr2=0x7f864014eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400b0ce0 0x7f864014eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 -- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8628004d40 msgr2=0x7f86400740b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8628004d40 0x7f86400740b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 -- 172.21.15.28:0/3099848093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86401f8ae0 con 0x7f8640073b30 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863effd640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8628004d40 0x7f86400740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863f7fe640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400b0ce0 0x7f864014eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f863e7fc640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8640073b30 0x7f864014e5f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8634011a80 tx=0x7f8634011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f86277fe640 1 -- 172.21.15.28:0/3099848093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863400fe10 con 0x7f8640073b30 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f86277fe640 1 -- 172.21.15.28:0/3099848093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8634016070 con 0x7f8640073b30 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86401f8dc0 con 0x7f8640073b30 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.142+0000 7f86277fe640 1 -- 172.21.15.28:0/3099848093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634012690 con 0x7f8640073b30 2024-09-10T20:39:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.143+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86401f9660 con 0x7f8640073b30 2024-09-10T20:39:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.144+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86400d0090 con 0x7f8640073b30 2024-09-10T20:39:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.144+0000 7f86277fe640 1 -- 172.21.15.28:0/3099848093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f8634012830 con 0x7f8640073b30 2024-09-10T20:39:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.145+0000 7f86277fe640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f862c078120 0x7f862c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.145+0000 7f86277fe640 1 -- 172.21.15.28:0/3099848093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f863409ad20 con 0x7f8640073b30 2024-09-10T20:39:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.145+0000 7f863effd640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f862c078120 0x7f862c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.146+0000 7f863effd640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f862c078120 0x7f862c07a5e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8630001c00 tx=0x7f8630001b90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.149+0000 7f86277fe640 1 -- 172.21.15.28:0/3099848093 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8634063520 con 0x7f8640073b30 2024-09-10T20:39:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.370+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86401f8f90 con 0x7f8640073b30 2024-09-10T20:39:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.371+0000 7f86277fe640 1 -- 172.21.15.28:0/3099848093 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5226 (secure 0 0 0) 0x7f8634062e20 con 0x7f8640073b30 2024-09-10T20:39:10.375 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:10.375 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"btime":"2024-09-10T20:39:02:975887+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1687267914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1687267914},{"type":"v1","addr":"172.21.15.159:6836","nonce":1687267914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/649109556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":649109556},{"type":"v1","addr":"172.21.15.159:6837","nonce":649109556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4851,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2646744984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2646744984},{"type":"v1","addr":"172.21.15.28:6834","nonce":2646744984}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:53.302329+0000","modified":"2024-09-10T20:38:56.323633+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4854":{"gid":4854,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1884641453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1884641453},{"type":"v1","addr":"172.21.15.28:6835","nonce":1884641453}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4854,"qdb_cluster":[4854]},"id":2}]} 2024-09-10T20:39:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f862c078120 msgr2=0x7f862c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f862c078120 0x7f862c07a5e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8630001c00 tx=0x7f8630001b90 comp rx=0 tx=0).stop 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8640073b30 msgr2=0x7f864014e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8640073b30 0x7f864014e5f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8634011a80 tx=0x7f8634011f40 comp rx=0 tx=0).stop 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 shutdown_connections 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f862c078120 0x7f862c07a5e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86400b0ce0 0x7f864014eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8640073b30 0x7f864014e5f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 --2- 172.21.15.28:0/3099848093 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8628004d40 0x7f86400740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 >> 172.21.15.28:0/3099848093 conn(0x7f864006cf00 msgr2=0x7f864015d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 shutdown_connections 2024-09-10T20:39:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.373+0000 7f864573d640 1 -- 172.21.15.28:0/3099848093 wait complete. 2024-09-10T20:39:10.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:10.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 -- 172.21.15.28:0/1131788207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 msgr2=0x7efe7c0b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 --2- 172.21.15.28:0/1131788207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c0b6e00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe7c0d4d80 tx=0x7efe7402aa30 comp rx=0 tx=0).stop 2024-09-10T20:39:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 -- 172.21.15.28:0/1131788207 shutdown_connections 2024-09-10T20:39:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 --2- 172.21.15.28:0/1131788207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c0b6e00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 --2- 172.21.15.28:0/1131788207 >> v2:172.21.15.159:3300/0 conn(0x7efe7c0d2b90 0x7efe7c161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 -- 172.21.15.28:0/1131788207 >> 172.21.15.28:0/1131788207 conn(0x7efe7c0d3fb0 msgr2=0x7efe7c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 -- 172.21.15.28:0/1131788207 shutdown_connections 2024-09-10T20:39:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.489+0000 7efe82033640 1 -- 172.21.15.28:0/1131788207 wait complete. 2024-09-10T20:39:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.491+0000 7efe82033640 1 Processor -- start 2024-09-10T20:39:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.491+0000 7efe82033640 1 -- start start 2024-09-10T20:39:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.491+0000 7efe82033640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7c0d2b90 0x7efe7c150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.491+0000 7efe82033640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe7c15de70 0x7efe7c1f5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe82033640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c1f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe82033640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe7c15e3d0 con 0x7efe7c162090 2024-09-10T20:39:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe82033640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe7c15e250 con 0x7efe7c15de70 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe82033640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe7c0b7340 con 0x7efe7c0d2b90 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c1f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c1f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56458/0 (socket says 172.21.15.28:56458) 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 -- 172.21.15.28:0/2112919414 learned_addr learned my addr 172.21.15.28:0/2112919414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7b7fe640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7c0d2b90 0x7efe7c150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7affd640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe7c15de70 0x7efe7c1f5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 -- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7c0d2b90 msgr2=0x7efe7c150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7c0d2b90 0x7efe7c150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 -- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe7c15de70 msgr2=0x7efe7c1f5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe7c15de70 0x7efe7c1f5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 -- 172.21.15.28:0/2112919414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7c1f5a70 con 0x7efe7c162090 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7b7fe640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7c0d2b90 0x7efe7c150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe7bfff640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c1f1ed0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efe74009d60 tx=0x7efe74003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe73fff640 1 -- 172.21.15.28:0/2112919414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe74033050 con 0x7efe7c162090 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe73fff640 1 -- 172.21.15.28:0/2112919414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe74004550 con 0x7efe7c162090 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe73fff640 1 -- 172.21.15.28:0/2112919414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe74038450 con 0x7efe7c162090 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe7c09ef80 con 0x7efe7c162090 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.492+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe7c09f500 con 0x7efe7c162090 2024-09-10T20:39:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.494+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe7c0b7830 con 0x7efe7c162090 2024-09-10T20:39:10.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.494+0000 7efe73fff640 1 -- 172.21.15.28:0/2112919414 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7efe74040070 con 0x7efe7c162090 2024-09-10T20:39:10.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.495+0000 7efe73fff640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe5c078120 0x7efe5c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.495+0000 7efe73fff640 1 -- 172.21.15.28:0/2112919414 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7efe740bb6a0 con 0x7efe7c162090 2024-09-10T20:39:10.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.495+0000 7efe7b7fe640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe5c078120 0x7efe5c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.496+0000 7efe7b7fe640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe5c078120 0x7efe5c07a5e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efe6c003eb0 tx=0x7efe6c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.499+0000 7efe73fff640 1 -- 172.21.15.28:0/2112919414 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efe7c1f5a70 con 0x7efe7c162090 2024-09-10T20:39:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.722+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe7c0b9f30 con 0x7efe7c162090 2024-09-10T20:39:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.722+0000 7efe73fff640 1 -- 172.21.15.28:0/2112919414 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5226 (secure 0 0 0) 0x7efe74083ea0 con 0x7efe7c162090 2024-09-10T20:39:10.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:10.727 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"btime":"2024-09-10T20:39:02:975887+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1687267914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1687267914},{"type":"v1","addr":"172.21.15.159:6836","nonce":1687267914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/649109556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":649109556},{"type":"v1","addr":"172.21.15.159:6837","nonce":649109556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4851,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2646744984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2646744984},{"type":"v1","addr":"172.21.15.28:6834","nonce":2646744984}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:38:53.302329+0000","modified":"2024-09-10T20:38:56.323633+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4854":{"gid":4854,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1884641453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1884641453},{"type":"v1","addr":"172.21.15.28:6835","nonce":1884641453}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4854,"qdb_cluster":[4854]},"id":2}]} 2024-09-10T20:39:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-10T20:39:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.724+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe5c078120 msgr2=0x7efe5c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.724+0000 7efe82033640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe5c078120 0x7efe5c07a5e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efe6c003eb0 tx=0x7efe6c009040 comp rx=0 tx=0).stop 2024-09-10T20:39:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 msgr2=0x7efe7c1f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c1f1ed0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efe74009d60 tx=0x7efe74003f10 comp rx=0 tx=0).stop 2024-09-10T20:39:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 shutdown_connections 2024-09-10T20:39:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe5c078120 0x7efe5c07a5e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe7c162090 0x7efe7c1f1ed0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe7c15de70 0x7efe7c1f5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 --2- 172.21.15.28:0/2112919414 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe7c0d2b90 0x7efe7c150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 >> 172.21.15.28:0/2112919414 conn(0x7efe7c0d3fb0 msgr2=0x7efe7c0b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 shutdown_connections 2024-09-10T20:39:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.725+0000 7efe82033640 1 -- 172.21.15.28:0/2112919414 wait complete. 2024-09-10T20:39:10.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 -- 172.21.15.28:0/1152402724 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 msgr2=0x7f9aa80cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1152402724 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 0x7f9aa80cc570 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4005fd0 tx=0x7f9aa402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 -- 172.21.15.28:0/1152402724 shutdown_connections 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1152402724 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa80d07a0 0x7f9aa80b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1152402724 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 0x7f9aa80cc570 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 -- 172.21.15.28:0/1152402724 >> 172.21.15.28:0/1152402724 conn(0x7f9aa806cf00 msgr2=0x7f9aa806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 -- 172.21.15.28:0/1152402724 shutdown_connections 2024-09-10T20:39:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.841+0000 7f9aafec6640 1 -- 172.21.15.28:0/1152402724 wait complete. 2024-09-10T20:39:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.842+0000 7f9aafec6640 1 Processor -- start 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.842+0000 7f9aafec6640 1 -- start start 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aafec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80b4340 0x7f9aa814f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aafec6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 0x7f9aa8072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aafec6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa80d07a0 0x7f9aa80b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aafec6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa80b33c0 con 0x7f9aa80b4340 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aafec6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa80b4af0 con 0x7f9aa80cd3d0 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aafec6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa80b3540 con 0x7f9aa80d07a0 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80b4340 0x7f9aa814f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80b4340 0x7f9aa814f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56464/0 (socket says 172.21.15.28:56464) 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 -- 172.21.15.28:0/1596836121 learned_addr learned my addr 172.21.15.28:0/1596836121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aad461640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 0x7f9aa8072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aae463640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa80d07a0 0x7f9aa80b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 -- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa80d07a0 msgr2=0x7f9aa80b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa80d07a0 0x7f9aa80b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 -- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 msgr2=0x7f9aa8072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 0x7f9aa8072850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 -- 172.21.15.28:0/1596836121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa8072d90 con 0x7f9aa80b4340 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9aadc62640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80b4340 0x7f9aa814f860 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.843+0000 7f9a9e7fc640 1 -- 172.21.15.28:0/1596836121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa403f070 con 0x7f9aa80b4340 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.844+0000 7f9a9e7fc640 1 -- 172.21.15.28:0/1596836121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9aa4003ce0 con 0x7f9aa80b4340 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.844+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa80734c0 con 0x7f9aa80b4340 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.844+0000 7f9a9e7fc640 1 -- 172.21.15.28:0/1596836121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4035030 con 0x7f9aa80b4340 2024-09-10T20:39:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.844+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa8073a80 con 0x7f9aa80b4340 2024-09-10T20:39:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.845+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa80b39f0 con 0x7f9aa80b4340 2024-09-10T20:39:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.845+0000 7f9a9e7fc640 1 -- 172.21.15.28:0/1596836121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f9aa4004a00 con 0x7f9aa80b4340 2024-09-10T20:39:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.846+0000 7f9a9e7fc640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9a8c078120 0x7f9a8c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.846+0000 7f9a9e7fc640 1 -- 172.21.15.28:0/1596836121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f9aa40baf90 con 0x7f9aa80b4340 2024-09-10T20:39:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.846+0000 7f9aad461640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9a8c078120 0x7f9a8c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.846+0000 7f9aad461640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9a8c078120 0x7f9a8c07a5e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9aa809b910 tx=0x7f9a98009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:10.850+0000 7f9a9e7fc640 1 -- 172.21.15.28:0/1596836121 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9aa4083790 con 0x7f9aa80b4340 2024-09-10T20:39:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.038+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9aa809ca10 con 0x7f9aa80b4340 2024-09-10T20:39:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.039+0000 7f9a9e7fc640 1 -- 172.21.15.28:0/1596836121 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16669 (secure 0 0 0) 0x7f9aa4083090 con 0x7f9aa80b4340 2024-09-10T20:39:11.043 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:11.043 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:38:54.948562+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:38:51.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-10T20:38:52.258677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:39:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9a8c078120 msgr2=0x7f9a8c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9a8c078120 0x7f9a8c07a5e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9aa809b910 tx=0x7f9a98009040 comp rx=0 tx=0).stop 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80b4340 msgr2=0x7f9aa814f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80b4340 0x7f9aa814f860 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa40035f0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 shutdown_connections 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9a8c078120 0x7f9a8c07a5e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa80d07a0 0x7f9aa80b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa80cd3d0 0x7f9aa8072850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 --2- 172.21.15.28:0/1596836121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80b4340 0x7f9aa814f860 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 >> 172.21.15.28:0/1596836121 conn(0x7f9aa806cf00 msgr2=0x7f9aa80cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 shutdown_connections 2024-09-10T20:39:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.042+0000 7f9aafec6640 1 -- 172.21.15.28:0/1596836121 wait complete. 2024-09-10T20:39:11.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:39:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.159+0000 7f1c6129b640 1 -- 172.21.15.28:0/1521914665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c44004d90 msgr2=0x7f1c44005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.159+0000 7f1c6129b640 1 --2- 172.21.15.28:0/1521914665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c44004d90 0x7f1c44005210 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1c54005fd0 tx=0x7f1c44005830 comp rx=0 tx=0).stop 2024-09-10T20:39:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.159+0000 7f1c5b7fe640 1 -- 172.21.15.28:0/1521914665 reap_dead start 2024-09-10T20:39:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.159+0000 7f1c6129b640 1 -- 172.21.15.28:0/1521914665 shutdown_connections 2024-09-10T20:39:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.159+0000 7f1c6129b640 1 -- 172.21.15.28:0/1521914665 >> 172.21.15.28:0/1521914665 conn(0x7f1c5c0d4040 msgr2=0x7f1c5c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.159+0000 7f1c6129b640 1 -- 172.21.15.28:0/1521914665 shutdown_connections 2024-09-10T20:39:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.159+0000 7f1c6129b640 1 -- 172.21.15.28:0/1521914665 wait complete. 2024-09-10T20:39:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.161+0000 7f1c6129b640 1 Processor -- start 2024-09-10T20:39:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.161+0000 7f1c6129b640 1 -- start start 2024-09-10T20:39:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c44004d90 0x7f1c5c14ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c5c1f93a0 0x7f1c5c154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5c0cdf80 0x7f1c5c155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5c1670b0 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5c166f30 con 0x7f1c44004d90 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c5c167230 con 0x7f1c5c1f93a0 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5c0cdf80 0x7f1c5c155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5c0cdf80 0x7f1c5c155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56488/0 (socket says 172.21.15.28:56488) 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 -- 172.21.15.28:0/1001997028 learned_addr learned my addr 172.21.15.28:0/1001997028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5affd640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c44004d90 0x7f1c5c14ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5a7fc640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c5c1f93a0 0x7f1c5c154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 -- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c5c1f93a0 msgr2=0x7f1c5c154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c5c1f93a0 0x7f1c5c154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 -- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c44004d90 msgr2=0x7f1c5c14ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c44004d90 0x7f1c5c14ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 -- 172.21.15.28:0/1001997028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5c1f4f40 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5affd640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c44004d90 0x7f1c5c14ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c5b7fe640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5c0cdf80 0x7f1c5c155380 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009f10 tx=0x7f1c5402bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c437fe640 1 -- 172.21.15.28:0/1001997028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c54042070 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c437fe640 1 -- 172.21.15.28:0/1001997028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c54033770 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c5c154a30 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c437fe640 1 -- 172.21.15.28:0/1001997028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c54038040 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.162+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c5c1f8ba0 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.163+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c5c0d27f0 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.164+0000 7f1c437fe640 1 -- 172.21.15.28:0/1001997028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f1c5403b070 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.164+0000 7f1c437fe640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c48078150 0x7f1c4807a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.165+0000 7f1c437fe640 1 -- 172.21.15.28:0/1001997028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f1c540b98f0 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.165+0000 7f1c5affd640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c48078150 0x7f1c4807a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.165+0000 7f1c5affd640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c48078150 0x7f1c4807a610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c006c10 tx=0x7f1c4c0067a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:11.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.169+0000 7f1c437fe640 1 -- 172.21.15.28:0/1001997028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c54081970 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.392+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1c5c0b9fc0 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.396 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:39:11.392+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:39:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.394+0000 7f1c437fe640 1 -- 172.21.15.28:0/1001997028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7f1c540863a0 con 0x7f1c5c0cdf80 2024-09-10T20:39:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:39:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c48078150 msgr2=0x7f1c4807a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c48078150 0x7f1c4807a610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c006c10 tx=0x7f1c4c0067a0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5c0cdf80 msgr2=0x7f1c5c155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5c0cdf80 0x7f1c5c155380 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009f10 tx=0x7f1c5402bf90 comp rx=0 tx=0).stop 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 shutdown_connections 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c48078150 0x7f1c4807a610 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5c0cdf80 0x7f1c5c155380 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c5c1f93a0 0x7f1c5c154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 --2- 172.21.15.28:0/1001997028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c44004d90 0x7f1c5c14ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.396+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 >> 172.21.15.28:0/1001997028 conn(0x7f1c5c0d4040 msgr2=0x7f1c5c159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.397+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 shutdown_connections 2024-09-10T20:39:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.397+0000 7f1c6129b640 1 -- 172.21.15.28:0/1001997028 wait complete. 2024-09-10T20:39:11.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:39:11.439 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:39:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 -- 172.21.15.28:0/1463211021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 msgr2=0x7faf9c0b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 --2- 172.21.15.28:0/1463211021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c0b3f60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7faf94005fd0 tx=0x7faf9402aa90 comp rx=0 tx=0).stop 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 -- 172.21.15.28:0/1463211021 shutdown_connections 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 --2- 172.21.15.28:0/1463211021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c0b3f60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 --2- 172.21.15.28:0/1463211021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf9c0b56e0 0x7faf9c0b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 -- 172.21.15.28:0/1463211021 >> 172.21.15.28:0/1463211021 conn(0x7faf9c0d40f0 msgr2=0x7faf9c0d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 -- 172.21.15.28:0/1463211021 shutdown_connections 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.517+0000 7fafa255d640 1 -- 172.21.15.28:0/1463211021 wait complete. 2024-09-10T20:39:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.519+0000 7fafa255d640 1 Processor -- start 2024-09-10T20:39:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.519+0000 7fafa255d640 1 -- start start 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.519+0000 7fafa255d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf9c0b56e0 0x7faf9c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa255d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf9c0d2140 0x7faf9c160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa255d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c1608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa255d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c0b2590 con 0x7faf9c0d2c30 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa255d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c0b2410 con 0x7faf9c0d2140 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa255d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faf9c0b2710 con 0x7faf9c0b56e0 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c1608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c1608d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56542/0 (socket says 172.21.15.28:56542) 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 -- 172.21.15.28:0/3640441261 learned_addr learned my addr 172.21.15.28:0/3640441261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7faf9bfff640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf9c0b56e0 0x7faf9c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7faf9b7fe640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf9c0d2140 0x7faf9c160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 -- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf9c0b56e0 msgr2=0x7faf9c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf9c0b56e0 0x7faf9c0cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 -- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf9c0d2140 msgr2=0x7faf9c160390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf9c0d2140 0x7faf9c160390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 -- 172.21.15.28:0/3640441261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c0ccbe0 con 0x7faf9c0d2c30 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa0afa640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c1608d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7faf94009d90 tx=0x7faf94003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7faf98ff9640 1 -- 172.21.15.28:0/3640441261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf94044070 con 0x7faf9c0d2c30 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7faf98ff9640 1 -- 172.21.15.28:0/3640441261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf94004250 con 0x7faf9c0d2c30 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9c0ccec0 con 0x7faf9c0d2c30 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7faf98ff9640 1 -- 172.21.15.28:0/3640441261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf94004680 con 0x7faf9c0d2c30 2024-09-10T20:39:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.520+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9c0cd760 con 0x7faf9c0d2c30 2024-09-10T20:39:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.521+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9c0cd960 con 0x7faf9c0d2c30 2024-09-10T20:39:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.522+0000 7faf98ff9640 1 -- 172.21.15.28:0/3640441261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7faf9402e970 con 0x7faf9c0d2c30 2024-09-10T20:39:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.523+0000 7faf98ff9640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faf88078120 0x7faf8807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.523+0000 7faf98ff9640 1 -- 172.21.15.28:0/3640441261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7faf940bb390 con 0x7faf9c0d2c30 2024-09-10T20:39:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.523+0000 7faf9bfff640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faf88078120 0x7faf8807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.523+0000 7faf9bfff640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faf88078120 0x7faf8807a5e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0017c0 tx=0x7faf8c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.527+0000 7faf98ff9640 1 -- 172.21.15.28:0/3640441261 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faf9c0cd960 con 0x7faf9c0d2c30 2024-09-10T20:39:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:11.750+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7faf9c0d2520 con 0x7faf9c0d2c30 2024-09-10T20:39:12.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.399+0000 7faf98ff9640 1 -- 172.21.15.28:0/3640441261 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7faf94032030 con 0x7faf9c0d2c30 2024-09-10T20:39:12.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faf88078120 msgr2=0x7faf8807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:12.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faf88078120 0x7faf8807a5e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0017c0 tx=0x7faf8c009040 comp rx=0 tx=0).stop 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 msgr2=0x7faf9c1608d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c1608d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7faf94009d90 tx=0x7faf94003f10 comp rx=0 tx=0).stop 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 shutdown_connections 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faf88078120 0x7faf8807a5e0 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0017c0 tx=0x7faf8c009040 comp rx=0 tx=0).stop 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9c0d2c30 0x7faf9c1608d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faf9c0d2140 0x7faf9c160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 --2- 172.21.15.28:0/3640441261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faf9c0b56e0 0x7faf9c0cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 >> 172.21.15.28:0/3640441261 conn(0x7faf9c0d40f0 msgr2=0x7faf9c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 shutdown_connections 2024-09-10T20:39:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.402+0000 7fafa255d640 1 -- 172.21.15.28:0/3640441261 wait complete. 2024-09-10T20:39:12.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:39:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.576+0000 7f5a652dd640 1 -- 172.21.15.28:0/1090131768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cda70 msgr2=0x7f5a600cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.576+0000 7f5a652dd640 1 --2- 172.21.15.28:0/1090131768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cda70 0x7f5a600cde80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5a58005fd0 tx=0x7f5a5802b8f0 comp rx=0 tx=0).stop 2024-09-10T20:39:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.576+0000 7f5a652dd640 1 -- 172.21.15.28:0/1090131768 shutdown_connections 2024-09-10T20:39:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.576+0000 7f5a652dd640 1 --2- 172.21.15.28:0/1090131768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cda70 0x7f5a600cde80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.576+0000 7f5a652dd640 1 -- 172.21.15.28:0/1090131768 >> 172.21.15.28:0/1090131768 conn(0x7f5a600cfc80 msgr2=0x7f5a600d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.576+0000 7f5a652dd640 1 -- 172.21.15.28:0/1090131768 shutdown_connections 2024-09-10T20:39:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.576+0000 7f5a652dd640 1 -- 172.21.15.28:0/1090131768 wait complete. 2024-09-10T20:39:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.577+0000 7f5a652dd640 1 Processor -- start 2024-09-10T20:39:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.577+0000 7f5a652dd640 1 -- start start 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a652dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cabc0 0x7f5a6015aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a652dd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a600cda70 0x7f5a6014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a652dd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6015a350 0x7f5a600c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a652dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6006e720 con 0x7f5a600cabc0 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a652dd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6006e5a0 con 0x7f5a600cda70 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a652dd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5a6006e8a0 con 0x7f5a6015a350 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cabc0 0x7f5a6015aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cabc0 0x7f5a6015aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56562/0 (socket says 172.21.15.28:56562) 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 -- 172.21.15.28:0/3232916286 learned_addr learned my addr 172.21.15.28:0/3232916286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5e7fc640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a600cda70 0x7f5a6014d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5f7fe640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6015a350 0x7f5a600c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 -- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6015a350 msgr2=0x7f5a600c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6015a350 0x7f5a600c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 -- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a600cda70 msgr2=0x7f5a6014d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a600cda70 0x7f5a6014d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 -- 172.21.15.28:0/3232916286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a600c6fa0 con 0x7f5a600cabc0 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5f7fe640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6015a350 0x7f5a600c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.578+0000 7f5a5effd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cabc0 0x7f5a6015aa40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000b3f0 tx=0x7f5a5000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.579+0000 7f5a477fe640 1 -- 172.21.15.28:0/3232916286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a50015020 con 0x7f5a600cabc0 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.579+0000 7f5a477fe640 1 -- 172.21.15.28:0/3232916286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a50011920 con 0x7f5a600cabc0 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.579+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a600c7280 con 0x7f5a600cabc0 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.579+0000 7f5a477fe640 1 -- 172.21.15.28:0/3232916286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a50011d70 con 0x7f5a600cabc0 2024-09-10T20:39:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.579+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a600c8930 con 0x7f5a600cabc0 2024-09-10T20:39:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.580+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a600cde80 con 0x7f5a600cabc0 2024-09-10T20:39:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.580+0000 7f5a477fe640 1 -- 172.21.15.28:0/3232916286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f5a5001a400 con 0x7f5a600cabc0 2024-09-10T20:39:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.581+0000 7f5a477fe640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a4c078120 0x7f5a4c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.581+0000 7f5a477fe640 1 -- 172.21.15.28:0/3232916286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f5a5009a0a0 con 0x7f5a600cabc0 2024-09-10T20:39:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.581+0000 7f5a5e7fc640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a4c078120 0x7f5a4c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.582+0000 7f5a5e7fc640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a4c078120 0x7f5a4c07a5e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a60155ea0 tx=0x7f5a54009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.585+0000 7f5a477fe640 1 -- 172.21.15.28:0/3232916286 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5a5001f030 con 0x7f5a600cabc0 2024-09-10T20:39:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:12.773+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5a600c7450 con 0x7f5a600cabc0 2024-09-10T20:39:13.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.400+0000 7f5a477fe640 1 -- 172.21.15.28:0/3232916286 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v34) ==== 181+0+0 (secure 0 0 0) 0x7f5a500627b0 con 0x7f5a600cabc0 2024-09-10T20:39:13.404 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a4c078120 msgr2=0x7f5a4c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a4c078120 0x7f5a4c07a5e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a60155ea0 tx=0x7f5a54009040 comp rx=0 tx=0).stop 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cabc0 msgr2=0x7f5a6015aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cabc0 0x7f5a6015aa40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000b3f0 tx=0x7f5a5000b8b0 comp rx=0 tx=0).stop 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 shutdown_connections 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a4c078120 0x7f5a4c07a5e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6015a350 0x7f5a600c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a600cda70 0x7f5a6014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 --2- 172.21.15.28:0/3232916286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a600cabc0 0x7f5a6015aa40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 >> 172.21.15.28:0/3232916286 conn(0x7f5a600cfc80 msgr2=0x7f5a600b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 shutdown_connections 2024-09-10T20:39:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.404+0000 7f5a652dd640 1 -- 172.21.15.28:0/3232916286 wait complete. 2024-09-10T20:39:13.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:39:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.535+0000 7f43bf3d3640 1 -- 172.21.15.28:0/846496783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 msgr2=0x7f43a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.535+0000 7f43bf3d3640 1 --2- 172.21.15.28:0/846496783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 0x7f43a4004c80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f43a8001630 tx=0x7f43a4005830 comp rx=0 tx=0).stop 2024-09-10T20:39:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.535+0000 7f43bd970640 1 -- 172.21.15.28:0/846496783 reap_dead start 2024-09-10T20:39:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.535+0000 7f43bf3d3640 1 -- 172.21.15.28:0/846496783 shutdown_connections 2024-09-10T20:39:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.535+0000 7f43bf3d3640 1 -- 172.21.15.28:0/846496783 >> 172.21.15.28:0/846496783 conn(0x7f43b80d0410 msgr2=0x7f43b80d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.535+0000 7f43bf3d3640 1 -- 172.21.15.28:0/846496783 shutdown_connections 2024-09-10T20:39:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.535+0000 7f43bf3d3640 1 -- 172.21.15.28:0/846496783 wait complete. 2024-09-10T20:39:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.536+0000 7f43bf3d3640 1 Processor -- start 2024-09-10T20:39:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.536+0000 7f43bf3d3640 1 -- start start 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bf3d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 0x7f43b80ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bf3d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43b8152000 0x7f43b8099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bf3d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43b80b1f90 0x7f43b80b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bf3d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43b80cf750 con 0x7f43a4004800 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bf3d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f43b80cf5d0 con 0x7f43b8152000 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bf3d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f43b80b07d0 con 0x7f43b80b1f90 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 0x7f43b80ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 0x7f43b80ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56582/0 (socket says 172.21.15.28:56582) 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 -- 172.21.15.28:0/2640387243 learned_addr learned my addr 172.21.15.28:0/2640387243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bc96e640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43b8152000 0x7f43b8099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd970640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43b80b1f90 0x7f43b80b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 -- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43b80b1f90 msgr2=0x7f43b80b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43b80b1f90 0x7f43b80b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 -- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43b8152000 msgr2=0x7f43b8099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43b8152000 0x7f43b8099910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 -- 172.21.15.28:0/2640387243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b80b9990 con 0x7f43a4004800 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd970640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43b80b1f90 0x7f43b80b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.537+0000 7f43bd16f640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 0x7f43b80ae830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f43b4004650 tx=0x7f43b400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.538+0000 7f43adffb640 1 -- 172.21.15.28:0/2640387243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b40107f0 con 0x7f43a4004800 2024-09-10T20:39:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.538+0000 7f43adffb640 1 -- 172.21.15.28:0/2640387243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43b4010990 con 0x7f43a4004800 2024-09-10T20:39:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.538+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b81f66c0 con 0x7f43a4004800 2024-09-10T20:39:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.538+0000 7f43adffb640 1 -- 172.21.15.28:0/2640387243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4010dc0 con 0x7f43a4004800 2024-09-10T20:39:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.538+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b8099e50 con 0x7f43a4004800 2024-09-10T20:39:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.539+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b81528c0 con 0x7f43a4004800 2024-09-10T20:39:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.539+0000 7f43adffb640 1 -- 172.21.15.28:0/2640387243 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f43b4023440 con 0x7f43a4004800 2024-09-10T20:39:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.540+0000 7f43adffb640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f439c078150 0x7f439c07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.540+0000 7f43adffb640 1 -- 172.21.15.28:0/2640387243 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5357+0+0 (secure 0 0 0) 0x7f43b409abd0 con 0x7f43a4004800 2024-09-10T20:39:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.540+0000 7f43bc96e640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f439c078150 0x7f439c07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.541+0000 7f43bc96e640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f439c078150 0x7f439c07a610 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f43a8032000 tx=0x7f43a802c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.544+0000 7f43adffb640 1 -- 172.21.15.28:0/2640387243 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f43b4063410 con 0x7f43a4004800 2024-09-10T20:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:13.729+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f43b815b5a0 con 0x7f43a4004800 2024-09-10T20:39:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.406+0000 7f43adffb640 1 -- 172.21.15.28:0/2640387243 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v35) ==== 169+0+0 (secure 0 0 0) 0x7f43b4062d10 con 0x7f43a4004800 2024-09-10T20:39:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:39:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.409+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f439c078150 msgr2=0x7f439c07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.409+0000 7f43bf3d3640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f439c078150 0x7f439c07a610 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f43a8032000 tx=0x7f43a802c040 comp rx=0 tx=0).stop 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.409+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 msgr2=0x7f43b80ae830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.409+0000 7f43bf3d3640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 0x7f43b80ae830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f43b4004650 tx=0x7f43b400ba40 comp rx=0 tx=0).stop 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 shutdown_connections 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f439c078150 0x7f439c07a610 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43b80b1f90 0x7f43b80b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43b8152000 0x7f43b8099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 --2- 172.21.15.28:0/2640387243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43a4004800 0x7f43b80ae830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 >> 172.21.15.28:0/2640387243 conn(0x7f43b80d0410 msgr2=0x7f43b80cf900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 shutdown_connections 2024-09-10T20:39:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.410+0000 7f43bf3d3640 1 -- 172.21.15.28:0/2640387243 wait complete. 2024-09-10T20:39:14.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 -- 172.21.15.28:0/1515930215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 msgr2=0x7f78d40cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/1515930215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40cb210 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f78cc005fd0 tx=0x7f78cc02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 -- 172.21.15.28:0/1515930215 shutdown_connections 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/1515930215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40cb210 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/1515930215 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d4151cc0 0x7f78d40cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 -- 172.21.15.28:0/1515930215 >> 172.21.15.28:0/1515930215 conn(0x7f78d40d0420 msgr2=0x7f78d40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 -- 172.21.15.28:0/1515930215 shutdown_connections 2024-09-10T20:39:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.533+0000 7f78dbe9b640 1 -- 172.21.15.28:0/1515930215 wait complete. 2024-09-10T20:39:14.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.534+0000 7f78dbe9b640 1 Processor -- start 2024-09-10T20:39:14.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.534+0000 7f78dbe9b640 1 -- start start 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78dbe9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d40738b0 0x7f78d41f8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78dbe9b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d4151cc0 0x7f78d40b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78dbe9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78dbe9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78d4073e70 con 0x7f78d415eef0 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78dbe9b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f78d4073cf0 con 0x7f78d40738b0 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78dbe9b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f78d4073ff0 con 0x7f78d4151cc0 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56600/0 (socket says 172.21.15.28:56600) 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 -- 172.21.15.28:0/40910319 learned_addr learned my addr 172.21.15.28:0/40910319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78d9c37640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d40738b0 0x7f78d41f8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78d9436640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d4151cc0 0x7f78d40b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 -- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d4151cc0 msgr2=0x7f78d40b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d4151cc0 0x7f78d40b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 -- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d40738b0 msgr2=0x7f78d41f8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d40738b0 0x7f78d41f8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78da438640 1 -- 172.21.15.28:0/40910319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d41f5de0 con 0x7f78d415eef0 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78d9c37640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d40738b0 0x7f78d41f8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.535+0000 7f78d9436640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d4151cc0 0x7f78d40b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.536+0000 7f78da438640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40c9770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f78cc006010 tx=0x7f78cc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.536+0000 7f78ca7fc640 1 -- 172.21.15.28:0/40910319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc031b90 con 0x7f78d415eef0 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.536+0000 7f78ca7fc640 1 -- 172.21.15.28:0/40910319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78cc031e20 con 0x7f78d415eef0 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.536+0000 7f78ca7fc640 1 -- 172.21.15.28:0/40910319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc033310 con 0x7f78d415eef0 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.536+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d40b0e00 con 0x7f78d415eef0 2024-09-10T20:39:14.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.536+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d40b13c0 con 0x7f78d415eef0 2024-09-10T20:39:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.537+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d40745f0 con 0x7f78d415eef0 2024-09-10T20:39:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.538+0000 7f78ca7fc640 1 -- 172.21.15.28:0/40910319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f78cc031660 con 0x7f78d415eef0 2024-09-10T20:39:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.538+0000 7f78ca7fc640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f78b8078120 0x7f78b807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.538+0000 7f78ca7fc640 1 -- 172.21.15.28:0/40910319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7f78cc0bb470 con 0x7f78d415eef0 2024-09-10T20:39:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.539+0000 7f78d9c37640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f78b8078120 0x7f78b807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.539+0000 7f78d9c37640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f78b8078120 0x7f78b807a5e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78d0005fd0 tx=0x7f78d0005f40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.542+0000 7f78ca7fc640 1 -- 172.21.15.28:0/40910319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f78d40745f0 con 0x7f78d415eef0 2024-09-10T20:39:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.730+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f78d40cba50 con 0x7f78d415eef0 2024-09-10T20:39:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.733+0000 7f78ca7fc640 1 -- 172.21.15.28:0/40910319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v29) ==== 71+0+0 (secure 0 0 0) 0x7f78cc03f030 con 0x7f78d415eef0 2024-09-10T20:39:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f78b8078120 msgr2=0x7f78b807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f78b8078120 0x7f78b807a5e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78d0005fd0 tx=0x7f78d0005f40 comp rx=0 tx=0).stop 2024-09-10T20:39:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 msgr2=0x7f78d40c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40c9770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f78cc006010 tx=0x7f78cc003ba0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 shutdown_connections 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f78b8078120 0x7f78b807a5e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d415eef0 0x7f78d40c9770 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78d4151cc0 0x7f78d40b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 --2- 172.21.15.28:0/40910319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78d40738b0 0x7f78d41f8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 >> 172.21.15.28:0/40910319 conn(0x7f78d40d0420 msgr2=0x7f78d415db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 shutdown_connections 2024-09-10T20:39:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.735+0000 7f78dbe9b640 1 -- 172.21.15.28:0/40910319 wait complete. 2024-09-10T20:39:14.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-09-10T20:39:14.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 -- 172.21.15.28:0/3152080325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee415e0f0 msgr2=0x7f9ee40d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/3152080325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee415e0f0 0x7f9ee40d2030 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9ee408c590 tx=0x7f9ed402a8b0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 -- 172.21.15.28:0/3152080325 shutdown_connections 2024-09-10T20:39:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/3152080325 >> v2:172.21.15.159:3300/0 conn(0x7f9ee40d4c40 0x7f9ee40d2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/3152080325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee415e0f0 0x7f9ee40d2030 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 -- 172.21.15.28:0/3152080325 >> 172.21.15.28:0/3152080325 conn(0x7f9ee40d4040 msgr2=0x7f9ee40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 -- 172.21.15.28:0/3152080325 shutdown_connections 2024-09-10T20:39:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.857+0000 7f9eec1a3640 1 -- 172.21.15.28:0/3152080325 wait complete. 2024-09-10T20:39:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.858+0000 7f9eec1a3640 1 Processor -- start 2024-09-10T20:39:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9eec1a3640 1 -- start start 2024-09-10T20:39:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9eec1a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee40d2ab0 0x7f9ee40cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9eec1a3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ee40d4c40 0x7f9ee415c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9eec1a3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ee415e0f0 0x7f9ee40c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9eec1a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee40cf8b0 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9eec1a3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee40b8a10 con 0x7f9ee415e0f0 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9eec1a3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee4162320 con 0x7f9ee40d4c40 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9ee9f3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee40d2ab0 0x7f9ee40cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9ee9f3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee40d2ab0 0x7f9ee40cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56612/0 (socket says 172.21.15.28:56612) 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.859+0000 7f9ee9f3f640 1 -- 172.21.15.28:0/477106910 learned_addr learned my addr 172.21.15.28:0/477106910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee973e640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ee40d4c40 0x7f9ee415c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee9f3f640 1 -- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ee40d4c40 msgr2=0x7f9ee415c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9eea740640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ee415e0f0 0x7f9ee40c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee9f3f640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ee40d4c40 0x7f9ee415c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee9f3f640 1 -- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ee415e0f0 msgr2=0x7f9ee40c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee9f3f640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ee415e0f0 0x7f9ee40c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee9f3f640 1 -- 172.21.15.28:0/477106910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee41f5050 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee973e640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ee40d4c40 0x7f9ee415c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9eea740640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ee415e0f0 0x7f9ee40c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee9f3f640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee40d2ab0 0x7f9ee40cc9d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009b30 tx=0x7f9ed4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee27fc640 1 -- 172.21.15.28:0/477106910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4031da0 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee27fc640 1 -- 172.21.15.28:0/477106910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ed4003e10 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee41f5330 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9ee27fc640 1 -- 172.21.15.28:0/477106910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4033310 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.860+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee41f58f0 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.861+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee4162840 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.862+0000 7f9ee27fc640 1 -- 172.21.15.28:0/477106910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f9ed40334b0 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.862+0000 7f9ee27fc640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ed0078120 0x7f9ed007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.862+0000 7f9ee27fc640 1 -- 172.21.15.28:0/477106910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7f9ed40bb4a0 con 0x7f9ee40d2ab0 2024-09-10T20:39:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.863+0000 7f9ee973e640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ed0078120 0x7f9ed007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.863+0000 7f9ee973e640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ed0078120 0x7f9ed007a5e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40ce920 tx=0x7f9ed8005d30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:14.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:14.866+0000 7f9ee27fc640 1 -- 172.21.15.28:0/477106910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ed403f030 con 0x7f9ee40d2ab0 2024-09-10T20:39:15.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.122+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f9ee406e2d0 con 0x7f9ee40d2ab0 2024-09-10T20:39:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.407+0000 7f9ee27fc640 1 -- 172.21.15.28:0/477106910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v80) ==== 158+0+0 (secure 0 0 0) 0x7f9ed40337d0 con 0x7f9ee40d2ab0 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ed0078120 msgr2=0x7f9ed007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ed0078120 0x7f9ed007a5e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40ce920 tx=0x7f9ed8005d30 comp rx=0 tx=0).stop 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee40d2ab0 msgr2=0x7f9ee40cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee40d2ab0 0x7f9ee40cc9d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009b30 tx=0x7f9ed4003c20 comp rx=0 tx=0).stop 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 shutdown_connections 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ed0078120 0x7f9ed007a5e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ee415e0f0 0x7f9ee40c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ee40d4c40 0x7f9ee415c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 --2- 172.21.15.28:0/477106910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee40d2ab0 0x7f9ee40cc9d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 >> 172.21.15.28:0/477106910 conn(0x7f9ee40d4040 msgr2=0x7f9ee406ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 shutdown_connections 2024-09-10T20:39:15.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.410+0000 7f9eec1a3640 1 -- 172.21.15.28:0/477106910 wait complete. 2024-09-10T20:39:15.425 INFO:tasks.cephfs_test_runner:Validate binary .meta file on legacy subvol root doesn't break the system ... ok 2024-09-10T20:39:15.426 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:39:15.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.528+0000 7f461df3d640 1 -- 172.21.15.28:0/3314041324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 msgr2=0x7f46180cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.528+0000 7f461df3d640 1 --2- 172.21.15.28:0/3314041324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180cf0f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4608009830 tx=0x7f460802a500 comp rx=0 tx=0).stop 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.529+0000 7f461df3d640 1 -- 172.21.15.28:0/3314041324 shutdown_connections 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.529+0000 7f461df3d640 1 --2- 172.21.15.28:0/3314041324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f461815eef0 0x7f46180cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.529+0000 7f461df3d640 1 --2- 172.21.15.28:0/3314041324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180cf0f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.529+0000 7f461df3d640 1 -- 172.21.15.28:0/3314041324 >> 172.21.15.28:0/3314041324 conn(0x7f46180d0420 msgr2=0x7f46180d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.529+0000 7f461df3d640 1 -- 172.21.15.28:0/3314041324 shutdown_connections 2024-09-10T20:39:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.529+0000 7f461df3d640 1 -- 172.21.15.28:0/3314041324 wait complete. 2024-09-10T20:39:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.530+0000 7f461df3d640 1 Processor -- start 2024-09-10T20:39:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.530+0000 7f461df3d640 1 -- start start 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f461df3d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46180738b0 0x7f46181f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f461df3d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f461df3d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f461815eef0 0x7f46181475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f461df3d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4618073e70 con 0x7f4618151cc0 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f461df3d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4618073cf0 con 0x7f461815eef0 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f461df3d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4618073ff0 con 0x7f46180738b0 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180b0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56624/0 (socket says 172.21.15.28:56624) 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 -- 172.21.15.28:0/213539876 learned_addr learned my addr 172.21.15.28:0/213539876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4617fff640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f461815eef0 0x7f46181475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f46177fe640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46180738b0 0x7f46181f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 -- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46180738b0 msgr2=0x7f46181f6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46180738b0 0x7f46181f6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 -- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f461815eef0 msgr2=0x7f46181475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f461815eef0 0x7f46181475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 -- 172.21.15.28:0/213539876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46181f3210 con 0x7f4618151cc0 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4617fff640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f461815eef0 0x7f46181475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f4616ffd640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180b0060 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f460c00c450 tx=0x7f460c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.531+0000 7f45fffff640 1 -- 172.21.15.28:0/213539876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c016020 con 0x7f4618151cc0 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.532+0000 7f45fffff640 1 -- 172.21.15.28:0/213539876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f460c012920 con 0x7f4618151cc0 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.532+0000 7f45fffff640 1 -- 172.21.15.28:0/213539876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c012d70 con 0x7f4618151cc0 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.532+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46181f34f0 con 0x7f4618151cc0 2024-09-10T20:39:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.532+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46181f3ab0 con 0x7f4618151cc0 2024-09-10T20:39:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.533+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46180cc1b0 con 0x7f4618151cc0 2024-09-10T20:39:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.533+0000 7f45fffff640 1 -- 172.21.15.28:0/213539876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f460c01b400 con 0x7f4618151cc0 2024-09-10T20:39:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.534+0000 7f45fffff640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4604078120 0x7f460407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.534+0000 7f45fffff640 1 -- 172.21.15.28:0/213539876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7f460c09a120 con 0x7f4618151cc0 2024-09-10T20:39:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.534+0000 7f46177fe640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4604078120 0x7f460407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.534+0000 7f46177fe640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4604078120 0x7f460407a5e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4608034000 tx=0x7f4608004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.538+0000 7f45fffff640 1 -- 172.21.15.28:0/213539876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f460c020020 con 0x7f4618151cc0 2024-09-10T20:39:15.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.726+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4618068a90 con 0x7f4618151cc0 2024-09-10T20:39:15.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.727+0000 7f45fffff640 1 -- 172.21.15.28:0/213539876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13032 (secure 0 0 0) 0x7f460c062c20 con 0x7f4618151cc0 2024-09-10T20:39:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:15.731 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:39:14.401657+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1884641453":"2024-09-11T20:39:11.393189+0000","172.21.15.28:6833/1884641453":"2024-09-11T20:39:11.393189+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4604078120 msgr2=0x7f460407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4604078120 0x7f460407a5e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4608034000 tx=0x7f4608004730 comp rx=0 tx=0).stop 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 msgr2=0x7f46180b0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180b0060 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f460c00c450 tx=0x7f460c00c910 comp rx=0 tx=0).stop 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 shutdown_connections 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4604078120 0x7f460407a5e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f461815eef0 0x7f46181475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618151cc0 0x7f46180b0060 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 --2- 172.21.15.28:0/213539876 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46180738b0 0x7f46181f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 >> 172.21.15.28:0/213539876 conn(0x7f46180d0420 msgr2=0x7f46180cd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:15.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 shutdown_connections 2024-09-10T20:39:15.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.729+0000 7f461df3d640 1 -- 172.21.15.28:0/213539876 wait complete. 2024-09-10T20:39:15.744 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:39:15.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:39:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.849+0000 7fb696b17640 1 -- 172.21.15.28:0/77197015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb69006fc20 msgr2=0x7fb690165af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.849+0000 7fb696b17640 1 --2- 172.21.15.28:0/77197015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb69006fc20 0x7fb690165af0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb688005fd0 tx=0x7fb68802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.849+0000 7fb696b17640 1 -- 172.21.15.28:0/77197015 shutdown_connections 2024-09-10T20:39:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.849+0000 7fb696b17640 1 --2- 172.21.15.28:0/77197015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb69006fc20 0x7fb690165af0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.849+0000 7fb696b17640 1 -- 172.21.15.28:0/77197015 >> 172.21.15.28:0/77197015 conn(0x7fb69006d110 msgr2=0x7fb6900b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.849+0000 7fb696b17640 1 -- 172.21.15.28:0/77197015 shutdown_connections 2024-09-10T20:39:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.849+0000 7fb696b17640 1 -- 172.21.15.28:0/77197015 wait complete. 2024-09-10T20:39:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.850+0000 7fb696b17640 1 Processor -- start 2024-09-10T20:39:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.850+0000 7fb696b17640 1 -- start start 2024-09-10T20:39:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb696b17640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb69006fc20 0x7fb690073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb696b17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6901587a0 0x7fb6901f8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb696b17640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb69015e7d0 0x7fb69014ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb696b17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6901661b0 con 0x7fb6901587a0 2024-09-10T20:39:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb696b17640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb690166030 con 0x7fb69006fc20 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb696b17640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb690166330 con 0x7fb69015e7d0 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6901587a0 0x7fb6901f8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6901587a0 0x7fb6901f8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56642/0 (socket says 172.21.15.28:56642) 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 -- 172.21.15.28:0/1842407581 learned_addr learned my addr 172.21.15.28:0/1842407581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb6948b3640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb69006fc20 0x7fb690073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb6950b4640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb69015e7d0 0x7fb69014ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 -- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb69015e7d0 msgr2=0x7fb69014ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb69015e7d0 0x7fb69014ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 -- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb69006fc20 msgr2=0x7fb690073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb69006fc20 0x7fb690073ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 -- 172.21.15.28:0/1842407581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb690072a20 con 0x7fb6901587a0 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb6948b3640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb69006fc20 0x7fb690073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.851+0000 7fb68ffff640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6901587a0 0x7fb6901f8500 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb684001eb0 tx=0x7fb68400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.852+0000 7fb68d7fa640 1 -- 172.21.15.28:0/1842407581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684011790 con 0x7fb6901587a0 2024-09-10T20:39:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.852+0000 7fb68d7fa640 1 -- 172.21.15.28:0/1842407581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb684017070 con 0x7fb6901587a0 2024-09-10T20:39:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.852+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb690072d00 con 0x7fb6901587a0 2024-09-10T20:39:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.852+0000 7fb68d7fa640 1 -- 172.21.15.28:0/1842407581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684011ca0 con 0x7fb6901587a0 2024-09-10T20:39:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.852+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6900732c0 con 0x7fb6901587a0 2024-09-10T20:39:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.853+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb690165e50 con 0x7fb6901587a0 2024-09-10T20:39:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.853+0000 7fb68d7fa640 1 -- 172.21.15.28:0/1842407581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fb68401c460 con 0x7fb6901587a0 2024-09-10T20:39:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.854+0000 7fb68d7fa640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb67c078120 0x7fb67c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.854+0000 7fb68d7fa640 1 -- 172.21.15.28:0/1842407581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7fb684099bc0 con 0x7fb6901587a0 2024-09-10T20:39:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.854+0000 7fb6948b3640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb67c078120 0x7fb67c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.855+0000 7fb6948b3640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb67c078120 0x7fb67c07a5e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb680001740 tx=0x7fb680009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:15.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:15.858+0000 7fb68d7fa640 1 -- 172.21.15.28:0/1842407581 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb690165e50 con 0x7fb6901587a0 2024-09-10T20:39:16.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.041+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb69015b170 con 0x7fb6901587a0 2024-09-10T20:39:16.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.041+0000 7fb68d7fa640 1 -- 172.21.15.28:0/1842407581 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7fb6840626c0 con 0x7fb6901587a0 2024-09-10T20:39:16.045 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb67c078120 msgr2=0x7fb67c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb67c078120 0x7fb67c07a5e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb680001740 tx=0x7fb680009040 comp rx=0 tx=0).stop 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6901587a0 msgr2=0x7fb6901f8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6901587a0 0x7fb6901f8500 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb684001eb0 tx=0x7fb68400c890 comp rx=0 tx=0).stop 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 shutdown_connections 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb67c078120 0x7fb67c07a5e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb69015e7d0 0x7fb69014ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6901587a0 0x7fb6901f8500 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 --2- 172.21.15.28:0/1842407581 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb69006fc20 0x7fb690073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:16.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 >> 172.21.15.28:0/1842407581 conn(0x7fb69006d110 msgr2=0x7fb6901592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:16.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 shutdown_connections 2024-09-10T20:39:16.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.044+0000 7fb696b17640 1 -- 172.21.15.28:0/1842407581 wait complete. 2024-09-10T20:39:16.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-09-10T20:39:16.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.161+0000 7fb47ba16640 1 -- 172.21.15.28:0/1593071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb460003a50 msgr2=0x7fb460003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:16.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.161+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1593071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb460003a50 0x7fb460003e70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb470001690 tx=0x7fb460005830 comp rx=0 tx=0).stop 2024-09-10T20:39:16.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.161+0000 7fb47ba16640 1 -- 172.21.15.28:0/1593071527 shutdown_connections 2024-09-10T20:39:16.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.161+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1593071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb460003a50 0x7fb460003e70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:16.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.161+0000 7fb47ba16640 1 -- 172.21.15.28:0/1593071527 >> 172.21.15.28:0/1593071527 conn(0x7fb4740d1e40 msgr2=0x7fb4740d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:16.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.161+0000 7fb47ba16640 1 -- 172.21.15.28:0/1593071527 shutdown_connections 2024-09-10T20:39:16.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.161+0000 7fb47ba16640 1 -- 172.21.15.28:0/1593071527 wait complete. 2024-09-10T20:39:16.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.162+0000 7fb47ba16640 1 Processor -- start 2024-09-10T20:39:16.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb47ba16640 1 -- start start 2024-09-10T20:39:16.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb47ba16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb460003a50 0x7fb4740af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:16.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb47ba16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4741f5d00 0x7fb4740c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:16.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb47ba16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb474155df0 0x7fb474156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:16.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb47ba16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4740d0010 con 0x7fb474155df0 2024-09-10T20:39:16.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb47ba16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb4740cfe90 con 0x7fb4741f5d00 2024-09-10T20:39:16.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb47ba16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb4740ce850 con 0x7fb460003a50 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb479fb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb474155df0 0x7fb474156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb479fb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb474155df0 0x7fb474156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56656/0 (socket says 172.21.15.28:56656) 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.163+0000 7fb479fb3640 1 -- 172.21.15.28:0/1847037953 learned_addr learned my addr 172.21.15.28:0/1847037953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb478fb1640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4741f5d00 0x7fb4740c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb4797b2640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb460003a50 0x7fb4740af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb479fb3640 1 -- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb460003a50 msgr2=0x7fb4740af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb479fb3640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb460003a50 0x7fb4740af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb479fb3640 1 -- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4741f5d00 msgr2=0x7fb4740c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb479fb3640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4741f5d00 0x7fb4740c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb479fb3640 1 -- 172.21.15.28:0/1847037953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4741f62a0 con 0x7fb474155df0 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb478fb1640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4741f5d00 0x7fb4740c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb479fb3640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb474155df0 0x7fb474156250 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb4740d1e10 tx=0x7fb46c00eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:16.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb469ffb640 1 -- 172.21.15.28:0/1847037953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00c920 con 0x7fb474155df0 2024-09-10T20:39:16.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb469ffb640 1 -- 172.21.15.28:0/1847037953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb46c017070 con 0x7fb474155df0 2024-09-10T20:39:16.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb469ffb640 1 -- 172.21.15.28:0/1847037953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00cdf0 con 0x7fb474155df0 2024-09-10T20:39:16.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4741f6580 con 0x7fb474155df0 2024-09-10T20:39:16.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.164+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4741f6b40 con 0x7fb474155df0 2024-09-10T20:39:16.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.165+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4740cedb0 con 0x7fb474155df0 2024-09-10T20:39:16.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.166+0000 7fb469ffb640 1 -- 172.21.15.28:0/1847037953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fb46c01c460 con 0x7fb474155df0 2024-09-10T20:39:16.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.166+0000 7fb469ffb640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb458078120 0x7fb45807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:16.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.166+0000 7fb469ffb640 1 -- 172.21.15.28:0/1847037953 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7fb46c09a2a0 con 0x7fb474155df0 2024-09-10T20:39:16.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.167+0000 7fb4797b2640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb458078120 0x7fb45807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:16.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.167+0000 7fb4797b2640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb458078120 0x7fb45807a5e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb470032000 tx=0x7fb470001190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:16.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.171+0000 7fb469ffb640 1 -- 172.21.15.28:0/1847037953 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb46c062c70 con 0x7fb474155df0 2024-09-10T20:39:16.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:16.428+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7fb4740b6e30 con 0x7fb474155df0 2024-09-10T20:39:17.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.412+0000 7fb469ffb640 1 -- 172.21.15.28:0/1847037953 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v82) ==== 143+0+0 (secure 0 0 0) 0x7fb46c06b390 con 0x7fb474155df0 2024-09-10T20:39:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb458078120 msgr2=0x7fb45807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb458078120 0x7fb45807a5e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb470032000 tx=0x7fb470001190 comp rx=0 tx=0).stop 2024-09-10T20:39:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb474155df0 msgr2=0x7fb474156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb474155df0 0x7fb474156250 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb4740d1e10 tx=0x7fb46c00eae0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 shutdown_connections 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb458078120 0x7fb45807a5e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb474155df0 0x7fb474156250 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4741f5d00 0x7fb4740c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 --2- 172.21.15.28:0/1847037953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb460003a50 0x7fb4740af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 >> 172.21.15.28:0/1847037953 conn(0x7fb4740d1e40 msgr2=0x7fb4740717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 shutdown_connections 2024-09-10T20:39:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.415+0000 7fb47ba16640 1 -- 172.21.15.28:0/1847037953 wait complete. 2024-09-10T20:39:17.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:39:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.540+0000 7f7be9008640 1 -- 172.21.15.28:0/1876970544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 msgr2=0x7f7bd0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.540+0000 7f7be9008640 1 --2- 172.21.15.28:0/1876970544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 0x7f7bd0004800 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009b80 tx=0x7f7bd0005830 comp rx=0 tx=0).stop 2024-09-10T20:39:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.540+0000 7f7be359e640 1 -- 172.21.15.28:0/1876970544 reap_dead start 2024-09-10T20:39:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.540+0000 7f7be9008640 1 -- 172.21.15.28:0/1876970544 shutdown_connections 2024-09-10T20:39:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.541+0000 7f7be9008640 1 -- 172.21.15.28:0/1876970544 >> 172.21.15.28:0/1876970544 conn(0x7f7be40d0420 msgr2=0x7f7be40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.541+0000 7f7be9008640 1 -- 172.21.15.28:0/1876970544 shutdown_connections 2024-09-10T20:39:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.541+0000 7f7be9008640 1 -- 172.21.15.28:0/1876970544 wait complete. 2024-09-10T20:39:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.542+0000 7f7be9008640 1 Processor -- start 2024-09-10T20:39:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.542+0000 7f7be9008640 1 -- start start 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be9008640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 0x7f7be41f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be9008640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7be41f28b0 0x7f7be41f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be9008640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7be40b4440 0x7f7be40b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be9008640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7be40cabf0 con 0x7f7bd0003fd0 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be9008640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7be40caa70 con 0x7f7be41f28b0 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be9008640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7be40cad70 con 0x7f7be40b4440 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 0x7f7be41f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 0x7f7be41f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56672/0 (socket says 172.21.15.28:56672) 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 -- 172.21.15.28:0/796468331 learned_addr learned my addr 172.21.15.28:0/796468331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be359e640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7be40b4440 0x7f7be40b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be259c640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7be41f28b0 0x7f7be41f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 -- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7be40b4440 msgr2=0x7f7be40b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7be40b4440 0x7f7be40b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 -- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7be41f28b0 msgr2=0x7f7be41f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7be41f28b0 0x7f7be41f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be2d9d640 1 -- 172.21.15.28:0/796468331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be41f5b80 con 0x7f7bd0003fd0 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.543+0000 7f7be259c640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7be41f28b0 0x7f7be41f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.544+0000 7f7be2d9d640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 0x7f7be41f3990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400e9f0 tx=0x7f7bd400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.544+0000 7f7bdb7fe640 1 -- 172.21.15.28:0/796468331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4015070 con 0x7f7bd0003fd0 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.544+0000 7f7bdb7fe640 1 -- 172.21.15.28:0/796468331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7bd400ce00 con 0x7f7bd0003fd0 2024-09-10T20:39:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.544+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be40b4d40 con 0x7f7bd0003fd0 2024-09-10T20:39:17.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.544+0000 7f7bdb7fe640 1 -- 172.21.15.28:0/796468331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4011680 con 0x7f7bd0003fd0 2024-09-10T20:39:17.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.544+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be40b5300 con 0x7f7bd0003fd0 2024-09-10T20:39:17.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.545+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be406ef30 con 0x7f7bd0003fd0 2024-09-10T20:39:17.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.546+0000 7f7bdb7fe640 1 -- 172.21.15.28:0/796468331 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f7bd4011820 con 0x7f7bd0003fd0 2024-09-10T20:39:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.546+0000 7f7bdb7fe640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7bc4078150 0x7f7bc407a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.546+0000 7f7bdb7fe640 1 -- 172.21.15.28:0/796468331 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7f7bd409a9c0 con 0x7f7bd0003fd0 2024-09-10T20:39:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.547+0000 7f7be259c640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7bc4078150 0x7f7bc407a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.547+0000 7f7be259c640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7bc4078150 0x7f7bc407a610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc035000 tx=0x7f7bcc02d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.551+0000 7f7bdb7fe640 1 -- 172.21.15.28:0/796468331 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7bd409ce20 con 0x7f7bd0003fd0 2024-09-10T20:39:17.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.735+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f7be40ca750 con 0x7f7bd0003fd0 2024-09-10T20:39:17.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.736+0000 7f7bdb7fe640 1 -- 172.21.15.28:0/796468331 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v29) ==== 59+0+1998 (secure 0 0 0) 0x7f7bd4063440 con 0x7f7bd0003fd0 2024-09-10T20:39:17.741 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:39:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:39:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:39:17.744 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:39:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7bc4078150 msgr2=0x7f7bc407a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7bc4078150 0x7f7bc407a610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc035000 tx=0x7f7bcc02d040 comp rx=0 tx=0).stop 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 msgr2=0x7f7be41f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 0x7f7be41f3990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400e9f0 tx=0x7f7bd400eeb0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 shutdown_connections 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7bc4078150 0x7f7bc407a610 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7be40b4440 0x7f7be40b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7be41f28b0 0x7f7be41f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 --2- 172.21.15.28:0/796468331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd0003fd0 0x7f7be41f3990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 >> 172.21.15.28:0/796468331 conn(0x7f7be40d0420 msgr2=0x7f7be415d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 shutdown_connections 2024-09-10T20:39:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.738+0000 7f7be9008640 1 -- 172.21.15.28:0/796468331 wait complete. 2024-09-10T20:39:17.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:39:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 -- 172.21.15.28:0/3835037628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415adf0 msgr2=0x7f03040cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 --2- 172.21.15.28:0/3835037628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415adf0 0x7f03040cb230 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009830 tx=0x7f02f402a380 comp rx=0 tx=0).stop 2024-09-10T20:39:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 -- 172.21.15.28:0/3835037628 shutdown_connections 2024-09-10T20:39:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 --2- 172.21.15.28:0/3835037628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f030415e7a0 0x7f03040d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 --2- 172.21.15.28:0/3835037628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415adf0 0x7f03040cb230 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 -- 172.21.15.28:0/3835037628 >> 172.21.15.28:0/3835037628 conn(0x7f030406cf00 msgr2=0x7f030406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 -- 172.21.15.28:0/3835037628 shutdown_connections 2024-09-10T20:39:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.857+0000 7f030acbd640 1 -- 172.21.15.28:0/3835037628 wait complete. 2024-09-10T20:39:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.858+0000 7f030acbd640 1 Processor -- start 2024-09-10T20:39:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.859+0000 7f030acbd640 1 -- start start 2024-09-10T20:39:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.859+0000 7f030acbd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f030415aa10 0x7f03040b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.859+0000 7f030acbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030415adf0 0x7f03040cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.859+0000 7f030acbd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415e7a0 0x7f0304150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.859+0000 7f030acbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0304070250 con 0x7f030415adf0 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.859+0000 7f030acbd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f03040700d0 con 0x7f030415e7a0 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.859+0000 7f030acbd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f03040703d0 con 0x7f030415aa10 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030415adf0 0x7f03040cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030415adf0 0x7f03040cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56688/0 (socket says 172.21.15.28:56688) 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 -- 172.21.15.28:0/3170343097 learned_addr learned my addr 172.21.15.28:0/3170343097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f0308a59640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f030415aa10 0x7f03040b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f030925a640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415e7a0 0x7f0304150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 -- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f030415aa10 msgr2=0x7f03040b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f030415aa10 0x7f03040b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 -- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415e7a0 msgr2=0x7f0304150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415e7a0 0x7f0304150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 -- 172.21.15.28:0/3170343097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03041f52e0 con 0x7f030415adf0 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f030925a640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415e7a0 0x7f0304150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:17.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02fbfff640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030415adf0 0x7f03040cd130 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009950 tx=0x7f02f4003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:17.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02f97fa640 1 -- 172.21.15.28:0/3170343097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f403f030 con 0x7f030415adf0 2024-09-10T20:39:17.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02f97fa640 1 -- 172.21.15.28:0/3170343097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02f4004510 con 0x7f030415adf0 2024-09-10T20:39:17.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f030415cc00 con 0x7f030415adf0 2024-09-10T20:39:17.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f02f97fa640 1 -- 172.21.15.28:0/3170343097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4037040 con 0x7f030415adf0 2024-09-10T20:39:17.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.860+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03040b3930 con 0x7f030415adf0 2024-09-10T20:39:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.861+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03040d0860 con 0x7f030415adf0 2024-09-10T20:39:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.862+0000 7f02f97fa640 1 -- 172.21.15.28:0/3170343097 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f02f403a030 con 0x7f030415adf0 2024-09-10T20:39:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.862+0000 7f02f97fa640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f02f0078120 0x7f02f007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.863+0000 7f02f97fa640 1 -- 172.21.15.28:0/3170343097 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7f02f40b9af0 con 0x7f030415adf0 2024-09-10T20:39:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.863+0000 7f0308a59640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f02f0078120 0x7f02f007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.863+0000 7f0308a59640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f02f0078120 0x7f02f007a5e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f03000078a0 tx=0x7f0300009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:17.867+0000 7f02f97fa640 1 -- 172.21.15.28:0/3170343097 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f02f4049070 con 0x7f030415adf0 2024-09-10T20:39:18.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.054+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f03040cc450 con 0x7f030415adf0 2024-09-10T20:39:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.057+0000 7f02f97fa640 1 -- 172.21.15.28:0/3170343097 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7f02f4082550 con 0x7f030415adf0 2024-09-10T20:39:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f02f0078120 msgr2=0x7f02f007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f02f0078120 0x7f02f007a5e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f03000078a0 tx=0x7f0300009040 comp rx=0 tx=0).stop 2024-09-10T20:39:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030415adf0 msgr2=0x7f03040cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030415adf0 0x7f03040cd130 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009950 tx=0x7f02f4003b60 comp rx=0 tx=0).stop 2024-09-10T20:39:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 shutdown_connections 2024-09-10T20:39:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f02f0078120 0x7f02f007a5e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f030415e7a0 0x7f0304150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f030415adf0 0x7f03040cd130 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 --2- 172.21.15.28:0/3170343097 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f030415aa10 0x7f03040b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 >> 172.21.15.28:0/3170343097 conn(0x7f030406cf00 msgr2=0x7f03040cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 shutdown_connections 2024-09-10T20:39:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.060+0000 7f030acbd640 1 -- 172.21.15.28:0/3170343097 wait complete. 2024-09-10T20:39:18.076 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:39:18.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:39:18.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:39:18.164 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:39:18.164 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:18.164 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:18.180 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:39:18.180 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:18.180 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:18.198 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:39:18.198 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:18.198 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:18.236 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:39:18.236 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:18.236 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:18.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:39:18.257 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:39:18.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 -- 172.21.15.28:0/2545909440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 msgr2=0x7f572c0cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 --2- 172.21.15.28:0/2545909440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c0cfbc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5720005fd0 tx=0x7f572002a530 comp rx=0 tx=0).stop 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 -- 172.21.15.28:0/2545909440 shutdown_connections 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 --2- 172.21.15.28:0/2545909440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c0cfbc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 --2- 172.21.15.28:0/2545909440 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f572c15de70 0x7f572c0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 -- 172.21.15.28:0/2545909440 >> 172.21.15.28:0/2545909440 conn(0x7f572c0d3fb0 msgr2=0x7f572c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 -- 172.21.15.28:0/2545909440 shutdown_connections 2024-09-10T20:39:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.378+0000 7f573299b640 1 -- 172.21.15.28:0/2545909440 wait complete. 2024-09-10T20:39:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.379+0000 7f573299b640 1 Processor -- start 2024-09-10T20:39:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.379+0000 7f573299b640 1 -- start start 2024-09-10T20:39:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f573299b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f573299b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f572c15de70 0x7f572c1543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f573299b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f572c161b50 0x7f572c1f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f573299b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f572c162350 con 0x7f572c0cf780 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f573299b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1621d0 con 0x7f572c15de70 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f573299b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f572c1624d0 con 0x7f572c161b50 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56708/0 (socket says 172.21.15.28:56708) 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 -- 172.21.15.28:0/813720361 learned_addr learned my addr 172.21.15.28:0/813720361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572b7fe640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f572c15de70 0x7f572c1543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f5730f38640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f572c161b50 0x7f572c1f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 -- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f572c161b50 msgr2=0x7f572c1f88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f572c161b50 0x7f572c1f88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 -- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f572c15de70 msgr2=0x7f572c1543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f572c15de70 0x7f572c1543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 -- 172.21.15.28:0/813720361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c1f8df0 con 0x7f572c0cf780 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572b7fe640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f572c15de70 0x7f572c1543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f5730f38640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f572c161b50 0x7f572c1f88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f572bfff640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c156b50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f571c00c890 tx=0x7f571c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f5728ff9640 1 -- 172.21.15.28:0/813720361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c004220 con 0x7f572c0cf780 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f5728ff9640 1 -- 172.21.15.28:0/813720361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f571c0079b0 con 0x7f572c0cf780 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.380+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c157210 con 0x7f572c0cf780 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.381+0000 7f5728ff9640 1 -- 172.21.15.28:0/813720361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c007de0 con 0x7f572c0cf780 2024-09-10T20:39:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.381+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c1f80b0 con 0x7f572c0cf780 2024-09-10T20:39:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.381+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572c157730 con 0x7f572c0cf780 2024-09-10T20:39:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.382+0000 7f5728ff9640 1 -- 172.21.15.28:0/813720361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f571c005020 con 0x7f572c0cf780 2024-09-10T20:39:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.383+0000 7f5728ff9640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5718078120 0x7f571807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.383+0000 7f5728ff9640 1 -- 172.21.15.28:0/813720361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7f571c09a3e0 con 0x7f572c0cf780 2024-09-10T20:39:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.383+0000 7f572b7fe640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5718078120 0x7f571807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.383+0000 7f572b7fe640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5718078120 0x7f571807a5e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5720033000 tx=0x7f5720004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.387+0000 7f5728ff9640 1 -- 172.21.15.28:0/813720361 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f572c1f8df0 con 0x7f572c0cf780 2024-09-10T20:39:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.610+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f572c0cfbc0 con 0x7f572c0cf780 2024-09-10T20:39:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.610+0000 7f5728ff9640 1 -- 172.21.15.28:0/813720361 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+3911 (secure 0 0 0) 0x7f571c014030 con 0x7f572c0cf780 2024-09-10T20:39:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:18.616 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"btime":"2024-09-10T20:39:12:395161+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1687267914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1687267914},{"type":"v1","addr":"172.21.15.159:6836","nonce":1687267914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4804,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/649109556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":649109556},{"type":"v1","addr":"172.21.15.159:6837","nonce":649109556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4851,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2646744984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2646744984},{"type":"v1","addr":"172.21.15.28:6834","nonce":2646744984}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5187,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/375497085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":375497085},{"type":"v1","addr":"172.21.15.28:6835","nonce":375497085}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-09-10T20:39:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 19 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5718078120 msgr2=0x7f571807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5718078120 0x7f571807a5e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5720033000 tx=0x7f5720004730 comp rx=0 tx=0).stop 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 msgr2=0x7f572c156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c156b50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f571c00c890 tx=0x7f571c00cd50 comp rx=0 tx=0).stop 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 shutdown_connections 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5718078120 0x7f571807a5e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f572c161b50 0x7f572c1f88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f572c15de70 0x7f572c1543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 --2- 172.21.15.28:0/813720361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f572c0cf780 0x7f572c156b50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 >> 172.21.15.28:0/813720361 conn(0x7f572c0d3fb0 msgr2=0x7f572c06edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:18.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 shutdown_connections 2024-09-10T20:39:18.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.613+0000 7f573299b640 1 -- 172.21.15.28:0/813720361 wait complete. 2024-09-10T20:39:18.629 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:39:18.629 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:39:18.629 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:39:18.629 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:39:18.629 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:39:18.630 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:39:18.630 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:39:18.630 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:39:18.630 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:39:18.630 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:39:18.630 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:39:18.630 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:39:18.631 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:39:18.631 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:39:18.631 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:39:18.631 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:39:18.639 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:39:18.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:39:18.640 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:39:18.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:39:18.643 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:39:18.643 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:39:18.643 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:39:18.644 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:39:18.682 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:39:18.683 DEBUG:teuthology.parallel:result is None 2024-09-10T20:39:18.683 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:39:18.683 DEBUG:teuthology.parallel:result is None 2024-09-10T20:39:18.685 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:39:18.686 DEBUG:teuthology.parallel:result is None 2024-09-10T20:39:18.686 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:39:18.686 DEBUG:teuthology.parallel:result is None 2024-09-10T20:39:18.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:39:18.760 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:39:18.761 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:39:18.767 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:39:18.767 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 -- 172.21.15.28:0/524431779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec15f660 msgr2=0x7feeec0ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 --2- 172.21.15.28:0/524431779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec15f660 0x7feeec0ce710 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7feee8005fd0 tx=0x7feee802a580 comp rx=0 tx=0).stop 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 -- 172.21.15.28:0/524431779 shutdown_connections 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 --2- 172.21.15.28:0/524431779 >> v2:172.21.15.159:3300/0 conn(0x7feeec0d29f0 0x7feeec0cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 --2- 172.21.15.28:0/524431779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec15f660 0x7feeec0ce710 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 -- 172.21.15.28:0/524431779 >> 172.21.15.28:0/524431779 conn(0x7feeec0d1ed0 msgr2=0x7feeec0d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 -- 172.21.15.28:0/524431779 shutdown_connections 2024-09-10T20:39:18.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.793+0000 7feef3a39640 1 -- 172.21.15.28:0/524431779 wait complete. 2024-09-10T20:39:18.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.794+0000 7feef3a39640 1 Processor -- start 2024-09-10T20:39:18.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.794+0000 7feef3a39640 1 -- start start 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef3a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec0b55c0 0x7feeec0b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef3a39640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feeec0d29f0 0x7feeec1f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef3a39640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feeec15f660 0x7feeec14c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef3a39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feeec0cdc70 con 0x7feeec0b55c0 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef3a39640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7feeec0cdaf0 con 0x7feeec0d29f0 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef3a39640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7feeec0cddf0 con 0x7feeec15f660 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec0b55c0 0x7feeec0b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec0b55c0 0x7feeec0b1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56748/0 (socket says 172.21.15.28:56748) 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 -- 172.21.15.28:0/110320980 learned_addr learned my addr 172.21.15.28:0/110320980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef0fd4640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feeec0d29f0 0x7feeec1f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef1fd6640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feeec15f660 0x7feeec14c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 -- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feeec15f660 msgr2=0x7feeec14c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feeec15f660 0x7feeec14c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 -- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feeec0d29f0 msgr2=0x7feeec1f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feeec0d29f0 0x7feeec1f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 -- 172.21.15.28:0/110320980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeec14d010 con 0x7feeec0b55c0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef1fd6640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feeec15f660 0x7feeec14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.795+0000 7feef17d5640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec0b55c0 0x7feeec0b1a30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7feee8034040 tx=0x7feee80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.796+0000 7feee1ffb640 1 -- 172.21.15.28:0/110320980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee803e030 con 0x7feeec0b55c0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.796+0000 7feee1ffb640 1 -- 172.21.15.28:0/110320980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feee8003d90 con 0x7feeec0b55c0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.796+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeec14ca50 con 0x7feeec0b55c0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.796+0000 7feee1ffb640 1 -- 172.21.15.28:0/110320980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8035040 con 0x7feeec0b55c0 2024-09-10T20:39:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.796+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeec1f61b0 con 0x7feeec0b55c0 2024-09-10T20:39:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.797+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeec0b5a10 con 0x7feeec0b55c0 2024-09-10T20:39:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.798+0000 7feee1ffb640 1 -- 172.21.15.28:0/110320980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7feee80049e0 con 0x7feeec0b55c0 2024-09-10T20:39:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.798+0000 7feee1ffb640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7feed0078120 0x7feed007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.798+0000 7feee1ffb640 1 -- 172.21.15.28:0/110320980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4943+0+0 (secure 0 0 0) 0x7feee80baaa0 con 0x7feeec0b55c0 2024-09-10T20:39:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.798+0000 7feef0fd4640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7feed0078120 0x7feed007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.799+0000 7feef0fd4640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7feed0078120 0x7feed007a5e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feeec14fb70 tx=0x7feedc00b390 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.802+0000 7feee1ffb640 1 -- 172.21.15.28:0/110320980 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feee8082ec0 con 0x7feeec0b55c0 2024-09-10T20:39:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:18.998+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7feeec15ebe0 con 0x7feeec0b55c0 2024-09-10T20:39:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.429+0000 7feee1ffb640 1 -- 172.21.15.28:0/110320980 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v37) ==== 97+0+0 (secure 0 0 0) 0x7feee80878f0 con 0x7feeec0b55c0 2024-09-10T20:39:19.434 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:39:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7feed0078120 msgr2=0x7feed007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7feed0078120 0x7feed007a5e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feeec14fb70 tx=0x7feedc00b390 comp rx=0 tx=0).stop 2024-09-10T20:39:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec0b55c0 msgr2=0x7feeec0b1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec0b55c0 0x7feeec0b1a30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7feee8034040 tx=0x7feee80038a0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 shutdown_connections 2024-09-10T20:39:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7feed0078120 0x7feed007a5e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feeec15f660 0x7feeec14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feeec0d29f0 0x7feeec1f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 --2- 172.21.15.28:0/110320980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeec0b55c0 0x7feeec0b1a30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 >> 172.21.15.28:0/110320980 conn(0x7feeec0d1ed0 msgr2=0x7feeec0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 shutdown_connections 2024-09-10T20:39:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.432+0000 7feef3a39640 1 -- 172.21.15.28:0/110320980 wait complete. 2024-09-10T20:39:19.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 -- 172.21.15.28:0/3234404282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b815de70 msgr2=0x7f34b80d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 --2- 172.21.15.28:0/3234404282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b815de70 0x7f34b80d2b90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f34b80d01d0 tx=0x7f34b402a530 comp rx=0 tx=0).stop 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 -- 172.21.15.28:0/3234404282 shutdown_connections 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 --2- 172.21.15.28:0/3234404282 >> v2:172.21.15.159:3300/0 conn(0x7f34b8162260 0x7f34b80cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 --2- 172.21.15.28:0/3234404282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b815de70 0x7f34b80d2b90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 -- 172.21.15.28:0/3234404282 >> 172.21.15.28:0/3234404282 conn(0x7f34b80d3fb0 msgr2=0x7f34b80d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 -- 172.21.15.28:0/3234404282 shutdown_connections 2024-09-10T20:39:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.551+0000 7f34bf26a640 1 -- 172.21.15.28:0/3234404282 wait complete. 2024-09-10T20:39:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 Processor -- start 2024-09-10T20:39:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 -- start start 2024-09-10T20:39:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b80cf780 0x7f34b81f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34b815de70 0x7f34b80a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34b8162260 0x7f34b80a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34b80b6f80 con 0x7f34b80cf780 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f34b80b6e00 con 0x7f34b815de70 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.553+0000 7f34bf26a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f34b80b7100 con 0x7f34b8162260 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b80cf780 0x7f34b81f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b80cf780 0x7f34b81f40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56780/0 (socket says 172.21.15.28:56780) 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 -- 172.21.15.28:0/591686027 learned_addr learned my addr 172.21.15.28:0/591686027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 -- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34b8162260 msgr2=0x7f34b80a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34b8162260 0x7f34b80a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 -- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34b815de70 msgr2=0x7f34b80a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34b815de70 0x7f34b80a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 -- 172.21.15.28:0/591686027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b8161270 con 0x7f34b80cf780 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bd006640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b80cf780 0x7f34b81f40c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009920 tx=0x7f34b4032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34adffb640 1 -- 172.21.15.28:0/591686027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4040070 con 0x7f34b80cf780 2024-09-10T20:39:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34adffb640 1 -- 172.21.15.28:0/591686027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34b4005310 con 0x7f34b80cf780 2024-09-10T20:39:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34adffb640 1 -- 172.21.15.28:0/591686027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4003800 con 0x7f34b80cf780 2024-09-10T20:39:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b809ecd0 con 0x7f34b80cf780 2024-09-10T20:39:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.554+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b809f290 con 0x7f34b80cf780 2024-09-10T20:39:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.555+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b80d2b90 con 0x7f34b80cf780 2024-09-10T20:39:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.556+0000 7f34adffb640 1 -- 172.21.15.28:0/591686027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f34b40039a0 con 0x7f34b80cf780 2024-09-10T20:39:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.556+0000 7f34adffb640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f349c078120 0x7f349c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.557+0000 7f34adffb640 1 -- 172.21.15.28:0/591686027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4857+0+0 (secure 0 0 0) 0x7f34b40bb390 con 0x7f34b80cf780 2024-09-10T20:39:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.557+0000 7f34bc805640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f349c078120 0x7f349c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.557+0000 7f34bc805640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f349c078120 0x7f349c07a5e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f34b80b4520 tx=0x7f34a8007a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:19.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.561+0000 7f34adffb640 1 -- 172.21.15.28:0/591686027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f34b4083e40 con 0x7f34b80cf780 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.814+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f34b809f460 con 0x7f34b80cf780 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.814+0000 7f34adffb640 1 -- 172.21.15.28:0/591686027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) ==== 80+0+5304 (secure 0 0 0) 0x7f34b4083740 con 0x7f34b80cf780 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:39:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:39:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:39:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:39:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:39:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:39:19.832 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:39:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f349c078120 msgr2=0x7f349c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f349c078120 0x7f349c07a5e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f34b80b4520 tx=0x7f34a8007a30 comp rx=0 tx=0).stop 2024-09-10T20:39:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b80cf780 msgr2=0x7f34b81f40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b80cf780 0x7f34b81f40c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009920 tx=0x7f34b4032d50 comp rx=0 tx=0).stop 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 shutdown_connections 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f349c078120 0x7f349c07a5e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34b8162260 0x7f34b80a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34b815de70 0x7f34b80a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 --2- 172.21.15.28:0/591686027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b80cf780 0x7f34b81f40c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 >> 172.21.15.28:0/591686027 conn(0x7f34b80d3fb0 msgr2=0x7f34b80b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 shutdown_connections 2024-09-10T20:39:19.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.816+0000 7f34bf26a640 1 -- 172.21.15.28:0/591686027 wait complete. 2024-09-10T20:39:19.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:39:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/229285189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 msgr2=0x7fc8e815e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/229285189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e815e7e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8068cb0 tx=0x7fc8dc02a560 comp rx=0 tx=0).stop 2024-09-10T20:39:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/229285189 shutdown_connections 2024-09-10T20:39:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/229285189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e815e7e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/229285189 >> v2:172.21.15.159:3300/0 conn(0x7fc8e815ed20 0x7fc8e80cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/229285189 >> 172.21.15.28:0/229285189 conn(0x7fc8e80d0420 msgr2=0x7fc8e80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/229285189 shutdown_connections 2024-09-10T20:39:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.936+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/229285189 wait complete. 2024-09-10T20:39:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.937+0000 7fc8ef8dd640 1 Processor -- start 2024-09-10T20:39:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.937+0000 7fc8ef8dd640 1 -- start start 2024-09-10T20:39:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ef8dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e81f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ef8dd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8e8151cc0 0x7fc8e80b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ef8dd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8e815ed20 0x7fc8e8153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ef8dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e815f5b0 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ef8dd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e815f430 con 0x7fc8e8151cc0 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ef8dd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e80cb750 con 0x7fc8e815ed20 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ed679640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e81f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ed679640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e81f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56800/0 (socket says 172.21.15.28:56800) 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ed679640 1 -- 172.21.15.28:0/2761717453 learned_addr learned my addr 172.21.15.28:0/2761717453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.938+0000 7fc8ece78640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8e8151cc0 0x7fc8e80b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ede7a640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8e815ed20 0x7fc8e8153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ed679640 1 -- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8e815ed20 msgr2=0x7fc8e8153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ed679640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8e815ed20 0x7fc8e8153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ed679640 1 -- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8e8151cc0 msgr2=0x7fc8e80b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ed679640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8e8151cc0 0x7fc8e80b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ed679640 1 -- 172.21.15.28:0/2761717453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e80b1e70 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ece78640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8e8151cc0 0x7fc8e80b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ed679640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e81f33a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000c460 tx=0x7fc8e000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8e5ffb640 1 -- 172.21.15.28:0/2761717453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0016020 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8e5ffb640 1 -- 172.21.15.28:0/2761717453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8e0012960 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e80b0dd0 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8e5ffb640 1 -- 172.21.15.28:0/2761717453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0012db0 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.939+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e80b1390 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.940+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e80cbb70 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.941+0000 7fc8e5ffb640 1 -- 172.21.15.28:0/2761717453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fc8e001b400 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.941+0000 7fc8e5ffb640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc8d4078120 0x7fc8d407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.941+0000 7fc8e5ffb640 1 -- 172.21.15.28:0/2761717453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4857+0+0 (secure 0 0 0) 0x7fc8e009a0c0 con 0x7fc8e80cf0f0 2024-09-10T20:39:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.942+0000 7fc8ece78640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc8d4078120 0x7fc8d407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.942+0000 7fc8ece78640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc8d4078120 0x7fc8d407a5e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc033000 tx=0x7fc8dc004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:19.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:19.946+0000 7fc8e5ffb640 1 -- 172.21.15.28:0/2761717453 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc8e0020020 con 0x7fc8e80cf0f0 2024-09-10T20:39:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.135+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc8e8068d10 con 0x7fc8e80cf0f0 2024-09-10T20:39:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.135+0000 7fc8e5ffb640 1 -- 172.21.15.28:0/2761717453 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v39) ==== 84+0+65 (secure 0 0 0) 0x7fc8e0062c10 con 0x7fc8e80cf0f0 2024-09-10T20:39:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:20.140 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:39:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:39:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:39:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:20.141 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:39:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.137+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc8d4078120 msgr2=0x7fc8d407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.137+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc8d4078120 0x7fc8d407a5e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc033000 tx=0x7fc8dc004730 comp rx=0 tx=0).stop 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.137+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 msgr2=0x7fc8e81f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.137+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e81f33a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000c460 tx=0x7fc8e000c920 comp rx=0 tx=0).stop 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 shutdown_connections 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc8d4078120 0x7fc8d407a5e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc8e815ed20 0x7fc8e8153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc8e8151cc0 0x7fc8e80b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 --2- 172.21.15.28:0/2761717453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e80cf0f0 0x7fc8e81f33a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 >> 172.21.15.28:0/2761717453 conn(0x7fc8e80d0420 msgr2=0x7fc8e814fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 shutdown_connections 2024-09-10T20:39:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.138+0000 7fc8ef8dd640 1 -- 172.21.15.28:0/2761717453 wait complete. 2024-09-10T20:39:20.153 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:39:20.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:39:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.259+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3673980683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe44004800 msgr2=0x7efe44004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.259+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3673980683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe44004800 0x7efe44004c80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efe40006f30 tx=0x7efe44005830 comp rx=0 tx=0).stop 2024-09-10T20:39:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.259+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3673980683 shutdown_connections 2024-09-10T20:39:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.259+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3673980683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe44004800 0x7efe44004c80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.259+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3673980683 >> 172.21.15.28:0/3673980683 conn(0x7efe580d1da0 msgr2=0x7efe580d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:20.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.259+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3673980683 shutdown_connections 2024-09-10T20:39:20.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.259+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3673980683 wait complete. 2024-09-10T20:39:20.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.260+0000 7efe5c9f8640 1 Processor -- start 2024-09-10T20:39:20.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.260+0000 7efe5c9f8640 1 -- start start 2024-09-10T20:39:20.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe5c9f8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe44004800 0x7efe5806dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:20.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe5c9f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe5806e130 0x7efe580caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:20.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe5c9f8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe5809f600 0x7efe580caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:20.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe5c9f8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe580cfd00 con 0x7efe5806e130 2024-09-10T20:39:20.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe5c9f8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe580cfb80 con 0x7efe44004800 2024-09-10T20:39:20.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe5c9f8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe580ceb90 con 0x7efe5809f600 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe5806e130 0x7efe580caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe5806e130 0x7efe580caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56806/0 (socket says 172.21.15.28:56806) 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 -- 172.21.15.28:0/3589949832 learned_addr learned my addr 172.21.15.28:0/3589949832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe5659c640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe44004800 0x7efe5806dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe56d9d640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe5809f600 0x7efe580caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 -- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe5809f600 msgr2=0x7efe580caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe5809f600 0x7efe580caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 -- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe44004800 msgr2=0x7efe5806dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe44004800 0x7efe5806dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.261+0000 7efe55d9b640 1 -- 172.21.15.28:0/3589949832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5806e670 con 0x7efe5806e130 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.262+0000 7efe55d9b640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe5806e130 0x7efe580caa10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efe40006f30 tx=0x7efe40001d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.262+0000 7efe4effd640 1 -- 172.21.15.28:0/3589949832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe40032030 con 0x7efe5806e130 2024-09-10T20:39:20.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.262+0000 7efe4effd640 1 -- 172.21.15.28:0/3589949832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe40003af0 con 0x7efe5806e130 2024-09-10T20:39:20.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.262+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe5806e950 con 0x7efe5806e130 2024-09-10T20:39:20.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.262+0000 7efe4effd640 1 -- 172.21.15.28:0/3589949832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4002e440 con 0x7efe5806e130 2024-09-10T20:39:20.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.262+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe5806ef10 con 0x7efe5806e130 2024-09-10T20:39:20.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.263+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe580b6e70 con 0x7efe5806e130 2024-09-10T20:39:20.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.264+0000 7efe4effd640 1 -- 172.21.15.28:0/3589949832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7efe4002e5e0 con 0x7efe5806e130 2024-09-10T20:39:20.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.264+0000 7efe4effd640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe38078150 0x7efe3807a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:20.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.264+0000 7efe4effd640 1 -- 172.21.15.28:0/3589949832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4857+0+0 (secure 0 0 0) 0x7efe400ba2f0 con 0x7efe5806e130 2024-09-10T20:39:20.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.264+0000 7efe5659c640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe38078150 0x7efe3807a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:20.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.265+0000 7efe5659c640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe38078150 0x7efe3807a610 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe580668e0 tx=0x7efe50005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:20.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.269+0000 7efe4effd640 1 -- 172.21.15.28:0/3589949832 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efe40047030 con 0x7efe5806e130 2024-09-10T20:39:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:20.453+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7efe580ce590 con 0x7efe5806e130 2024-09-10T20:39:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.439+0000 7efe4effd640 1 -- 172.21.15.28:0/3589949832 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v40) ==== 138+0+0 (secure 0 0 0) 0x7efe400827f0 con 0x7efe5806e130 2024-09-10T20:39:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:39:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe38078150 msgr2=0x7efe3807a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe38078150 0x7efe3807a610 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe580668e0 tx=0x7efe50005f90 comp rx=0 tx=0).stop 2024-09-10T20:39:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe5806e130 msgr2=0x7efe580caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe5806e130 0x7efe580caa10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efe40006f30 tx=0x7efe40001d10 comp rx=0 tx=0).stop 2024-09-10T20:39:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 shutdown_connections 2024-09-10T20:39:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe38078150 0x7efe3807a610 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe5809f600 0x7efe580caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe5806e130 0x7efe580caa10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 --2- 172.21.15.28:0/3589949832 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe44004800 0x7efe5806dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 >> 172.21.15.28:0/3589949832 conn(0x7efe580d1da0 msgr2=0x7efe58071890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 shutdown_connections 2024-09-10T20:39:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.442+0000 7efe5c9f8640 1 -- 172.21.15.28:0/3589949832 wait complete. 2024-09-10T20:39:21.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:39:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.578+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2537735556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 msgr2=0x7fbff80cf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.578+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2537735556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff80cf540 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbff808e230 tx=0x7fbfec02a460 comp rx=0 tx=0).stop 2024-09-10T20:39:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.578+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2537735556 shutdown_connections 2024-09-10T20:39:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.578+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2537735556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff80cf540 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.578+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2537735556 >> 172.21.15.28:0/2537735556 conn(0x7fbff80d4040 msgr2=0x7fbff80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.578+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2537735556 shutdown_connections 2024-09-10T20:39:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.578+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2537735556 wait complete. 2024-09-10T20:39:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.580+0000 7fbffdbc6640 1 Processor -- start 2024-09-10T20:39:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.580+0000 7fbffdbc6640 1 -- start start 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbffdbc6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbff80cfa80 0x7fbff80c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbffdbc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbffdbc6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbff815e130 0x7fbff815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbffdbc6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81624a0 con 0x7fbff80d4b60 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbffdbc6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8162320 con 0x7fbff815e130 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbffdbc6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbff8162620 con 0x7fbff80cfa80 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff815e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49580/0 (socket says 172.21.15.28:49580) 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 -- 172.21.15.28:0/2697565532 learned_addr learned my addr 172.21.15.28:0/2697565532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff77fe640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbff80cfa80 0x7fbff80c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff7fff640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbff815e130 0x7fbff815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 -- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbff80cfa80 msgr2=0x7fbff80c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbff80cfa80 0x7fbff80c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 -- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbff815e130 msgr2=0x7fbff815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbff815e130 0x7fbff815ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 -- 172.21.15.28:0/2697565532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff80c8f20 con 0x7fbff80d4b60 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbff6ffd640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff815e710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009920 tx=0x7fbfec032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbfdffff640 1 -- 172.21.15.28:0/2697565532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec040070 con 0x7fbff80d4b60 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff80c9200 con 0x7fbff80d4b60 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.581+0000 7fbfdffff640 1 -- 172.21.15.28:0/2697565532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfec005310 con 0x7fbff80d4b60 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.582+0000 7fbfdffff640 1 -- 172.21.15.28:0/2697565532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec003800 con 0x7fbff80d4b60 2024-09-10T20:39:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.582+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff80c97c0 con 0x7fbff80d4b60 2024-09-10T20:39:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.583+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff80cf540 con 0x7fbff80d4b60 2024-09-10T20:39:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.583+0000 7fbfdffff640 1 -- 172.21.15.28:0/2697565532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fbfec0039a0 con 0x7fbff80d4b60 2024-09-10T20:39:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.584+0000 7fbfdffff640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfe4078150 0x7fbfe407a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.584+0000 7fbfdffff640 1 -- 172.21.15.28:0/2697565532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5227+0+0 (secure 0 0 0) 0x7fbfec0bb5f0 con 0x7fbff80d4b60 2024-09-10T20:39:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.584+0000 7fbff77fe640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfe4078150 0x7fbfe407a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.585+0000 7fbff77fe640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfe4078150 0x7fbfe407a610 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800a750 tx=0x7fbfe8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:21.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.589+0000 7fbfdffff640 1 -- 172.21.15.28:0/2697565532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbfec083ec0 con 0x7fbff80d4b60 2024-09-10T20:39:21.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:21.773+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbff80b6350 con 0x7fbff80d4b60 2024-09-10T20:39:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.448+0000 7fbfdffff640 1 -- 172.21.15.28:0/2697565532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v41) ==== 170+0+0 (secure 0 0 0) 0x7fbfec0837c0 con 0x7fbff80d4b60 2024-09-10T20:39:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:39:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfe4078150 msgr2=0x7fbfe407a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfe4078150 0x7fbfe407a610 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800a750 tx=0x7fbfe8008040 comp rx=0 tx=0).stop 2024-09-10T20:39:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 msgr2=0x7fbff815e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff815e710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009920 tx=0x7fbfec032d50 comp rx=0 tx=0).stop 2024-09-10T20:39:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 shutdown_connections 2024-09-10T20:39:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfe4078150 0x7fbfe407a610 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbff815e130 0x7fbff815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff80d4b60 0x7fbff815e710 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 --2- 172.21.15.28:0/2697565532 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbff80cfa80 0x7fbff80c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.453+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 >> 172.21.15.28:0/2697565532 conn(0x7fbff80d4040 msgr2=0x7fbff806d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.454+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 shutdown_connections 2024-09-10T20:39:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.459+0000 7fbffdbc6640 1 -- 172.21.15.28:0/2697565532 wait complete. 2024-09-10T20:39:22.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:39:22.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.595+0000 7fbe14423640 1 -- 172.21.15.28:0/2708157926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c15e7d0 msgr2=0x7fbe0c06f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:22.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.595+0000 7fbe14423640 1 --2- 172.21.15.28:0/2708157926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c15e7d0 0x7fbe0c06f510 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009830 tx=0x7fbdfc02a530 comp rx=0 tx=0).stop 2024-09-10T20:39:22.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.595+0000 7fbe14423640 1 -- 172.21.15.28:0/2708157926 shutdown_connections 2024-09-10T20:39:22.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.595+0000 7fbe14423640 1 --2- 172.21.15.28:0/2708157926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c15e7d0 0x7fbe0c06f510 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:22.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.595+0000 7fbe14423640 1 -- 172.21.15.28:0/2708157926 >> 172.21.15.28:0/2708157926 conn(0x7fbe0c06d110 msgr2=0x7fbe0c0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:22.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.595+0000 7fbe14423640 1 -- 172.21.15.28:0/2708157926 shutdown_connections 2024-09-10T20:39:22.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.595+0000 7fbe14423640 1 -- 172.21.15.28:0/2708157926 wait complete. 2024-09-10T20:39:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.596+0000 7fbe14423640 1 Processor -- start 2024-09-10T20:39:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.596+0000 7fbe14423640 1 -- start start 2024-09-10T20:39:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe14423640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe0c06fa50 0x7fbe0c154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe14423640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c1587a0 0x7fbe0c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe14423640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe0c15e7d0 0x7fbe0c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe14423640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c15d670 con 0x7fbe0c1587a0 2024-09-10T20:39:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe14423640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c15d4f0 con 0x7fbe0c15e7d0 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe14423640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0c15d7f0 con 0x7fbe0c06fa50 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c1587a0 0x7fbe0c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c1587a0 0x7fbe0c0ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49600/0 (socket says 172.21.15.28:49600) 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 -- 172.21.15.28:0/4120171594 learned_addr learned my addr 172.21.15.28:0/4120171594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe129c0640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe0c15e7d0 0x7fbe0c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe121bf640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe0c06fa50 0x7fbe0c154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 -- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe0c06fa50 msgr2=0x7fbe0c154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe0c06fa50 0x7fbe0c154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 -- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe0c15e7d0 msgr2=0x7fbe0c0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe0c15e7d0 0x7fbe0c0ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe119be640 1 -- 172.21.15.28:0/4120171594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c0b05f0 con 0x7fbe0c1587a0 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.597+0000 7fbe121bf640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe0c06fa50 0x7fbe0c154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.598+0000 7fbe119be640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c1587a0 0x7fbe0c0ce850 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000c00 tx=0x7fbdfc0316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:22.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.598+0000 7fbe02ffd640 1 -- 172.21.15.28:0/4120171594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc003b00 con 0x7fbe0c1587a0 2024-09-10T20:39:22.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.598+0000 7fbe02ffd640 1 -- 172.21.15.28:0/4120171594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdfc003d90 con 0x7fbe0c1587a0 2024-09-10T20:39:22.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.598+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0c1f1210 con 0x7fbe0c1587a0 2024-09-10T20:39:22.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.598+0000 7fbe02ffd640 1 -- 172.21.15.28:0/4120171594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc03d690 con 0x7fbe0c1587a0 2024-09-10T20:39:22.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.598+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0c1f1ab0 con 0x7fbe0c1587a0 2024-09-10T20:39:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.599+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe0c15dca0 con 0x7fbe0c1587a0 2024-09-10T20:39:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.599+0000 7fbe02ffd640 1 -- 172.21.15.28:0/4120171594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fbdfc03d830 con 0x7fbe0c1587a0 2024-09-10T20:39:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.600+0000 7fbe02ffd640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbdf0078120 0x7fbdf007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.600+0000 7fbe02ffd640 1 -- 172.21.15.28:0/4120171594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5609+0+0 (secure 0 0 0) 0x7fbdfc0bdcd0 con 0x7fbe0c1587a0 2024-09-10T20:39:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.600+0000 7fbe121bf640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbdf0078120 0x7fbdf007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.601+0000 7fbe121bf640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbdf0078120 0x7fbdf007a5e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe08007fd0 tx=0x7fbe08009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:22.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.604+0000 7fbe02ffd640 1 -- 172.21.15.28:0/4120171594 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbdfc041030 con 0x7fbe0c1587a0 2024-09-10T20:39:22.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:22.827+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbe0c0b2540 con 0x7fbe0c1587a0 2024-09-10T20:39:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.450+0000 7fbe02ffd640 1 -- 172.21.15.28:0/4120171594 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) ==== 172+0+0 (secure 0 0 0) 0x7fbdfc0864a0 con 0x7fbe0c1587a0 2024-09-10T20:39:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 6 and data pool 7 2024-09-10T20:39:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbdf0078120 msgr2=0x7fbdf007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbdf0078120 0x7fbdf007a5e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe08007fd0 tx=0x7fbe08009040 comp rx=0 tx=0).stop 2024-09-10T20:39:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c1587a0 msgr2=0x7fbe0c0ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c1587a0 0x7fbe0c0ce850 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000c00 tx=0x7fbdfc0316b0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 shutdown_connections 2024-09-10T20:39:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbdf0078120 0x7fbdf007a5e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe0c15e7d0 0x7fbe0c0ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe0c1587a0 0x7fbe0c0ce850 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 --2- 172.21.15.28:0/4120171594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe0c06fa50 0x7fbe0c154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.456+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 >> 172.21.15.28:0/4120171594 conn(0x7fbe0c06d110 msgr2=0x7fbe0c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.457+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 shutdown_connections 2024-09-10T20:39:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.457+0000 7fbe14423640 1 -- 172.21.15.28:0/4120171594 wait complete. 2024-09-10T20:39:23.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:39:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 -- 172.21.15.28:0/3189882897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 msgr2=0x7f97540cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 --2- 172.21.15.28:0/3189882897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 0x7f97540cb210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f974c009830 tx=0x7f974c02a510 comp rx=0 tx=0).stop 2024-09-10T20:39:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 -- 172.21.15.28:0/3189882897 shutdown_connections 2024-09-10T20:39:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 --2- 172.21.15.28:0/3189882897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97540cf0f0 0x7f975415e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 --2- 172.21.15.28:0/3189882897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 0x7f97540cb210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 -- 172.21.15.28:0/3189882897 >> 172.21.15.28:0/3189882897 conn(0x7f97540d0420 msgr2=0x7f97540d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 -- 172.21.15.28:0/3189882897 shutdown_connections 2024-09-10T20:39:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.602+0000 7f975bcdd640 1 -- 172.21.15.28:0/3189882897 wait complete. 2024-09-10T20:39:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.603+0000 7f975bcdd640 1 Processor -- start 2024-09-10T20:39:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.603+0000 7f975bcdd640 1 -- start start 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f975bcdd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97540cf0f0 0x7f97541558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f975bcdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9754151cc0 0x7f9754141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f975bcdd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 0x7f97541441c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f975bcdd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f975415f5b0 con 0x7f9754151cc0 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f975bcdd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f975415f430 con 0x7f975415ed20 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f975bcdd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f97540cb750 con 0x7f97540cf0f0 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9754151cc0 0x7f9754141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9754151cc0 0x7f9754141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49602/0 (socket says 172.21.15.28:49602) 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 -- 172.21.15.28:0/2329691472 learned_addr learned my addr 172.21.15.28:0/2329691472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759a79640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97540cf0f0 0x7f97541558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f975a27a640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 0x7f97541441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 -- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97540cf0f0 msgr2=0x7f97541558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97540cf0f0 0x7f97541558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 -- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 msgr2=0x7f97541441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 0x7f97541441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.604+0000 7f9759278640 1 -- 172.21.15.28:0/2329691472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9754155df0 con 0x7f9754151cc0 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.605+0000 7f9759278640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9754151cc0 0x7f9754141880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f974800e9a0 tx=0x7f974800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.605+0000 7f97527fc640 1 -- 172.21.15.28:0/2329691472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974800cd80 con 0x7f9754151cc0 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.605+0000 7f97527fc640 1 -- 172.21.15.28:0/2329691472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9748010430 con 0x7f9754151cc0 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.605+0000 7f97527fc640 1 -- 172.21.15.28:0/2329691472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9748010880 con 0x7f9754151cc0 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.605+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9754154980 con 0x7f9754151cc0 2024-09-10T20:39:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.605+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9754154f40 con 0x7f9754151cc0 2024-09-10T20:39:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.606+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97540cbb70 con 0x7f9754151cc0 2024-09-10T20:39:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.607+0000 7f97527fc640 1 -- 172.21.15.28:0/2329691472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f9748010a20 con 0x7f9754151cc0 2024-09-10T20:39:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.607+0000 7f97527fc640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9740078120 0x7f974007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.607+0000 7f97527fc640 1 -- 172.21.15.28:0/2329691472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5709+0+0 (secure 0 0 0) 0x7f974809aa90 con 0x7f9754151cc0 2024-09-10T20:39:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.607+0000 7f9759a79640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9740078120 0x7f974007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.608+0000 7f9759a79640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9740078120 0x7f974007a5e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9744001f00 tx=0x7f9744009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.611+0000 7f97527fc640 1 -- 172.21.15.28:0/2329691472 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9748062c70 con 0x7f9754151cc0 2024-09-10T20:39:23.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.797+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9754068d10 con 0x7f9754151cc0 2024-09-10T20:39:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.798+0000 7f97527fc640 1 -- 172.21.15.28:0/2329691472 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+16682 (secure 0 0 0) 0x7f97480676a0 con 0x7f9754151cc0 2024-09-10T20:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:39:22.829958+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:39:20.458884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-10T20:39:21.778513+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9740078120 msgr2=0x7f974007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9740078120 0x7f974007a5e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9744001f00 tx=0x7f9744009040 comp rx=0 tx=0).stop 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9754151cc0 msgr2=0x7f9754141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9754151cc0 0x7f9754141880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f974800e9a0 tx=0x7f974800ee60 comp rx=0 tx=0).stop 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 shutdown_connections 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9740078120 0x7f974007a5e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f975415ed20 0x7f97541441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9754151cc0 0x7f9754141880 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 --2- 172.21.15.28:0/2329691472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97540cf0f0 0x7f97541558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 >> 172.21.15.28:0/2329691472 conn(0x7f97540d0420 msgr2=0x7f9754073920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 shutdown_connections 2024-09-10T20:39:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.801+0000 7f975bcdd640 1 -- 172.21.15.28:0/2329691472 wait complete. 2024-09-10T20:39:23.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.916+0000 7fdfae433640 1 -- 172.21.15.28:0/1879178914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80738b0 msgr2=0x7fdfa815e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 --2- 172.21.15.28:0/1879178914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80738b0 0x7fdfa815e7e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8074360 tx=0x7fdf9c02a4d0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 -- 172.21.15.28:0/1879178914 shutdown_connections 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 --2- 172.21.15.28:0/1879178914 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfa8073cf0 0x7fdfa815ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 --2- 172.21.15.28:0/1879178914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80738b0 0x7fdfa815e7e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 --2- 172.21.15.28:0/1879178914 >> v2:172.21.15.159:3300/0 conn(0x7fdfa8151cc0 0x7fdfa80cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 -- 172.21.15.28:0/1879178914 >> 172.21.15.28:0/1879178914 conn(0x7fdfa80d0420 msgr2=0x7fdfa80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 -- 172.21.15.28:0/1879178914 shutdown_connections 2024-09-10T20:39:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.917+0000 7fdfae433640 1 -- 172.21.15.28:0/1879178914 wait complete. 2024-09-10T20:39:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.918+0000 7fdfae433640 1 Processor -- start 2024-09-10T20:39:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.918+0000 7fdfae433640 1 -- start start 2024-09-10T20:39:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfae433640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfa80738b0 0x7fdfa814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfae433640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa8073cf0 0x7fdfa8146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfae433640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfa8151cc0 0x7fdfa806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfae433640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa80cb210 con 0x7fdfa8073cf0 2024-09-10T20:39:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfae433640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa815f640 con 0x7fdfa8151cc0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfae433640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa80cb390 con 0x7fdfa80738b0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa8073cf0 0x7fdfa8146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa8073cf0 0x7fdfa8146030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49622/0 (socket says 172.21.15.28:49622) 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 -- 172.21.15.28:0/2569696396 learned_addr learned my addr 172.21.15.28:0/2569696396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa7fff640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfa80738b0 0x7fdfa814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfac9d0640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfa8151cc0 0x7fdfa806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 -- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfa80738b0 msgr2=0x7fdfa814cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfa80738b0 0x7fdfa814cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 -- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfa8151cc0 msgr2=0x7fdfa806cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfa8151cc0 0x7fdfa806cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 -- 172.21.15.28:0/2569696396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa80b3570 con 0x7fdfa8073cf0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.919+0000 7fdfa77fe640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa8073cf0 0x7fdfa8146030 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009850 tx=0x7fdf9c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.920+0000 7fdfa4ff9640 1 -- 172.21.15.28:0/2569696396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c03f070 con 0x7fdfa8073cf0 2024-09-10T20:39:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.920+0000 7fdfa4ff9640 1 -- 172.21.15.28:0/2569696396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf9c004890 con 0x7fdfa8073cf0 2024-09-10T20:39:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.920+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa80b4310 con 0x7fdfa8073cf0 2024-09-10T20:39:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.920+0000 7fdfa4ff9640 1 -- 172.21.15.28:0/2569696396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c035030 con 0x7fdfa8073cf0 2024-09-10T20:39:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.920+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa80b3b10 con 0x7fdfa8073cf0 2024-09-10T20:39:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.921+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa80cf0f0 con 0x7fdfa8073cf0 2024-09-10T20:39:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.921+0000 7fdfa4ff9640 1 -- 172.21.15.28:0/2569696396 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fdf9c004ca0 con 0x7fdfa8073cf0 2024-09-10T20:39:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.922+0000 7fdfa4ff9640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdf94078120 0x7fdf9407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.922+0000 7fdfa4ff9640 1 -- 172.21.15.28:0/2569696396 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5709+0+0 (secure 0 0 0) 0x7fdf9c0bb1b0 con 0x7fdfa8073cf0 2024-09-10T20:39:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.922+0000 7fdfa7fff640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdf94078120 0x7fdf9407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.923+0000 7fdfa7fff640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdf94078120 0x7fdf9407a5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf98005fd0 tx=0x7fdf9800a250 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:23.926+0000 7fdfa4ff9640 1 -- 172.21.15.28:0/2569696396 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdf9c083410 con 0x7fdfa8073cf0 2024-09-10T20:39:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.112+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdfa8068d10 con 0x7fdfa8073cf0 2024-09-10T20:39:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.113+0000 7fdfa4ff9640 1 -- 172.21.15.28:0/2569696396 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+16682 (secure 0 0 0) 0x7fdf9c087e40 con 0x7fdfa8073cf0 2024-09-10T20:39:24.118 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:24.118 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:39:22.829958+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:39:20.458884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-10T20:39:21.778513+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:39:24.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdf94078120 msgr2=0x7fdf9407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:24.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdf94078120 0x7fdf9407a5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf98005fd0 tx=0x7fdf9800a250 comp rx=0 tx=0).stop 2024-09-10T20:39:24.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa8073cf0 msgr2=0x7fdfa8146030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:24.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa8073cf0 0x7fdfa8146030 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009850 tx=0x7fdf9c0039e0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 shutdown_connections 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdf94078120 0x7fdf9407a5e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfa8151cc0 0x7fdfa806cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa8073cf0 0x7fdfa8146030 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 --2- 172.21.15.28:0/2569696396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfa80738b0 0x7fdfa814cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 >> 172.21.15.28:0/2569696396 conn(0x7fdfa80d0420 msgr2=0x7fdfa8149990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 shutdown_connections 2024-09-10T20:39:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.115+0000 7fdfae433640 1 -- 172.21.15.28:0/2569696396 wait complete. 2024-09-10T20:39:24.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:39:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.234+0000 7f78a409b640 1 -- 172.21.15.28:0/1426150996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c0cb3a0 msgr2=0x7f789c0cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.234+0000 7f78a409b640 1 --2- 172.21.15.28:0/1426150996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c0cb3a0 0x7f789c0cb830 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7894009830 tx=0x7f789402a5c0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.234+0000 7f78a409b640 1 -- 172.21.15.28:0/1426150996 shutdown_connections 2024-09-10T20:39:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.234+0000 7f78a409b640 1 --2- 172.21.15.28:0/1426150996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c0cb3a0 0x7f789c0cb830 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.234+0000 7f78a409b640 1 -- 172.21.15.28:0/1426150996 >> 172.21.15.28:0/1426150996 conn(0x7f789c0d0d50 msgr2=0x7f789c0d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.234+0000 7f78a409b640 1 -- 172.21.15.28:0/1426150996 shutdown_connections 2024-09-10T20:39:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.234+0000 7f78a409b640 1 -- 172.21.15.28:0/1426150996 wait complete. 2024-09-10T20:39:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.235+0000 7f78a409b640 1 Processor -- start 2024-09-10T20:39:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.235+0000 7f78a409b640 1 -- start start 2024-09-10T20:39:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a409b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f789c0cb3a0 0x7f789c0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a409b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f789c0d1840 0x7f789c0c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a409b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c152df0 0x7f789c0c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a409b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f789c06ce70 con 0x7f789c152df0 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a409b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f789c06ccf0 con 0x7f789c0cb3a0 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a409b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f789c06cff0 con 0x7f789c0d1840 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a2638640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c152df0 0x7f789c0c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a2638640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c152df0 0x7f789c0c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49634/0 (socket says 172.21.15.28:49634) 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.236+0000 7f78a2638640 1 -- 172.21.15.28:0/442379759 learned_addr learned my addr 172.21.15.28:0/442379759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a1e37640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f789c0cb3a0 0x7f789c0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a1636640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f789c0d1840 0x7f789c0c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a2638640 1 -- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f789c0d1840 msgr2=0x7f789c0c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a2638640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f789c0d1840 0x7f789c0c4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a2638640 1 -- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f789c0cb3a0 msgr2=0x7f789c0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a2638640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f789c0cb3a0 0x7f789c0c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a2638640 1 -- 172.21.15.28:0/442379759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789c0b00b0 con 0x7f789c152df0 2024-09-10T20:39:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a1e37640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f789c0cb3a0 0x7f789c0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a1636640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f789c0d1840 0x7f789c0c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a2638640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c152df0 0x7f789c0c54c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f789402abb0 tx=0x7f7894033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78927fc640 1 -- 172.21.15.28:0/442379759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894031b50 con 0x7f789c152df0 2024-09-10T20:39:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78927fc640 1 -- 172.21.15.28:0/442379759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7894031de0 con 0x7f789c152df0 2024-09-10T20:39:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f789c0b0390 con 0x7f789c152df0 2024-09-10T20:39:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78927fc640 1 -- 172.21.15.28:0/442379759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894005280 con 0x7f789c152df0 2024-09-10T20:39:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.237+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f789c0c4750 con 0x7f789c152df0 2024-09-10T20:39:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.238+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f789c06a3b0 con 0x7f789c152df0 2024-09-10T20:39:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.239+0000 7f78927fc640 1 -- 172.21.15.28:0/442379759 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f7894005420 con 0x7f789c152df0 2024-09-10T20:39:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.239+0000 7f78927fc640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7880078120 0x7f788007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.240+0000 7f78927fc640 1 -- 172.21.15.28:0/442379759 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5709+0+0 (secure 0 0 0) 0x7f78940bb420 con 0x7f789c152df0 2024-09-10T20:39:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.240+0000 7f78a1e37640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7880078120 0x7f788007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.240+0000 7f78a1e37640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7880078120 0x7f788007a5e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7898001db0 tx=0x7f7898009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.243+0000 7f78927fc640 1 -- 172.21.15.28:0/442379759 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f789c06a3b0 con 0x7f789c152df0 2024-09-10T20:39:24.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:24.469+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f789c06fb80 con 0x7f789c152df0 2024-09-10T20:39:25.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.473+0000 7f78927fc640 1 -- 172.21.15.28:0/442379759 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7f789403f030 con 0x7f789c152df0 2024-09-10T20:39:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7880078120 msgr2=0x7f788007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7880078120 0x7f788007a5e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7898001db0 tx=0x7f7898009040 comp rx=0 tx=0).stop 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c152df0 msgr2=0x7f789c0c54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c152df0 0x7f789c0c54c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f789402abb0 tx=0x7f7894033ba0 comp rx=0 tx=0).stop 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 shutdown_connections 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7880078120 0x7f788007a5e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f789c152df0 0x7f789c0c54c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f789c0d1840 0x7f789c0c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 --2- 172.21.15.28:0/442379759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f789c0cb3a0 0x7f789c0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 >> 172.21.15.28:0/442379759 conn(0x7f789c0d0d50 msgr2=0x7f789c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 shutdown_connections 2024-09-10T20:39:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.477+0000 7f78a409b640 1 -- 172.21.15.28:0/442379759 wait complete. 2024-09-10T20:39:25.495 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:39:25.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:39:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.606+0000 7fbe38247640 1 -- 172.21.15.28:0/1258109902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe1c0043b0 msgr2=0x7fbe1c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.606+0000 7fbe38247640 1 --2- 172.21.15.28:0/1258109902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe1c0043b0 0x7fbe1c004860 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000bb20 tx=0x7fbe1c005830 comp rx=0 tx=0).stop 2024-09-10T20:39:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.606+0000 7fbe367e4640 1 -- 172.21.15.28:0/1258109902 reap_dead start 2024-09-10T20:39:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.606+0000 7fbe38247640 1 -- 172.21.15.28:0/1258109902 shutdown_connections 2024-09-10T20:39:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.606+0000 7fbe38247640 1 -- 172.21.15.28:0/1258109902 >> 172.21.15.28:0/1258109902 conn(0x7fbe300d0d50 msgr2=0x7fbe300d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.606+0000 7fbe38247640 1 -- 172.21.15.28:0/1258109902 shutdown_connections 2024-09-10T20:39:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.606+0000 7fbe38247640 1 -- 172.21.15.28:0/1258109902 wait complete. 2024-09-10T20:39:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 Processor -- start 2024-09-10T20:39:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 -- start start 2024-09-10T20:39:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe1c0043b0 0x7fbe3015e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe300c5e60 0x7fbe300ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe300cab90 0x7fbe301f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30073b20 con 0x7fbe300c5e60 2024-09-10T20:39:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbe300739a0 con 0x7fbe1c0043b0 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe38247640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3015f000 con 0x7fbe300cab90 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.608+0000 7fbe357e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe300c5e60 0x7fbe300ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe300c5e60 0x7fbe300ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49644/0 (socket says 172.21.15.28:49644) 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 -- 172.21.15.28:0/2366239996 learned_addr learned my addr 172.21.15.28:0/2366239996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe35fe3640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe1c0043b0 0x7fbe3015e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe367e4640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe300cab90 0x7fbe301f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 -- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe300cab90 msgr2=0x7fbe301f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe300cab90 0x7fbe301f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 -- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe1c0043b0 msgr2=0x7fbe3015e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe1c0043b0 0x7fbe3015e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 -- 172.21.15.28:0/2366239996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe301f0600 con 0x7fbe300c5e60 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe357e2640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe300c5e60 0x7fbe300ca650 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000bc40 tx=0x7fbe20031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe267fc640 1 -- 172.21.15.28:0/2366239996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20040070 con 0x7fbe300c5e60 2024-09-10T20:39:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe267fc640 1 -- 172.21.15.28:0/2366239996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe20004030 con 0x7fbe300c5e60 2024-09-10T20:39:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe301f08e0 con 0x7fbe300c5e60 2024-09-10T20:39:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe267fc640 1 -- 172.21.15.28:0/2366239996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20035040 con 0x7fbe300c5e60 2024-09-10T20:39:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.609+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe301f0ea0 con 0x7fbe300c5e60 2024-09-10T20:39:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.610+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe3015f4f0 con 0x7fbe300c5e60 2024-09-10T20:39:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.611+0000 7fbe267fc640 1 -- 172.21.15.28:0/2366239996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fbe2003a630 con 0x7fbe300c5e60 2024-09-10T20:39:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.612+0000 7fbe267fc640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbe14078120 0x7fbe1407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.612+0000 7fbe267fc640 1 -- 172.21.15.28:0/2366239996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fbe200bb080 con 0x7fbe300c5e60 2024-09-10T20:39:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.612+0000 7fbe35fe3640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbe14078120 0x7fbe1407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.613+0000 7fbe35fe3640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbe14078120 0x7fbe1407a5e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbe300b0650 tx=0x7fbe2c00b260 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:25.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.616+0000 7fbe267fc640 1 -- 172.21.15.28:0/2366239996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbe200836c0 con 0x7fbe300c5e60 2024-09-10T20:39:25.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:25.839+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fbe300749a0 con 0x7fbe300c5e60 2024-09-10T20:39:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.478+0000 7fbe267fc640 1 -- 172.21.15.28:0/2366239996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) ==== 122+0+0 (secure 0 0 0) 0x7fbe2008ae20 con 0x7fbe300c5e60 2024-09-10T20:39:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbe14078120 msgr2=0x7fbe1407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbe14078120 0x7fbe1407a5e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbe300b0650 tx=0x7fbe2c00b260 comp rx=0 tx=0).stop 2024-09-10T20:39:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe300c5e60 msgr2=0x7fbe300ca650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe300c5e60 0x7fbe300ca650 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000bc40 tx=0x7fbe20031f90 comp rx=0 tx=0).stop 2024-09-10T20:39:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 shutdown_connections 2024-09-10T20:39:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbe14078120 0x7fbe1407a5e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbe300cab90 0x7fbe301f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe300c5e60 0x7fbe300ca650 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 --2- 172.21.15.28:0/2366239996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbe1c0043b0 0x7fbe3015e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 >> 172.21.15.28:0/2366239996 conn(0x7fbe300d0d50 msgr2=0x7fbe300cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 shutdown_connections 2024-09-10T20:39:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.481+0000 7fbe38247640 1 -- 172.21.15.28:0/2366239996 wait complete. 2024-09-10T20:39:26.500 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:39:26.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.606+0000 7f5066ca2640 1 -- 172.21.15.28:0/2772419772 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 msgr2=0x7f506015e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.606+0000 7f5066ca2640 1 --2- 172.21.15.28:0/2772419772 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 0x7f506015e930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5054005fd0 tx=0x7f505402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.607+0000 7f5066ca2640 1 -- 172.21.15.28:0/2772419772 shutdown_connections 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.607+0000 7f5066ca2640 1 --2- 172.21.15.28:0/2772419772 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f506015ee70 0x7f50600d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.607+0000 7f5066ca2640 1 --2- 172.21.15.28:0/2772419772 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 0x7f506015e930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.607+0000 7f5066ca2640 1 -- 172.21.15.28:0/2772419772 >> 172.21.15.28:0/2772419772 conn(0x7f506006cf00 msgr2=0x7f506006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.607+0000 7f5066ca2640 1 -- 172.21.15.28:0/2772419772 shutdown_connections 2024-09-10T20:39:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.607+0000 7f5066ca2640 1 -- 172.21.15.28:0/2772419772 wait complete. 2024-09-10T20:39:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.608+0000 7f5066ca2640 1 Processor -- start 2024-09-10T20:39:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.608+0000 7f5066ca2640 1 -- start start 2024-09-10T20:39:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5060155100 0x7f50601f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 0x7f506014c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506015ee70 0x7f5060137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5060072de0 con 0x7f506015ee70 2024-09-10T20:39:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f50600d0af0 con 0x7f50601554e0 2024-09-10T20:39:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5060072f60 con 0x7f5060155100 2024-09-10T20:39:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506015ee70 0x7f5060137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506015ee70 0x7f5060137260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49646/0 (socket says 172.21.15.28:49646) 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 -- 172.21.15.28:0/1176055871 learned_addr learned my addr 172.21.15.28:0/1176055871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5064a3e640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5060155100 0x7f50601f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f505ffff640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 0x7f506014c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 -- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5060155100 msgr2=0x7f50601f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5060155100 0x7f50601f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 -- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 msgr2=0x7f506014c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 0x7f506014c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 -- 172.21.15.28:0/1176055871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506014c5b0 con 0x7f506015ee70 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5064a3e640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5060155100 0x7f50601f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f506523f640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506015ee70 0x7f5060137260 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f505800efd0 tx=0x7f505800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f505d7fa640 1 -- 172.21.15.28:0/1176055871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058003b70 con 0x7f506015ee70 2024-09-10T20:39:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f505d7fa640 1 -- 172.21.15.28:0/1176055871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5058003e00 con 0x7f506015ee70 2024-09-10T20:39:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50601377a0 con 0x7f506015ee70 2024-09-10T20:39:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f505d7fa640 1 -- 172.21.15.28:0/1176055871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058004280 con 0x7f506015ee70 2024-09-10T20:39:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.610+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5060137d20 con 0x7f506015ee70 2024-09-10T20:39:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.611+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f506015e930 con 0x7f506015ee70 2024-09-10T20:39:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.612+0000 7f505d7fa640 1 -- 172.21.15.28:0/1176055871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f5058004420 con 0x7f506015ee70 2024-09-10T20:39:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.612+0000 7f505d7fa640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f504c078120 0x7f504c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.612+0000 7f505d7fa640 1 -- 172.21.15.28:0/1176055871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f505809b0d0 con 0x7f506015ee70 2024-09-10T20:39:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.613+0000 7f5064a3e640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f504c078120 0x7f504c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.613+0000 7f5064a3e640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f504c078120 0x7f504c07a5e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5050007fd0 tx=0x7f5050009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.617+0000 7f505d7fa640 1 -- 172.21.15.28:0/1176055871 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f50580638d0 con 0x7f506015ee70 2024-09-10T20:39:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.839+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f506015def0 con 0x7f506015ee70 2024-09-10T20:39:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.839+0000 7f505d7fa640 1 -- 172.21.15.28:0/1176055871 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f50580631d0 con 0x7f506015ee70 2024-09-10T20:39:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:26.845 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-10T20:39:26:475950+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f504c078120 msgr2=0x7f504c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f504c078120 0x7f504c07a5e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5050007fd0 tx=0x7f5050009040 comp rx=0 tx=0).stop 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506015ee70 msgr2=0x7f5060137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506015ee70 0x7f5060137260 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f505800efd0 tx=0x7f505800c6a0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 shutdown_connections 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f504c078120 0x7f504c07a5e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506015ee70 0x7f5060137260 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f50601554e0 0x7f506014c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 --2- 172.21.15.28:0/1176055871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5060155100 0x7f50601f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 >> 172.21.15.28:0/1176055871 conn(0x7f506006cf00 msgr2=0x7f506006fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 shutdown_connections 2024-09-10T20:39:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.842+0000 7f5066ca2640 1 -- 172.21.15.28:0/1176055871 wait complete. 2024-09-10T20:39:26.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:39:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.959+0000 7faa00c7f640 1 -- 172.21.15.28:0/605444751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0d07a0 msgr2=0x7fa9fc0b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.959+0000 7faa00c7f640 1 --2- 172.21.15.28:0/605444751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0d07a0 0x7fa9fc0b2e80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009be0 tx=0x7fa9f402a960 comp rx=0 tx=0).stop 2024-09-10T20:39:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.960+0000 7faa00c7f640 1 -- 172.21.15.28:0/605444751 shutdown_connections 2024-09-10T20:39:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.960+0000 7faa00c7f640 1 --2- 172.21.15.28:0/605444751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0d07a0 0x7fa9fc0b2e80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.960+0000 7faa00c7f640 1 -- 172.21.15.28:0/605444751 >> 172.21.15.28:0/605444751 conn(0x7fa9fc06cf00 msgr2=0x7fa9fc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.960+0000 7faa00c7f640 1 -- 172.21.15.28:0/605444751 shutdown_connections 2024-09-10T20:39:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.960+0000 7faa00c7f640 1 -- 172.21.15.28:0/605444751 wait complete. 2024-09-10T20:39:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.961+0000 7faa00c7f640 1 Processor -- start 2024-09-10T20:39:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.961+0000 7faa00c7f640 1 -- start start 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7faa00c7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0cc570 0x7fa9fc0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7faa00c7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9fc0cd3d0 0x7fa9fc1f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7faa00c7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9fc0d07a0 0x7fa9fc14b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7faa00c7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9fc0b3540 con 0x7fa9fc0cc570 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7faa00c7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa9fc0b33c0 con 0x7fa9fc0cd3d0 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7faa00c7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa9fc0b36c0 con 0x7fa9fc0d07a0 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0cc570 0x7fa9fc0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0cc570 0x7fa9fc0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49670/0 (socket says 172.21.15.28:49670) 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 -- 172.21.15.28:0/4160164071 learned_addr learned my addr 172.21.15.28:0/4160164071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fad9d640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9fc0d07a0 0x7fa9fc14b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9f1d9b640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9fc0cd3d0 0x7fa9fc1f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 -- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9fc0d07a0 msgr2=0x7fa9fc14b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9fc0d07a0 0x7fa9fc14b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 -- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9fc0cd3d0 msgr2=0x7fa9fc1f18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9fc0cd3d0 0x7fa9fc1f18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 -- 172.21.15.28:0/4160164071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9fc0af940 con 0x7fa9fc0cc570 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9f1d9b640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9fc0cd3d0 0x7fa9fc1f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fad9d640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9fc0d07a0 0x7fa9fc14b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.962+0000 7fa9fa59c640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0cc570 0x7fa9fc0cb7b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00c450 tx=0x7fa9ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.963+0000 7fa9f37fe640 1 -- 172.21.15.28:0/4160164071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec0052d0 con 0x7fa9fc0cc570 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.963+0000 7fa9f37fe640 1 -- 172.21.15.28:0/4160164071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9ec005470 con 0x7fa9fc0cc570 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.963+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9fc1f4ff0 con 0x7fa9fc0cc570 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.963+0000 7fa9f37fe640 1 -- 172.21.15.28:0/4160164071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec012870 con 0x7fa9fc0cc570 2024-09-10T20:39:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.963+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9fc1f1df0 con 0x7fa9fc0cc570 2024-09-10T20:39:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.964+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9fc0b3ae0 con 0x7fa9fc0cc570 2024-09-10T20:39:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.964+0000 7fa9f37fe640 1 -- 172.21.15.28:0/4160164071 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fa9ec012a10 con 0x7fa9fc0cc570 2024-09-10T20:39:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.965+0000 7fa9f37fe640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9dc078120 0x7fa9dc07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.965+0000 7fa9f37fe640 1 -- 172.21.15.28:0/4160164071 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fa9ec099c50 con 0x7fa9fc0cc570 2024-09-10T20:39:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.965+0000 7fa9f1d9b640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9dc078120 0x7fa9dc07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.965+0000 7fa9f1d9b640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9dc078120 0x7fa9dc07a5e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40046b0 tx=0x7fa9e4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:26.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:26.969+0000 7fa9f37fe640 1 -- 172.21.15.28:0/4160164071 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa9ec09f020 con 0x7fa9fc0cc570 2024-09-10T20:39:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.156+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9fc1f4da0 con 0x7fa9fc0cc570 2024-09-10T20:39:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.157+0000 7fa9f37fe640 1 -- 172.21.15.28:0/4160164071 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16669 (secure 0 0 0) 0x7fa9ec062450 con 0x7fa9fc0cc570 2024-09-10T20:39:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:27.161 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":43,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:39:24.463365+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:39:20.458884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-10T20:39:21.778513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9dc078120 msgr2=0x7fa9dc07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9dc078120 0x7fa9dc07a5e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40046b0 tx=0x7fa9e4009040 comp rx=0 tx=0).stop 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0cc570 msgr2=0x7fa9fc0cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0cc570 0x7fa9fc0cb7b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00c450 tx=0x7fa9ec00c910 comp rx=0 tx=0).stop 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 shutdown_connections 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9dc078120 0x7fa9dc07a5e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9fc0d07a0 0x7fa9fc14b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9fc0cd3d0 0x7fa9fc1f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 --2- 172.21.15.28:0/4160164071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9fc0cc570 0x7fa9fc0cb7b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 >> 172.21.15.28:0/4160164071 conn(0x7fa9fc06cf00 msgr2=0x7fa9fc0acdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 shutdown_connections 2024-09-10T20:39:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.159+0000 7faa00c7f640 1 -- 172.21.15.28:0/4160164071 wait complete. 2024-09-10T20:39:27.175 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:39:27.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:39:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 -- 172.21.15.28:0/3752057547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100d2960 msgr2=0x7f1c100ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 --2- 172.21.15.28:0/3752057547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100d2960 0x7f1c100ce850 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1c100b7ed0 tx=0x7f1bf802a560 comp rx=0 tx=0).stop 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 -- 172.21.15.28:0/3752057547 shutdown_connections 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 --2- 172.21.15.28:0/3752057547 >> v2:172.21.15.159:3300/0 conn(0x7f1c100cd9b0 0x7f1c100ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 --2- 172.21.15.28:0/3752057547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100d2960 0x7f1c100ce850 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 --2- 172.21.15.28:0/3752057547 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c1015c050 0x7f1c100cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 -- 172.21.15.28:0/3752057547 >> 172.21.15.28:0/3752057547 conn(0x7f1c100d1e40 msgr2=0x7f1c100d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 -- 172.21.15.28:0/3752057547 shutdown_connections 2024-09-10T20:39:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.276+0000 7f1c15285640 1 -- 172.21.15.28:0/3752057547 wait complete. 2024-09-10T20:39:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.277+0000 7f1c15285640 1 Processor -- start 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c15285640 1 -- start start 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c15285640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100cd9b0 0x7f1c101e55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c15285640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c100d2960 0x7f1c10150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c15285640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c1015c050 0x7f1c101e72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c15285640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c100b1820 con 0x7f1c100cd9b0 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c15285640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1c100b16a0 con 0x7f1c1015c050 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c15285640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c100b19a0 con 0x7f1c100d2960 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100cd9b0 0x7f1c101e55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100cd9b0 0x7f1c101e55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49702/0 (socket says 172.21.15.28:49702) 2024-09-10T20:39:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.278+0000 7f1c0effd640 1 -- 172.21.15.28:0/3574556060 learned_addr learned my addr 172.21.15.28:0/3574556060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c0f7fe640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c1015c050 0x7f1c101e72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c07fff640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c100d2960 0x7f1c10150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c0effd640 1 -- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c100d2960 msgr2=0x7f1c10150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c0effd640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c100d2960 0x7f1c10150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c0effd640 1 -- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c1015c050 msgr2=0x7f1c101e72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c0effd640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c1015c050 0x7f1c101e72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c0effd640 1 -- 172.21.15.28:0/3574556060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c101f4ea0 con 0x7f1c100cd9b0 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c07fff640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c100d2960 0x7f1c10150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c0effd640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100cd9b0 0x7f1c101e55a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c970 tx=0x7f1c0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:27.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c077fe640 1 -- 172.21.15.28:0/3574556060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00007bb0 con 0x7f1c100cd9b0 2024-09-10T20:39:27.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c077fe640 1 -- 172.21.15.28:0/3574556060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c00007e40 con 0x7f1c100cd9b0 2024-09-10T20:39:27.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c101f5180 con 0x7f1c100cd9b0 2024-09-10T20:39:27.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c077fe640 1 -- 172.21.15.28:0/3574556060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000f690 con 0x7f1c100cd9b0 2024-09-10T20:39:27.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.279+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c101f5740 con 0x7f1c100cd9b0 2024-09-10T20:39:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.280+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c100b1f30 con 0x7f1c100cd9b0 2024-09-10T20:39:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.281+0000 7f1c077fe640 1 -- 172.21.15.28:0/3574556060 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f1c00020030 con 0x7f1c100cd9b0 2024-09-10T20:39:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.281+0000 7f1c077fe640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf0078120 0x7f1bf007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.281+0000 7f1c077fe640 1 -- 172.21.15.28:0/3574556060 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f1c0009a6e0 con 0x7f1c100cd9b0 2024-09-10T20:39:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.281+0000 7f1c07fff640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf0078120 0x7f1bf007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.282+0000 7f1c07fff640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf0078120 0x7f1bf007a5e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8033000 tx=0x7f1bf8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.286+0000 7f1c077fe640 1 -- 172.21.15.28:0/3574556060 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c00014030 con 0x7f1c100cd9b0 2024-09-10T20:39:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.466+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1c100b2d10 con 0x7f1bf0078120 2024-09-10T20:39:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.471+0000 7f1c077fe640 1 -- 172.21.15.28:0/3574556060 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+244688 (secure 0 0 0) 0x7f1c100b2d10 con 0x7f1bf0078120 2024-09-10T20:39:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:39:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf0078120 msgr2=0x7f1bf007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf0078120 0x7f1bf007a5e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8033000 tx=0x7f1bf8004730 comp rx=0 tx=0).stop 2024-09-10T20:39:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100cd9b0 msgr2=0x7f1c101e55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100cd9b0 0x7f1c101e55a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c970 tx=0x7f1c0000ce30 comp rx=0 tx=0).stop 2024-09-10T20:39:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 shutdown_connections 2024-09-10T20:39:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1bf0078120 0x7f1bf007a5e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c1015c050 0x7f1c101e72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c100d2960 0x7f1c10150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 --2- 172.21.15.28:0/3574556060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c100cd9b0 0x7f1c101e55a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 >> 172.21.15.28:0/3574556060 conn(0x7f1c100d1e40 msgr2=0x7f1c100b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 shutdown_connections 2024-09-10T20:39:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.474+0000 7f1c15285640 1 -- 172.21.15.28:0/3574556060 wait complete. 2024-09-10T20:39:27.489 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-10T20:39:25.991250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592618,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":131,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":100,"acting":100,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":237724,"kb_used_data":4840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736996,"statfs":{"total":773094113280,"available":772850683904,"internally_reserved":0,"allocated":4956160,"data_stored":2654960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2250,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.547476"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471413+0000","last_change":"2024-09-10T20:39:23.474482+0000","last_active":"2024-09-10T20:39:24.471413+0000","last_peered":"2024-09-10T20:39:24.471413+0000","last_clean":"2024-09-10T20:39:24.471413+0000","last_became_active":"2024-09-10T20:39:23.469443+0000","last_became_peered":"2024-09-10T20:39:23.469443+0000","last_unstale":"2024-09-10T20:39:24.471413+0000","last_undegraded":"2024-09-10T20:39:24.471413+0000","last_fullsized":"2024-09-10T20:39:24.471413+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470522+0000","last_change":"2024-09-10T20:39:23.478562+0000","last_active":"2024-09-10T20:39:24.470522+0000","last_peered":"2024-09-10T20:39:24.470522+0000","last_clean":"2024-09-10T20:39:24.470522+0000","last_became_active":"2024-09-10T20:39:23.478273+0000","last_became_peered":"2024-09-10T20:39:23.478273+0000","last_unstale":"2024-09-10T20:39:24.470522+0000","last_undegraded":"2024-09-10T20:39:24.470522+0000","last_fullsized":"2024-09-10T20:39:24.470522+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:06:13.043952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471730+0000","last_change":"2024-09-10T20:39:23.477481+0000","last_active":"2024-09-10T20:39:24.471730+0000","last_peered":"2024-09-10T20:39:24.471730+0000","last_clean":"2024-09-10T20:39:24.471730+0000","last_became_active":"2024-09-10T20:39:23.477194+0000","last_became_peered":"2024-09-10T20:39:23.477194+0000","last_unstale":"2024-09-10T20:39:24.471730+0000","last_undegraded":"2024-09-10T20:39:24.471730+0000","last_fullsized":"2024-09-10T20:39:24.471730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:16.051559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470973+0000","last_change":"2024-09-10T20:39:23.465809+0000","last_active":"2024-09-10T20:39:24.470973+0000","last_peered":"2024-09-10T20:39:24.470973+0000","last_clean":"2024-09-10T20:39:24.470973+0000","last_became_active":"2024-09-10T20:39:23.465540+0000","last_became_peered":"2024-09-10T20:39:23.465540+0000","last_unstale":"2024-09-10T20:39:24.470973+0000","last_undegraded":"2024-09-10T20:39:24.470973+0000","last_fullsized":"2024-09-10T20:39:24.470973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.481077+0000","last_change":"2024-09-10T20:39:23.480200+0000","last_active":"2024-09-10T20:39:23.481077+0000","last_peered":"2024-09-10T20:39:23.481077+0000","last_clean":"2024-09-10T20:39:23.481077+0000","last_became_active":"2024-09-10T20:39:23.479928+0000","last_became_peered":"2024-09-10T20:39:23.479928+0000","last_unstale":"2024-09-10T20:39:23.481077+0000","last_undegraded":"2024-09-10T20:39:23.481077+0000","last_fullsized":"2024-09-10T20:39:23.481077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:20.100800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471901+0000","last_change":"2024-09-10T20:39:23.472464+0000","last_active":"2024-09-10T20:39:24.471901+0000","last_peered":"2024-09-10T20:39:24.471901+0000","last_clean":"2024-09-10T20:39:24.471901+0000","last_became_active":"2024-09-10T20:39:23.472185+0000","last_became_peered":"2024-09-10T20:39:23.472185+0000","last_unstale":"2024-09-10T20:39:24.471901+0000","last_undegraded":"2024-09-10T20:39:24.471901+0000","last_fullsized":"2024-09-10T20:39:24.471901+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:16:50.093970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470189+0000","last_change":"2024-09-10T20:39:23.479455+0000","last_active":"2024-09-10T20:39:24.470189+0000","last_peered":"2024-09-10T20:39:24.470189+0000","last_clean":"2024-09-10T20:39:24.470189+0000","last_became_active":"2024-09-10T20:39:23.479185+0000","last_became_peered":"2024-09-10T20:39:23.479185+0000","last_unstale":"2024-09-10T20:39:24.470189+0000","last_undegraded":"2024-09-10T20:39:24.470189+0000","last_fullsized":"2024-09-10T20:39:24.470189+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471366+0000","last_change":"2024-09-10T20:39:23.472528+0000","last_active":"2024-09-10T20:39:24.471366+0000","last_peered":"2024-09-10T20:39:24.471366+0000","last_clean":"2024-09-10T20:39:24.471366+0000","last_became_active":"2024-09-10T20:39:23.472247+0000","last_became_peered":"2024-09-10T20:39:23.472247+0000","last_unstale":"2024-09-10T20:39:24.471366+0000","last_undegraded":"2024-09-10T20:39:24.471366+0000","last_fullsized":"2024-09-10T20:39:24.471366+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471460+0000","last_change":"2024-09-10T20:39:23.477260+0000","last_active":"2024-09-10T20:39:24.471460+0000","last_peered":"2024-09-10T20:39:24.471460+0000","last_clean":"2024-09-10T20:39:24.471460+0000","last_became_active":"2024-09-10T20:39:23.476962+0000","last_became_peered":"2024-09-10T20:39:23.476962+0000","last_unstale":"2024-09-10T20:39:24.471460+0000","last_undegraded":"2024-09-10T20:39:24.471460+0000","last_fullsized":"2024-09-10T20:39:24.471460+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475335+0000","last_change":"2024-09-10T20:39:23.467260+0000","last_active":"2024-09-10T20:39:23.475335+0000","last_peered":"2024-09-10T20:39:23.475335+0000","last_clean":"2024-09-10T20:39:23.475335+0000","last_became_active":"2024-09-10T20:39:23.465867+0000","last_became_peered":"2024-09-10T20:39:23.465867+0000","last_unstale":"2024-09-10T20:39:23.475335+0000","last_undegraded":"2024-09-10T20:39:23.475335+0000","last_fullsized":"2024-09-10T20:39:23.475335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:51.591296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471738+0000","last_change":"2024-09-10T20:39:23.479313+0000","last_active":"2024-09-10T20:39:24.471738+0000","last_peered":"2024-09-10T20:39:24.471738+0000","last_clean":"2024-09-10T20:39:24.471738+0000","last_became_active":"2024-09-10T20:39:23.478782+0000","last_became_peered":"2024-09-10T20:39:23.478782+0000","last_unstale":"2024-09-10T20:39:24.471738+0000","last_undegraded":"2024-09-10T20:39:24.471738+0000","last_fullsized":"2024-09-10T20:39:24.471738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:32:50.720508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473143+0000","last_change":"2024-09-10T20:39:23.471294+0000","last_active":"2024-09-10T20:39:23.473143+0000","last_peered":"2024-09-10T20:39:23.473143+0000","last_clean":"2024-09-10T20:39:23.473143+0000","last_became_active":"2024-09-10T20:39:23.471002+0000","last_became_peered":"2024-09-10T20:39:23.471002+0000","last_unstale":"2024-09-10T20:39:23.473143+0000","last_undegraded":"2024-09-10T20:39:23.473143+0000","last_fullsized":"2024-09-10T20:39:23.473143+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:53.982272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471486+0000","last_change":"2024-09-10T20:39:23.471724+0000","last_active":"2024-09-10T20:39:24.471486+0000","last_peered":"2024-09-10T20:39:24.471486+0000","last_clean":"2024-09-10T20:39:24.471486+0000","last_became_active":"2024-09-10T20:39:23.471453+0000","last_became_peered":"2024-09-10T20:39:23.471453+0000","last_unstale":"2024-09-10T20:39:24.471486+0000","last_undegraded":"2024-09-10T20:39:24.471486+0000","last_fullsized":"2024-09-10T20:39:24.471486+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:52.838472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469972+0000","last_change":"2024-09-10T20:39:23.469522+0000","last_active":"2024-09-10T20:39:24.469972+0000","last_peered":"2024-09-10T20:39:24.469972+0000","last_clean":"2024-09-10T20:39:24.469972+0000","last_became_active":"2024-09-10T20:39:23.469129+0000","last_became_peered":"2024-09-10T20:39:23.469129+0000","last_unstale":"2024-09-10T20:39:24.469972+0000","last_undegraded":"2024-09-10T20:39:24.469972+0000","last_fullsized":"2024-09-10T20:39:24.469972+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:28.627939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471378+0000","last_change":"2024-09-10T20:39:23.470278+0000","last_active":"2024-09-10T20:39:24.471378+0000","last_peered":"2024-09-10T20:39:24.471378+0000","last_clean":"2024-09-10T20:39:24.471378+0000","last_became_active":"2024-09-10T20:39:23.469632+0000","last_became_peered":"2024-09-10T20:39:23.469632+0000","last_unstale":"2024-09-10T20:39:24.471378+0000","last_undegraded":"2024-09-10T20:39:24.471378+0000","last_fullsized":"2024-09-10T20:39:24.471378+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.476954+0000","last_change":"2024-09-10T20:39:23.476296+0000","last_active":"2024-09-10T20:39:23.476954+0000","last_peered":"2024-09-10T20:39:23.476954+0000","last_clean":"2024-09-10T20:39:23.476954+0000","last_became_active":"2024-09-10T20:39:23.476009+0000","last_became_peered":"2024-09-10T20:39:23.476009+0000","last_unstale":"2024-09-10T20:39:23.476954+0000","last_undegraded":"2024-09-10T20:39:23.476954+0000","last_fullsized":"2024-09-10T20:39:23.476954+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:29:17.845671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":19,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470688+0000","last_change":"2024-09-10T20:39:22.471060+0000","last_active":"2024-09-10T20:39:24.470688+0000","last_peered":"2024-09-10T20:39:24.470688+0000","last_clean":"2024-09-10T20:39:24.470688+0000","last_became_active":"2024-09-10T20:39:22.470753+0000","last_became_peered":"2024-09-10T20:39:22.470753+0000","last_unstale":"2024-09-10T20:39:24.470688+0000","last_undegraded":"2024-09-10T20:39:24.470688+0000","last_fullsized":"2024-09-10T20:39:24.470688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:22.626422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470292+0000","last_change":"2024-09-10T20:39:22.467354+0000","last_active":"2024-09-10T20:39:24.470292+0000","last_peered":"2024-09-10T20:39:24.470292+0000","last_clean":"2024-09-10T20:39:24.470292+0000","last_became_active":"2024-09-10T20:39:22.467105+0000","last_became_peered":"2024-09-10T20:39:22.467105+0000","last_unstale":"2024-09-10T20:39:24.470292+0000","last_undegraded":"2024-09-10T20:39:24.470292+0000","last_fullsized":"2024-09-10T20:39:24.470292+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:42.142331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471995+0000","last_change":"2024-09-10T20:39:23.465129+0000","last_active":"2024-09-10T20:39:24.471995+0000","last_peered":"2024-09-10T20:39:24.471995+0000","last_clean":"2024-09-10T20:39:24.471995+0000","last_became_active":"2024-09-10T20:39:23.464816+0000","last_became_peered":"2024-09-10T20:39:23.464816+0000","last_unstale":"2024-09-10T20:39:24.471995+0000","last_undegraded":"2024-09-10T20:39:24.471995+0000","last_fullsized":"2024-09-10T20:39:24.471995+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:06.505948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470439+0000","last_change":"2024-09-10T20:39:22.468131+0000","last_active":"2024-09-10T20:39:24.470439+0000","last_peered":"2024-09-10T20:39:24.470439+0000","last_clean":"2024-09-10T20:39:24.470439+0000","last_became_active":"2024-09-10T20:39:22.467846+0000","last_became_peered":"2024-09-10T20:39:22.467846+0000","last_unstale":"2024-09-10T20:39:24.470439+0000","last_undegraded":"2024-09-10T20:39:24.470439+0000","last_fullsized":"2024-09-10T20:39:24.470439+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:40.914501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.480621+0000","last_change":"2024-09-10T20:39:23.477169+0000","last_active":"2024-09-10T20:39:23.480621+0000","last_peered":"2024-09-10T20:39:23.480621+0000","last_clean":"2024-09-10T20:39:23.480621+0000","last_became_active":"2024-09-10T20:39:23.476830+0000","last_became_peered":"2024-09-10T20:39:23.476830+0000","last_unstale":"2024-09-10T20:39:23.480621+0000","last_undegraded":"2024-09-10T20:39:23.480621+0000","last_fullsized":"2024-09-10T20:39:23.480621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:20:58.616528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475067+0000","last_change":"2024-09-10T20:39:23.466496+0000","last_active":"2024-09-10T20:39:23.475067+0000","last_peered":"2024-09-10T20:39:23.475067+0000","last_clean":"2024-09-10T20:39:23.475067+0000","last_became_active":"2024-09-10T20:39:23.466162+0000","last_became_peered":"2024-09-10T20:39:23.466162+0000","last_unstale":"2024-09-10T20:39:23.475067+0000","last_undegraded":"2024-09-10T20:39:23.475067+0000","last_fullsized":"2024-09-10T20:39:23.475067+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:36.107439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460912+0000","last_change":"2024-09-10T20:39:22.464013+0000","last_active":"2024-09-10T20:39:23.460912+0000","last_peered":"2024-09-10T20:39:23.460912+0000","last_clean":"2024-09-10T20:39:23.460912+0000","last_became_active":"2024-09-10T20:39:22.463745+0000","last_became_peered":"2024-09-10T20:39:22.463745+0000","last_unstale":"2024-09-10T20:39:23.460912+0000","last_undegraded":"2024-09-10T20:39:23.460912+0000","last_fullsized":"2024-09-10T20:39:23.460912+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:17.407343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460348+0000","last_change":"2024-09-10T20:39:22.470606+0000","last_active":"2024-09-10T20:39:23.460348+0000","last_peered":"2024-09-10T20:39:23.460348+0000","last_clean":"2024-09-10T20:39:23.460348+0000","last_became_active":"2024-09-10T20:39:22.470313+0000","last_became_peered":"2024-09-10T20:39:22.470313+0000","last_unstale":"2024-09-10T20:39:23.460348+0000","last_undegraded":"2024-09-10T20:39:23.460348+0000","last_fullsized":"2024-09-10T20:39:23.460348+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:13.562757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471621+0000","last_change":"2024-09-10T20:39:23.478464+0000","last_active":"2024-09-10T20:39:24.471621+0000","last_peered":"2024-09-10T20:39:24.471621+0000","last_clean":"2024-09-10T20:39:24.471621+0000","last_became_active":"2024-09-10T20:39:23.477764+0000","last_became_peered":"2024-09-10T20:39:23.477764+0000","last_unstale":"2024-09-10T20:39:24.471621+0000","last_undegraded":"2024-09-10T20:39:24.471621+0000","last_fullsized":"2024-09-10T20:39:24.471621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:07.988361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":19,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.487847+0000","last_change":"2024-09-10T20:39:22.457688+0000","last_active":"2024-09-10T20:39:23.487847+0000","last_peered":"2024-09-10T20:39:23.487847+0000","last_clean":"2024-09-10T20:39:23.487847+0000","last_became_active":"2024-09-10T20:39:22.457326+0000","last_became_peered":"2024-09-10T20:39:22.457326+0000","last_unstale":"2024-09-10T20:39:23.487847+0000","last_undegraded":"2024-09-10T20:39:23.487847+0000","last_fullsized":"2024-09-10T20:39:23.487847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:51.098445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473204+0000","last_change":"2024-09-10T20:39:22.463919+0000","last_active":"2024-09-10T20:39:23.473204+0000","last_peered":"2024-09-10T20:39:23.473204+0000","last_clean":"2024-09-10T20:39:23.473204+0000","last_became_active":"2024-09-10T20:39:22.463650+0000","last_became_peered":"2024-09-10T20:39:22.463650+0000","last_unstale":"2024-09-10T20:39:23.473204+0000","last_undegraded":"2024-09-10T20:39:23.473204+0000","last_fullsized":"2024-09-10T20:39:23.473204+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:07:05.483324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484089+0000","last_change":"2024-09-10T20:39:23.482909+0000","last_active":"2024-09-10T20:39:23.484089+0000","last_peered":"2024-09-10T20:39:23.484089+0000","last_clean":"2024-09-10T20:39:23.484089+0000","last_became_active":"2024-09-10T20:39:23.482551+0000","last_became_peered":"2024-09-10T20:39:23.482551+0000","last_unstale":"2024-09-10T20:39:23.484089+0000","last_undegraded":"2024-09-10T20:39:23.484089+0000","last_fullsized":"2024-09-10T20:39:23.484089+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:10.837228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.467882+0000","last_change":"2024-09-10T20:39:23.465467+0000","last_active":"2024-09-10T20:39:23.467882+0000","last_peered":"2024-09-10T20:39:23.467882+0000","last_clean":"2024-09-10T20:39:23.467882+0000","last_became_active":"2024-09-10T20:39:23.465165+0000","last_became_peered":"2024-09-10T20:39:23.465165+0000","last_unstale":"2024-09-10T20:39:23.467882+0000","last_undegraded":"2024-09-10T20:39:23.467882+0000","last_fullsized":"2024-09-10T20:39:23.467882+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:15.544019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469888+0000","last_change":"2024-09-10T20:39:22.471774+0000","last_active":"2024-09-10T20:39:24.469888+0000","last_peered":"2024-09-10T20:39:24.469888+0000","last_clean":"2024-09-10T20:39:24.469888+0000","last_became_active":"2024-09-10T20:39:22.471455+0000","last_became_peered":"2024-09-10T20:39:22.471455+0000","last_unstale":"2024-09-10T20:39:24.469888+0000","last_undegraded":"2024-09-10T20:39:24.469888+0000","last_fullsized":"2024-09-10T20:39:24.469888+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:31.755937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":21,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.485737+0000","last_change":"2024-09-10T20:39:22.468116+0000","last_active":"2024-09-10T20:39:23.485737+0000","last_peered":"2024-09-10T20:39:23.485737+0000","last_clean":"2024-09-10T20:39:23.485737+0000","last_became_active":"2024-09-10T20:39:22.467712+0000","last_became_peered":"2024-09-10T20:39:22.467712+0000","last_unstale":"2024-09-10T20:39:23.485737+0000","last_undegraded":"2024-09-10T20:39:23.485737+0000","last_fullsized":"2024-09-10T20:39:23.485737+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:18:01.811366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.474598+0000","last_change":"2024-09-10T20:39:23.473728+0000","last_active":"2024-09-10T20:39:23.474598+0000","last_peered":"2024-09-10T20:39:23.474598+0000","last_clean":"2024-09-10T20:39:23.474598+0000","last_became_active":"2024-09-10T20:39:23.473482+0000","last_became_peered":"2024-09-10T20:39:23.473482+0000","last_unstale":"2024-09-10T20:39:23.474598+0000","last_undegraded":"2024-09-10T20:39:23.474598+0000","last_fullsized":"2024-09-10T20:39:23.474598+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:42:24.635057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471265+0000","last_change":"2024-09-10T20:39:22.464648+0000","last_active":"2024-09-10T20:39:24.471265+0000","last_peered":"2024-09-10T20:39:24.471265+0000","last_clean":"2024-09-10T20:39:24.471265+0000","last_became_active":"2024-09-10T20:39:22.464381+0000","last_became_peered":"2024-09-10T20:39:22.464381+0000","last_unstale":"2024-09-10T20:39:24.471265+0000","last_undegraded":"2024-09-10T20:39:24.471265+0000","last_fullsized":"2024-09-10T20:39:24.471265+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:34.596334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.469081+0000","last_change":"2024-09-10T20:39:23.466893+0000","last_active":"2024-09-10T20:39:23.469081+0000","last_peered":"2024-09-10T20:39:23.469081+0000","last_clean":"2024-09-10T20:39:23.469081+0000","last_became_active":"2024-09-10T20:39:23.466577+0000","last_became_peered":"2024-09-10T20:39:23.466577+0000","last_unstale":"2024-09-10T20:39:23.469081+0000","last_undegraded":"2024-09-10T20:39:23.469081+0000","last_fullsized":"2024-09-10T20:39:23.469081+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:50.331650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470125+0000","last_change":"2024-09-10T20:39:22.471454+0000","last_active":"2024-09-10T20:39:24.470125+0000","last_peered":"2024-09-10T20:39:24.470125+0000","last_clean":"2024-09-10T20:39:24.470125+0000","last_became_active":"2024-09-10T20:39:22.471186+0000","last_became_peered":"2024-09-10T20:39:22.471186+0000","last_unstale":"2024-09-10T20:39:24.470125+0000","last_undegraded":"2024-09-10T20:39:24.470125+0000","last_fullsized":"2024-09-10T20:39:24.470125+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:11.967787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479095+0000","last_change":"2024-09-10T20:39:23.474293+0000","last_active":"2024-09-10T20:39:23.479095+0000","last_peered":"2024-09-10T20:39:23.479095+0000","last_clean":"2024-09-10T20:39:23.479095+0000","last_became_active":"2024-09-10T20:39:23.474013+0000","last_became_peered":"2024-09-10T20:39:23.474013+0000","last_unstale":"2024-09-10T20:39:23.479095+0000","last_undegraded":"2024-09-10T20:39:23.479095+0000","last_fullsized":"2024-09-10T20:39:23.479095+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:39:00.145526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484950+0000","last_change":"2024-09-10T20:39:22.467010+0000","last_active":"2024-09-10T20:39:23.484950+0000","last_peered":"2024-09-10T20:39:23.484950+0000","last_clean":"2024-09-10T20:39:23.484950+0000","last_became_active":"2024-09-10T20:39:22.466604+0000","last_became_peered":"2024-09-10T20:39:22.466604+0000","last_unstale":"2024-09-10T20:39:23.484950+0000","last_undegraded":"2024-09-10T20:39:23.484950+0000","last_fullsized":"2024-09-10T20:39:23.484950+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:33:38.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470167+0000","last_change":"2024-09-10T20:39:23.483419+0000","last_active":"2024-09-10T20:39:24.470167+0000","last_peered":"2024-09-10T20:39:24.470167+0000","last_clean":"2024-09-10T20:39:24.470167+0000","last_became_active":"2024-09-10T20:39:23.482573+0000","last_became_peered":"2024-09-10T20:39:23.482573+0000","last_unstale":"2024-09-10T20:39:24.470167+0000","last_undegraded":"2024-09-10T20:39:24.470167+0000","last_fullsized":"2024-09-10T20:39:24.470167+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471576+0000","last_change":"2024-09-10T20:39:23.470437+0000","last_active":"2024-09-10T20:39:24.471576+0000","last_peered":"2024-09-10T20:39:24.471576+0000","last_clean":"2024-09-10T20:39:24.471576+0000","last_became_active":"2024-09-10T20:39:23.470018+0000","last_became_peered":"2024-09-10T20:39:23.470018+0000","last_unstale":"2024-09-10T20:39:24.471576+0000","last_undegraded":"2024-09-10T20:39:24.471576+0000","last_fullsized":"2024-09-10T20:39:24.471576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:19.864809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.462467+0000","last_change":"2024-09-10T20:39:22.470294+0000","last_active":"2024-09-10T20:39:23.462467+0000","last_peered":"2024-09-10T20:39:23.462467+0000","last_clean":"2024-09-10T20:39:23.462467+0000","last_became_active":"2024-09-10T20:39:22.469974+0000","last_became_peered":"2024-09-10T20:39:22.469974+0000","last_unstale":"2024-09-10T20:39:23.462467+0000","last_undegraded":"2024-09-10T20:39:23.462467+0000","last_fullsized":"2024-09-10T20:39:23.462467+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:14.321360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471788+0000","last_change":"2024-09-10T20:39:23.467237+0000","last_active":"2024-09-10T20:39:24.471788+0000","last_peered":"2024-09-10T20:39:24.471788+0000","last_clean":"2024-09-10T20:39:24.471788+0000","last_became_active":"2024-09-10T20:39:23.466955+0000","last_became_peered":"2024-09-10T20:39:23.466955+0000","last_unstale":"2024-09-10T20:39:24.471788+0000","last_undegraded":"2024-09-10T20:39:24.471788+0000","last_fullsized":"2024-09-10T20:39:24.471788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:17:53.617323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471588+0000","last_change":"2024-09-10T20:39:22.466270+0000","last_active":"2024-09-10T20:39:24.471588+0000","last_peered":"2024-09-10T20:39:24.471588+0000","last_clean":"2024-09-10T20:39:24.471588+0000","last_became_active":"2024-09-10T20:39:22.465951+0000","last_became_peered":"2024-09-10T20:39:22.465951+0000","last_unstale":"2024-09-10T20:39:24.471588+0000","last_undegraded":"2024-09-10T20:39:24.471588+0000","last_fullsized":"2024-09-10T20:39:24.471588+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475360+0000","last_change":"2024-09-10T20:39:23.468716+0000","last_active":"2024-09-10T20:39:23.475360+0000","last_peered":"2024-09-10T20:39:23.475360+0000","last_clean":"2024-09-10T20:39:23.475360+0000","last_became_active":"2024-09-10T20:39:23.468450+0000","last_became_peered":"2024-09-10T20:39:23.468450+0000","last_unstale":"2024-09-10T20:39:23.475360+0000","last_undegraded":"2024-09-10T20:39:23.475360+0000","last_fullsized":"2024-09-10T20:39:23.475360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:00.101321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460877+0000","last_change":"2024-09-10T20:39:22.467181+0000","last_active":"2024-09-10T20:39:23.460877+0000","last_peered":"2024-09-10T20:39:23.460877+0000","last_clean":"2024-09-10T20:39:23.460877+0000","last_became_active":"2024-09-10T20:39:22.466859+0000","last_became_peered":"2024-09-10T20:39:22.466859+0000","last_unstale":"2024-09-10T20:39:23.460877+0000","last_undegraded":"2024-09-10T20:39:23.460877+0000","last_fullsized":"2024-09-10T20:39:23.460877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:54.024056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479036+0000","last_change":"2024-09-10T20:39:23.473568+0000","last_active":"2024-09-10T20:39:23.479036+0000","last_peered":"2024-09-10T20:39:23.479036+0000","last_clean":"2024-09-10T20:39:23.479036+0000","last_became_active":"2024-09-10T20:39:23.473257+0000","last_became_peered":"2024-09-10T20:39:23.473257+0000","last_unstale":"2024-09-10T20:39:23.479036+0000","last_undegraded":"2024-09-10T20:39:23.479036+0000","last_fullsized":"2024-09-10T20:39:23.479036+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:02.160411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471424+0000","last_change":"2024-09-10T20:39:23.482739+0000","last_active":"2024-09-10T20:39:24.471424+0000","last_peered":"2024-09-10T20:39:24.471424+0000","last_clean":"2024-09-10T20:39:24.471424+0000","last_became_active":"2024-09-10T20:39:23.482483+0000","last_became_peered":"2024-09-10T20:39:23.482483+0000","last_unstale":"2024-09-10T20:39:24.471424+0000","last_undegraded":"2024-09-10T20:39:24.471424+0000","last_fullsized":"2024-09-10T20:39:24.471424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:53.453242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469982+0000","last_change":"2024-09-10T20:39:22.464748+0000","last_active":"2024-09-10T20:39:24.469982+0000","last_peered":"2024-09-10T20:39:24.469982+0000","last_clean":"2024-09-10T20:39:24.469982+0000","last_became_active":"2024-09-10T20:39:22.464248+0000","last_became_peered":"2024-09-10T20:39:22.464248+0000","last_unstale":"2024-09-10T20:39:24.469982+0000","last_undegraded":"2024-09-10T20:39:24.469982+0000","last_fullsized":"2024-09-10T20:39:24.469982+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:34:33.491750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.459867+0000","last_change":"2024-09-10T20:39:22.465000+0000","last_active":"2024-09-10T20:39:23.459867+0000","last_peered":"2024-09-10T20:39:23.459867+0000","last_clean":"2024-09-10T20:39:23.459867+0000","last_became_active":"2024-09-10T20:39:22.464669+0000","last_became_peered":"2024-09-10T20:39:22.464669+0000","last_unstale":"2024-09-10T20:39:23.459867+0000","last_undegraded":"2024-09-10T20:39:23.459867+0000","last_fullsized":"2024-09-10T20:39:23.459867+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:43:06.255942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471722+0000","last_change":"2024-09-10T20:39:23.476662+0000","last_active":"2024-09-10T20:39:24.471722+0000","last_peered":"2024-09-10T20:39:24.471722+0000","last_clean":"2024-09-10T20:39:24.471722+0000","last_became_active":"2024-09-10T20:39:23.476375+0000","last_became_peered":"2024-09-10T20:39:23.476375+0000","last_unstale":"2024-09-10T20:39:24.471722+0000","last_undegraded":"2024-09-10T20:39:24.471722+0000","last_fullsized":"2024-09-10T20:39:24.471722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:46.142798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":141,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T20:39:20.047668+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:39:20.047668+0000","last_peered":"2024-09-10T20:39:20.047668+0000","last_clean":"2024-09-10T20:39:20.047668+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:39:20.047668+0000","last_undegraded":"2024-09-10T20:39:20.047668+0000","last_fullsized":"2024-09-10T20:39:20.047668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2250,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":34,"acting":34,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509460,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31632,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":446464,"data_stored":173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509459,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32208,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339632,"statfs":{"total":96636764160,"available":96603783168,"internally_reserved":0,"allocated":1040384,"data_stored":764128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":491520,"data_stored":193243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1093632,"data_stored":789317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":503808,"data_stored":194331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:39:27.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 -- 172.21.15.28:0/3004246328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1901554e0 msgr2=0x7ff19015e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 --2- 172.21.15.28:0/3004246328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1901554e0 0x7ff19015e930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff1900ced80 tx=0x7ff18002a560 comp rx=0 tx=0).stop 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 -- 172.21.15.28:0/3004246328 shutdown_connections 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 --2- 172.21.15.28:0/3004246328 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff19015ee70 0x7ff1900d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 --2- 172.21.15.28:0/3004246328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1901554e0 0x7ff19015e930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 --2- 172.21.15.28:0/3004246328 >> v2:172.21.15.159:3300/0 conn(0x7ff190155100 0x7ff19015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 -- 172.21.15.28:0/3004246328 >> 172.21.15.28:0/3004246328 conn(0x7ff19006cf00 msgr2=0x7ff19006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 -- 172.21.15.28:0/3004246328 shutdown_connections 2024-09-10T20:39:27.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.631+0000 7ff19767f640 1 -- 172.21.15.28:0/3004246328 wait complete. 2024-09-10T20:39:27.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.632+0000 7ff19767f640 1 Processor -- start 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.632+0000 7ff19767f640 1 -- start start 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff19767f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff190155100 0x7ff1900b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff19767f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1901554e0 0x7ff1901f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff19767f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ee70 0x7ff190143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff19767f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff190072bd0 con 0x7ff19015ee70 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff19767f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff1900d0ab0 con 0x7ff1901554e0 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff19767f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff190072d50 con 0x7ff190155100 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ee70 0x7ff190143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ee70 0x7ff190143320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49722/0 (socket says 172.21.15.28:49722) 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 -- 172.21.15.28:0/1025660055 learned_addr learned my addr 172.21.15.28:0/1025660055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff19541b640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff190155100 0x7ff1900b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff194c1a640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1901554e0 0x7ff1901f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 -- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff190155100 msgr2=0x7ff1900b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff190155100 0x7ff1900b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 -- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1901554e0 msgr2=0x7ff1901f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1901554e0 0x7ff1901f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 -- 172.21.15.28:0/1025660055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff190143860 con 0x7ff19015ee70 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff195c1c640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ee70 0x7ff190143320 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff18800eaa0 tx=0x7ff18800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.633+0000 7ff185ffb640 1 -- 172.21.15.28:0/1025660055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18800cdb0 con 0x7ff19015ee70 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.634+0000 7ff185ffb640 1 -- 172.21.15.28:0/1025660055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff188004c10 con 0x7ff19015ee70 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.634+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1900b1e90 con 0x7ff19015ee70 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.634+0000 7ff185ffb640 1 -- 172.21.15.28:0/1025660055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188005020 con 0x7ff19015ee70 2024-09-10T20:39:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.634+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1900ad040 con 0x7ff19015ee70 2024-09-10T20:39:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.635+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff19015e930 con 0x7ff19015ee70 2024-09-10T20:39:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.636+0000 7ff185ffb640 1 -- 172.21.15.28:0/1025660055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7ff18800f040 con 0x7ff19015ee70 2024-09-10T20:39:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.636+0000 7ff185ffb640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff174078120 0x7ff17407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.636+0000 7ff185ffb640 1 -- 172.21.15.28:0/1025660055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7ff18809a690 con 0x7ff19015ee70 2024-09-10T20:39:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.636+0000 7ff19541b640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff174078120 0x7ff17407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.637+0000 7ff19541b640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff174078120 0x7ff17407a5e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00a4d0 tx=0x7ff18c00a460 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.641+0000 7ff185ffb640 1 -- 172.21.15.28:0/1025660055 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff188016030 con 0x7ff19015ee70 2024-09-10T20:39:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.823+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1900d03b0 con 0x7ff174078120 2024-09-10T20:39:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.827+0000 7ff185ffb640 1 -- 172.21.15.28:0/1025660055 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+244688 (secure 0 0 0) 0x7ff1900d03b0 con 0x7ff174078120 2024-09-10T20:39:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:39:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff174078120 msgr2=0x7ff17407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff174078120 0x7ff17407a5e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00a4d0 tx=0x7ff18c00a460 comp rx=0 tx=0).stop 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ee70 msgr2=0x7ff190143320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ee70 0x7ff190143320 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff18800eaa0 tx=0x7ff18800ef60 comp rx=0 tx=0).stop 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 shutdown_connections 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff174078120 0x7ff17407a5e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ee70 0x7ff190143320 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff1901554e0 0x7ff1901f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 --2- 172.21.15.28:0/1025660055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff190155100 0x7ff1900b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 >> 172.21.15.28:0/1025660055 conn(0x7ff19006cf00 msgr2=0x7ff19006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 shutdown_connections 2024-09-10T20:39:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:27.831+0000 7ff19767f640 1 -- 172.21.15.28:0/1025660055 wait complete. 2024-09-10T20:39:27.845 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-10T20:39:25.991250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592618,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":131,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":100,"acting":100,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":237724,"kb_used_data":4840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736996,"statfs":{"total":773094113280,"available":772850683904,"internally_reserved":0,"allocated":4956160,"data_stored":2654960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2250,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.547476"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471413+0000","last_change":"2024-09-10T20:39:23.474482+0000","last_active":"2024-09-10T20:39:24.471413+0000","last_peered":"2024-09-10T20:39:24.471413+0000","last_clean":"2024-09-10T20:39:24.471413+0000","last_became_active":"2024-09-10T20:39:23.469443+0000","last_became_peered":"2024-09-10T20:39:23.469443+0000","last_unstale":"2024-09-10T20:39:24.471413+0000","last_undegraded":"2024-09-10T20:39:24.471413+0000","last_fullsized":"2024-09-10T20:39:24.471413+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470522+0000","last_change":"2024-09-10T20:39:23.478562+0000","last_active":"2024-09-10T20:39:24.470522+0000","last_peered":"2024-09-10T20:39:24.470522+0000","last_clean":"2024-09-10T20:39:24.470522+0000","last_became_active":"2024-09-10T20:39:23.478273+0000","last_became_peered":"2024-09-10T20:39:23.478273+0000","last_unstale":"2024-09-10T20:39:24.470522+0000","last_undegraded":"2024-09-10T20:39:24.470522+0000","last_fullsized":"2024-09-10T20:39:24.470522+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:06:13.043952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471730+0000","last_change":"2024-09-10T20:39:23.477481+0000","last_active":"2024-09-10T20:39:24.471730+0000","last_peered":"2024-09-10T20:39:24.471730+0000","last_clean":"2024-09-10T20:39:24.471730+0000","last_became_active":"2024-09-10T20:39:23.477194+0000","last_became_peered":"2024-09-10T20:39:23.477194+0000","last_unstale":"2024-09-10T20:39:24.471730+0000","last_undegraded":"2024-09-10T20:39:24.471730+0000","last_fullsized":"2024-09-10T20:39:24.471730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:16.051559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470973+0000","last_change":"2024-09-10T20:39:23.465809+0000","last_active":"2024-09-10T20:39:24.470973+0000","last_peered":"2024-09-10T20:39:24.470973+0000","last_clean":"2024-09-10T20:39:24.470973+0000","last_became_active":"2024-09-10T20:39:23.465540+0000","last_became_peered":"2024-09-10T20:39:23.465540+0000","last_unstale":"2024-09-10T20:39:24.470973+0000","last_undegraded":"2024-09-10T20:39:24.470973+0000","last_fullsized":"2024-09-10T20:39:24.470973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.481077+0000","last_change":"2024-09-10T20:39:23.480200+0000","last_active":"2024-09-10T20:39:23.481077+0000","last_peered":"2024-09-10T20:39:23.481077+0000","last_clean":"2024-09-10T20:39:23.481077+0000","last_became_active":"2024-09-10T20:39:23.479928+0000","last_became_peered":"2024-09-10T20:39:23.479928+0000","last_unstale":"2024-09-10T20:39:23.481077+0000","last_undegraded":"2024-09-10T20:39:23.481077+0000","last_fullsized":"2024-09-10T20:39:23.481077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:20.100800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471901+0000","last_change":"2024-09-10T20:39:23.472464+0000","last_active":"2024-09-10T20:39:24.471901+0000","last_peered":"2024-09-10T20:39:24.471901+0000","last_clean":"2024-09-10T20:39:24.471901+0000","last_became_active":"2024-09-10T20:39:23.472185+0000","last_became_peered":"2024-09-10T20:39:23.472185+0000","last_unstale":"2024-09-10T20:39:24.471901+0000","last_undegraded":"2024-09-10T20:39:24.471901+0000","last_fullsized":"2024-09-10T20:39:24.471901+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:16:50.093970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470189+0000","last_change":"2024-09-10T20:39:23.479455+0000","last_active":"2024-09-10T20:39:24.470189+0000","last_peered":"2024-09-10T20:39:24.470189+0000","last_clean":"2024-09-10T20:39:24.470189+0000","last_became_active":"2024-09-10T20:39:23.479185+0000","last_became_peered":"2024-09-10T20:39:23.479185+0000","last_unstale":"2024-09-10T20:39:24.470189+0000","last_undegraded":"2024-09-10T20:39:24.470189+0000","last_fullsized":"2024-09-10T20:39:24.470189+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471366+0000","last_change":"2024-09-10T20:39:23.472528+0000","last_active":"2024-09-10T20:39:24.471366+0000","last_peered":"2024-09-10T20:39:24.471366+0000","last_clean":"2024-09-10T20:39:24.471366+0000","last_became_active":"2024-09-10T20:39:23.472247+0000","last_became_peered":"2024-09-10T20:39:23.472247+0000","last_unstale":"2024-09-10T20:39:24.471366+0000","last_undegraded":"2024-09-10T20:39:24.471366+0000","last_fullsized":"2024-09-10T20:39:24.471366+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471460+0000","last_change":"2024-09-10T20:39:23.477260+0000","last_active":"2024-09-10T20:39:24.471460+0000","last_peered":"2024-09-10T20:39:24.471460+0000","last_clean":"2024-09-10T20:39:24.471460+0000","last_became_active":"2024-09-10T20:39:23.476962+0000","last_became_peered":"2024-09-10T20:39:23.476962+0000","last_unstale":"2024-09-10T20:39:24.471460+0000","last_undegraded":"2024-09-10T20:39:24.471460+0000","last_fullsized":"2024-09-10T20:39:24.471460+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475335+0000","last_change":"2024-09-10T20:39:23.467260+0000","last_active":"2024-09-10T20:39:23.475335+0000","last_peered":"2024-09-10T20:39:23.475335+0000","last_clean":"2024-09-10T20:39:23.475335+0000","last_became_active":"2024-09-10T20:39:23.465867+0000","last_became_peered":"2024-09-10T20:39:23.465867+0000","last_unstale":"2024-09-10T20:39:23.475335+0000","last_undegraded":"2024-09-10T20:39:23.475335+0000","last_fullsized":"2024-09-10T20:39:23.475335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:51.591296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471738+0000","last_change":"2024-09-10T20:39:23.479313+0000","last_active":"2024-09-10T20:39:24.471738+0000","last_peered":"2024-09-10T20:39:24.471738+0000","last_clean":"2024-09-10T20:39:24.471738+0000","last_became_active":"2024-09-10T20:39:23.478782+0000","last_became_peered":"2024-09-10T20:39:23.478782+0000","last_unstale":"2024-09-10T20:39:24.471738+0000","last_undegraded":"2024-09-10T20:39:24.471738+0000","last_fullsized":"2024-09-10T20:39:24.471738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:32:50.720508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473143+0000","last_change":"2024-09-10T20:39:23.471294+0000","last_active":"2024-09-10T20:39:23.473143+0000","last_peered":"2024-09-10T20:39:23.473143+0000","last_clean":"2024-09-10T20:39:23.473143+0000","last_became_active":"2024-09-10T20:39:23.471002+0000","last_became_peered":"2024-09-10T20:39:23.471002+0000","last_unstale":"2024-09-10T20:39:23.473143+0000","last_undegraded":"2024-09-10T20:39:23.473143+0000","last_fullsized":"2024-09-10T20:39:23.473143+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:53.982272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471486+0000","last_change":"2024-09-10T20:39:23.471724+0000","last_active":"2024-09-10T20:39:24.471486+0000","last_peered":"2024-09-10T20:39:24.471486+0000","last_clean":"2024-09-10T20:39:24.471486+0000","last_became_active":"2024-09-10T20:39:23.471453+0000","last_became_peered":"2024-09-10T20:39:23.471453+0000","last_unstale":"2024-09-10T20:39:24.471486+0000","last_undegraded":"2024-09-10T20:39:24.471486+0000","last_fullsized":"2024-09-10T20:39:24.471486+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:52.838472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469972+0000","last_change":"2024-09-10T20:39:23.469522+0000","last_active":"2024-09-10T20:39:24.469972+0000","last_peered":"2024-09-10T20:39:24.469972+0000","last_clean":"2024-09-10T20:39:24.469972+0000","last_became_active":"2024-09-10T20:39:23.469129+0000","last_became_peered":"2024-09-10T20:39:23.469129+0000","last_unstale":"2024-09-10T20:39:24.469972+0000","last_undegraded":"2024-09-10T20:39:24.469972+0000","last_fullsized":"2024-09-10T20:39:24.469972+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:28.627939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471378+0000","last_change":"2024-09-10T20:39:23.470278+0000","last_active":"2024-09-10T20:39:24.471378+0000","last_peered":"2024-09-10T20:39:24.471378+0000","last_clean":"2024-09-10T20:39:24.471378+0000","last_became_active":"2024-09-10T20:39:23.469632+0000","last_became_peered":"2024-09-10T20:39:23.469632+0000","last_unstale":"2024-09-10T20:39:24.471378+0000","last_undegraded":"2024-09-10T20:39:24.471378+0000","last_fullsized":"2024-09-10T20:39:24.471378+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.476954+0000","last_change":"2024-09-10T20:39:23.476296+0000","last_active":"2024-09-10T20:39:23.476954+0000","last_peered":"2024-09-10T20:39:23.476954+0000","last_clean":"2024-09-10T20:39:23.476954+0000","last_became_active":"2024-09-10T20:39:23.476009+0000","last_became_peered":"2024-09-10T20:39:23.476009+0000","last_unstale":"2024-09-10T20:39:23.476954+0000","last_undegraded":"2024-09-10T20:39:23.476954+0000","last_fullsized":"2024-09-10T20:39:23.476954+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:29:17.845671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":19,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470688+0000","last_change":"2024-09-10T20:39:22.471060+0000","last_active":"2024-09-10T20:39:24.470688+0000","last_peered":"2024-09-10T20:39:24.470688+0000","last_clean":"2024-09-10T20:39:24.470688+0000","last_became_active":"2024-09-10T20:39:22.470753+0000","last_became_peered":"2024-09-10T20:39:22.470753+0000","last_unstale":"2024-09-10T20:39:24.470688+0000","last_undegraded":"2024-09-10T20:39:24.470688+0000","last_fullsized":"2024-09-10T20:39:24.470688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:22.626422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470292+0000","last_change":"2024-09-10T20:39:22.467354+0000","last_active":"2024-09-10T20:39:24.470292+0000","last_peered":"2024-09-10T20:39:24.470292+0000","last_clean":"2024-09-10T20:39:24.470292+0000","last_became_active":"2024-09-10T20:39:22.467105+0000","last_became_peered":"2024-09-10T20:39:22.467105+0000","last_unstale":"2024-09-10T20:39:24.470292+0000","last_undegraded":"2024-09-10T20:39:24.470292+0000","last_fullsized":"2024-09-10T20:39:24.470292+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:42.142331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471995+0000","last_change":"2024-09-10T20:39:23.465129+0000","last_active":"2024-09-10T20:39:24.471995+0000","last_peered":"2024-09-10T20:39:24.471995+0000","last_clean":"2024-09-10T20:39:24.471995+0000","last_became_active":"2024-09-10T20:39:23.464816+0000","last_became_peered":"2024-09-10T20:39:23.464816+0000","last_unstale":"2024-09-10T20:39:24.471995+0000","last_undegraded":"2024-09-10T20:39:24.471995+0000","last_fullsized":"2024-09-10T20:39:24.471995+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:06.505948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470439+0000","last_change":"2024-09-10T20:39:22.468131+0000","last_active":"2024-09-10T20:39:24.470439+0000","last_peered":"2024-09-10T20:39:24.470439+0000","last_clean":"2024-09-10T20:39:24.470439+0000","last_became_active":"2024-09-10T20:39:22.467846+0000","last_became_peered":"2024-09-10T20:39:22.467846+0000","last_unstale":"2024-09-10T20:39:24.470439+0000","last_undegraded":"2024-09-10T20:39:24.470439+0000","last_fullsized":"2024-09-10T20:39:24.470439+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:40.914501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.480621+0000","last_change":"2024-09-10T20:39:23.477169+0000","last_active":"2024-09-10T20:39:23.480621+0000","last_peered":"2024-09-10T20:39:23.480621+0000","last_clean":"2024-09-10T20:39:23.480621+0000","last_became_active":"2024-09-10T20:39:23.476830+0000","last_became_peered":"2024-09-10T20:39:23.476830+0000","last_unstale":"2024-09-10T20:39:23.480621+0000","last_undegraded":"2024-09-10T20:39:23.480621+0000","last_fullsized":"2024-09-10T20:39:23.480621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:20:58.616528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475067+0000","last_change":"2024-09-10T20:39:23.466496+0000","last_active":"2024-09-10T20:39:23.475067+0000","last_peered":"2024-09-10T20:39:23.475067+0000","last_clean":"2024-09-10T20:39:23.475067+0000","last_became_active":"2024-09-10T20:39:23.466162+0000","last_became_peered":"2024-09-10T20:39:23.466162+0000","last_unstale":"2024-09-10T20:39:23.475067+0000","last_undegraded":"2024-09-10T20:39:23.475067+0000","last_fullsized":"2024-09-10T20:39:23.475067+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:36.107439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460912+0000","last_change":"2024-09-10T20:39:22.464013+0000","last_active":"2024-09-10T20:39:23.460912+0000","last_peered":"2024-09-10T20:39:23.460912+0000","last_clean":"2024-09-10T20:39:23.460912+0000","last_became_active":"2024-09-10T20:39:22.463745+0000","last_became_peered":"2024-09-10T20:39:22.463745+0000","last_unstale":"2024-09-10T20:39:23.460912+0000","last_undegraded":"2024-09-10T20:39:23.460912+0000","last_fullsized":"2024-09-10T20:39:23.460912+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:17.407343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460348+0000","last_change":"2024-09-10T20:39:22.470606+0000","last_active":"2024-09-10T20:39:23.460348+0000","last_peered":"2024-09-10T20:39:23.460348+0000","last_clean":"2024-09-10T20:39:23.460348+0000","last_became_active":"2024-09-10T20:39:22.470313+0000","last_became_peered":"2024-09-10T20:39:22.470313+0000","last_unstale":"2024-09-10T20:39:23.460348+0000","last_undegraded":"2024-09-10T20:39:23.460348+0000","last_fullsized":"2024-09-10T20:39:23.460348+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:13.562757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471621+0000","last_change":"2024-09-10T20:39:23.478464+0000","last_active":"2024-09-10T20:39:24.471621+0000","last_peered":"2024-09-10T20:39:24.471621+0000","last_clean":"2024-09-10T20:39:24.471621+0000","last_became_active":"2024-09-10T20:39:23.477764+0000","last_became_peered":"2024-09-10T20:39:23.477764+0000","last_unstale":"2024-09-10T20:39:24.471621+0000","last_undegraded":"2024-09-10T20:39:24.471621+0000","last_fullsized":"2024-09-10T20:39:24.471621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:07.988361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":19,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.487847+0000","last_change":"2024-09-10T20:39:22.457688+0000","last_active":"2024-09-10T20:39:23.487847+0000","last_peered":"2024-09-10T20:39:23.487847+0000","last_clean":"2024-09-10T20:39:23.487847+0000","last_became_active":"2024-09-10T20:39:22.457326+0000","last_became_peered":"2024-09-10T20:39:22.457326+0000","last_unstale":"2024-09-10T20:39:23.487847+0000","last_undegraded":"2024-09-10T20:39:23.487847+0000","last_fullsized":"2024-09-10T20:39:23.487847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:51.098445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473204+0000","last_change":"2024-09-10T20:39:22.463919+0000","last_active":"2024-09-10T20:39:23.473204+0000","last_peered":"2024-09-10T20:39:23.473204+0000","last_clean":"2024-09-10T20:39:23.473204+0000","last_became_active":"2024-09-10T20:39:22.463650+0000","last_became_peered":"2024-09-10T20:39:22.463650+0000","last_unstale":"2024-09-10T20:39:23.473204+0000","last_undegraded":"2024-09-10T20:39:23.473204+0000","last_fullsized":"2024-09-10T20:39:23.473204+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:07:05.483324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484089+0000","last_change":"2024-09-10T20:39:23.482909+0000","last_active":"2024-09-10T20:39:23.484089+0000","last_peered":"2024-09-10T20:39:23.484089+0000","last_clean":"2024-09-10T20:39:23.484089+0000","last_became_active":"2024-09-10T20:39:23.482551+0000","last_became_peered":"2024-09-10T20:39:23.482551+0000","last_unstale":"2024-09-10T20:39:23.484089+0000","last_undegraded":"2024-09-10T20:39:23.484089+0000","last_fullsized":"2024-09-10T20:39:23.484089+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:10.837228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.467882+0000","last_change":"2024-09-10T20:39:23.465467+0000","last_active":"2024-09-10T20:39:23.467882+0000","last_peered":"2024-09-10T20:39:23.467882+0000","last_clean":"2024-09-10T20:39:23.467882+0000","last_became_active":"2024-09-10T20:39:23.465165+0000","last_became_peered":"2024-09-10T20:39:23.465165+0000","last_unstale":"2024-09-10T20:39:23.467882+0000","last_undegraded":"2024-09-10T20:39:23.467882+0000","last_fullsized":"2024-09-10T20:39:23.467882+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:15.544019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469888+0000","last_change":"2024-09-10T20:39:22.471774+0000","last_active":"2024-09-10T20:39:24.469888+0000","last_peered":"2024-09-10T20:39:24.469888+0000","last_clean":"2024-09-10T20:39:24.469888+0000","last_became_active":"2024-09-10T20:39:22.471455+0000","last_became_peered":"2024-09-10T20:39:22.471455+0000","last_unstale":"2024-09-10T20:39:24.469888+0000","last_undegraded":"2024-09-10T20:39:24.469888+0000","last_fullsized":"2024-09-10T20:39:24.469888+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:31.755937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":21,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.485737+0000","last_change":"2024-09-10T20:39:22.468116+0000","last_active":"2024-09-10T20:39:23.485737+0000","last_peered":"2024-09-10T20:39:23.485737+0000","last_clean":"2024-09-10T20:39:23.485737+0000","last_became_active":"2024-09-10T20:39:22.467712+0000","last_became_peered":"2024-09-10T20:39:22.467712+0000","last_unstale":"2024-09-10T20:39:23.485737+0000","last_undegraded":"2024-09-10T20:39:23.485737+0000","last_fullsized":"2024-09-10T20:39:23.485737+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:18:01.811366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.474598+0000","last_change":"2024-09-10T20:39:23.473728+0000","last_active":"2024-09-10T20:39:23.474598+0000","last_peered":"2024-09-10T20:39:23.474598+0000","last_clean":"2024-09-10T20:39:23.474598+0000","last_became_active":"2024-09-10T20:39:23.473482+0000","last_became_peered":"2024-09-10T20:39:23.473482+0000","last_unstale":"2024-09-10T20:39:23.474598+0000","last_undegraded":"2024-09-10T20:39:23.474598+0000","last_fullsized":"2024-09-10T20:39:23.474598+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:42:24.635057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471265+0000","last_change":"2024-09-10T20:39:22.464648+0000","last_active":"2024-09-10T20:39:24.471265+0000","last_peered":"2024-09-10T20:39:24.471265+0000","last_clean":"2024-09-10T20:39:24.471265+0000","last_became_active":"2024-09-10T20:39:22.464381+0000","last_became_peered":"2024-09-10T20:39:22.464381+0000","last_unstale":"2024-09-10T20:39:24.471265+0000","last_undegraded":"2024-09-10T20:39:24.471265+0000","last_fullsized":"2024-09-10T20:39:24.471265+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:34.596334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.469081+0000","last_change":"2024-09-10T20:39:23.466893+0000","last_active":"2024-09-10T20:39:23.469081+0000","last_peered":"2024-09-10T20:39:23.469081+0000","last_clean":"2024-09-10T20:39:23.469081+0000","last_became_active":"2024-09-10T20:39:23.466577+0000","last_became_peered":"2024-09-10T20:39:23.466577+0000","last_unstale":"2024-09-10T20:39:23.469081+0000","last_undegraded":"2024-09-10T20:39:23.469081+0000","last_fullsized":"2024-09-10T20:39:23.469081+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:50.331650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470125+0000","last_change":"2024-09-10T20:39:22.471454+0000","last_active":"2024-09-10T20:39:24.470125+0000","last_peered":"2024-09-10T20:39:24.470125+0000","last_clean":"2024-09-10T20:39:24.470125+0000","last_became_active":"2024-09-10T20:39:22.471186+0000","last_became_peered":"2024-09-10T20:39:22.471186+0000","last_unstale":"2024-09-10T20:39:24.470125+0000","last_undegraded":"2024-09-10T20:39:24.470125+0000","last_fullsized":"2024-09-10T20:39:24.470125+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:11.967787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479095+0000","last_change":"2024-09-10T20:39:23.474293+0000","last_active":"2024-09-10T20:39:23.479095+0000","last_peered":"2024-09-10T20:39:23.479095+0000","last_clean":"2024-09-10T20:39:23.479095+0000","last_became_active":"2024-09-10T20:39:23.474013+0000","last_became_peered":"2024-09-10T20:39:23.474013+0000","last_unstale":"2024-09-10T20:39:23.479095+0000","last_undegraded":"2024-09-10T20:39:23.479095+0000","last_fullsized":"2024-09-10T20:39:23.479095+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:39:00.145526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484950+0000","last_change":"2024-09-10T20:39:22.467010+0000","last_active":"2024-09-10T20:39:23.484950+0000","last_peered":"2024-09-10T20:39:23.484950+0000","last_clean":"2024-09-10T20:39:23.484950+0000","last_became_active":"2024-09-10T20:39:22.466604+0000","last_became_peered":"2024-09-10T20:39:22.466604+0000","last_unstale":"2024-09-10T20:39:23.484950+0000","last_undegraded":"2024-09-10T20:39:23.484950+0000","last_fullsized":"2024-09-10T20:39:23.484950+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:33:38.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470167+0000","last_change":"2024-09-10T20:39:23.483419+0000","last_active":"2024-09-10T20:39:24.470167+0000","last_peered":"2024-09-10T20:39:24.470167+0000","last_clean":"2024-09-10T20:39:24.470167+0000","last_became_active":"2024-09-10T20:39:23.482573+0000","last_became_peered":"2024-09-10T20:39:23.482573+0000","last_unstale":"2024-09-10T20:39:24.470167+0000","last_undegraded":"2024-09-10T20:39:24.470167+0000","last_fullsized":"2024-09-10T20:39:24.470167+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471576+0000","last_change":"2024-09-10T20:39:23.470437+0000","last_active":"2024-09-10T20:39:24.471576+0000","last_peered":"2024-09-10T20:39:24.471576+0000","last_clean":"2024-09-10T20:39:24.471576+0000","last_became_active":"2024-09-10T20:39:23.470018+0000","last_became_peered":"2024-09-10T20:39:23.470018+0000","last_unstale":"2024-09-10T20:39:24.471576+0000","last_undegraded":"2024-09-10T20:39:24.471576+0000","last_fullsized":"2024-09-10T20:39:24.471576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:19.864809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.462467+0000","last_change":"2024-09-10T20:39:22.470294+0000","last_active":"2024-09-10T20:39:23.462467+0000","last_peered":"2024-09-10T20:39:23.462467+0000","last_clean":"2024-09-10T20:39:23.462467+0000","last_became_active":"2024-09-10T20:39:22.469974+0000","last_became_peered":"2024-09-10T20:39:22.469974+0000","last_unstale":"2024-09-10T20:39:23.462467+0000","last_undegraded":"2024-09-10T20:39:23.462467+0000","last_fullsized":"2024-09-10T20:39:23.462467+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:14.321360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471788+0000","last_change":"2024-09-10T20:39:23.467237+0000","last_active":"2024-09-10T20:39:24.471788+0000","last_peered":"2024-09-10T20:39:24.471788+0000","last_clean":"2024-09-10T20:39:24.471788+0000","last_became_active":"2024-09-10T20:39:23.466955+0000","last_became_peered":"2024-09-10T20:39:23.466955+0000","last_unstale":"2024-09-10T20:39:24.471788+0000","last_undegraded":"2024-09-10T20:39:24.471788+0000","last_fullsized":"2024-09-10T20:39:24.471788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:17:53.617323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471588+0000","last_change":"2024-09-10T20:39:22.466270+0000","last_active":"2024-09-10T20:39:24.471588+0000","last_peered":"2024-09-10T20:39:24.471588+0000","last_clean":"2024-09-10T20:39:24.471588+0000","last_became_active":"2024-09-10T20:39:22.465951+0000","last_became_peered":"2024-09-10T20:39:22.465951+0000","last_unstale":"2024-09-10T20:39:24.471588+0000","last_undegraded":"2024-09-10T20:39:24.471588+0000","last_fullsized":"2024-09-10T20:39:24.471588+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475360+0000","last_change":"2024-09-10T20:39:23.468716+0000","last_active":"2024-09-10T20:39:23.475360+0000","last_peered":"2024-09-10T20:39:23.475360+0000","last_clean":"2024-09-10T20:39:23.475360+0000","last_became_active":"2024-09-10T20:39:23.468450+0000","last_became_peered":"2024-09-10T20:39:23.468450+0000","last_unstale":"2024-09-10T20:39:23.475360+0000","last_undegraded":"2024-09-10T20:39:23.475360+0000","last_fullsized":"2024-09-10T20:39:23.475360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:00.101321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460877+0000","last_change":"2024-09-10T20:39:22.467181+0000","last_active":"2024-09-10T20:39:23.460877+0000","last_peered":"2024-09-10T20:39:23.460877+0000","last_clean":"2024-09-10T20:39:23.460877+0000","last_became_active":"2024-09-10T20:39:22.466859+0000","last_became_peered":"2024-09-10T20:39:22.466859+0000","last_unstale":"2024-09-10T20:39:23.460877+0000","last_undegraded":"2024-09-10T20:39:23.460877+0000","last_fullsized":"2024-09-10T20:39:23.460877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:54.024056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479036+0000","last_change":"2024-09-10T20:39:23.473568+0000","last_active":"2024-09-10T20:39:23.479036+0000","last_peered":"2024-09-10T20:39:23.479036+0000","last_clean":"2024-09-10T20:39:23.479036+0000","last_became_active":"2024-09-10T20:39:23.473257+0000","last_became_peered":"2024-09-10T20:39:23.473257+0000","last_unstale":"2024-09-10T20:39:23.479036+0000","last_undegraded":"2024-09-10T20:39:23.479036+0000","last_fullsized":"2024-09-10T20:39:23.479036+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:02.160411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471424+0000","last_change":"2024-09-10T20:39:23.482739+0000","last_active":"2024-09-10T20:39:24.471424+0000","last_peered":"2024-09-10T20:39:24.471424+0000","last_clean":"2024-09-10T20:39:24.471424+0000","last_became_active":"2024-09-10T20:39:23.482483+0000","last_became_peered":"2024-09-10T20:39:23.482483+0000","last_unstale":"2024-09-10T20:39:24.471424+0000","last_undegraded":"2024-09-10T20:39:24.471424+0000","last_fullsized":"2024-09-10T20:39:24.471424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:53.453242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469982+0000","last_change":"2024-09-10T20:39:22.464748+0000","last_active":"2024-09-10T20:39:24.469982+0000","last_peered":"2024-09-10T20:39:24.469982+0000","last_clean":"2024-09-10T20:39:24.469982+0000","last_became_active":"2024-09-10T20:39:22.464248+0000","last_became_peered":"2024-09-10T20:39:22.464248+0000","last_unstale":"2024-09-10T20:39:24.469982+0000","last_undegraded":"2024-09-10T20:39:24.469982+0000","last_fullsized":"2024-09-10T20:39:24.469982+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:34:33.491750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:22.441589+0000","last_change":"2024-09-10T20:39:22.441589+0000","last_active":"2024-09-10T20:39:22.441589+0000","last_peered":"2024-09-10T20:39:22.441589+0000","last_clean":"2024-09-10T20:39:22.441589+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:22.441589+0000","last_undegraded":"2024-09-10T20:39:22.441589+0000","last_fullsized":"2024-09-10T20:39:22.441589+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.459867+0000","last_change":"2024-09-10T20:39:22.465000+0000","last_active":"2024-09-10T20:39:23.459867+0000","last_peered":"2024-09-10T20:39:23.459867+0000","last_clean":"2024-09-10T20:39:23.459867+0000","last_became_active":"2024-09-10T20:39:22.464669+0000","last_became_peered":"2024-09-10T20:39:22.464669+0000","last_unstale":"2024-09-10T20:39:23.459867+0000","last_undegraded":"2024-09-10T20:39:23.459867+0000","last_fullsized":"2024-09-10T20:39:23.459867+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:43:06.255942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471722+0000","last_change":"2024-09-10T20:39:23.476662+0000","last_active":"2024-09-10T20:39:24.471722+0000","last_peered":"2024-09-10T20:39:24.471722+0000","last_clean":"2024-09-10T20:39:24.471722+0000","last_became_active":"2024-09-10T20:39:23.476375+0000","last_became_peered":"2024-09-10T20:39:23.476375+0000","last_unstale":"2024-09-10T20:39:24.471722+0000","last_undegraded":"2024-09-10T20:39:24.471722+0000","last_fullsized":"2024-09-10T20:39:24.471722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:46.142798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:39:21.439516+0000","last_change":"2024-09-10T20:39:21.439516+0000","last_active":"2024-09-10T20:39:21.439516+0000","last_peered":"2024-09-10T20:39:21.439516+0000","last_clean":"2024-09-10T20:39:21.439516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:39:21.439516+0000","last_undegraded":"2024-09-10T20:39:21.439516+0000","last_fullsized":"2024-09-10T20:39:21.439516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":141,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T20:39:20.047668+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:39:20.047668+0000","last_peered":"2024-09-10T20:39:20.047668+0000","last_clean":"2024-09-10T20:39:20.047668+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:39:20.047668+0000","last_undegraded":"2024-09-10T20:39:20.047668+0000","last_fullsized":"2024-09-10T20:39:20.047668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2250,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":34,"acting":34,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509460,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31632,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":446464,"data_stored":173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509459,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32208,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339632,"statfs":{"total":96636764160,"available":96603783168,"internally_reserved":0,"allocated":1040384,"data_stored":764128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":491520,"data_stored":193243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1093632,"data_stored":789317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":503808,"data_stored":194331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:39:27.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 -- 172.21.15.28:0/1662223940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 msgr2=0x7f82800d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 --2- 172.21.15.28:0/1662223940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82800d2960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f82800d0f70 tx=0x7f827402a540 comp rx=0 tx=0).stop 2024-09-10T20:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 -- 172.21.15.28:0/1662223940 shutdown_connections 2024-09-10T20:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 --2- 172.21.15.28:0/1662223940 >> v2:172.21.15.159:3300/0 conn(0x7f82800b5610 0x7f82800b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 --2- 172.21.15.28:0/1662223940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82800d2960 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 -- 172.21.15.28:0/1662223940 >> 172.21.15.28:0/1662223940 conn(0x7f82800d1e40 msgr2=0x7f82800d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 -- 172.21.15.28:0/1662223940 shutdown_connections 2024-09-10T20:39:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.020+0000 7f8286190640 1 -- 172.21.15.28:0/1662223940 wait complete. 2024-09-10T20:39:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.021+0000 7f8286190640 1 Processor -- start 2024-09-10T20:39:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.021+0000 7f8286190640 1 -- start start 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f8286190640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82800b5610 0x7f82801f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f8286190640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82801f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f8286190640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f828015bf30 0x7f8280159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f8286190640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82800b5fa0 con 0x7f82800d05d0 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f8286190640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f82800d0d50 con 0x7f828015bf30 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f8286190640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f82800b6120 con 0x7f82800b5610 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f827effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82801f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f827effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82801f1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49732/0 (socket says 172.21.15.28:49732) 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f827effd640 1 -- 172.21.15.28:0/4125591818 learned_addr learned my addr 172.21.15.28:0/4125591818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f827ffff640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f828015bf30 0x7f8280159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f827effd640 1 -- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82800b5610 msgr2=0x7f82801f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f827effd640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82800b5610 0x7f82801f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.022+0000 7f827effd640 1 -- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f828015bf30 msgr2=0x7f8280159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f827effd640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f828015bf30 0x7f8280159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f827effd640 1 -- 172.21.15.28:0/4125591818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8280159570 con 0x7f82800d05d0 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f827effd640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82801f1f60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f827402ab30 tx=0x7f8274032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f8267fff640 1 -- 172.21.15.28:0/4125591818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8274040070 con 0x7f82800d05d0 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f8267fff640 1 -- 172.21.15.28:0/4125591818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8274005310 con 0x7f82800d05d0 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82800c8fb0 con 0x7f82800d05d0 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f8267fff640 1 -- 172.21.15.28:0/4125591818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8274005c90 con 0x7f82800d05d0 2024-09-10T20:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.023+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82800c9530 con 0x7f82800d05d0 2024-09-10T20:39:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.024+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82800b65d0 con 0x7f82800d05d0 2024-09-10T20:39:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.025+0000 7f8267fff640 1 -- 172.21.15.28:0/4125591818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f827404c030 con 0x7f82800d05d0 2024-09-10T20:39:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.025+0000 7f8267fff640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f826c078150 0x7f826c07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.025+0000 7f8267fff640 1 -- 172.21.15.28:0/4125591818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f82740bb3e0 con 0x7f82800d05d0 2024-09-10T20:39:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.025+0000 7f827f7fe640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f826c078150 0x7f826c07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.026+0000 7f827f7fe640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f826c078150 0x7f826c07a610 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8270007fd0 tx=0x7f8270007900 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.029+0000 7f8267fff640 1 -- 172.21.15.28:0/4125591818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8274083be0 con 0x7f82800d05d0 2024-09-10T20:39:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.211+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82800b32d0 con 0x7f826c078150 2024-09-10T20:39:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.215+0000 7f8267fff640 1 -- 172.21.15.28:0/4125591818 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f82800b32d0 con 0x7f826c078150 2024-09-10T20:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:39:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f826c078150 msgr2=0x7f826c07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f826c078150 0x7f826c07a610 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8270007fd0 tx=0x7f8270007900 comp rx=0 tx=0).stop 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 msgr2=0x7f82801f1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82801f1f60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f827402ab30 tx=0x7f8274032c10 comp rx=0 tx=0).stop 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 shutdown_connections 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f826c078150 0x7f826c07a610 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f828015bf30 0x7f8280159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800d05d0 0x7f82801f1f60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 --2- 172.21.15.28:0/4125591818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82800b5610 0x7f82801f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 >> 172.21.15.28:0/4125591818 conn(0x7f82800d1e40 msgr2=0x7f82800c8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 shutdown_connections 2024-09-10T20:39:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.218+0000 7f8286190640 1 -- 172.21.15.28:0/4125591818 wait complete. 2024-09-10T20:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-10T20:39:27.991829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237944,"kb_used_data":5060,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736776,"statfs":{"total":773094113280,"available":772850458624,"internally_reserved":0,"allocated":5181440,"data_stored":2758072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001436"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814513+0000","last_change":"2024-09-10T20:39:23.483860+0000","last_active":"2024-09-10T20:39:24.814513+0000","last_peered":"2024-09-10T20:39:24.814513+0000","last_clean":"2024-09-10T20:39:24.814513+0000","last_became_active":"2024-09-10T20:39:23.483590+0000","last_became_peered":"2024-09-10T20:39:23.483590+0000","last_unstale":"2024-09-10T20:39:24.814513+0000","last_undegraded":"2024-09-10T20:39:24.814513+0000","last_fullsized":"2024-09-10T20:39:24.814513+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251574+0000","last_change":"2024-09-10T20:39:23.476103+0000","last_active":"2024-09-10T20:39:25.251574+0000","last_peered":"2024-09-10T20:39:25.251574+0000","last_clean":"2024-09-10T20:39:25.251574+0000","last_became_active":"2024-09-10T20:39:23.475659+0000","last_became_peered":"2024-09-10T20:39:23.475659+0000","last_unstale":"2024-09-10T20:39:25.251574+0000","last_undegraded":"2024-09-10T20:39:25.251574+0000","last_fullsized":"2024-09-10T20:39:25.251574+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:01.324112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469676+0000","last_change":"2024-09-10T20:39:23.479367+0000","last_active":"2024-09-10T20:39:24.469676+0000","last_peered":"2024-09-10T20:39:24.469676+0000","last_clean":"2024-09-10T20:39:24.469676+0000","last_became_active":"2024-09-10T20:39:23.479089+0000","last_became_peered":"2024-09-10T20:39:23.479089+0000","last_unstale":"2024-09-10T20:39:24.469676+0000","last_undegraded":"2024-09-10T20:39:24.469676+0000","last_fullsized":"2024-09-10T20:39:24.469676+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:43:14.607190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471413+0000","last_change":"2024-09-10T20:39:23.474482+0000","last_active":"2024-09-10T20:39:24.471413+0000","last_peered":"2024-09-10T20:39:24.471413+0000","last_clean":"2024-09-10T20:39:24.471413+0000","last_became_active":"2024-09-10T20:39:23.469443+0000","last_became_peered":"2024-09-10T20:39:23.469443+0000","last_unstale":"2024-09-10T20:39:24.471413+0000","last_undegraded":"2024-09-10T20:39:24.471413+0000","last_fullsized":"2024-09-10T20:39:24.471413+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814280+0000","last_change":"2024-09-10T20:39:23.469934+0000","last_active":"2024-09-10T20:39:24.814280+0000","last_peered":"2024-09-10T20:39:24.814280+0000","last_clean":"2024-09-10T20:39:24.814280+0000","last_became_active":"2024-09-10T20:39:23.469620+0000","last_became_peered":"2024-09-10T20:39:23.469620+0000","last_unstale":"2024-09-10T20:39:24.814280+0000","last_undegraded":"2024-09-10T20:39:24.814280+0000","last_fullsized":"2024-09-10T20:39:24.814280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:43.501230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470522+0000","last_change":"2024-09-10T20:39:23.478562+0000","last_active":"2024-09-10T20:39:24.470522+0000","last_peered":"2024-09-10T20:39:24.470522+0000","last_clean":"2024-09-10T20:39:24.470522+0000","last_became_active":"2024-09-10T20:39:23.478273+0000","last_became_peered":"2024-09-10T20:39:23.478273+0000","last_unstale":"2024-09-10T20:39:24.470522+0000","last_undegraded":"2024-09-10T20:39:24.470522+0000","last_fullsized":"2024-09-10T20:39:24.470522+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:06:13.043952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251327+0000","last_change":"2024-09-10T20:39:23.475249+0000","last_active":"2024-09-10T20:39:25.251327+0000","last_peered":"2024-09-10T20:39:25.251327+0000","last_clean":"2024-09-10T20:39:25.251327+0000","last_became_active":"2024-09-10T20:39:23.474917+0000","last_became_peered":"2024-09-10T20:39:23.474917+0000","last_unstale":"2024-09-10T20:39:25.251327+0000","last_undegraded":"2024-09-10T20:39:25.251327+0000","last_fullsized":"2024-09-10T20:39:25.251327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:32.157058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471730+0000","last_change":"2024-09-10T20:39:23.477481+0000","last_active":"2024-09-10T20:39:24.471730+0000","last_peered":"2024-09-10T20:39:24.471730+0000","last_clean":"2024-09-10T20:39:24.471730+0000","last_became_active":"2024-09-10T20:39:23.477194+0000","last_became_peered":"2024-09-10T20:39:23.477194+0000","last_unstale":"2024-09-10T20:39:24.471730+0000","last_undegraded":"2024-09-10T20:39:24.471730+0000","last_fullsized":"2024-09-10T20:39:24.471730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:16.051559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470973+0000","last_change":"2024-09-10T20:39:23.465809+0000","last_active":"2024-09-10T20:39:24.470973+0000","last_peered":"2024-09-10T20:39:24.470973+0000","last_clean":"2024-09-10T20:39:24.470973+0000","last_became_active":"2024-09-10T20:39:23.465540+0000","last_became_peered":"2024-09-10T20:39:23.465540+0000","last_unstale":"2024-09-10T20:39:24.470973+0000","last_undegraded":"2024-09-10T20:39:24.470973+0000","last_fullsized":"2024-09-10T20:39:24.470973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.481077+0000","last_change":"2024-09-10T20:39:23.480200+0000","last_active":"2024-09-10T20:39:23.481077+0000","last_peered":"2024-09-10T20:39:23.481077+0000","last_clean":"2024-09-10T20:39:23.481077+0000","last_became_active":"2024-09-10T20:39:23.479928+0000","last_became_peered":"2024-09-10T20:39:23.479928+0000","last_unstale":"2024-09-10T20:39:23.481077+0000","last_undegraded":"2024-09-10T20:39:23.481077+0000","last_fullsized":"2024-09-10T20:39:23.481077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:20.100800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471901+0000","last_change":"2024-09-10T20:39:23.472464+0000","last_active":"2024-09-10T20:39:24.471901+0000","last_peered":"2024-09-10T20:39:24.471901+0000","last_clean":"2024-09-10T20:39:24.471901+0000","last_became_active":"2024-09-10T20:39:23.472185+0000","last_became_peered":"2024-09-10T20:39:23.472185+0000","last_unstale":"2024-09-10T20:39:24.471901+0000","last_undegraded":"2024-09-10T20:39:24.471901+0000","last_fullsized":"2024-09-10T20:39:24.471901+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:16:50.093970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251720+0000","last_change":"2024-09-10T20:39:23.481383+0000","last_active":"2024-09-10T20:39:25.251720+0000","last_peered":"2024-09-10T20:39:25.251720+0000","last_clean":"2024-09-10T20:39:25.251720+0000","last_became_active":"2024-09-10T20:39:23.480519+0000","last_became_peered":"2024-09-10T20:39:23.480519+0000","last_unstale":"2024-09-10T20:39:25.251720+0000","last_undegraded":"2024-09-10T20:39:25.251720+0000","last_fullsized":"2024-09-10T20:39:25.251720+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:06:51.852363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470189+0000","last_change":"2024-09-10T20:39:23.479455+0000","last_active":"2024-09-10T20:39:24.470189+0000","last_peered":"2024-09-10T20:39:24.470189+0000","last_clean":"2024-09-10T20:39:24.470189+0000","last_became_active":"2024-09-10T20:39:23.479185+0000","last_became_peered":"2024-09-10T20:39:23.479185+0000","last_unstale":"2024-09-10T20:39:24.470189+0000","last_undegraded":"2024-09-10T20:39:24.470189+0000","last_fullsized":"2024-09-10T20:39:24.470189+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471366+0000","last_change":"2024-09-10T20:39:23.472528+0000","last_active":"2024-09-10T20:39:24.471366+0000","last_peered":"2024-09-10T20:39:24.471366+0000","last_clean":"2024-09-10T20:39:24.471366+0000","last_became_active":"2024-09-10T20:39:23.472247+0000","last_became_peered":"2024-09-10T20:39:23.472247+0000","last_unstale":"2024-09-10T20:39:24.471366+0000","last_undegraded":"2024-09-10T20:39:24.471366+0000","last_fullsized":"2024-09-10T20:39:24.471366+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471460+0000","last_change":"2024-09-10T20:39:23.477260+0000","last_active":"2024-09-10T20:39:24.471460+0000","last_peered":"2024-09-10T20:39:24.471460+0000","last_clean":"2024-09-10T20:39:24.471460+0000","last_became_active":"2024-09-10T20:39:23.476962+0000","last_became_peered":"2024-09-10T20:39:23.476962+0000","last_unstale":"2024-09-10T20:39:24.471460+0000","last_undegraded":"2024-09-10T20:39:24.471460+0000","last_fullsized":"2024-09-10T20:39:24.471460+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814151+0000","last_change":"2024-09-10T20:39:23.482868+0000","last_active":"2024-09-10T20:39:24.814151+0000","last_peered":"2024-09-10T20:39:24.814151+0000","last_clean":"2024-09-10T20:39:24.814151+0000","last_became_active":"2024-09-10T20:39:23.482352+0000","last_became_peered":"2024-09-10T20:39:23.482352+0000","last_unstale":"2024-09-10T20:39:24.814151+0000","last_undegraded":"2024-09-10T20:39:24.814151+0000","last_fullsized":"2024-09-10T20:39:24.814151+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:10.014292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251846+0000","last_change":"2024-09-10T20:39:23.481835+0000","last_active":"2024-09-10T20:39:25.251846+0000","last_peered":"2024-09-10T20:39:25.251846+0000","last_clean":"2024-09-10T20:39:25.251846+0000","last_became_active":"2024-09-10T20:39:23.481462+0000","last_became_peered":"2024-09-10T20:39:23.481462+0000","last_unstale":"2024-09-10T20:39:25.251846+0000","last_undegraded":"2024-09-10T20:39:25.251846+0000","last_fullsized":"2024-09-10T20:39:25.251846+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:37:09.072417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469826+0000","last_change":"2024-09-10T20:39:23.478525+0000","last_active":"2024-09-10T20:39:24.469826+0000","last_peered":"2024-09-10T20:39:24.469826+0000","last_clean":"2024-09-10T20:39:24.469826+0000","last_became_active":"2024-09-10T20:39:23.478217+0000","last_became_peered":"2024-09-10T20:39:23.478217+0000","last_unstale":"2024-09-10T20:39:24.469826+0000","last_undegraded":"2024-09-10T20:39:24.469826+0000","last_fullsized":"2024-09-10T20:39:24.469826+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:28:10.445472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814465+0000","last_change":"2024-09-10T20:39:23.471412+0000","last_active":"2024-09-10T20:39:24.814465+0000","last_peered":"2024-09-10T20:39:24.814465+0000","last_clean":"2024-09-10T20:39:24.814465+0000","last_became_active":"2024-09-10T20:39:23.471147+0000","last_became_peered":"2024-09-10T20:39:23.471147+0000","last_unstale":"2024-09-10T20:39:24.814465+0000","last_undegraded":"2024-09-10T20:39:24.814465+0000","last_fullsized":"2024-09-10T20:39:24.814465+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475335+0000","last_change":"2024-09-10T20:39:23.467260+0000","last_active":"2024-09-10T20:39:23.475335+0000","last_peered":"2024-09-10T20:39:23.475335+0000","last_clean":"2024-09-10T20:39:23.475335+0000","last_became_active":"2024-09-10T20:39:23.465867+0000","last_became_peered":"2024-09-10T20:39:23.465867+0000","last_unstale":"2024-09-10T20:39:23.475335+0000","last_undegraded":"2024-09-10T20:39:23.475335+0000","last_fullsized":"2024-09-10T20:39:23.475335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:51.591296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471738+0000","last_change":"2024-09-10T20:39:23.479313+0000","last_active":"2024-09-10T20:39:24.471738+0000","last_peered":"2024-09-10T20:39:24.471738+0000","last_clean":"2024-09-10T20:39:24.471738+0000","last_became_active":"2024-09-10T20:39:23.478782+0000","last_became_peered":"2024-09-10T20:39:23.478782+0000","last_unstale":"2024-09-10T20:39:24.471738+0000","last_undegraded":"2024-09-10T20:39:24.471738+0000","last_fullsized":"2024-09-10T20:39:24.471738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:32:50.720508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473143+0000","last_change":"2024-09-10T20:39:23.471294+0000","last_active":"2024-09-10T20:39:23.473143+0000","last_peered":"2024-09-10T20:39:23.473143+0000","last_clean":"2024-09-10T20:39:23.473143+0000","last_became_active":"2024-09-10T20:39:23.471002+0000","last_became_peered":"2024-09-10T20:39:23.471002+0000","last_unstale":"2024-09-10T20:39:23.473143+0000","last_undegraded":"2024-09-10T20:39:23.473143+0000","last_fullsized":"2024-09-10T20:39:23.473143+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:53.982272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252253+0000","last_change":"2024-09-10T20:39:23.472791+0000","last_active":"2024-09-10T20:39:25.252253+0000","last_peered":"2024-09-10T20:39:25.252253+0000","last_clean":"2024-09-10T20:39:25.252253+0000","last_became_active":"2024-09-10T20:39:23.472469+0000","last_became_peered":"2024-09-10T20:39:23.472469+0000","last_unstale":"2024-09-10T20:39:25.252253+0000","last_undegraded":"2024-09-10T20:39:25.252253+0000","last_fullsized":"2024-09-10T20:39:25.252253+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:37.756616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251230+0000","last_change":"2024-09-10T20:39:23.483420+0000","last_active":"2024-09-10T20:39:25.251230+0000","last_peered":"2024-09-10T20:39:25.251230+0000","last_clean":"2024-09-10T20:39:25.251230+0000","last_became_active":"2024-09-10T20:39:23.483170+0000","last_became_peered":"2024-09-10T20:39:23.483170+0000","last_unstale":"2024-09-10T20:39:25.251230+0000","last_undegraded":"2024-09-10T20:39:25.251230+0000","last_fullsized":"2024-09-10T20:39:25.251230+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814592+0000","last_change":"2024-09-10T20:39:23.483079+0000","last_active":"2024-09-10T20:39:24.814592+0000","last_peered":"2024-09-10T20:39:24.814592+0000","last_clean":"2024-09-10T20:39:24.814592+0000","last_became_active":"2024-09-10T20:39:23.482769+0000","last_became_peered":"2024-09-10T20:39:23.482769+0000","last_unstale":"2024-09-10T20:39:24.814592+0000","last_undegraded":"2024-09-10T20:39:24.814592+0000","last_fullsized":"2024-09-10T20:39:24.814592+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471486+0000","last_change":"2024-09-10T20:39:23.471724+0000","last_active":"2024-09-10T20:39:24.471486+0000","last_peered":"2024-09-10T20:39:24.471486+0000","last_clean":"2024-09-10T20:39:24.471486+0000","last_became_active":"2024-09-10T20:39:23.471453+0000","last_became_peered":"2024-09-10T20:39:23.471453+0000","last_unstale":"2024-09-10T20:39:24.471486+0000","last_undegraded":"2024-09-10T20:39:24.471486+0000","last_fullsized":"2024-09-10T20:39:24.471486+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:52.838472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252396+0000","last_change":"2024-09-10T20:39:23.474509+0000","last_active":"2024-09-10T20:39:25.252396+0000","last_peered":"2024-09-10T20:39:25.252396+0000","last_clean":"2024-09-10T20:39:25.252396+0000","last_became_active":"2024-09-10T20:39:23.474180+0000","last_became_peered":"2024-09-10T20:39:23.474180+0000","last_unstale":"2024-09-10T20:39:25.252396+0000","last_undegraded":"2024-09-10T20:39:25.252396+0000","last_fullsized":"2024-09-10T20:39:25.252396+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:19.254485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252368+0000","last_change":"2024-09-10T20:39:23.482090+0000","last_active":"2024-09-10T20:39:25.252368+0000","last_peered":"2024-09-10T20:39:25.252368+0000","last_clean":"2024-09-10T20:39:25.252368+0000","last_became_active":"2024-09-10T20:39:23.481803+0000","last_became_peered":"2024-09-10T20:39:23.481803+0000","last_unstale":"2024-09-10T20:39:25.252368+0000","last_undegraded":"2024-09-10T20:39:25.252368+0000","last_fullsized":"2024-09-10T20:39:25.252368+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:52.741446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814356+0000","last_change":"2024-09-10T20:39:23.473090+0000","last_active":"2024-09-10T20:39:24.814356+0000","last_peered":"2024-09-10T20:39:24.814356+0000","last_clean":"2024-09-10T20:39:24.814356+0000","last_became_active":"2024-09-10T20:39:23.472803+0000","last_became_peered":"2024-09-10T20:39:23.472803+0000","last_unstale":"2024-09-10T20:39:24.814356+0000","last_undegraded":"2024-09-10T20:39:24.814356+0000","last_fullsized":"2024-09-10T20:39:24.814356+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:22.229236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251535+0000","last_change":"2024-09-10T20:39:23.479620+0000","last_active":"2024-09-10T20:39:25.251535+0000","last_peered":"2024-09-10T20:39:25.251535+0000","last_clean":"2024-09-10T20:39:25.251535+0000","last_became_active":"2024-09-10T20:39:23.479319+0000","last_became_peered":"2024-09-10T20:39:23.479319+0000","last_unstale":"2024-09-10T20:39:25.251535+0000","last_undegraded":"2024-09-10T20:39:25.251535+0000","last_fullsized":"2024-09-10T20:39:25.251535+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469972+0000","last_change":"2024-09-10T20:39:23.469522+0000","last_active":"2024-09-10T20:39:24.469972+0000","last_peered":"2024-09-10T20:39:24.469972+0000","last_clean":"2024-09-10T20:39:24.469972+0000","last_became_active":"2024-09-10T20:39:23.469129+0000","last_became_peered":"2024-09-10T20:39:23.469129+0000","last_unstale":"2024-09-10T20:39:24.469972+0000","last_undegraded":"2024-09-10T20:39:24.469972+0000","last_fullsized":"2024-09-10T20:39:24.469972+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:28.627939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471378+0000","last_change":"2024-09-10T20:39:23.470278+0000","last_active":"2024-09-10T20:39:24.471378+0000","last_peered":"2024-09-10T20:39:24.471378+0000","last_clean":"2024-09-10T20:39:24.471378+0000","last_became_active":"2024-09-10T20:39:23.469632+0000","last_became_peered":"2024-09-10T20:39:23.469632+0000","last_unstale":"2024-09-10T20:39:24.471378+0000","last_undegraded":"2024-09-10T20:39:24.471378+0000","last_fullsized":"2024-09-10T20:39:24.471378+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.476954+0000","last_change":"2024-09-10T20:39:23.476296+0000","last_active":"2024-09-10T20:39:23.476954+0000","last_peered":"2024-09-10T20:39:23.476954+0000","last_clean":"2024-09-10T20:39:23.476954+0000","last_became_active":"2024-09-10T20:39:23.476009+0000","last_became_peered":"2024-09-10T20:39:23.476009+0000","last_unstale":"2024-09-10T20:39:23.476954+0000","last_undegraded":"2024-09-10T20:39:23.476954+0000","last_fullsized":"2024-09-10T20:39:23.476954+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:29:17.845671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":19,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470688+0000","last_change":"2024-09-10T20:39:22.471060+0000","last_active":"2024-09-10T20:39:24.470688+0000","last_peered":"2024-09-10T20:39:24.470688+0000","last_clean":"2024-09-10T20:39:24.470688+0000","last_became_active":"2024-09-10T20:39:22.470753+0000","last_became_peered":"2024-09-10T20:39:22.470753+0000","last_unstale":"2024-09-10T20:39:24.470688+0000","last_undegraded":"2024-09-10T20:39:24.470688+0000","last_fullsized":"2024-09-10T20:39:24.470688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:22.626422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814639+0000","last_change":"2024-09-10T20:39:23.470015+0000","last_active":"2024-09-10T20:39:24.814639+0000","last_peered":"2024-09-10T20:39:24.814639+0000","last_clean":"2024-09-10T20:39:24.814639+0000","last_became_active":"2024-09-10T20:39:23.469693+0000","last_became_peered":"2024-09-10T20:39:23.469693+0000","last_unstale":"2024-09-10T20:39:24.814639+0000","last_undegraded":"2024-09-10T20:39:24.814639+0000","last_fullsized":"2024-09-10T20:39:24.814639+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252307+0000","last_change":"2024-09-10T20:39:22.461136+0000","last_active":"2024-09-10T20:39:25.252307+0000","last_peered":"2024-09-10T20:39:25.252307+0000","last_clean":"2024-09-10T20:39:25.252307+0000","last_became_active":"2024-09-10T20:39:22.460848+0000","last_became_peered":"2024-09-10T20:39:22.460848+0000","last_unstale":"2024-09-10T20:39:25.252307+0000","last_undegraded":"2024-09-10T20:39:25.252307+0000","last_fullsized":"2024-09-10T20:39:25.252307+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:18.873047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251912+0000","last_change":"2024-09-10T20:39:23.481110+0000","last_active":"2024-09-10T20:39:25.251912+0000","last_peered":"2024-09-10T20:39:25.251912+0000","last_clean":"2024-09-10T20:39:25.251912+0000","last_became_active":"2024-09-10T20:39:23.480241+0000","last_became_peered":"2024-09-10T20:39:23.480241+0000","last_unstale":"2024-09-10T20:39:25.251912+0000","last_undegraded":"2024-09-10T20:39:25.251912+0000","last_fullsized":"2024-09-10T20:39:25.251912+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:29.489296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470292+0000","last_change":"2024-09-10T20:39:22.467354+0000","last_active":"2024-09-10T20:39:24.470292+0000","last_peered":"2024-09-10T20:39:24.470292+0000","last_clean":"2024-09-10T20:39:24.470292+0000","last_became_active":"2024-09-10T20:39:22.467105+0000","last_became_peered":"2024-09-10T20:39:22.467105+0000","last_unstale":"2024-09-10T20:39:24.470292+0000","last_undegraded":"2024-09-10T20:39:24.470292+0000","last_fullsized":"2024-09-10T20:39:24.470292+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:42.142331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471995+0000","last_change":"2024-09-10T20:39:23.465129+0000","last_active":"2024-09-10T20:39:24.471995+0000","last_peered":"2024-09-10T20:39:24.471995+0000","last_clean":"2024-09-10T20:39:24.471995+0000","last_became_active":"2024-09-10T20:39:23.464816+0000","last_became_peered":"2024-09-10T20:39:23.464816+0000","last_unstale":"2024-09-10T20:39:24.471995+0000","last_undegraded":"2024-09-10T20:39:24.471995+0000","last_fullsized":"2024-09-10T20:39:24.471995+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:06.505948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252472+0000","last_change":"2024-09-10T20:39:22.463145+0000","last_active":"2024-09-10T20:39:25.252472+0000","last_peered":"2024-09-10T20:39:25.252472+0000","last_clean":"2024-09-10T20:39:25.252472+0000","last_became_active":"2024-09-10T20:39:22.462844+0000","last_became_peered":"2024-09-10T20:39:22.462844+0000","last_unstale":"2024-09-10T20:39:25.252472+0000","last_undegraded":"2024-09-10T20:39:25.252472+0000","last_fullsized":"2024-09-10T20:39:25.252472+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:42:40.187952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251873+0000","last_change":"2024-09-10T20:39:23.471689+0000","last_active":"2024-09-10T20:39:25.251873+0000","last_peered":"2024-09-10T20:39:25.251873+0000","last_clean":"2024-09-10T20:39:25.251873+0000","last_became_active":"2024-09-10T20:39:23.471333+0000","last_became_peered":"2024-09-10T20:39:23.471333+0000","last_unstale":"2024-09-10T20:39:25.251873+0000","last_undegraded":"2024-09-10T20:39:25.251873+0000","last_fullsized":"2024-09-10T20:39:25.251873+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:01:24.286087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470439+0000","last_change":"2024-09-10T20:39:22.468131+0000","last_active":"2024-09-10T20:39:24.470439+0000","last_peered":"2024-09-10T20:39:24.470439+0000","last_clean":"2024-09-10T20:39:24.470439+0000","last_became_active":"2024-09-10T20:39:22.467846+0000","last_became_peered":"2024-09-10T20:39:22.467846+0000","last_unstale":"2024-09-10T20:39:24.470439+0000","last_undegraded":"2024-09-10T20:39:24.470439+0000","last_fullsized":"2024-09-10T20:39:24.470439+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:40.914501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.480621+0000","last_change":"2024-09-10T20:39:23.477169+0000","last_active":"2024-09-10T20:39:23.480621+0000","last_peered":"2024-09-10T20:39:23.480621+0000","last_clean":"2024-09-10T20:39:23.480621+0000","last_became_active":"2024-09-10T20:39:23.476830+0000","last_became_peered":"2024-09-10T20:39:23.476830+0000","last_unstale":"2024-09-10T20:39:23.480621+0000","last_undegraded":"2024-09-10T20:39:23.480621+0000","last_fullsized":"2024-09-10T20:39:23.480621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:20:58.616528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251795+0000","last_change":"2024-09-10T20:39:22.468056+0000","last_active":"2024-09-10T20:39:25.251795+0000","last_peered":"2024-09-10T20:39:25.251795+0000","last_clean":"2024-09-10T20:39:25.251795+0000","last_became_active":"2024-09-10T20:39:22.467477+0000","last_became_peered":"2024-09-10T20:39:22.467477+0000","last_unstale":"2024-09-10T20:39:25.251795+0000","last_undegraded":"2024-09-10T20:39:25.251795+0000","last_fullsized":"2024-09-10T20:39:25.251795+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:45.140725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475067+0000","last_change":"2024-09-10T20:39:23.466496+0000","last_active":"2024-09-10T20:39:23.475067+0000","last_peered":"2024-09-10T20:39:23.475067+0000","last_clean":"2024-09-10T20:39:23.475067+0000","last_became_active":"2024-09-10T20:39:23.466162+0000","last_became_peered":"2024-09-10T20:39:23.466162+0000","last_unstale":"2024-09-10T20:39:23.475067+0000","last_undegraded":"2024-09-10T20:39:23.475067+0000","last_fullsized":"2024-09-10T20:39:23.475067+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:36.107439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814173+0000","last_change":"2024-09-10T20:39:22.469484+0000","last_active":"2024-09-10T20:39:24.814173+0000","last_peered":"2024-09-10T20:39:24.814173+0000","last_clean":"2024-09-10T20:39:24.814173+0000","last_became_active":"2024-09-10T20:39:22.469218+0000","last_became_peered":"2024-09-10T20:39:22.469218+0000","last_unstale":"2024-09-10T20:39:24.814173+0000","last_undegraded":"2024-09-10T20:39:24.814173+0000","last_fullsized":"2024-09-10T20:39:24.814173+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:42:09.052814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252173+0000","last_change":"2024-09-10T20:39:23.466343+0000","last_active":"2024-09-10T20:39:25.252173+0000","last_peered":"2024-09-10T20:39:25.252173+0000","last_clean":"2024-09-10T20:39:25.252173+0000","last_became_active":"2024-09-10T20:39:23.466061+0000","last_became_peered":"2024-09-10T20:39:23.466061+0000","last_unstale":"2024-09-10T20:39:25.252173+0000","last_undegraded":"2024-09-10T20:39:25.252173+0000","last_fullsized":"2024-09-10T20:39:25.252173+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.847458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460912+0000","last_change":"2024-09-10T20:39:22.464013+0000","last_active":"2024-09-10T20:39:23.460912+0000","last_peered":"2024-09-10T20:39:23.460912+0000","last_clean":"2024-09-10T20:39:23.460912+0000","last_became_active":"2024-09-10T20:39:22.463745+0000","last_became_peered":"2024-09-10T20:39:22.463745+0000","last_unstale":"2024-09-10T20:39:23.460912+0000","last_undegraded":"2024-09-10T20:39:23.460912+0000","last_fullsized":"2024-09-10T20:39:23.460912+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:17.407343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469738+0000","last_change":"2024-09-10T20:39:23.470667+0000","last_active":"2024-09-10T20:39:24.469738+0000","last_peered":"2024-09-10T20:39:24.469738+0000","last_clean":"2024-09-10T20:39:24.469738+0000","last_became_active":"2024-09-10T20:39:23.470425+0000","last_became_peered":"2024-09-10T20:39:23.470425+0000","last_unstale":"2024-09-10T20:39:24.469738+0000","last_undegraded":"2024-09-10T20:39:24.469738+0000","last_fullsized":"2024-09-10T20:39:24.469738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:08:02.455782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469712+0000","last_change":"2024-09-10T20:39:22.466186+0000","last_active":"2024-09-10T20:39:24.469712+0000","last_peered":"2024-09-10T20:39:24.469712+0000","last_clean":"2024-09-10T20:39:24.469712+0000","last_became_active":"2024-09-10T20:39:22.465718+0000","last_became_peered":"2024-09-10T20:39:22.465718+0000","last_unstale":"2024-09-10T20:39:24.469712+0000","last_undegraded":"2024-09-10T20:39:24.469712+0000","last_fullsized":"2024-09-10T20:39:24.469712+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:09.130187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469604+0000","last_change":"2024-09-10T20:39:23.474536+0000","last_active":"2024-09-10T20:39:24.469604+0000","last_peered":"2024-09-10T20:39:24.469604+0000","last_clean":"2024-09-10T20:39:24.469604+0000","last_became_active":"2024-09-10T20:39:23.474234+0000","last_became_peered":"2024-09-10T20:39:23.474234+0000","last_unstale":"2024-09-10T20:39:24.469604+0000","last_undegraded":"2024-09-10T20:39:24.469604+0000","last_fullsized":"2024-09-10T20:39:24.469604+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:38.533241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460348+0000","last_change":"2024-09-10T20:39:22.470606+0000","last_active":"2024-09-10T20:39:23.460348+0000","last_peered":"2024-09-10T20:39:23.460348+0000","last_clean":"2024-09-10T20:39:23.460348+0000","last_became_active":"2024-09-10T20:39:22.470313+0000","last_became_peered":"2024-09-10T20:39:22.470313+0000","last_unstale":"2024-09-10T20:39:23.460348+0000","last_undegraded":"2024-09-10T20:39:23.460348+0000","last_fullsized":"2024-09-10T20:39:23.460348+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:13.562757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471621+0000","last_change":"2024-09-10T20:39:23.478464+0000","last_active":"2024-09-10T20:39:24.471621+0000","last_peered":"2024-09-10T20:39:24.471621+0000","last_clean":"2024-09-10T20:39:24.471621+0000","last_became_active":"2024-09-10T20:39:23.477764+0000","last_became_peered":"2024-09-10T20:39:23.477764+0000","last_unstale":"2024-09-10T20:39:24.471621+0000","last_undegraded":"2024-09-10T20:39:24.471621+0000","last_fullsized":"2024-09-10T20:39:24.471621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:07.988361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"42'3","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251237+0000","last_change":"2024-09-10T20:39:22.466977+0000","last_active":"2024-09-10T20:39:25.251237+0000","last_peered":"2024-09-10T20:39:25.251237+0000","last_clean":"2024-09-10T20:39:25.251237+0000","last_became_active":"2024-09-10T20:39:22.466685+0000","last_became_peered":"2024-09-10T20:39:22.466685+0000","last_unstale":"2024-09-10T20:39:25.251237+0000","last_undegraded":"2024-09-10T20:39:25.251237+0000","last_fullsized":"2024-09-10T20:39:25.251237+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:44:01.537714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252311+0000","last_change":"2024-09-10T20:39:23.472408+0000","last_active":"2024-09-10T20:39:25.252311+0000","last_peered":"2024-09-10T20:39:25.252311+0000","last_clean":"2024-09-10T20:39:25.252311+0000","last_became_active":"2024-09-10T20:39:23.472123+0000","last_became_peered":"2024-09-10T20:39:23.472123+0000","last_unstale":"2024-09-10T20:39:25.252311+0000","last_undegraded":"2024-09-10T20:39:25.252311+0000","last_fullsized":"2024-09-10T20:39:25.252311+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:58:19.163029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":19,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.487847+0000","last_change":"2024-09-10T20:39:22.457688+0000","last_active":"2024-09-10T20:39:23.487847+0000","last_peered":"2024-09-10T20:39:23.487847+0000","last_clean":"2024-09-10T20:39:23.487847+0000","last_became_active":"2024-09-10T20:39:22.457326+0000","last_became_peered":"2024-09-10T20:39:22.457326+0000","last_unstale":"2024-09-10T20:39:23.487847+0000","last_undegraded":"2024-09-10T20:39:23.487847+0000","last_fullsized":"2024-09-10T20:39:23.487847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:51.098445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469742+0000","last_change":"2024-09-10T20:39:23.478920+0000","last_active":"2024-09-10T20:39:24.469742+0000","last_peered":"2024-09-10T20:39:24.469742+0000","last_clean":"2024-09-10T20:39:24.469742+0000","last_became_active":"2024-09-10T20:39:23.478555+0000","last_became_peered":"2024-09-10T20:39:23.478555+0000","last_unstale":"2024-09-10T20:39:24.469742+0000","last_undegraded":"2024-09-10T20:39:24.469742+0000","last_fullsized":"2024-09-10T20:39:24.469742+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:12:21.695423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473204+0000","last_change":"2024-09-10T20:39:22.463919+0000","last_active":"2024-09-10T20:39:23.473204+0000","last_peered":"2024-09-10T20:39:23.473204+0000","last_clean":"2024-09-10T20:39:23.473204+0000","last_became_active":"2024-09-10T20:39:22.463650+0000","last_became_peered":"2024-09-10T20:39:22.463650+0000","last_unstale":"2024-09-10T20:39:23.473204+0000","last_undegraded":"2024-09-10T20:39:23.473204+0000","last_fullsized":"2024-09-10T20:39:23.473204+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:07:05.483324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484089+0000","last_change":"2024-09-10T20:39:23.482909+0000","last_active":"2024-09-10T20:39:23.484089+0000","last_peered":"2024-09-10T20:39:23.484089+0000","last_clean":"2024-09-10T20:39:23.484089+0000","last_became_active":"2024-09-10T20:39:23.482551+0000","last_became_peered":"2024-09-10T20:39:23.482551+0000","last_unstale":"2024-09-10T20:39:23.484089+0000","last_undegraded":"2024-09-10T20:39:23.484089+0000","last_fullsized":"2024-09-10T20:39:23.484089+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:10.837228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814152+0000","last_change":"2024-09-10T20:39:22.471268+0000","last_active":"2024-09-10T20:39:24.814152+0000","last_peered":"2024-09-10T20:39:24.814152+0000","last_clean":"2024-09-10T20:39:24.814152+0000","last_became_active":"2024-09-10T20:39:22.470115+0000","last_became_peered":"2024-09-10T20:39:22.470115+0000","last_unstale":"2024-09-10T20:39:24.814152+0000","last_undegraded":"2024-09-10T20:39:24.814152+0000","last_fullsized":"2024-09-10T20:39:24.814152+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.467882+0000","last_change":"2024-09-10T20:39:23.465467+0000","last_active":"2024-09-10T20:39:23.467882+0000","last_peered":"2024-09-10T20:39:23.467882+0000","last_clean":"2024-09-10T20:39:23.467882+0000","last_became_active":"2024-09-10T20:39:23.465165+0000","last_became_peered":"2024-09-10T20:39:23.465165+0000","last_unstale":"2024-09-10T20:39:23.467882+0000","last_undegraded":"2024-09-10T20:39:23.467882+0000","last_fullsized":"2024-09-10T20:39:23.467882+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:15.544019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469888+0000","last_change":"2024-09-10T20:39:22.471774+0000","last_active":"2024-09-10T20:39:24.469888+0000","last_peered":"2024-09-10T20:39:24.469888+0000","last_clean":"2024-09-10T20:39:24.469888+0000","last_became_active":"2024-09-10T20:39:22.471455+0000","last_became_peered":"2024-09-10T20:39:22.471455+0000","last_unstale":"2024-09-10T20:39:24.469888+0000","last_undegraded":"2024-09-10T20:39:24.469888+0000","last_fullsized":"2024-09-10T20:39:24.469888+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:31.755937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251215+0000","last_change":"2024-09-10T20:39:23.482713+0000","last_active":"2024-09-10T20:39:25.251215+0000","last_peered":"2024-09-10T20:39:25.251215+0000","last_clean":"2024-09-10T20:39:25.251215+0000","last_became_active":"2024-09-10T20:39:23.482442+0000","last_became_peered":"2024-09-10T20:39:23.482442+0000","last_unstale":"2024-09-10T20:39:25.251215+0000","last_undegraded":"2024-09-10T20:39:25.251215+0000","last_fullsized":"2024-09-10T20:39:25.251215+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":21,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.485737+0000","last_change":"2024-09-10T20:39:22.468116+0000","last_active":"2024-09-10T20:39:23.485737+0000","last_peered":"2024-09-10T20:39:23.485737+0000","last_clean":"2024-09-10T20:39:23.485737+0000","last_became_active":"2024-09-10T20:39:22.467712+0000","last_became_peered":"2024-09-10T20:39:22.467712+0000","last_unstale":"2024-09-10T20:39:23.485737+0000","last_undegraded":"2024-09-10T20:39:23.485737+0000","last_fullsized":"2024-09-10T20:39:23.485737+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:18:01.811366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.474598+0000","last_change":"2024-09-10T20:39:23.473728+0000","last_active":"2024-09-10T20:39:23.474598+0000","last_peered":"2024-09-10T20:39:23.474598+0000","last_clean":"2024-09-10T20:39:23.474598+0000","last_became_active":"2024-09-10T20:39:23.473482+0000","last_became_peered":"2024-09-10T20:39:23.473482+0000","last_unstale":"2024-09-10T20:39:23.474598+0000","last_undegraded":"2024-09-10T20:39:23.474598+0000","last_fullsized":"2024-09-10T20:39:23.474598+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:42:24.635057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471265+0000","last_change":"2024-09-10T20:39:22.464648+0000","last_active":"2024-09-10T20:39:24.471265+0000","last_peered":"2024-09-10T20:39:24.471265+0000","last_clean":"2024-09-10T20:39:24.471265+0000","last_became_active":"2024-09-10T20:39:22.464381+0000","last_became_peered":"2024-09-10T20:39:22.464381+0000","last_unstale":"2024-09-10T20:39:24.471265+0000","last_undegraded":"2024-09-10T20:39:24.471265+0000","last_fullsized":"2024-09-10T20:39:24.471265+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:34.596334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.469081+0000","last_change":"2024-09-10T20:39:23.466893+0000","last_active":"2024-09-10T20:39:23.469081+0000","last_peered":"2024-09-10T20:39:23.469081+0000","last_clean":"2024-09-10T20:39:23.469081+0000","last_became_active":"2024-09-10T20:39:23.466577+0000","last_became_peered":"2024-09-10T20:39:23.466577+0000","last_unstale":"2024-09-10T20:39:23.469081+0000","last_undegraded":"2024-09-10T20:39:23.469081+0000","last_fullsized":"2024-09-10T20:39:23.469081+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:50.331650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469893+0000","last_change":"2024-09-10T20:39:22.472535+0000","last_active":"2024-09-10T20:39:24.469893+0000","last_peered":"2024-09-10T20:39:24.469893+0000","last_clean":"2024-09-10T20:39:24.469893+0000","last_became_active":"2024-09-10T20:39:22.472220+0000","last_became_peered":"2024-09-10T20:39:22.472220+0000","last_unstale":"2024-09-10T20:39:24.469893+0000","last_undegraded":"2024-09-10T20:39:24.469893+0000","last_fullsized":"2024-09-10T20:39:24.469893+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:49.040049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251980+0000","last_change":"2024-09-10T20:39:23.480836+0000","last_active":"2024-09-10T20:39:25.251980+0000","last_peered":"2024-09-10T20:39:25.251980+0000","last_clean":"2024-09-10T20:39:25.251980+0000","last_became_active":"2024-09-10T20:39:23.479966+0000","last_became_peered":"2024-09-10T20:39:23.479966+0000","last_unstale":"2024-09-10T20:39:25.251980+0000","last_undegraded":"2024-09-10T20:39:25.251980+0000","last_fullsized":"2024-09-10T20:39:25.251980+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:35.149497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469794+0000","last_change":"2024-09-10T20:39:22.468576+0000","last_active":"2024-09-10T20:39:24.469794+0000","last_peered":"2024-09-10T20:39:24.469794+0000","last_clean":"2024-09-10T20:39:24.469794+0000","last_became_active":"2024-09-10T20:39:22.468301+0000","last_became_peered":"2024-09-10T20:39:22.468301+0000","last_unstale":"2024-09-10T20:39:24.469794+0000","last_undegraded":"2024-09-10T20:39:24.469794+0000","last_fullsized":"2024-09-10T20:39:24.469794+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:38.187418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469705+0000","last_change":"2024-09-10T20:39:23.480119+0000","last_active":"2024-09-10T20:39:24.469705+0000","last_peered":"2024-09-10T20:39:24.469705+0000","last_clean":"2024-09-10T20:39:24.469705+0000","last_became_active":"2024-09-10T20:39:23.479859+0000","last_became_peered":"2024-09-10T20:39:23.479859+0000","last_unstale":"2024-09-10T20:39:24.469705+0000","last_undegraded":"2024-09-10T20:39:24.469705+0000","last_fullsized":"2024-09-10T20:39:24.469705+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:45.569351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470125+0000","last_change":"2024-09-10T20:39:22.471454+0000","last_active":"2024-09-10T20:39:24.470125+0000","last_peered":"2024-09-10T20:39:24.470125+0000","last_clean":"2024-09-10T20:39:24.470125+0000","last_became_active":"2024-09-10T20:39:22.471186+0000","last_became_peered":"2024-09-10T20:39:22.471186+0000","last_unstale":"2024-09-10T20:39:24.470125+0000","last_undegraded":"2024-09-10T20:39:24.470125+0000","last_fullsized":"2024-09-10T20:39:24.470125+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:11.967787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479095+0000","last_change":"2024-09-10T20:39:23.474293+0000","last_active":"2024-09-10T20:39:23.479095+0000","last_peered":"2024-09-10T20:39:23.479095+0000","last_clean":"2024-09-10T20:39:23.479095+0000","last_became_active":"2024-09-10T20:39:23.474013+0000","last_became_peered":"2024-09-10T20:39:23.474013+0000","last_unstale":"2024-09-10T20:39:23.479095+0000","last_undegraded":"2024-09-10T20:39:23.479095+0000","last_fullsized":"2024-09-10T20:39:23.479095+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:39:00.145526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484950+0000","last_change":"2024-09-10T20:39:22.467010+0000","last_active":"2024-09-10T20:39:23.484950+0000","last_peered":"2024-09-10T20:39:23.484950+0000","last_clean":"2024-09-10T20:39:23.484950+0000","last_became_active":"2024-09-10T20:39:22.466604+0000","last_became_peered":"2024-09-10T20:39:22.466604+0000","last_unstale":"2024-09-10T20:39:23.484950+0000","last_undegraded":"2024-09-10T20:39:23.484950+0000","last_fullsized":"2024-09-10T20:39:23.484950+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:33:38.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470167+0000","last_change":"2024-09-10T20:39:23.483419+0000","last_active":"2024-09-10T20:39:24.470167+0000","last_peered":"2024-09-10T20:39:24.470167+0000","last_clean":"2024-09-10T20:39:24.470167+0000","last_became_active":"2024-09-10T20:39:23.482573+0000","last_became_peered":"2024-09-10T20:39:23.482573+0000","last_unstale":"2024-09-10T20:39:24.470167+0000","last_undegraded":"2024-09-10T20:39:24.470167+0000","last_fullsized":"2024-09-10T20:39:24.470167+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814400+0000","last_change":"2024-09-10T20:39:22.472184+0000","last_active":"2024-09-10T20:39:24.814400+0000","last_peered":"2024-09-10T20:39:24.814400+0000","last_clean":"2024-09-10T20:39:24.814400+0000","last_became_active":"2024-09-10T20:39:22.471851+0000","last_became_peered":"2024-09-10T20:39:22.471851+0000","last_unstale":"2024-09-10T20:39:24.814400+0000","last_undegraded":"2024-09-10T20:39:24.814400+0000","last_fullsized":"2024-09-10T20:39:24.814400+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:06.279859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471576+0000","last_change":"2024-09-10T20:39:23.470437+0000","last_active":"2024-09-10T20:39:24.471576+0000","last_peered":"2024-09-10T20:39:24.471576+0000","last_clean":"2024-09-10T20:39:24.471576+0000","last_became_active":"2024-09-10T20:39:23.470018+0000","last_became_peered":"2024-09-10T20:39:23.470018+0000","last_unstale":"2024-09-10T20:39:24.471576+0000","last_undegraded":"2024-09-10T20:39:24.471576+0000","last_fullsized":"2024-09-10T20:39:24.471576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:19.864809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.462467+0000","last_change":"2024-09-10T20:39:22.470294+0000","last_active":"2024-09-10T20:39:23.462467+0000","last_peered":"2024-09-10T20:39:23.462467+0000","last_clean":"2024-09-10T20:39:23.462467+0000","last_became_active":"2024-09-10T20:39:22.469974+0000","last_became_peered":"2024-09-10T20:39:22.469974+0000","last_unstale":"2024-09-10T20:39:23.462467+0000","last_undegraded":"2024-09-10T20:39:23.462467+0000","last_fullsized":"2024-09-10T20:39:23.462467+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:14.321360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471788+0000","last_change":"2024-09-10T20:39:23.467237+0000","last_active":"2024-09-10T20:39:24.471788+0000","last_peered":"2024-09-10T20:39:24.471788+0000","last_clean":"2024-09-10T20:39:24.471788+0000","last_became_active":"2024-09-10T20:39:23.466955+0000","last_became_peered":"2024-09-10T20:39:23.466955+0000","last_unstale":"2024-09-10T20:39:24.471788+0000","last_undegraded":"2024-09-10T20:39:24.471788+0000","last_fullsized":"2024-09-10T20:39:24.471788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:17:53.617323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251301+0000","last_change":"2024-09-10T20:39:22.463388+0000","last_active":"2024-09-10T20:39:25.251301+0000","last_peered":"2024-09-10T20:39:25.251301+0000","last_clean":"2024-09-10T20:39:25.251301+0000","last_became_active":"2024-09-10T20:39:22.463050+0000","last_became_peered":"2024-09-10T20:39:22.463050+0000","last_unstale":"2024-09-10T20:39:25.251301+0000","last_undegraded":"2024-09-10T20:39:25.251301+0000","last_fullsized":"2024-09-10T20:39:25.251301+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:49.479271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252422+0000","last_change":"2024-09-10T20:39:23.466822+0000","last_active":"2024-09-10T20:39:25.252422+0000","last_peered":"2024-09-10T20:39:25.252422+0000","last_clean":"2024-09-10T20:39:25.252422+0000","last_became_active":"2024-09-10T20:39:23.466056+0000","last_became_peered":"2024-09-10T20:39:23.466056+0000","last_unstale":"2024-09-10T20:39:25.252422+0000","last_undegraded":"2024-09-10T20:39:25.252422+0000","last_fullsized":"2024-09-10T20:39:25.252422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:03.707588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471588+0000","last_change":"2024-09-10T20:39:22.466270+0000","last_active":"2024-09-10T20:39:24.471588+0000","last_peered":"2024-09-10T20:39:24.471588+0000","last_clean":"2024-09-10T20:39:24.471588+0000","last_became_active":"2024-09-10T20:39:22.465951+0000","last_became_peered":"2024-09-10T20:39:22.465951+0000","last_unstale":"2024-09-10T20:39:24.471588+0000","last_undegraded":"2024-09-10T20:39:24.471588+0000","last_fullsized":"2024-09-10T20:39:24.471588+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475360+0000","last_change":"2024-09-10T20:39:23.468716+0000","last_active":"2024-09-10T20:39:23.475360+0000","last_peered":"2024-09-10T20:39:23.475360+0000","last_clean":"2024-09-10T20:39:23.475360+0000","last_became_active":"2024-09-10T20:39:23.468450+0000","last_became_peered":"2024-09-10T20:39:23.468450+0000","last_unstale":"2024-09-10T20:39:23.475360+0000","last_undegraded":"2024-09-10T20:39:23.475360+0000","last_fullsized":"2024-09-10T20:39:23.475360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:00.101321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814372+0000","last_change":"2024-09-10T20:39:22.470970+0000","last_active":"2024-09-10T20:39:24.814372+0000","last_peered":"2024-09-10T20:39:24.814372+0000","last_clean":"2024-09-10T20:39:24.814372+0000","last_became_active":"2024-09-10T20:39:22.470662+0000","last_became_peered":"2024-09-10T20:39:22.470662+0000","last_unstale":"2024-09-10T20:39:24.814372+0000","last_undegraded":"2024-09-10T20:39:24.814372+0000","last_fullsized":"2024-09-10T20:39:24.814372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:34:54.628648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252557+0000","last_change":"2024-09-10T20:39:23.476329+0000","last_active":"2024-09-10T20:39:25.252557+0000","last_peered":"2024-09-10T20:39:25.252557+0000","last_clean":"2024-09-10T20:39:25.252557+0000","last_became_active":"2024-09-10T20:39:23.476062+0000","last_became_peered":"2024-09-10T20:39:23.476062+0000","last_unstale":"2024-09-10T20:39:25.252557+0000","last_undegraded":"2024-09-10T20:39:25.252557+0000","last_fullsized":"2024-09-10T20:39:25.252557+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:16:42.873397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460877+0000","last_change":"2024-09-10T20:39:22.467181+0000","last_active":"2024-09-10T20:39:23.460877+0000","last_peered":"2024-09-10T20:39:23.460877+0000","last_clean":"2024-09-10T20:39:23.460877+0000","last_became_active":"2024-09-10T20:39:22.466859+0000","last_became_peered":"2024-09-10T20:39:22.466859+0000","last_unstale":"2024-09-10T20:39:23.460877+0000","last_undegraded":"2024-09-10T20:39:23.460877+0000","last_fullsized":"2024-09-10T20:39:23.460877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:54.024056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479036+0000","last_change":"2024-09-10T20:39:23.473568+0000","last_active":"2024-09-10T20:39:23.479036+0000","last_peered":"2024-09-10T20:39:23.479036+0000","last_clean":"2024-09-10T20:39:23.479036+0000","last_became_active":"2024-09-10T20:39:23.473257+0000","last_became_peered":"2024-09-10T20:39:23.473257+0000","last_unstale":"2024-09-10T20:39:23.479036+0000","last_undegraded":"2024-09-10T20:39:23.479036+0000","last_fullsized":"2024-09-10T20:39:23.479036+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:02.160411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814407+0000","last_change":"2024-09-10T20:39:22.467492+0000","last_active":"2024-09-10T20:39:24.814407+0000","last_peered":"2024-09-10T20:39:24.814407+0000","last_clean":"2024-09-10T20:39:24.814407+0000","last_became_active":"2024-09-10T20:39:22.467217+0000","last_became_peered":"2024-09-10T20:39:22.467217+0000","last_unstale":"2024-09-10T20:39:24.814407+0000","last_undegraded":"2024-09-10T20:39:24.814407+0000","last_fullsized":"2024-09-10T20:39:24.814407+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:32.173179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471424+0000","last_change":"2024-09-10T20:39:23.482739+0000","last_active":"2024-09-10T20:39:24.471424+0000","last_peered":"2024-09-10T20:39:24.471424+0000","last_clean":"2024-09-10T20:39:24.471424+0000","last_became_active":"2024-09-10T20:39:23.482483+0000","last_became_peered":"2024-09-10T20:39:23.482483+0000","last_unstale":"2024-09-10T20:39:24.471424+0000","last_undegraded":"2024-09-10T20:39:24.471424+0000","last_fullsized":"2024-09-10T20:39:24.471424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:53.453242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251921+0000","last_change":"2024-09-10T20:39:22.466109+0000","last_active":"2024-09-10T20:39:25.251921+0000","last_peered":"2024-09-10T20:39:25.251921+0000","last_clean":"2024-09-10T20:39:25.251921+0000","last_became_active":"2024-09-10T20:39:22.463916+0000","last_became_peered":"2024-09-10T20:39:22.463916+0000","last_unstale":"2024-09-10T20:39:25.251921+0000","last_undegraded":"2024-09-10T20:39:25.251921+0000","last_fullsized":"2024-09-10T20:39:25.251921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:19.681143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469354+0000","last_change":"2024-09-10T20:39:23.479485+0000","last_active":"2024-09-10T20:39:24.469354+0000","last_peered":"2024-09-10T20:39:24.469354+0000","last_clean":"2024-09-10T20:39:24.469354+0000","last_became_active":"2024-09-10T20:39:23.479219+0000","last_became_peered":"2024-09-10T20:39:23.479219+0000","last_unstale":"2024-09-10T20:39:24.469354+0000","last_undegraded":"2024-09-10T20:39:24.469354+0000","last_fullsized":"2024-09-10T20:39:24.469354+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:22.900530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469982+0000","last_change":"2024-09-10T20:39:22.464748+0000","last_active":"2024-09-10T20:39:24.469982+0000","last_peered":"2024-09-10T20:39:24.469982+0000","last_clean":"2024-09-10T20:39:24.469982+0000","last_became_active":"2024-09-10T20:39:22.464248+0000","last_became_peered":"2024-09-10T20:39:22.464248+0000","last_unstale":"2024-09-10T20:39:24.469982+0000","last_undegraded":"2024-09-10T20:39:24.469982+0000","last_fullsized":"2024-09-10T20:39:24.469982+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:34:33.491750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251582+0000","last_change":"2024-09-10T20:39:23.482269+0000","last_active":"2024-09-10T20:39:25.251582+0000","last_peered":"2024-09-10T20:39:25.251582+0000","last_clean":"2024-09-10T20:39:25.251582+0000","last_became_active":"2024-09-10T20:39:23.481990+0000","last_became_peered":"2024-09-10T20:39:23.481990+0000","last_unstale":"2024-09-10T20:39:25.251582+0000","last_undegraded":"2024-09-10T20:39:25.251582+0000","last_fullsized":"2024-09-10T20:39:25.251582+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:40.821209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.459867+0000","last_change":"2024-09-10T20:39:22.465000+0000","last_active":"2024-09-10T20:39:23.459867+0000","last_peered":"2024-09-10T20:39:23.459867+0000","last_clean":"2024-09-10T20:39:23.459867+0000","last_became_active":"2024-09-10T20:39:22.464669+0000","last_became_peered":"2024-09-10T20:39:22.464669+0000","last_unstale":"2024-09-10T20:39:23.459867+0000","last_undegraded":"2024-09-10T20:39:23.459867+0000","last_fullsized":"2024-09-10T20:39:23.459867+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:43:06.255942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471722+0000","last_change":"2024-09-10T20:39:23.476662+0000","last_active":"2024-09-10T20:39:24.471722+0000","last_peered":"2024-09-10T20:39:24.471722+0000","last_clean":"2024-09-10T20:39:24.471722+0000","last_became_active":"2024-09-10T20:39:23.476375+0000","last_became_peered":"2024-09-10T20:39:23.476375+0000","last_unstale":"2024-09-10T20:39:24.471722+0000","last_undegraded":"2024-09-10T20:39:24.471722+0000","last_fullsized":"2024-09-10T20:39:24.471722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:46.142798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252456+0000","last_change":"2024-09-10T20:39:22.464857+0000","last_active":"2024-09-10T20:39:25.252456+0000","last_peered":"2024-09-10T20:39:25.252456+0000","last_clean":"2024-09-10T20:39:25.252456+0000","last_became_active":"2024-09-10T20:39:22.464583+0000","last_became_peered":"2024-09-10T20:39:22.464583+0000","last_unstale":"2024-09-10T20:39:25.252456+0000","last_undegraded":"2024-09-10T20:39:25.252456+0000","last_fullsized":"2024-09-10T20:39:25.252456+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:15.049485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":149,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469244+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:39:24.469244+0000","last_peered":"2024-09-10T20:39:24.469244+0000","last_clean":"2024-09-10T20:39:24.469244+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:39:24.469244+0000","last_undegraded":"2024-09-10T20:39:24.469244+0000","last_fullsized":"2024-09-10T20:39:24.469244+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509461,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":507904,"data_stored":200539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509460,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":499712,"data_stored":199017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1093632,"data_stored":789365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509460,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":503808,"data_stored":199599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":491520,"data_stored":193243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1093632,"data_stored":789317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":503808,"data_stored":194331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:39:28.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:39:28.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.374+0000 7f796c5c2640 1 -- 172.21.15.28:0/2030160239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7950004d40 msgr2=0x7f79500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.374+0000 7f796c5c2640 1 --2- 172.21.15.28:0/2030160239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7950004d40 0x7f79500051c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f795c006fd0 tx=0x7f7950005830 comp rx=0 tx=0).stop 2024-09-10T20:39:28.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.374+0000 7f796ab5f640 1 -- 172.21.15.28:0/2030160239 reap_dead start 2024-09-10T20:39:28.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.374+0000 7f796c5c2640 1 -- 172.21.15.28:0/2030160239 shutdown_connections 2024-09-10T20:39:28.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.374+0000 7f796c5c2640 1 -- 172.21.15.28:0/2030160239 >> 172.21.15.28:0/2030160239 conn(0x7f79640d4090 msgr2=0x7f79640d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:28.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.374+0000 7f796c5c2640 1 -- 172.21.15.28:0/2030160239 shutdown_connections 2024-09-10T20:39:28.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.374+0000 7f796c5c2640 1 -- 172.21.15.28:0/2030160239 wait complete. 2024-09-10T20:39:28.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.375+0000 7f796c5c2640 1 Processor -- start 2024-09-10T20:39:28.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.375+0000 7f796c5c2640 1 -- start start 2024-09-10T20:39:28.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796c5c2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7950004d40 0x7f79640a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796c5c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79640a0330 0x7f79641f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796c5c2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79640b3d20 0x7f7964154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796c5c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79640d2640 con 0x7f79640a0330 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796c5c2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f79640b2e90 con 0x7f79640b3d20 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796c5c2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f79640d27c0 con 0x7f7950004d40 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f7969b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79640a0330 0x7f79641f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f7969b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79640a0330 0x7f79641f58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49750/0 (socket says 172.21.15.28:49750) 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f7969b5d640 1 -- 172.21.15.28:0/3668732792 learned_addr learned my addr 172.21.15.28:0/3668732792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796a35e640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7950004d40 0x7f79640a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.376+0000 7f796ab5f640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79640b3d20 0x7f7964154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f7969b5d640 1 -- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7950004d40 msgr2=0x7f79640a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f7969b5d640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7950004d40 0x7f79640a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f7969b5d640 1 -- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79640b3d20 msgr2=0x7f7964154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f7969b5d640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79640b3d20 0x7f7964154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f7969b5d640 1 -- 172.21.15.28:0/3668732792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79640a07c0 con 0x7f79640a0330 2024-09-10T20:39:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f7969b5d640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79640a0330 0x7f79641f58f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7954003ef0 tx=0x7f795400b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f795affd640 1 -- 172.21.15.28:0/3668732792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540107d0 con 0x7f79640a0330 2024-09-10T20:39:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f795affd640 1 -- 172.21.15.28:0/3668732792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7954010970 con 0x7f79640a0330 2024-09-10T20:39:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79640a0aa0 con 0x7f79640a0330 2024-09-10T20:39:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f795affd640 1 -- 172.21.15.28:0/3668732792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954010da0 con 0x7f79640a0330 2024-09-10T20:39:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.377+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79640a1340 con 0x7f79640a0330 2024-09-10T20:39:28.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.378+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79641612f0 con 0x7f79640a0330 2024-09-10T20:39:28.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.379+0000 7f795affd640 1 -- 172.21.15.28:0/3668732792 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f7954023440 con 0x7f79640a0330 2024-09-10T20:39:28.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.380+0000 7f795affd640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7948078120 0x7f794807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.380+0000 7f795affd640 1 -- 172.21.15.28:0/3668732792 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f795409aea0 con 0x7f79640a0330 2024-09-10T20:39:28.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.380+0000 7f796a35e640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7948078120 0x7f794807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.380+0000 7f796a35e640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7948078120 0x7f794807a5e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f796000bc40 tx=0x7f796000b2e0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:28.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.384+0000 7f795affd640 1 -- 172.21.15.28:0/3668732792 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f79641612f0 con 0x7f79640a0330 2024-09-10T20:39:28.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.647+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f79640d2bb0 con 0x7f79640a0330 2024-09-10T20:39:28.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.647+0000 7f795affd640 1 -- 172.21.15.28:0/3668732792 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1474 (secure 0 0 0) 0x7f79640d2bb0 con 0x7f79640a0330 2024-09-10T20:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":101,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":8,"num_up_osds":8,"osd_up_since":1726000680,"num_in_osds":8,"osd_in_since":1726000670,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":243654656,"bytes_avail":772850458624,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":26,"btime":"2024-09-10T20:39:26:475950+0000","id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5286}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-10T20:39:11.988296+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:39:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.649+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7948078120 msgr2=0x7f794807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.649+0000 7f796c5c2640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7948078120 0x7f794807a5e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f796000bc40 tx=0x7f796000b2e0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.649+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79640a0330 msgr2=0x7f79641f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.649+0000 7f796c5c2640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79640a0330 0x7f79641f58f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7954003ef0 tx=0x7f795400b9a0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 shutdown_connections 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7948078120 0x7f794807a5e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79640b3d20 0x7f7964154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79640a0330 0x7f79641f58f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 --2- 172.21.15.28:0/3668732792 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7950004d40 0x7f79640a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 >> 172.21.15.28:0/3668732792 conn(0x7f79640d4090 msgr2=0x7f79640d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 shutdown_connections 2024-09-10T20:39:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.650+0000 7f796c5c2640 1 -- 172.21.15.28:0/3668732792 wait complete. 2024-09-10T20:39:28.664 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T20:39:28.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:39:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.767+0000 7f0084071640 1 -- 172.21.15.28:0/2686492561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0064004d40 msgr2=0x7f00640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.767+0000 7f0084071640 1 --2- 172.21.15.28:0/2686492561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0064004d40 0x7f00640051c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f006c00c630 tx=0x7f0064005830 comp rx=0 tx=0).stop 2024-09-10T20:39:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.767+0000 7f008260e640 1 -- 172.21.15.28:0/2686492561 reap_dead start 2024-09-10T20:39:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.767+0000 7f0084071640 1 -- 172.21.15.28:0/2686492561 shutdown_connections 2024-09-10T20:39:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.767+0000 7f0084071640 1 -- 172.21.15.28:0/2686492561 >> 172.21.15.28:0/2686492561 conn(0x7f007c0d3fb0 msgr2=0x7f007c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.767+0000 7f0084071640 1 -- 172.21.15.28:0/2686492561 shutdown_connections 2024-09-10T20:39:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.767+0000 7f0084071640 1 -- 172.21.15.28:0/2686492561 wait complete. 2024-09-10T20:39:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.768+0000 7f0084071640 1 Processor -- start 2024-09-10T20:39:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.768+0000 7f0084071640 1 -- start start 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f0084071640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0064004d40 0x7f007c1613a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f0084071640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f007c0a15b0 0x7f007c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f0084071640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f007c09dbf0 0x7f007c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f0084071640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f007c15e3d0 con 0x7f007c0a15b0 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f0084071640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f007c15e250 con 0x7f007c09dbf0 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f0084071640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f007c161b50 con 0x7f0064004d40 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f007c0a15b0 0x7f007c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f007c0a15b0 0x7f007c158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49766/0 (socket says 172.21.15.28:49766) 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 -- 172.21.15.28:0/3902635813 learned_addr learned my addr 172.21.15.28:0/3902635813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008260e640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f007c09dbf0 0x7f007c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f0081e0d640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0064004d40 0x7f007c1613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 -- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0064004d40 msgr2=0x7f007c1613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0064004d40 0x7f007c1613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 -- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f007c09dbf0 msgr2=0x7f007c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f007c09dbf0 0x7f007c09e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008160c640 1 -- 172.21.15.28:0/3902635813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007c1f4590 con 0x7f007c0a15b0 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.769+0000 7f008260e640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f007c09dbf0 0x7f007c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.770+0000 7f008160c640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f007c0a15b0 0x7f007c158e20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0070011b10 tx=0x7f0070009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.770+0000 7f007a7fc640 1 -- 172.21.15.28:0/3902635813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007001e070 con 0x7f007c0a15b0 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.770+0000 7f007a7fc640 1 -- 172.21.15.28:0/3902635813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0070004050 con 0x7f007c0a15b0 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.770+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007c1f7da0 con 0x7f007c0a15b0 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.770+0000 7f007a7fc640 1 -- 172.21.15.28:0/3902635813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007000fea0 con 0x7f007c0a15b0 2024-09-10T20:39:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.770+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f007c1f8360 con 0x7f007c0a15b0 2024-09-10T20:39:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.771+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f007c0cf780 con 0x7f007c0a15b0 2024-09-10T20:39:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.772+0000 7f007a7fc640 1 -- 172.21.15.28:0/3902635813 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f00700035f0 con 0x7f007c0a15b0 2024-09-10T20:39:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.772+0000 7f007a7fc640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0068078150 0x7f006807a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.772+0000 7f007a7fc640 1 -- 172.21.15.28:0/3902635813 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f007009aef0 con 0x7f007c0a15b0 2024-09-10T20:39:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.772+0000 7f0081e0d640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0068078150 0x7f006807a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.773+0000 7f0081e0d640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0068078150 0x7f006807a610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0074000c00 tx=0x7f0074006b70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:28.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.776+0000 7f007a7fc640 1 -- 172.21.15.28:0/3902635813 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f00700630d0 con 0x7f007c0a15b0 2024-09-10T20:39:28.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.958+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f007c0b9f30 con 0x7f0068078150 2024-09-10T20:39:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.962+0000 7f007a7fc640 1 -- 172.21.15.28:0/3902635813 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f007c0b9f30 con 0x7f0068078150 2024-09-10T20:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:39:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0068078150 msgr2=0x7f006807a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0068078150 0x7f006807a610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0074000c00 tx=0x7f0074006b70 comp rx=0 tx=0).stop 2024-09-10T20:39:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f007c0a15b0 msgr2=0x7f007c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f007c0a15b0 0x7f007c158e20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0070011b10 tx=0x7f0070009e90 comp rx=0 tx=0).stop 2024-09-10T20:39:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 shutdown_connections 2024-09-10T20:39:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0068078150 0x7f006807a610 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f007c09dbf0 0x7f007c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f007c0a15b0 0x7f007c158e20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 --2- 172.21.15.28:0/3902635813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0064004d40 0x7f007c1613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 >> 172.21.15.28:0/3902635813 conn(0x7f007c0d3fb0 msgr2=0x7f007c0b91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 shutdown_connections 2024-09-10T20:39:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:28.965+0000 7f0084071640 1 -- 172.21.15.28:0/3902635813 wait complete. 2024-09-10T20:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-10T20:39:27.991829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237944,"kb_used_data":5060,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736776,"statfs":{"total":773094113280,"available":772850458624,"internally_reserved":0,"allocated":5181440,"data_stored":2758072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001436"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814513+0000","last_change":"2024-09-10T20:39:23.483860+0000","last_active":"2024-09-10T20:39:24.814513+0000","last_peered":"2024-09-10T20:39:24.814513+0000","last_clean":"2024-09-10T20:39:24.814513+0000","last_became_active":"2024-09-10T20:39:23.483590+0000","last_became_peered":"2024-09-10T20:39:23.483590+0000","last_unstale":"2024-09-10T20:39:24.814513+0000","last_undegraded":"2024-09-10T20:39:24.814513+0000","last_fullsized":"2024-09-10T20:39:24.814513+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251574+0000","last_change":"2024-09-10T20:39:23.476103+0000","last_active":"2024-09-10T20:39:25.251574+0000","last_peered":"2024-09-10T20:39:25.251574+0000","last_clean":"2024-09-10T20:39:25.251574+0000","last_became_active":"2024-09-10T20:39:23.475659+0000","last_became_peered":"2024-09-10T20:39:23.475659+0000","last_unstale":"2024-09-10T20:39:25.251574+0000","last_undegraded":"2024-09-10T20:39:25.251574+0000","last_fullsized":"2024-09-10T20:39:25.251574+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:01.324112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469676+0000","last_change":"2024-09-10T20:39:23.479367+0000","last_active":"2024-09-10T20:39:24.469676+0000","last_peered":"2024-09-10T20:39:24.469676+0000","last_clean":"2024-09-10T20:39:24.469676+0000","last_became_active":"2024-09-10T20:39:23.479089+0000","last_became_peered":"2024-09-10T20:39:23.479089+0000","last_unstale":"2024-09-10T20:39:24.469676+0000","last_undegraded":"2024-09-10T20:39:24.469676+0000","last_fullsized":"2024-09-10T20:39:24.469676+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:43:14.607190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471413+0000","last_change":"2024-09-10T20:39:23.474482+0000","last_active":"2024-09-10T20:39:24.471413+0000","last_peered":"2024-09-10T20:39:24.471413+0000","last_clean":"2024-09-10T20:39:24.471413+0000","last_became_active":"2024-09-10T20:39:23.469443+0000","last_became_peered":"2024-09-10T20:39:23.469443+0000","last_unstale":"2024-09-10T20:39:24.471413+0000","last_undegraded":"2024-09-10T20:39:24.471413+0000","last_fullsized":"2024-09-10T20:39:24.471413+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814280+0000","last_change":"2024-09-10T20:39:23.469934+0000","last_active":"2024-09-10T20:39:24.814280+0000","last_peered":"2024-09-10T20:39:24.814280+0000","last_clean":"2024-09-10T20:39:24.814280+0000","last_became_active":"2024-09-10T20:39:23.469620+0000","last_became_peered":"2024-09-10T20:39:23.469620+0000","last_unstale":"2024-09-10T20:39:24.814280+0000","last_undegraded":"2024-09-10T20:39:24.814280+0000","last_fullsized":"2024-09-10T20:39:24.814280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:43.501230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470522+0000","last_change":"2024-09-10T20:39:23.478562+0000","last_active":"2024-09-10T20:39:24.470522+0000","last_peered":"2024-09-10T20:39:24.470522+0000","last_clean":"2024-09-10T20:39:24.470522+0000","last_became_active":"2024-09-10T20:39:23.478273+0000","last_became_peered":"2024-09-10T20:39:23.478273+0000","last_unstale":"2024-09-10T20:39:24.470522+0000","last_undegraded":"2024-09-10T20:39:24.470522+0000","last_fullsized":"2024-09-10T20:39:24.470522+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:06:13.043952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251327+0000","last_change":"2024-09-10T20:39:23.475249+0000","last_active":"2024-09-10T20:39:25.251327+0000","last_peered":"2024-09-10T20:39:25.251327+0000","last_clean":"2024-09-10T20:39:25.251327+0000","last_became_active":"2024-09-10T20:39:23.474917+0000","last_became_peered":"2024-09-10T20:39:23.474917+0000","last_unstale":"2024-09-10T20:39:25.251327+0000","last_undegraded":"2024-09-10T20:39:25.251327+0000","last_fullsized":"2024-09-10T20:39:25.251327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:32.157058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471730+0000","last_change":"2024-09-10T20:39:23.477481+0000","last_active":"2024-09-10T20:39:24.471730+0000","last_peered":"2024-09-10T20:39:24.471730+0000","last_clean":"2024-09-10T20:39:24.471730+0000","last_became_active":"2024-09-10T20:39:23.477194+0000","last_became_peered":"2024-09-10T20:39:23.477194+0000","last_unstale":"2024-09-10T20:39:24.471730+0000","last_undegraded":"2024-09-10T20:39:24.471730+0000","last_fullsized":"2024-09-10T20:39:24.471730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:16.051559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470973+0000","last_change":"2024-09-10T20:39:23.465809+0000","last_active":"2024-09-10T20:39:24.470973+0000","last_peered":"2024-09-10T20:39:24.470973+0000","last_clean":"2024-09-10T20:39:24.470973+0000","last_became_active":"2024-09-10T20:39:23.465540+0000","last_became_peered":"2024-09-10T20:39:23.465540+0000","last_unstale":"2024-09-10T20:39:24.470973+0000","last_undegraded":"2024-09-10T20:39:24.470973+0000","last_fullsized":"2024-09-10T20:39:24.470973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.481077+0000","last_change":"2024-09-10T20:39:23.480200+0000","last_active":"2024-09-10T20:39:23.481077+0000","last_peered":"2024-09-10T20:39:23.481077+0000","last_clean":"2024-09-10T20:39:23.481077+0000","last_became_active":"2024-09-10T20:39:23.479928+0000","last_became_peered":"2024-09-10T20:39:23.479928+0000","last_unstale":"2024-09-10T20:39:23.481077+0000","last_undegraded":"2024-09-10T20:39:23.481077+0000","last_fullsized":"2024-09-10T20:39:23.481077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:20.100800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471901+0000","last_change":"2024-09-10T20:39:23.472464+0000","last_active":"2024-09-10T20:39:24.471901+0000","last_peered":"2024-09-10T20:39:24.471901+0000","last_clean":"2024-09-10T20:39:24.471901+0000","last_became_active":"2024-09-10T20:39:23.472185+0000","last_became_peered":"2024-09-10T20:39:23.472185+0000","last_unstale":"2024-09-10T20:39:24.471901+0000","last_undegraded":"2024-09-10T20:39:24.471901+0000","last_fullsized":"2024-09-10T20:39:24.471901+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:16:50.093970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251720+0000","last_change":"2024-09-10T20:39:23.481383+0000","last_active":"2024-09-10T20:39:25.251720+0000","last_peered":"2024-09-10T20:39:25.251720+0000","last_clean":"2024-09-10T20:39:25.251720+0000","last_became_active":"2024-09-10T20:39:23.480519+0000","last_became_peered":"2024-09-10T20:39:23.480519+0000","last_unstale":"2024-09-10T20:39:25.251720+0000","last_undegraded":"2024-09-10T20:39:25.251720+0000","last_fullsized":"2024-09-10T20:39:25.251720+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:06:51.852363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470189+0000","last_change":"2024-09-10T20:39:23.479455+0000","last_active":"2024-09-10T20:39:24.470189+0000","last_peered":"2024-09-10T20:39:24.470189+0000","last_clean":"2024-09-10T20:39:24.470189+0000","last_became_active":"2024-09-10T20:39:23.479185+0000","last_became_peered":"2024-09-10T20:39:23.479185+0000","last_unstale":"2024-09-10T20:39:24.470189+0000","last_undegraded":"2024-09-10T20:39:24.470189+0000","last_fullsized":"2024-09-10T20:39:24.470189+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471366+0000","last_change":"2024-09-10T20:39:23.472528+0000","last_active":"2024-09-10T20:39:24.471366+0000","last_peered":"2024-09-10T20:39:24.471366+0000","last_clean":"2024-09-10T20:39:24.471366+0000","last_became_active":"2024-09-10T20:39:23.472247+0000","last_became_peered":"2024-09-10T20:39:23.472247+0000","last_unstale":"2024-09-10T20:39:24.471366+0000","last_undegraded":"2024-09-10T20:39:24.471366+0000","last_fullsized":"2024-09-10T20:39:24.471366+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471460+0000","last_change":"2024-09-10T20:39:23.477260+0000","last_active":"2024-09-10T20:39:24.471460+0000","last_peered":"2024-09-10T20:39:24.471460+0000","last_clean":"2024-09-10T20:39:24.471460+0000","last_became_active":"2024-09-10T20:39:23.476962+0000","last_became_peered":"2024-09-10T20:39:23.476962+0000","last_unstale":"2024-09-10T20:39:24.471460+0000","last_undegraded":"2024-09-10T20:39:24.471460+0000","last_fullsized":"2024-09-10T20:39:24.471460+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814151+0000","last_change":"2024-09-10T20:39:23.482868+0000","last_active":"2024-09-10T20:39:24.814151+0000","last_peered":"2024-09-10T20:39:24.814151+0000","last_clean":"2024-09-10T20:39:24.814151+0000","last_became_active":"2024-09-10T20:39:23.482352+0000","last_became_peered":"2024-09-10T20:39:23.482352+0000","last_unstale":"2024-09-10T20:39:24.814151+0000","last_undegraded":"2024-09-10T20:39:24.814151+0000","last_fullsized":"2024-09-10T20:39:24.814151+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:10.014292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251846+0000","last_change":"2024-09-10T20:39:23.481835+0000","last_active":"2024-09-10T20:39:25.251846+0000","last_peered":"2024-09-10T20:39:25.251846+0000","last_clean":"2024-09-10T20:39:25.251846+0000","last_became_active":"2024-09-10T20:39:23.481462+0000","last_became_peered":"2024-09-10T20:39:23.481462+0000","last_unstale":"2024-09-10T20:39:25.251846+0000","last_undegraded":"2024-09-10T20:39:25.251846+0000","last_fullsized":"2024-09-10T20:39:25.251846+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:37:09.072417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469826+0000","last_change":"2024-09-10T20:39:23.478525+0000","last_active":"2024-09-10T20:39:24.469826+0000","last_peered":"2024-09-10T20:39:24.469826+0000","last_clean":"2024-09-10T20:39:24.469826+0000","last_became_active":"2024-09-10T20:39:23.478217+0000","last_became_peered":"2024-09-10T20:39:23.478217+0000","last_unstale":"2024-09-10T20:39:24.469826+0000","last_undegraded":"2024-09-10T20:39:24.469826+0000","last_fullsized":"2024-09-10T20:39:24.469826+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:28:10.445472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814465+0000","last_change":"2024-09-10T20:39:23.471412+0000","last_active":"2024-09-10T20:39:24.814465+0000","last_peered":"2024-09-10T20:39:24.814465+0000","last_clean":"2024-09-10T20:39:24.814465+0000","last_became_active":"2024-09-10T20:39:23.471147+0000","last_became_peered":"2024-09-10T20:39:23.471147+0000","last_unstale":"2024-09-10T20:39:24.814465+0000","last_undegraded":"2024-09-10T20:39:24.814465+0000","last_fullsized":"2024-09-10T20:39:24.814465+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475335+0000","last_change":"2024-09-10T20:39:23.467260+0000","last_active":"2024-09-10T20:39:23.475335+0000","last_peered":"2024-09-10T20:39:23.475335+0000","last_clean":"2024-09-10T20:39:23.475335+0000","last_became_active":"2024-09-10T20:39:23.465867+0000","last_became_peered":"2024-09-10T20:39:23.465867+0000","last_unstale":"2024-09-10T20:39:23.475335+0000","last_undegraded":"2024-09-10T20:39:23.475335+0000","last_fullsized":"2024-09-10T20:39:23.475335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:51.591296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471738+0000","last_change":"2024-09-10T20:39:23.479313+0000","last_active":"2024-09-10T20:39:24.471738+0000","last_peered":"2024-09-10T20:39:24.471738+0000","last_clean":"2024-09-10T20:39:24.471738+0000","last_became_active":"2024-09-10T20:39:23.478782+0000","last_became_peered":"2024-09-10T20:39:23.478782+0000","last_unstale":"2024-09-10T20:39:24.471738+0000","last_undegraded":"2024-09-10T20:39:24.471738+0000","last_fullsized":"2024-09-10T20:39:24.471738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:32:50.720508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473143+0000","last_change":"2024-09-10T20:39:23.471294+0000","last_active":"2024-09-10T20:39:23.473143+0000","last_peered":"2024-09-10T20:39:23.473143+0000","last_clean":"2024-09-10T20:39:23.473143+0000","last_became_active":"2024-09-10T20:39:23.471002+0000","last_became_peered":"2024-09-10T20:39:23.471002+0000","last_unstale":"2024-09-10T20:39:23.473143+0000","last_undegraded":"2024-09-10T20:39:23.473143+0000","last_fullsized":"2024-09-10T20:39:23.473143+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:53.982272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252253+0000","last_change":"2024-09-10T20:39:23.472791+0000","last_active":"2024-09-10T20:39:25.252253+0000","last_peered":"2024-09-10T20:39:25.252253+0000","last_clean":"2024-09-10T20:39:25.252253+0000","last_became_active":"2024-09-10T20:39:23.472469+0000","last_became_peered":"2024-09-10T20:39:23.472469+0000","last_unstale":"2024-09-10T20:39:25.252253+0000","last_undegraded":"2024-09-10T20:39:25.252253+0000","last_fullsized":"2024-09-10T20:39:25.252253+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:37.756616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251230+0000","last_change":"2024-09-10T20:39:23.483420+0000","last_active":"2024-09-10T20:39:25.251230+0000","last_peered":"2024-09-10T20:39:25.251230+0000","last_clean":"2024-09-10T20:39:25.251230+0000","last_became_active":"2024-09-10T20:39:23.483170+0000","last_became_peered":"2024-09-10T20:39:23.483170+0000","last_unstale":"2024-09-10T20:39:25.251230+0000","last_undegraded":"2024-09-10T20:39:25.251230+0000","last_fullsized":"2024-09-10T20:39:25.251230+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814592+0000","last_change":"2024-09-10T20:39:23.483079+0000","last_active":"2024-09-10T20:39:24.814592+0000","last_peered":"2024-09-10T20:39:24.814592+0000","last_clean":"2024-09-10T20:39:24.814592+0000","last_became_active":"2024-09-10T20:39:23.482769+0000","last_became_peered":"2024-09-10T20:39:23.482769+0000","last_unstale":"2024-09-10T20:39:24.814592+0000","last_undegraded":"2024-09-10T20:39:24.814592+0000","last_fullsized":"2024-09-10T20:39:24.814592+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471486+0000","last_change":"2024-09-10T20:39:23.471724+0000","last_active":"2024-09-10T20:39:24.471486+0000","last_peered":"2024-09-10T20:39:24.471486+0000","last_clean":"2024-09-10T20:39:24.471486+0000","last_became_active":"2024-09-10T20:39:23.471453+0000","last_became_peered":"2024-09-10T20:39:23.471453+0000","last_unstale":"2024-09-10T20:39:24.471486+0000","last_undegraded":"2024-09-10T20:39:24.471486+0000","last_fullsized":"2024-09-10T20:39:24.471486+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:52.838472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252396+0000","last_change":"2024-09-10T20:39:23.474509+0000","last_active":"2024-09-10T20:39:25.252396+0000","last_peered":"2024-09-10T20:39:25.252396+0000","last_clean":"2024-09-10T20:39:25.252396+0000","last_became_active":"2024-09-10T20:39:23.474180+0000","last_became_peered":"2024-09-10T20:39:23.474180+0000","last_unstale":"2024-09-10T20:39:25.252396+0000","last_undegraded":"2024-09-10T20:39:25.252396+0000","last_fullsized":"2024-09-10T20:39:25.252396+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:19.254485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252368+0000","last_change":"2024-09-10T20:39:23.482090+0000","last_active":"2024-09-10T20:39:25.252368+0000","last_peered":"2024-09-10T20:39:25.252368+0000","last_clean":"2024-09-10T20:39:25.252368+0000","last_became_active":"2024-09-10T20:39:23.481803+0000","last_became_peered":"2024-09-10T20:39:23.481803+0000","last_unstale":"2024-09-10T20:39:25.252368+0000","last_undegraded":"2024-09-10T20:39:25.252368+0000","last_fullsized":"2024-09-10T20:39:25.252368+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:52.741446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814356+0000","last_change":"2024-09-10T20:39:23.473090+0000","last_active":"2024-09-10T20:39:24.814356+0000","last_peered":"2024-09-10T20:39:24.814356+0000","last_clean":"2024-09-10T20:39:24.814356+0000","last_became_active":"2024-09-10T20:39:23.472803+0000","last_became_peered":"2024-09-10T20:39:23.472803+0000","last_unstale":"2024-09-10T20:39:24.814356+0000","last_undegraded":"2024-09-10T20:39:24.814356+0000","last_fullsized":"2024-09-10T20:39:24.814356+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:22.229236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251535+0000","last_change":"2024-09-10T20:39:23.479620+0000","last_active":"2024-09-10T20:39:25.251535+0000","last_peered":"2024-09-10T20:39:25.251535+0000","last_clean":"2024-09-10T20:39:25.251535+0000","last_became_active":"2024-09-10T20:39:23.479319+0000","last_became_peered":"2024-09-10T20:39:23.479319+0000","last_unstale":"2024-09-10T20:39:25.251535+0000","last_undegraded":"2024-09-10T20:39:25.251535+0000","last_fullsized":"2024-09-10T20:39:25.251535+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469972+0000","last_change":"2024-09-10T20:39:23.469522+0000","last_active":"2024-09-10T20:39:24.469972+0000","last_peered":"2024-09-10T20:39:24.469972+0000","last_clean":"2024-09-10T20:39:24.469972+0000","last_became_active":"2024-09-10T20:39:23.469129+0000","last_became_peered":"2024-09-10T20:39:23.469129+0000","last_unstale":"2024-09-10T20:39:24.469972+0000","last_undegraded":"2024-09-10T20:39:24.469972+0000","last_fullsized":"2024-09-10T20:39:24.469972+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:28.627939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471378+0000","last_change":"2024-09-10T20:39:23.470278+0000","last_active":"2024-09-10T20:39:24.471378+0000","last_peered":"2024-09-10T20:39:24.471378+0000","last_clean":"2024-09-10T20:39:24.471378+0000","last_became_active":"2024-09-10T20:39:23.469632+0000","last_became_peered":"2024-09-10T20:39:23.469632+0000","last_unstale":"2024-09-10T20:39:24.471378+0000","last_undegraded":"2024-09-10T20:39:24.471378+0000","last_fullsized":"2024-09-10T20:39:24.471378+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.476954+0000","last_change":"2024-09-10T20:39:23.476296+0000","last_active":"2024-09-10T20:39:23.476954+0000","last_peered":"2024-09-10T20:39:23.476954+0000","last_clean":"2024-09-10T20:39:23.476954+0000","last_became_active":"2024-09-10T20:39:23.476009+0000","last_became_peered":"2024-09-10T20:39:23.476009+0000","last_unstale":"2024-09-10T20:39:23.476954+0000","last_undegraded":"2024-09-10T20:39:23.476954+0000","last_fullsized":"2024-09-10T20:39:23.476954+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:29:17.845671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":19,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470688+0000","last_change":"2024-09-10T20:39:22.471060+0000","last_active":"2024-09-10T20:39:24.470688+0000","last_peered":"2024-09-10T20:39:24.470688+0000","last_clean":"2024-09-10T20:39:24.470688+0000","last_became_active":"2024-09-10T20:39:22.470753+0000","last_became_peered":"2024-09-10T20:39:22.470753+0000","last_unstale":"2024-09-10T20:39:24.470688+0000","last_undegraded":"2024-09-10T20:39:24.470688+0000","last_fullsized":"2024-09-10T20:39:24.470688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:22.626422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814639+0000","last_change":"2024-09-10T20:39:23.470015+0000","last_active":"2024-09-10T20:39:24.814639+0000","last_peered":"2024-09-10T20:39:24.814639+0000","last_clean":"2024-09-10T20:39:24.814639+0000","last_became_active":"2024-09-10T20:39:23.469693+0000","last_became_peered":"2024-09-10T20:39:23.469693+0000","last_unstale":"2024-09-10T20:39:24.814639+0000","last_undegraded":"2024-09-10T20:39:24.814639+0000","last_fullsized":"2024-09-10T20:39:24.814639+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252307+0000","last_change":"2024-09-10T20:39:22.461136+0000","last_active":"2024-09-10T20:39:25.252307+0000","last_peered":"2024-09-10T20:39:25.252307+0000","last_clean":"2024-09-10T20:39:25.252307+0000","last_became_active":"2024-09-10T20:39:22.460848+0000","last_became_peered":"2024-09-10T20:39:22.460848+0000","last_unstale":"2024-09-10T20:39:25.252307+0000","last_undegraded":"2024-09-10T20:39:25.252307+0000","last_fullsized":"2024-09-10T20:39:25.252307+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:18.873047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251912+0000","last_change":"2024-09-10T20:39:23.481110+0000","last_active":"2024-09-10T20:39:25.251912+0000","last_peered":"2024-09-10T20:39:25.251912+0000","last_clean":"2024-09-10T20:39:25.251912+0000","last_became_active":"2024-09-10T20:39:23.480241+0000","last_became_peered":"2024-09-10T20:39:23.480241+0000","last_unstale":"2024-09-10T20:39:25.251912+0000","last_undegraded":"2024-09-10T20:39:25.251912+0000","last_fullsized":"2024-09-10T20:39:25.251912+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:29.489296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470292+0000","last_change":"2024-09-10T20:39:22.467354+0000","last_active":"2024-09-10T20:39:24.470292+0000","last_peered":"2024-09-10T20:39:24.470292+0000","last_clean":"2024-09-10T20:39:24.470292+0000","last_became_active":"2024-09-10T20:39:22.467105+0000","last_became_peered":"2024-09-10T20:39:22.467105+0000","last_unstale":"2024-09-10T20:39:24.470292+0000","last_undegraded":"2024-09-10T20:39:24.470292+0000","last_fullsized":"2024-09-10T20:39:24.470292+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:42.142331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471995+0000","last_change":"2024-09-10T20:39:23.465129+0000","last_active":"2024-09-10T20:39:24.471995+0000","last_peered":"2024-09-10T20:39:24.471995+0000","last_clean":"2024-09-10T20:39:24.471995+0000","last_became_active":"2024-09-10T20:39:23.464816+0000","last_became_peered":"2024-09-10T20:39:23.464816+0000","last_unstale":"2024-09-10T20:39:24.471995+0000","last_undegraded":"2024-09-10T20:39:24.471995+0000","last_fullsized":"2024-09-10T20:39:24.471995+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:06.505948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252472+0000","last_change":"2024-09-10T20:39:22.463145+0000","last_active":"2024-09-10T20:39:25.252472+0000","last_peered":"2024-09-10T20:39:25.252472+0000","last_clean":"2024-09-10T20:39:25.252472+0000","last_became_active":"2024-09-10T20:39:22.462844+0000","last_became_peered":"2024-09-10T20:39:22.462844+0000","last_unstale":"2024-09-10T20:39:25.252472+0000","last_undegraded":"2024-09-10T20:39:25.252472+0000","last_fullsized":"2024-09-10T20:39:25.252472+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:42:40.187952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251873+0000","last_change":"2024-09-10T20:39:23.471689+0000","last_active":"2024-09-10T20:39:25.251873+0000","last_peered":"2024-09-10T20:39:25.251873+0000","last_clean":"2024-09-10T20:39:25.251873+0000","last_became_active":"2024-09-10T20:39:23.471333+0000","last_became_peered":"2024-09-10T20:39:23.471333+0000","last_unstale":"2024-09-10T20:39:25.251873+0000","last_undegraded":"2024-09-10T20:39:25.251873+0000","last_fullsized":"2024-09-10T20:39:25.251873+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:01:24.286087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470439+0000","last_change":"2024-09-10T20:39:22.468131+0000","last_active":"2024-09-10T20:39:24.470439+0000","last_peered":"2024-09-10T20:39:24.470439+0000","last_clean":"2024-09-10T20:39:24.470439+0000","last_became_active":"2024-09-10T20:39:22.467846+0000","last_became_peered":"2024-09-10T20:39:22.467846+0000","last_unstale":"2024-09-10T20:39:24.470439+0000","last_undegraded":"2024-09-10T20:39:24.470439+0000","last_fullsized":"2024-09-10T20:39:24.470439+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:40.914501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.480621+0000","last_change":"2024-09-10T20:39:23.477169+0000","last_active":"2024-09-10T20:39:23.480621+0000","last_peered":"2024-09-10T20:39:23.480621+0000","last_clean":"2024-09-10T20:39:23.480621+0000","last_became_active":"2024-09-10T20:39:23.476830+0000","last_became_peered":"2024-09-10T20:39:23.476830+0000","last_unstale":"2024-09-10T20:39:23.480621+0000","last_undegraded":"2024-09-10T20:39:23.480621+0000","last_fullsized":"2024-09-10T20:39:23.480621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:20:58.616528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251795+0000","last_change":"2024-09-10T20:39:22.468056+0000","last_active":"2024-09-10T20:39:25.251795+0000","last_peered":"2024-09-10T20:39:25.251795+0000","last_clean":"2024-09-10T20:39:25.251795+0000","last_became_active":"2024-09-10T20:39:22.467477+0000","last_became_peered":"2024-09-10T20:39:22.467477+0000","last_unstale":"2024-09-10T20:39:25.251795+0000","last_undegraded":"2024-09-10T20:39:25.251795+0000","last_fullsized":"2024-09-10T20:39:25.251795+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:45.140725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475067+0000","last_change":"2024-09-10T20:39:23.466496+0000","last_active":"2024-09-10T20:39:23.475067+0000","last_peered":"2024-09-10T20:39:23.475067+0000","last_clean":"2024-09-10T20:39:23.475067+0000","last_became_active":"2024-09-10T20:39:23.466162+0000","last_became_peered":"2024-09-10T20:39:23.466162+0000","last_unstale":"2024-09-10T20:39:23.475067+0000","last_undegraded":"2024-09-10T20:39:23.475067+0000","last_fullsized":"2024-09-10T20:39:23.475067+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:36.107439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814173+0000","last_change":"2024-09-10T20:39:22.469484+0000","last_active":"2024-09-10T20:39:24.814173+0000","last_peered":"2024-09-10T20:39:24.814173+0000","last_clean":"2024-09-10T20:39:24.814173+0000","last_became_active":"2024-09-10T20:39:22.469218+0000","last_became_peered":"2024-09-10T20:39:22.469218+0000","last_unstale":"2024-09-10T20:39:24.814173+0000","last_undegraded":"2024-09-10T20:39:24.814173+0000","last_fullsized":"2024-09-10T20:39:24.814173+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:42:09.052814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252173+0000","last_change":"2024-09-10T20:39:23.466343+0000","last_active":"2024-09-10T20:39:25.252173+0000","last_peered":"2024-09-10T20:39:25.252173+0000","last_clean":"2024-09-10T20:39:25.252173+0000","last_became_active":"2024-09-10T20:39:23.466061+0000","last_became_peered":"2024-09-10T20:39:23.466061+0000","last_unstale":"2024-09-10T20:39:25.252173+0000","last_undegraded":"2024-09-10T20:39:25.252173+0000","last_fullsized":"2024-09-10T20:39:25.252173+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.847458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460912+0000","last_change":"2024-09-10T20:39:22.464013+0000","last_active":"2024-09-10T20:39:23.460912+0000","last_peered":"2024-09-10T20:39:23.460912+0000","last_clean":"2024-09-10T20:39:23.460912+0000","last_became_active":"2024-09-10T20:39:22.463745+0000","last_became_peered":"2024-09-10T20:39:22.463745+0000","last_unstale":"2024-09-10T20:39:23.460912+0000","last_undegraded":"2024-09-10T20:39:23.460912+0000","last_fullsized":"2024-09-10T20:39:23.460912+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:17.407343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469738+0000","last_change":"2024-09-10T20:39:23.470667+0000","last_active":"2024-09-10T20:39:24.469738+0000","last_peered":"2024-09-10T20:39:24.469738+0000","last_clean":"2024-09-10T20:39:24.469738+0000","last_became_active":"2024-09-10T20:39:23.470425+0000","last_became_peered":"2024-09-10T20:39:23.470425+0000","last_unstale":"2024-09-10T20:39:24.469738+0000","last_undegraded":"2024-09-10T20:39:24.469738+0000","last_fullsized":"2024-09-10T20:39:24.469738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:08:02.455782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469712+0000","last_change":"2024-09-10T20:39:22.466186+0000","last_active":"2024-09-10T20:39:24.469712+0000","last_peered":"2024-09-10T20:39:24.469712+0000","last_clean":"2024-09-10T20:39:24.469712+0000","last_became_active":"2024-09-10T20:39:22.465718+0000","last_became_peered":"2024-09-10T20:39:22.465718+0000","last_unstale":"2024-09-10T20:39:24.469712+0000","last_undegraded":"2024-09-10T20:39:24.469712+0000","last_fullsized":"2024-09-10T20:39:24.469712+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:09.130187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469604+0000","last_change":"2024-09-10T20:39:23.474536+0000","last_active":"2024-09-10T20:39:24.469604+0000","last_peered":"2024-09-10T20:39:24.469604+0000","last_clean":"2024-09-10T20:39:24.469604+0000","last_became_active":"2024-09-10T20:39:23.474234+0000","last_became_peered":"2024-09-10T20:39:23.474234+0000","last_unstale":"2024-09-10T20:39:24.469604+0000","last_undegraded":"2024-09-10T20:39:24.469604+0000","last_fullsized":"2024-09-10T20:39:24.469604+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:38.533241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460348+0000","last_change":"2024-09-10T20:39:22.470606+0000","last_active":"2024-09-10T20:39:23.460348+0000","last_peered":"2024-09-10T20:39:23.460348+0000","last_clean":"2024-09-10T20:39:23.460348+0000","last_became_active":"2024-09-10T20:39:22.470313+0000","last_became_peered":"2024-09-10T20:39:22.470313+0000","last_unstale":"2024-09-10T20:39:23.460348+0000","last_undegraded":"2024-09-10T20:39:23.460348+0000","last_fullsized":"2024-09-10T20:39:23.460348+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:13.562757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471621+0000","last_change":"2024-09-10T20:39:23.478464+0000","last_active":"2024-09-10T20:39:24.471621+0000","last_peered":"2024-09-10T20:39:24.471621+0000","last_clean":"2024-09-10T20:39:24.471621+0000","last_became_active":"2024-09-10T20:39:23.477764+0000","last_became_peered":"2024-09-10T20:39:23.477764+0000","last_unstale":"2024-09-10T20:39:24.471621+0000","last_undegraded":"2024-09-10T20:39:24.471621+0000","last_fullsized":"2024-09-10T20:39:24.471621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:07.988361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"42'3","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251237+0000","last_change":"2024-09-10T20:39:22.466977+0000","last_active":"2024-09-10T20:39:25.251237+0000","last_peered":"2024-09-10T20:39:25.251237+0000","last_clean":"2024-09-10T20:39:25.251237+0000","last_became_active":"2024-09-10T20:39:22.466685+0000","last_became_peered":"2024-09-10T20:39:22.466685+0000","last_unstale":"2024-09-10T20:39:25.251237+0000","last_undegraded":"2024-09-10T20:39:25.251237+0000","last_fullsized":"2024-09-10T20:39:25.251237+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:44:01.537714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252311+0000","last_change":"2024-09-10T20:39:23.472408+0000","last_active":"2024-09-10T20:39:25.252311+0000","last_peered":"2024-09-10T20:39:25.252311+0000","last_clean":"2024-09-10T20:39:25.252311+0000","last_became_active":"2024-09-10T20:39:23.472123+0000","last_became_peered":"2024-09-10T20:39:23.472123+0000","last_unstale":"2024-09-10T20:39:25.252311+0000","last_undegraded":"2024-09-10T20:39:25.252311+0000","last_fullsized":"2024-09-10T20:39:25.252311+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:58:19.163029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":19,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.487847+0000","last_change":"2024-09-10T20:39:22.457688+0000","last_active":"2024-09-10T20:39:23.487847+0000","last_peered":"2024-09-10T20:39:23.487847+0000","last_clean":"2024-09-10T20:39:23.487847+0000","last_became_active":"2024-09-10T20:39:22.457326+0000","last_became_peered":"2024-09-10T20:39:22.457326+0000","last_unstale":"2024-09-10T20:39:23.487847+0000","last_undegraded":"2024-09-10T20:39:23.487847+0000","last_fullsized":"2024-09-10T20:39:23.487847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:51.098445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469742+0000","last_change":"2024-09-10T20:39:23.478920+0000","last_active":"2024-09-10T20:39:24.469742+0000","last_peered":"2024-09-10T20:39:24.469742+0000","last_clean":"2024-09-10T20:39:24.469742+0000","last_became_active":"2024-09-10T20:39:23.478555+0000","last_became_peered":"2024-09-10T20:39:23.478555+0000","last_unstale":"2024-09-10T20:39:24.469742+0000","last_undegraded":"2024-09-10T20:39:24.469742+0000","last_fullsized":"2024-09-10T20:39:24.469742+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:12:21.695423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.473204+0000","last_change":"2024-09-10T20:39:22.463919+0000","last_active":"2024-09-10T20:39:23.473204+0000","last_peered":"2024-09-10T20:39:23.473204+0000","last_clean":"2024-09-10T20:39:23.473204+0000","last_became_active":"2024-09-10T20:39:22.463650+0000","last_became_peered":"2024-09-10T20:39:22.463650+0000","last_unstale":"2024-09-10T20:39:23.473204+0000","last_undegraded":"2024-09-10T20:39:23.473204+0000","last_fullsized":"2024-09-10T20:39:23.473204+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:07:05.483324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484089+0000","last_change":"2024-09-10T20:39:23.482909+0000","last_active":"2024-09-10T20:39:23.484089+0000","last_peered":"2024-09-10T20:39:23.484089+0000","last_clean":"2024-09-10T20:39:23.484089+0000","last_became_active":"2024-09-10T20:39:23.482551+0000","last_became_peered":"2024-09-10T20:39:23.482551+0000","last_unstale":"2024-09-10T20:39:23.484089+0000","last_undegraded":"2024-09-10T20:39:23.484089+0000","last_fullsized":"2024-09-10T20:39:23.484089+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:10.837228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814152+0000","last_change":"2024-09-10T20:39:22.471268+0000","last_active":"2024-09-10T20:39:24.814152+0000","last_peered":"2024-09-10T20:39:24.814152+0000","last_clean":"2024-09-10T20:39:24.814152+0000","last_became_active":"2024-09-10T20:39:22.470115+0000","last_became_peered":"2024-09-10T20:39:22.470115+0000","last_unstale":"2024-09-10T20:39:24.814152+0000","last_undegraded":"2024-09-10T20:39:24.814152+0000","last_fullsized":"2024-09-10T20:39:24.814152+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.467882+0000","last_change":"2024-09-10T20:39:23.465467+0000","last_active":"2024-09-10T20:39:23.467882+0000","last_peered":"2024-09-10T20:39:23.467882+0000","last_clean":"2024-09-10T20:39:23.467882+0000","last_became_active":"2024-09-10T20:39:23.465165+0000","last_became_peered":"2024-09-10T20:39:23.465165+0000","last_unstale":"2024-09-10T20:39:23.467882+0000","last_undegraded":"2024-09-10T20:39:23.467882+0000","last_fullsized":"2024-09-10T20:39:23.467882+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:15.544019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469888+0000","last_change":"2024-09-10T20:39:22.471774+0000","last_active":"2024-09-10T20:39:24.469888+0000","last_peered":"2024-09-10T20:39:24.469888+0000","last_clean":"2024-09-10T20:39:24.469888+0000","last_became_active":"2024-09-10T20:39:22.471455+0000","last_became_peered":"2024-09-10T20:39:22.471455+0000","last_unstale":"2024-09-10T20:39:24.469888+0000","last_undegraded":"2024-09-10T20:39:24.469888+0000","last_fullsized":"2024-09-10T20:39:24.469888+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:31.755937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251215+0000","last_change":"2024-09-10T20:39:23.482713+0000","last_active":"2024-09-10T20:39:25.251215+0000","last_peered":"2024-09-10T20:39:25.251215+0000","last_clean":"2024-09-10T20:39:25.251215+0000","last_became_active":"2024-09-10T20:39:23.482442+0000","last_became_peered":"2024-09-10T20:39:23.482442+0000","last_unstale":"2024-09-10T20:39:25.251215+0000","last_undegraded":"2024-09-10T20:39:25.251215+0000","last_fullsized":"2024-09-10T20:39:25.251215+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":21,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.485737+0000","last_change":"2024-09-10T20:39:22.468116+0000","last_active":"2024-09-10T20:39:23.485737+0000","last_peered":"2024-09-10T20:39:23.485737+0000","last_clean":"2024-09-10T20:39:23.485737+0000","last_became_active":"2024-09-10T20:39:22.467712+0000","last_became_peered":"2024-09-10T20:39:22.467712+0000","last_unstale":"2024-09-10T20:39:23.485737+0000","last_undegraded":"2024-09-10T20:39:23.485737+0000","last_fullsized":"2024-09-10T20:39:23.485737+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:18:01.811366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.474598+0000","last_change":"2024-09-10T20:39:23.473728+0000","last_active":"2024-09-10T20:39:23.474598+0000","last_peered":"2024-09-10T20:39:23.474598+0000","last_clean":"2024-09-10T20:39:23.474598+0000","last_became_active":"2024-09-10T20:39:23.473482+0000","last_became_peered":"2024-09-10T20:39:23.473482+0000","last_unstale":"2024-09-10T20:39:23.474598+0000","last_undegraded":"2024-09-10T20:39:23.474598+0000","last_fullsized":"2024-09-10T20:39:23.474598+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:42:24.635057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471265+0000","last_change":"2024-09-10T20:39:22.464648+0000","last_active":"2024-09-10T20:39:24.471265+0000","last_peered":"2024-09-10T20:39:24.471265+0000","last_clean":"2024-09-10T20:39:24.471265+0000","last_became_active":"2024-09-10T20:39:22.464381+0000","last_became_peered":"2024-09-10T20:39:22.464381+0000","last_unstale":"2024-09-10T20:39:24.471265+0000","last_undegraded":"2024-09-10T20:39:24.471265+0000","last_fullsized":"2024-09-10T20:39:24.471265+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:34.596334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.469081+0000","last_change":"2024-09-10T20:39:23.466893+0000","last_active":"2024-09-10T20:39:23.469081+0000","last_peered":"2024-09-10T20:39:23.469081+0000","last_clean":"2024-09-10T20:39:23.469081+0000","last_became_active":"2024-09-10T20:39:23.466577+0000","last_became_peered":"2024-09-10T20:39:23.466577+0000","last_unstale":"2024-09-10T20:39:23.469081+0000","last_undegraded":"2024-09-10T20:39:23.469081+0000","last_fullsized":"2024-09-10T20:39:23.469081+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:50.331650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469893+0000","last_change":"2024-09-10T20:39:22.472535+0000","last_active":"2024-09-10T20:39:24.469893+0000","last_peered":"2024-09-10T20:39:24.469893+0000","last_clean":"2024-09-10T20:39:24.469893+0000","last_became_active":"2024-09-10T20:39:22.472220+0000","last_became_peered":"2024-09-10T20:39:22.472220+0000","last_unstale":"2024-09-10T20:39:24.469893+0000","last_undegraded":"2024-09-10T20:39:24.469893+0000","last_fullsized":"2024-09-10T20:39:24.469893+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:49.040049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251980+0000","last_change":"2024-09-10T20:39:23.480836+0000","last_active":"2024-09-10T20:39:25.251980+0000","last_peered":"2024-09-10T20:39:25.251980+0000","last_clean":"2024-09-10T20:39:25.251980+0000","last_became_active":"2024-09-10T20:39:23.479966+0000","last_became_peered":"2024-09-10T20:39:23.479966+0000","last_unstale":"2024-09-10T20:39:25.251980+0000","last_undegraded":"2024-09-10T20:39:25.251980+0000","last_fullsized":"2024-09-10T20:39:25.251980+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:35.149497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469794+0000","last_change":"2024-09-10T20:39:22.468576+0000","last_active":"2024-09-10T20:39:24.469794+0000","last_peered":"2024-09-10T20:39:24.469794+0000","last_clean":"2024-09-10T20:39:24.469794+0000","last_became_active":"2024-09-10T20:39:22.468301+0000","last_became_peered":"2024-09-10T20:39:22.468301+0000","last_unstale":"2024-09-10T20:39:24.469794+0000","last_undegraded":"2024-09-10T20:39:24.469794+0000","last_fullsized":"2024-09-10T20:39:24.469794+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:38.187418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469705+0000","last_change":"2024-09-10T20:39:23.480119+0000","last_active":"2024-09-10T20:39:24.469705+0000","last_peered":"2024-09-10T20:39:24.469705+0000","last_clean":"2024-09-10T20:39:24.469705+0000","last_became_active":"2024-09-10T20:39:23.479859+0000","last_became_peered":"2024-09-10T20:39:23.479859+0000","last_unstale":"2024-09-10T20:39:24.469705+0000","last_undegraded":"2024-09-10T20:39:24.469705+0000","last_fullsized":"2024-09-10T20:39:24.469705+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:45.569351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470125+0000","last_change":"2024-09-10T20:39:22.471454+0000","last_active":"2024-09-10T20:39:24.470125+0000","last_peered":"2024-09-10T20:39:24.470125+0000","last_clean":"2024-09-10T20:39:24.470125+0000","last_became_active":"2024-09-10T20:39:22.471186+0000","last_became_peered":"2024-09-10T20:39:22.471186+0000","last_unstale":"2024-09-10T20:39:24.470125+0000","last_undegraded":"2024-09-10T20:39:24.470125+0000","last_fullsized":"2024-09-10T20:39:24.470125+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:11.967787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479095+0000","last_change":"2024-09-10T20:39:23.474293+0000","last_active":"2024-09-10T20:39:23.479095+0000","last_peered":"2024-09-10T20:39:23.479095+0000","last_clean":"2024-09-10T20:39:23.479095+0000","last_became_active":"2024-09-10T20:39:23.474013+0000","last_became_peered":"2024-09-10T20:39:23.474013+0000","last_unstale":"2024-09-10T20:39:23.479095+0000","last_undegraded":"2024-09-10T20:39:23.479095+0000","last_fullsized":"2024-09-10T20:39:23.479095+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:39:00.145526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.484950+0000","last_change":"2024-09-10T20:39:22.467010+0000","last_active":"2024-09-10T20:39:23.484950+0000","last_peered":"2024-09-10T20:39:23.484950+0000","last_clean":"2024-09-10T20:39:23.484950+0000","last_became_active":"2024-09-10T20:39:22.466604+0000","last_became_peered":"2024-09-10T20:39:22.466604+0000","last_unstale":"2024-09-10T20:39:23.484950+0000","last_undegraded":"2024-09-10T20:39:23.484950+0000","last_fullsized":"2024-09-10T20:39:23.484950+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:33:38.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470167+0000","last_change":"2024-09-10T20:39:23.483419+0000","last_active":"2024-09-10T20:39:24.470167+0000","last_peered":"2024-09-10T20:39:24.470167+0000","last_clean":"2024-09-10T20:39:24.470167+0000","last_became_active":"2024-09-10T20:39:23.482573+0000","last_became_peered":"2024-09-10T20:39:23.482573+0000","last_unstale":"2024-09-10T20:39:24.470167+0000","last_undegraded":"2024-09-10T20:39:24.470167+0000","last_fullsized":"2024-09-10T20:39:24.470167+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814400+0000","last_change":"2024-09-10T20:39:22.472184+0000","last_active":"2024-09-10T20:39:24.814400+0000","last_peered":"2024-09-10T20:39:24.814400+0000","last_clean":"2024-09-10T20:39:24.814400+0000","last_became_active":"2024-09-10T20:39:22.471851+0000","last_became_peered":"2024-09-10T20:39:22.471851+0000","last_unstale":"2024-09-10T20:39:24.814400+0000","last_undegraded":"2024-09-10T20:39:24.814400+0000","last_fullsized":"2024-09-10T20:39:24.814400+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:06.279859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471576+0000","last_change":"2024-09-10T20:39:23.470437+0000","last_active":"2024-09-10T20:39:24.471576+0000","last_peered":"2024-09-10T20:39:24.471576+0000","last_clean":"2024-09-10T20:39:24.471576+0000","last_became_active":"2024-09-10T20:39:23.470018+0000","last_became_peered":"2024-09-10T20:39:23.470018+0000","last_unstale":"2024-09-10T20:39:24.471576+0000","last_undegraded":"2024-09-10T20:39:24.471576+0000","last_fullsized":"2024-09-10T20:39:24.471576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:19.864809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.462467+0000","last_change":"2024-09-10T20:39:22.470294+0000","last_active":"2024-09-10T20:39:23.462467+0000","last_peered":"2024-09-10T20:39:23.462467+0000","last_clean":"2024-09-10T20:39:23.462467+0000","last_became_active":"2024-09-10T20:39:22.469974+0000","last_became_peered":"2024-09-10T20:39:22.469974+0000","last_unstale":"2024-09-10T20:39:23.462467+0000","last_undegraded":"2024-09-10T20:39:23.462467+0000","last_fullsized":"2024-09-10T20:39:23.462467+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:14.321360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471788+0000","last_change":"2024-09-10T20:39:23.467237+0000","last_active":"2024-09-10T20:39:24.471788+0000","last_peered":"2024-09-10T20:39:24.471788+0000","last_clean":"2024-09-10T20:39:24.471788+0000","last_became_active":"2024-09-10T20:39:23.466955+0000","last_became_peered":"2024-09-10T20:39:23.466955+0000","last_unstale":"2024-09-10T20:39:24.471788+0000","last_undegraded":"2024-09-10T20:39:24.471788+0000","last_fullsized":"2024-09-10T20:39:24.471788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:17:53.617323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251301+0000","last_change":"2024-09-10T20:39:22.463388+0000","last_active":"2024-09-10T20:39:25.251301+0000","last_peered":"2024-09-10T20:39:25.251301+0000","last_clean":"2024-09-10T20:39:25.251301+0000","last_became_active":"2024-09-10T20:39:22.463050+0000","last_became_peered":"2024-09-10T20:39:22.463050+0000","last_unstale":"2024-09-10T20:39:25.251301+0000","last_undegraded":"2024-09-10T20:39:25.251301+0000","last_fullsized":"2024-09-10T20:39:25.251301+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:49.479271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252422+0000","last_change":"2024-09-10T20:39:23.466822+0000","last_active":"2024-09-10T20:39:25.252422+0000","last_peered":"2024-09-10T20:39:25.252422+0000","last_clean":"2024-09-10T20:39:25.252422+0000","last_became_active":"2024-09-10T20:39:23.466056+0000","last_became_peered":"2024-09-10T20:39:23.466056+0000","last_unstale":"2024-09-10T20:39:25.252422+0000","last_undegraded":"2024-09-10T20:39:25.252422+0000","last_fullsized":"2024-09-10T20:39:25.252422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:03.707588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471588+0000","last_change":"2024-09-10T20:39:22.466270+0000","last_active":"2024-09-10T20:39:24.471588+0000","last_peered":"2024-09-10T20:39:24.471588+0000","last_clean":"2024-09-10T20:39:24.471588+0000","last_became_active":"2024-09-10T20:39:22.465951+0000","last_became_peered":"2024-09-10T20:39:22.465951+0000","last_unstale":"2024-09-10T20:39:24.471588+0000","last_undegraded":"2024-09-10T20:39:24.471588+0000","last_fullsized":"2024-09-10T20:39:24.471588+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.475360+0000","last_change":"2024-09-10T20:39:23.468716+0000","last_active":"2024-09-10T20:39:23.475360+0000","last_peered":"2024-09-10T20:39:23.475360+0000","last_clean":"2024-09-10T20:39:23.475360+0000","last_became_active":"2024-09-10T20:39:23.468450+0000","last_became_peered":"2024-09-10T20:39:23.468450+0000","last_unstale":"2024-09-10T20:39:23.475360+0000","last_undegraded":"2024-09-10T20:39:23.475360+0000","last_fullsized":"2024-09-10T20:39:23.475360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:00.101321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814372+0000","last_change":"2024-09-10T20:39:22.470970+0000","last_active":"2024-09-10T20:39:24.814372+0000","last_peered":"2024-09-10T20:39:24.814372+0000","last_clean":"2024-09-10T20:39:24.814372+0000","last_became_active":"2024-09-10T20:39:22.470662+0000","last_became_peered":"2024-09-10T20:39:22.470662+0000","last_unstale":"2024-09-10T20:39:24.814372+0000","last_undegraded":"2024-09-10T20:39:24.814372+0000","last_fullsized":"2024-09-10T20:39:24.814372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:34:54.628648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252557+0000","last_change":"2024-09-10T20:39:23.476329+0000","last_active":"2024-09-10T20:39:25.252557+0000","last_peered":"2024-09-10T20:39:25.252557+0000","last_clean":"2024-09-10T20:39:25.252557+0000","last_became_active":"2024-09-10T20:39:23.476062+0000","last_became_peered":"2024-09-10T20:39:23.476062+0000","last_unstale":"2024-09-10T20:39:25.252557+0000","last_undegraded":"2024-09-10T20:39:25.252557+0000","last_fullsized":"2024-09-10T20:39:25.252557+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:16:42.873397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.460877+0000","last_change":"2024-09-10T20:39:22.467181+0000","last_active":"2024-09-10T20:39:23.460877+0000","last_peered":"2024-09-10T20:39:23.460877+0000","last_clean":"2024-09-10T20:39:23.460877+0000","last_became_active":"2024-09-10T20:39:22.466859+0000","last_became_peered":"2024-09-10T20:39:22.466859+0000","last_unstale":"2024-09-10T20:39:23.460877+0000","last_undegraded":"2024-09-10T20:39:23.460877+0000","last_fullsized":"2024-09-10T20:39:23.460877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:54.024056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.479036+0000","last_change":"2024-09-10T20:39:23.473568+0000","last_active":"2024-09-10T20:39:23.479036+0000","last_peered":"2024-09-10T20:39:23.479036+0000","last_clean":"2024-09-10T20:39:23.479036+0000","last_became_active":"2024-09-10T20:39:23.473257+0000","last_became_peered":"2024-09-10T20:39:23.473257+0000","last_unstale":"2024-09-10T20:39:23.479036+0000","last_undegraded":"2024-09-10T20:39:23.479036+0000","last_fullsized":"2024-09-10T20:39:23.479036+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:02.160411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814407+0000","last_change":"2024-09-10T20:39:22.467492+0000","last_active":"2024-09-10T20:39:24.814407+0000","last_peered":"2024-09-10T20:39:24.814407+0000","last_clean":"2024-09-10T20:39:24.814407+0000","last_became_active":"2024-09-10T20:39:22.467217+0000","last_became_peered":"2024-09-10T20:39:22.467217+0000","last_unstale":"2024-09-10T20:39:24.814407+0000","last_undegraded":"2024-09-10T20:39:24.814407+0000","last_fullsized":"2024-09-10T20:39:24.814407+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:32.173179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471424+0000","last_change":"2024-09-10T20:39:23.482739+0000","last_active":"2024-09-10T20:39:24.471424+0000","last_peered":"2024-09-10T20:39:24.471424+0000","last_clean":"2024-09-10T20:39:24.471424+0000","last_became_active":"2024-09-10T20:39:23.482483+0000","last_became_peered":"2024-09-10T20:39:23.482483+0000","last_unstale":"2024-09-10T20:39:24.471424+0000","last_undegraded":"2024-09-10T20:39:24.471424+0000","last_fullsized":"2024-09-10T20:39:24.471424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:53.453242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251921+0000","last_change":"2024-09-10T20:39:22.466109+0000","last_active":"2024-09-10T20:39:25.251921+0000","last_peered":"2024-09-10T20:39:25.251921+0000","last_clean":"2024-09-10T20:39:25.251921+0000","last_became_active":"2024-09-10T20:39:22.463916+0000","last_became_peered":"2024-09-10T20:39:22.463916+0000","last_unstale":"2024-09-10T20:39:25.251921+0000","last_undegraded":"2024-09-10T20:39:25.251921+0000","last_fullsized":"2024-09-10T20:39:25.251921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:19.681143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469354+0000","last_change":"2024-09-10T20:39:23.479485+0000","last_active":"2024-09-10T20:39:24.469354+0000","last_peered":"2024-09-10T20:39:24.469354+0000","last_clean":"2024-09-10T20:39:24.469354+0000","last_became_active":"2024-09-10T20:39:23.479219+0000","last_became_peered":"2024-09-10T20:39:23.479219+0000","last_unstale":"2024-09-10T20:39:24.469354+0000","last_undegraded":"2024-09-10T20:39:24.469354+0000","last_fullsized":"2024-09-10T20:39:24.469354+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:22.900530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469982+0000","last_change":"2024-09-10T20:39:22.464748+0000","last_active":"2024-09-10T20:39:24.469982+0000","last_peered":"2024-09-10T20:39:24.469982+0000","last_clean":"2024-09-10T20:39:24.469982+0000","last_became_active":"2024-09-10T20:39:22.464248+0000","last_became_peered":"2024-09-10T20:39:22.464248+0000","last_unstale":"2024-09-10T20:39:24.469982+0000","last_undegraded":"2024-09-10T20:39:24.469982+0000","last_fullsized":"2024-09-10T20:39:24.469982+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:34:33.491750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251582+0000","last_change":"2024-09-10T20:39:23.482269+0000","last_active":"2024-09-10T20:39:25.251582+0000","last_peered":"2024-09-10T20:39:25.251582+0000","last_clean":"2024-09-10T20:39:25.251582+0000","last_became_active":"2024-09-10T20:39:23.481990+0000","last_became_peered":"2024-09-10T20:39:23.481990+0000","last_unstale":"2024-09-10T20:39:25.251582+0000","last_undegraded":"2024-09-10T20:39:25.251582+0000","last_fullsized":"2024-09-10T20:39:25.251582+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:40.821209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-10T20:39:23.459867+0000","last_change":"2024-09-10T20:39:22.465000+0000","last_active":"2024-09-10T20:39:23.459867+0000","last_peered":"2024-09-10T20:39:23.459867+0000","last_clean":"2024-09-10T20:39:23.459867+0000","last_became_active":"2024-09-10T20:39:22.464669+0000","last_became_peered":"2024-09-10T20:39:22.464669+0000","last_unstale":"2024-09-10T20:39:23.459867+0000","last_undegraded":"2024-09-10T20:39:23.459867+0000","last_fullsized":"2024-09-10T20:39:23.459867+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:43:06.255942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471722+0000","last_change":"2024-09-10T20:39:23.476662+0000","last_active":"2024-09-10T20:39:24.471722+0000","last_peered":"2024-09-10T20:39:24.471722+0000","last_clean":"2024-09-10T20:39:24.471722+0000","last_became_active":"2024-09-10T20:39:23.476375+0000","last_became_peered":"2024-09-10T20:39:23.476375+0000","last_unstale":"2024-09-10T20:39:24.471722+0000","last_undegraded":"2024-09-10T20:39:24.471722+0000","last_fullsized":"2024-09-10T20:39:24.471722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:46.142798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252456+0000","last_change":"2024-09-10T20:39:22.464857+0000","last_active":"2024-09-10T20:39:25.252456+0000","last_peered":"2024-09-10T20:39:25.252456+0000","last_clean":"2024-09-10T20:39:25.252456+0000","last_became_active":"2024-09-10T20:39:22.464583+0000","last_became_peered":"2024-09-10T20:39:22.464583+0000","last_unstale":"2024-09-10T20:39:25.252456+0000","last_undegraded":"2024-09-10T20:39:25.252456+0000","last_fullsized":"2024-09-10T20:39:25.252456+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:15.049485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":149,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469244+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:39:24.469244+0000","last_peered":"2024-09-10T20:39:24.469244+0000","last_clean":"2024-09-10T20:39:24.469244+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:39:24.469244+0000","last_undegraded":"2024-09-10T20:39:24.469244+0000","last_fullsized":"2024-09-10T20:39:24.469244+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509461,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":507904,"data_stored":200539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509460,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":499712,"data_stored":199017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1093632,"data_stored":789365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509460,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":503808,"data_stored":199599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":491520,"data_stored":193243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1093632,"data_stored":789317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":503808,"data_stored":194331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:39:29.429 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-10T20:39:29.425+0000 7f7ffd39f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:39:30.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:39:30.940 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:39:31.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:39:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/769409565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 msgr2=0x7f4f6815c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/769409565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c200 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4f680b2fd0 tx=0x7f4f5802a560 comp rx=0 tx=0).stop 2024-09-10T20:39:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/769409565 shutdown_connections 2024-09-10T20:39:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/769409565 >> v2:172.21.15.159:3300/0 conn(0x7f4f680acf20 0x7f4f6806eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/769409565 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f681564e0 0x7f4f68072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/769409565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c200 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/769409565 >> 172.21.15.28:0/769409565 conn(0x7f4f680b1c40 msgr2=0x7f4f680b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/769409565 shutdown_connections 2024-09-10T20:39:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.102+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/769409565 wait complete. 2024-09-10T20:39:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.104+0000 7f4f6cd9c640 1 Processor -- start 2024-09-10T20:39:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.104+0000 7f4f6cd9c640 1 -- start start 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6cd9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6cd9c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f680acf20 0x7f4f680b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6cd9c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f681564e0 0x7f4f681e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6cd9c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f680aefb0 con 0x7f4f68072b50 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6cd9c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6806f260 con 0x7f4f681564e0 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6cd9c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4f680af130 con 0x7f4f680acf20 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46342/0 (socket says 172.21.15.28:46342) 2024-09-10T20:39:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 -- 172.21.15.28:0/3090006010 learned_addr learned my addr 172.21.15.28:0/3090006010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f66d9d640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f681564e0 0x7f4f681e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f65d9b640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f680acf20 0x7f4f680b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 -- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f680acf20 msgr2=0x7f4f680b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f680acf20 0x7f4f680b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 -- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f681564e0 msgr2=0x7f4f681e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f681564e0 0x7f4f681e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 -- 172.21.15.28:0/3090006010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f681f4e50 con 0x7f4f68072b50 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f66d9d640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f681564e0 0x7f4f681e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f6659c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c1d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4f58000c00 tx=0x7f4f58003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.105+0000 7f4f56ffd640 1 -- 172.21.15.28:0/3090006010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5803f070 con 0x7f4f68072b50 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.106+0000 7f4f56ffd640 1 -- 172.21.15.28:0/3090006010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f580047a0 con 0x7f4f68072b50 2024-09-10T20:39:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.106+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f681f5130 con 0x7f4f68072b50 2024-09-10T20:39:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.106+0000 7f4f56ffd640 1 -- 172.21.15.28:0/3090006010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58035030 con 0x7f4f68072b50 2024-09-10T20:39:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.106+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f681f56f0 con 0x7f4f68072b50 2024-09-10T20:39:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.107+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f680af690 con 0x7f4f68072b50 2024-09-10T20:39:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.107+0000 7f4f56ffd640 1 -- 172.21.15.28:0/3090006010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f4f58004bb0 con 0x7f4f68072b50 2024-09-10T20:39:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.108+0000 7f4f56ffd640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4f48078120 0x7f4f4807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.108+0000 7f4f56ffd640 1 -- 172.21.15.28:0/3090006010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f4f580bb330 con 0x7f4f68072b50 2024-09-10T20:39:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.108+0000 7f4f65d9b640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4f48078120 0x7f4f4807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.109+0000 7f4f65d9b640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4f48078120 0x7f4f4807a5e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c005fd0 tx=0x7f4f5c005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:32.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.112+0000 7f4f56ffd640 1 -- 172.21.15.28:0/3090006010 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4f58083aa0 con 0x7f4f68072b50 2024-09-10T20:39:32.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.294+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4f68070960 con 0x7f4f48078120 2024-09-10T20:39:32.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.299+0000 7f4f56ffd640 1 -- 172.21.15.28:0/3090006010 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251859 (secure 0 0 0) 0x7f4f68070960 con 0x7f4f48078120 2024-09-10T20:39:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:32.306 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4f48078120 msgr2=0x7f4f4807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4f48078120 0x7f4f4807a5e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c005fd0 tx=0x7f4f5c005850 comp rx=0 tx=0).stop 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 msgr2=0x7f4f6815c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c1d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4f58000c00 tx=0x7f4f58003910 comp rx=0 tx=0).stop 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 shutdown_connections 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4f48078120 0x7f4f4807a5e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f681564e0 0x7f4f681e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f680acf20 0x7f4f680b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 --2- 172.21.15.28:0/3090006010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f68072b50 0x7f4f6815c1d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 >> 172.21.15.28:0/3090006010 conn(0x7f4f680b1c40 msgr2=0x7f4f6815a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 shutdown_connections 2024-09-10T20:39:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.302+0000 7f4f6cd9c640 1 -- 172.21.15.28:0/3090006010 wait complete. 2024-09-10T20:39:32.317 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-10T20:39:31.992692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237976,"kb_used_data":5096,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736744,"statfs":{"total":773094113280,"available":772850425856,"internally_reserved":0,"allocated":5218304,"data_stored":2776972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.539170"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814513+0000","last_change":"2024-09-10T20:39:23.483860+0000","last_active":"2024-09-10T20:39:24.814513+0000","last_peered":"2024-09-10T20:39:24.814513+0000","last_clean":"2024-09-10T20:39:24.814513+0000","last_became_active":"2024-09-10T20:39:23.483590+0000","last_became_peered":"2024-09-10T20:39:23.483590+0000","last_unstale":"2024-09-10T20:39:24.814513+0000","last_undegraded":"2024-09-10T20:39:24.814513+0000","last_fullsized":"2024-09-10T20:39:24.814513+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251574+0000","last_change":"2024-09-10T20:39:23.476103+0000","last_active":"2024-09-10T20:39:25.251574+0000","last_peered":"2024-09-10T20:39:25.251574+0000","last_clean":"2024-09-10T20:39:25.251574+0000","last_became_active":"2024-09-10T20:39:23.475659+0000","last_became_peered":"2024-09-10T20:39:23.475659+0000","last_unstale":"2024-09-10T20:39:25.251574+0000","last_undegraded":"2024-09-10T20:39:25.251574+0000","last_fullsized":"2024-09-10T20:39:25.251574+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:01.324112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469676+0000","last_change":"2024-09-10T20:39:23.479367+0000","last_active":"2024-09-10T20:39:24.469676+0000","last_peered":"2024-09-10T20:39:24.469676+0000","last_clean":"2024-09-10T20:39:24.469676+0000","last_became_active":"2024-09-10T20:39:23.479089+0000","last_became_peered":"2024-09-10T20:39:23.479089+0000","last_unstale":"2024-09-10T20:39:24.469676+0000","last_undegraded":"2024-09-10T20:39:24.469676+0000","last_fullsized":"2024-09-10T20:39:24.469676+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:43:14.607190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471413+0000","last_change":"2024-09-10T20:39:23.474482+0000","last_active":"2024-09-10T20:39:24.471413+0000","last_peered":"2024-09-10T20:39:24.471413+0000","last_clean":"2024-09-10T20:39:24.471413+0000","last_became_active":"2024-09-10T20:39:23.469443+0000","last_became_peered":"2024-09-10T20:39:23.469443+0000","last_unstale":"2024-09-10T20:39:24.471413+0000","last_undegraded":"2024-09-10T20:39:24.471413+0000","last_fullsized":"2024-09-10T20:39:24.471413+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814280+0000","last_change":"2024-09-10T20:39:23.469934+0000","last_active":"2024-09-10T20:39:24.814280+0000","last_peered":"2024-09-10T20:39:24.814280+0000","last_clean":"2024-09-10T20:39:24.814280+0000","last_became_active":"2024-09-10T20:39:23.469620+0000","last_became_peered":"2024-09-10T20:39:23.469620+0000","last_unstale":"2024-09-10T20:39:24.814280+0000","last_undegraded":"2024-09-10T20:39:24.814280+0000","last_fullsized":"2024-09-10T20:39:24.814280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:43.501230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470522+0000","last_change":"2024-09-10T20:39:23.478562+0000","last_active":"2024-09-10T20:39:24.470522+0000","last_peered":"2024-09-10T20:39:24.470522+0000","last_clean":"2024-09-10T20:39:24.470522+0000","last_became_active":"2024-09-10T20:39:23.478273+0000","last_became_peered":"2024-09-10T20:39:23.478273+0000","last_unstale":"2024-09-10T20:39:24.470522+0000","last_undegraded":"2024-09-10T20:39:24.470522+0000","last_fullsized":"2024-09-10T20:39:24.470522+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:06:13.043952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251327+0000","last_change":"2024-09-10T20:39:23.475249+0000","last_active":"2024-09-10T20:39:25.251327+0000","last_peered":"2024-09-10T20:39:25.251327+0000","last_clean":"2024-09-10T20:39:25.251327+0000","last_became_active":"2024-09-10T20:39:23.474917+0000","last_became_peered":"2024-09-10T20:39:23.474917+0000","last_unstale":"2024-09-10T20:39:25.251327+0000","last_undegraded":"2024-09-10T20:39:25.251327+0000","last_fullsized":"2024-09-10T20:39:25.251327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:32.157058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471730+0000","last_change":"2024-09-10T20:39:23.477481+0000","last_active":"2024-09-10T20:39:24.471730+0000","last_peered":"2024-09-10T20:39:24.471730+0000","last_clean":"2024-09-10T20:39:24.471730+0000","last_became_active":"2024-09-10T20:39:23.477194+0000","last_became_peered":"2024-09-10T20:39:23.477194+0000","last_unstale":"2024-09-10T20:39:24.471730+0000","last_undegraded":"2024-09-10T20:39:24.471730+0000","last_fullsized":"2024-09-10T20:39:24.471730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:16.051559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470973+0000","last_change":"2024-09-10T20:39:23.465809+0000","last_active":"2024-09-10T20:39:24.470973+0000","last_peered":"2024-09-10T20:39:24.470973+0000","last_clean":"2024-09-10T20:39:24.470973+0000","last_became_active":"2024-09-10T20:39:23.465540+0000","last_became_peered":"2024-09-10T20:39:23.465540+0000","last_unstale":"2024-09-10T20:39:24.470973+0000","last_undegraded":"2024-09-10T20:39:24.470973+0000","last_fullsized":"2024-09-10T20:39:24.470973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252465+0000","last_change":"2024-09-10T20:39:23.480200+0000","last_active":"2024-09-10T20:39:25.252465+0000","last_peered":"2024-09-10T20:39:25.252465+0000","last_clean":"2024-09-10T20:39:25.252465+0000","last_became_active":"2024-09-10T20:39:23.479928+0000","last_became_peered":"2024-09-10T20:39:23.479928+0000","last_unstale":"2024-09-10T20:39:25.252465+0000","last_undegraded":"2024-09-10T20:39:25.252465+0000","last_fullsized":"2024-09-10T20:39:25.252465+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:59:35.212040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471901+0000","last_change":"2024-09-10T20:39:23.472464+0000","last_active":"2024-09-10T20:39:24.471901+0000","last_peered":"2024-09-10T20:39:24.471901+0000","last_clean":"2024-09-10T20:39:24.471901+0000","last_became_active":"2024-09-10T20:39:23.472185+0000","last_became_peered":"2024-09-10T20:39:23.472185+0000","last_unstale":"2024-09-10T20:39:24.471901+0000","last_undegraded":"2024-09-10T20:39:24.471901+0000","last_fullsized":"2024-09-10T20:39:24.471901+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:16:50.093970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251720+0000","last_change":"2024-09-10T20:39:23.481383+0000","last_active":"2024-09-10T20:39:25.251720+0000","last_peered":"2024-09-10T20:39:25.251720+0000","last_clean":"2024-09-10T20:39:25.251720+0000","last_became_active":"2024-09-10T20:39:23.480519+0000","last_became_peered":"2024-09-10T20:39:23.480519+0000","last_unstale":"2024-09-10T20:39:25.251720+0000","last_undegraded":"2024-09-10T20:39:25.251720+0000","last_fullsized":"2024-09-10T20:39:25.251720+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:06:51.852363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470189+0000","last_change":"2024-09-10T20:39:23.479455+0000","last_active":"2024-09-10T20:39:24.470189+0000","last_peered":"2024-09-10T20:39:24.470189+0000","last_clean":"2024-09-10T20:39:24.470189+0000","last_became_active":"2024-09-10T20:39:23.479185+0000","last_became_peered":"2024-09-10T20:39:23.479185+0000","last_unstale":"2024-09-10T20:39:24.470189+0000","last_undegraded":"2024-09-10T20:39:24.470189+0000","last_fullsized":"2024-09-10T20:39:24.470189+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471366+0000","last_change":"2024-09-10T20:39:23.472528+0000","last_active":"2024-09-10T20:39:24.471366+0000","last_peered":"2024-09-10T20:39:24.471366+0000","last_clean":"2024-09-10T20:39:24.471366+0000","last_became_active":"2024-09-10T20:39:23.472247+0000","last_became_peered":"2024-09-10T20:39:23.472247+0000","last_unstale":"2024-09-10T20:39:24.471366+0000","last_undegraded":"2024-09-10T20:39:24.471366+0000","last_fullsized":"2024-09-10T20:39:24.471366+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:19.989819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471460+0000","last_change":"2024-09-10T20:39:23.477260+0000","last_active":"2024-09-10T20:39:24.471460+0000","last_peered":"2024-09-10T20:39:24.471460+0000","last_clean":"2024-09-10T20:39:24.471460+0000","last_became_active":"2024-09-10T20:39:23.476962+0000","last_became_peered":"2024-09-10T20:39:23.476962+0000","last_unstale":"2024-09-10T20:39:24.471460+0000","last_undegraded":"2024-09-10T20:39:24.471460+0000","last_fullsized":"2024-09-10T20:39:24.471460+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814151+0000","last_change":"2024-09-10T20:39:23.482868+0000","last_active":"2024-09-10T20:39:24.814151+0000","last_peered":"2024-09-10T20:39:24.814151+0000","last_clean":"2024-09-10T20:39:24.814151+0000","last_became_active":"2024-09-10T20:39:23.482352+0000","last_became_peered":"2024-09-10T20:39:23.482352+0000","last_unstale":"2024-09-10T20:39:24.814151+0000","last_undegraded":"2024-09-10T20:39:24.814151+0000","last_fullsized":"2024-09-10T20:39:24.814151+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:10.014292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251846+0000","last_change":"2024-09-10T20:39:23.481835+0000","last_active":"2024-09-10T20:39:25.251846+0000","last_peered":"2024-09-10T20:39:25.251846+0000","last_clean":"2024-09-10T20:39:25.251846+0000","last_became_active":"2024-09-10T20:39:23.481462+0000","last_became_peered":"2024-09-10T20:39:23.481462+0000","last_unstale":"2024-09-10T20:39:25.251846+0000","last_undegraded":"2024-09-10T20:39:25.251846+0000","last_fullsized":"2024-09-10T20:39:25.251846+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:37:09.072417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469826+0000","last_change":"2024-09-10T20:39:23.478525+0000","last_active":"2024-09-10T20:39:24.469826+0000","last_peered":"2024-09-10T20:39:24.469826+0000","last_clean":"2024-09-10T20:39:24.469826+0000","last_became_active":"2024-09-10T20:39:23.478217+0000","last_became_peered":"2024-09-10T20:39:23.478217+0000","last_unstale":"2024-09-10T20:39:24.469826+0000","last_undegraded":"2024-09-10T20:39:24.469826+0000","last_fullsized":"2024-09-10T20:39:24.469826+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:28:10.445472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814465+0000","last_change":"2024-09-10T20:39:23.471412+0000","last_active":"2024-09-10T20:39:24.814465+0000","last_peered":"2024-09-10T20:39:24.814465+0000","last_clean":"2024-09-10T20:39:24.814465+0000","last_became_active":"2024-09-10T20:39:23.471147+0000","last_became_peered":"2024-09-10T20:39:23.471147+0000","last_unstale":"2024-09-10T20:39:24.814465+0000","last_undegraded":"2024-09-10T20:39:24.814465+0000","last_fullsized":"2024-09-10T20:39:24.814465+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:23:50.789861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252651+0000","last_change":"2024-09-10T20:39:23.467260+0000","last_active":"2024-09-10T20:39:25.252651+0000","last_peered":"2024-09-10T20:39:25.252651+0000","last_clean":"2024-09-10T20:39:25.252651+0000","last_became_active":"2024-09-10T20:39:23.465867+0000","last_became_peered":"2024-09-10T20:39:23.465867+0000","last_unstale":"2024-09-10T20:39:25.252651+0000","last_undegraded":"2024-09-10T20:39:25.252651+0000","last_fullsized":"2024-09-10T20:39:25.252651+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:32:53.573414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471738+0000","last_change":"2024-09-10T20:39:23.479313+0000","last_active":"2024-09-10T20:39:24.471738+0000","last_peered":"2024-09-10T20:39:24.471738+0000","last_clean":"2024-09-10T20:39:24.471738+0000","last_became_active":"2024-09-10T20:39:23.478782+0000","last_became_peered":"2024-09-10T20:39:23.478782+0000","last_unstale":"2024-09-10T20:39:24.471738+0000","last_undegraded":"2024-09-10T20:39:24.471738+0000","last_fullsized":"2024-09-10T20:39:24.471738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:32:50.720508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252058+0000","last_change":"2024-09-10T20:39:23.471294+0000","last_active":"2024-09-10T20:39:25.252058+0000","last_peered":"2024-09-10T20:39:25.252058+0000","last_clean":"2024-09-10T20:39:25.252058+0000","last_became_active":"2024-09-10T20:39:23.471002+0000","last_became_peered":"2024-09-10T20:39:23.471002+0000","last_unstale":"2024-09-10T20:39:25.252058+0000","last_undegraded":"2024-09-10T20:39:25.252058+0000","last_fullsized":"2024-09-10T20:39:25.252058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:17.519055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252253+0000","last_change":"2024-09-10T20:39:23.472791+0000","last_active":"2024-09-10T20:39:25.252253+0000","last_peered":"2024-09-10T20:39:25.252253+0000","last_clean":"2024-09-10T20:39:25.252253+0000","last_became_active":"2024-09-10T20:39:23.472469+0000","last_became_peered":"2024-09-10T20:39:23.472469+0000","last_unstale":"2024-09-10T20:39:25.252253+0000","last_undegraded":"2024-09-10T20:39:25.252253+0000","last_fullsized":"2024-09-10T20:39:25.252253+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:37.756616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251230+0000","last_change":"2024-09-10T20:39:23.483420+0000","last_active":"2024-09-10T20:39:25.251230+0000","last_peered":"2024-09-10T20:39:25.251230+0000","last_clean":"2024-09-10T20:39:25.251230+0000","last_became_active":"2024-09-10T20:39:23.483170+0000","last_became_peered":"2024-09-10T20:39:23.483170+0000","last_unstale":"2024-09-10T20:39:25.251230+0000","last_undegraded":"2024-09-10T20:39:25.251230+0000","last_fullsized":"2024-09-10T20:39:25.251230+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:51.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814592+0000","last_change":"2024-09-10T20:39:23.483079+0000","last_active":"2024-09-10T20:39:24.814592+0000","last_peered":"2024-09-10T20:39:24.814592+0000","last_clean":"2024-09-10T20:39:24.814592+0000","last_became_active":"2024-09-10T20:39:23.482769+0000","last_became_peered":"2024-09-10T20:39:23.482769+0000","last_unstale":"2024-09-10T20:39:24.814592+0000","last_undegraded":"2024-09-10T20:39:24.814592+0000","last_fullsized":"2024-09-10T20:39:24.814592+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471486+0000","last_change":"2024-09-10T20:39:23.471724+0000","last_active":"2024-09-10T20:39:24.471486+0000","last_peered":"2024-09-10T20:39:24.471486+0000","last_clean":"2024-09-10T20:39:24.471486+0000","last_became_active":"2024-09-10T20:39:23.471453+0000","last_became_peered":"2024-09-10T20:39:23.471453+0000","last_unstale":"2024-09-10T20:39:24.471486+0000","last_undegraded":"2024-09-10T20:39:24.471486+0000","last_fullsized":"2024-09-10T20:39:24.471486+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:52.838472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252396+0000","last_change":"2024-09-10T20:39:23.474509+0000","last_active":"2024-09-10T20:39:25.252396+0000","last_peered":"2024-09-10T20:39:25.252396+0000","last_clean":"2024-09-10T20:39:25.252396+0000","last_became_active":"2024-09-10T20:39:23.474180+0000","last_became_peered":"2024-09-10T20:39:23.474180+0000","last_unstale":"2024-09-10T20:39:25.252396+0000","last_undegraded":"2024-09-10T20:39:25.252396+0000","last_fullsized":"2024-09-10T20:39:25.252396+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:19.254485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252368+0000","last_change":"2024-09-10T20:39:23.482090+0000","last_active":"2024-09-10T20:39:25.252368+0000","last_peered":"2024-09-10T20:39:25.252368+0000","last_clean":"2024-09-10T20:39:25.252368+0000","last_became_active":"2024-09-10T20:39:23.481803+0000","last_became_peered":"2024-09-10T20:39:23.481803+0000","last_unstale":"2024-09-10T20:39:25.252368+0000","last_undegraded":"2024-09-10T20:39:25.252368+0000","last_fullsized":"2024-09-10T20:39:25.252368+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:52.741446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814356+0000","last_change":"2024-09-10T20:39:23.473090+0000","last_active":"2024-09-10T20:39:24.814356+0000","last_peered":"2024-09-10T20:39:24.814356+0000","last_clean":"2024-09-10T20:39:24.814356+0000","last_became_active":"2024-09-10T20:39:23.472803+0000","last_became_peered":"2024-09-10T20:39:23.472803+0000","last_unstale":"2024-09-10T20:39:24.814356+0000","last_undegraded":"2024-09-10T20:39:24.814356+0000","last_fullsized":"2024-09-10T20:39:24.814356+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:22.229236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251535+0000","last_change":"2024-09-10T20:39:23.479620+0000","last_active":"2024-09-10T20:39:25.251535+0000","last_peered":"2024-09-10T20:39:25.251535+0000","last_clean":"2024-09-10T20:39:25.251535+0000","last_became_active":"2024-09-10T20:39:23.479319+0000","last_became_peered":"2024-09-10T20:39:23.479319+0000","last_unstale":"2024-09-10T20:39:25.251535+0000","last_undegraded":"2024-09-10T20:39:25.251535+0000","last_fullsized":"2024-09-10T20:39:25.251535+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:13.760939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469972+0000","last_change":"2024-09-10T20:39:23.469522+0000","last_active":"2024-09-10T20:39:24.469972+0000","last_peered":"2024-09-10T20:39:24.469972+0000","last_clean":"2024-09-10T20:39:24.469972+0000","last_became_active":"2024-09-10T20:39:23.469129+0000","last_became_peered":"2024-09-10T20:39:23.469129+0000","last_unstale":"2024-09-10T20:39:24.469972+0000","last_undegraded":"2024-09-10T20:39:24.469972+0000","last_fullsized":"2024-09-10T20:39:24.469972+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:28.627939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471378+0000","last_change":"2024-09-10T20:39:23.470278+0000","last_active":"2024-09-10T20:39:24.471378+0000","last_peered":"2024-09-10T20:39:24.471378+0000","last_clean":"2024-09-10T20:39:24.471378+0000","last_became_active":"2024-09-10T20:39:23.469632+0000","last_became_peered":"2024-09-10T20:39:23.469632+0000","last_unstale":"2024-09-10T20:39:24.471378+0000","last_undegraded":"2024-09-10T20:39:24.471378+0000","last_fullsized":"2024-09-10T20:39:24.471378+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:46:36.733749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251826+0000","last_change":"2024-09-10T20:39:23.476296+0000","last_active":"2024-09-10T20:39:25.251826+0000","last_peered":"2024-09-10T20:39:25.251826+0000","last_clean":"2024-09-10T20:39:25.251826+0000","last_became_active":"2024-09-10T20:39:23.476009+0000","last_became_peered":"2024-09-10T20:39:23.476009+0000","last_unstale":"2024-09-10T20:39:25.251826+0000","last_undegraded":"2024-09-10T20:39:25.251826+0000","last_fullsized":"2024-09-10T20:39:25.251826+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:54:10.316107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":19,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470688+0000","last_change":"2024-09-10T20:39:22.471060+0000","last_active":"2024-09-10T20:39:24.470688+0000","last_peered":"2024-09-10T20:39:24.470688+0000","last_clean":"2024-09-10T20:39:24.470688+0000","last_became_active":"2024-09-10T20:39:22.470753+0000","last_became_peered":"2024-09-10T20:39:22.470753+0000","last_unstale":"2024-09-10T20:39:24.470688+0000","last_undegraded":"2024-09-10T20:39:24.470688+0000","last_fullsized":"2024-09-10T20:39:24.470688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:22.626422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814639+0000","last_change":"2024-09-10T20:39:23.470015+0000","last_active":"2024-09-10T20:39:24.814639+0000","last_peered":"2024-09-10T20:39:24.814639+0000","last_clean":"2024-09-10T20:39:24.814639+0000","last_became_active":"2024-09-10T20:39:23.469693+0000","last_became_peered":"2024-09-10T20:39:23.469693+0000","last_unstale":"2024-09-10T20:39:24.814639+0000","last_undegraded":"2024-09-10T20:39:24.814639+0000","last_fullsized":"2024-09-10T20:39:24.814639+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:25.557517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252307+0000","last_change":"2024-09-10T20:39:22.461136+0000","last_active":"2024-09-10T20:39:25.252307+0000","last_peered":"2024-09-10T20:39:25.252307+0000","last_clean":"2024-09-10T20:39:25.252307+0000","last_became_active":"2024-09-10T20:39:22.460848+0000","last_became_peered":"2024-09-10T20:39:22.460848+0000","last_unstale":"2024-09-10T20:39:25.252307+0000","last_undegraded":"2024-09-10T20:39:25.252307+0000","last_fullsized":"2024-09-10T20:39:25.252307+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:18.873047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251912+0000","last_change":"2024-09-10T20:39:23.481110+0000","last_active":"2024-09-10T20:39:25.251912+0000","last_peered":"2024-09-10T20:39:25.251912+0000","last_clean":"2024-09-10T20:39:25.251912+0000","last_became_active":"2024-09-10T20:39:23.480241+0000","last_became_peered":"2024-09-10T20:39:23.480241+0000","last_unstale":"2024-09-10T20:39:25.251912+0000","last_undegraded":"2024-09-10T20:39:25.251912+0000","last_fullsized":"2024-09-10T20:39:25.251912+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:29.489296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470292+0000","last_change":"2024-09-10T20:39:22.467354+0000","last_active":"2024-09-10T20:39:24.470292+0000","last_peered":"2024-09-10T20:39:24.470292+0000","last_clean":"2024-09-10T20:39:24.470292+0000","last_became_active":"2024-09-10T20:39:22.467105+0000","last_became_peered":"2024-09-10T20:39:22.467105+0000","last_unstale":"2024-09-10T20:39:24.470292+0000","last_undegraded":"2024-09-10T20:39:24.470292+0000","last_fullsized":"2024-09-10T20:39:24.470292+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:42.142331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471995+0000","last_change":"2024-09-10T20:39:23.465129+0000","last_active":"2024-09-10T20:39:24.471995+0000","last_peered":"2024-09-10T20:39:24.471995+0000","last_clean":"2024-09-10T20:39:24.471995+0000","last_became_active":"2024-09-10T20:39:23.464816+0000","last_became_peered":"2024-09-10T20:39:23.464816+0000","last_unstale":"2024-09-10T20:39:24.471995+0000","last_undegraded":"2024-09-10T20:39:24.471995+0000","last_fullsized":"2024-09-10T20:39:24.471995+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:06.505948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252472+0000","last_change":"2024-09-10T20:39:22.463145+0000","last_active":"2024-09-10T20:39:25.252472+0000","last_peered":"2024-09-10T20:39:25.252472+0000","last_clean":"2024-09-10T20:39:25.252472+0000","last_became_active":"2024-09-10T20:39:22.462844+0000","last_became_peered":"2024-09-10T20:39:22.462844+0000","last_unstale":"2024-09-10T20:39:25.252472+0000","last_undegraded":"2024-09-10T20:39:25.252472+0000","last_fullsized":"2024-09-10T20:39:25.252472+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:42:40.187952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251873+0000","last_change":"2024-09-10T20:39:23.471689+0000","last_active":"2024-09-10T20:39:25.251873+0000","last_peered":"2024-09-10T20:39:25.251873+0000","last_clean":"2024-09-10T20:39:25.251873+0000","last_became_active":"2024-09-10T20:39:23.471333+0000","last_became_peered":"2024-09-10T20:39:23.471333+0000","last_unstale":"2024-09-10T20:39:25.251873+0000","last_undegraded":"2024-09-10T20:39:25.251873+0000","last_fullsized":"2024-09-10T20:39:25.251873+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:01:24.286087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470439+0000","last_change":"2024-09-10T20:39:22.468131+0000","last_active":"2024-09-10T20:39:24.470439+0000","last_peered":"2024-09-10T20:39:24.470439+0000","last_clean":"2024-09-10T20:39:24.470439+0000","last_became_active":"2024-09-10T20:39:22.467846+0000","last_became_peered":"2024-09-10T20:39:22.467846+0000","last_unstale":"2024-09-10T20:39:24.470439+0000","last_undegraded":"2024-09-10T20:39:24.470439+0000","last_fullsized":"2024-09-10T20:39:24.470439+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:40.914501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252191+0000","last_change":"2024-09-10T20:39:23.477169+0000","last_active":"2024-09-10T20:39:25.252191+0000","last_peered":"2024-09-10T20:39:25.252191+0000","last_clean":"2024-09-10T20:39:25.252191+0000","last_became_active":"2024-09-10T20:39:23.476830+0000","last_became_peered":"2024-09-10T20:39:23.476830+0000","last_unstale":"2024-09-10T20:39:25.252191+0000","last_undegraded":"2024-09-10T20:39:25.252191+0000","last_fullsized":"2024-09-10T20:39:25.252191+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:45:49.013026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251795+0000","last_change":"2024-09-10T20:39:22.468056+0000","last_active":"2024-09-10T20:39:25.251795+0000","last_peered":"2024-09-10T20:39:25.251795+0000","last_clean":"2024-09-10T20:39:25.251795+0000","last_became_active":"2024-09-10T20:39:22.467477+0000","last_became_peered":"2024-09-10T20:39:22.467477+0000","last_unstale":"2024-09-10T20:39:25.251795+0000","last_undegraded":"2024-09-10T20:39:25.251795+0000","last_fullsized":"2024-09-10T20:39:25.251795+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:45.140725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252324+0000","last_change":"2024-09-10T20:39:23.466496+0000","last_active":"2024-09-10T20:39:25.252324+0000","last_peered":"2024-09-10T20:39:25.252324+0000","last_clean":"2024-09-10T20:39:25.252324+0000","last_became_active":"2024-09-10T20:39:23.466162+0000","last_became_peered":"2024-09-10T20:39:23.466162+0000","last_unstale":"2024-09-10T20:39:25.252324+0000","last_undegraded":"2024-09-10T20:39:25.252324+0000","last_fullsized":"2024-09-10T20:39:25.252324+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:01.129927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814173+0000","last_change":"2024-09-10T20:39:22.469484+0000","last_active":"2024-09-10T20:39:24.814173+0000","last_peered":"2024-09-10T20:39:24.814173+0000","last_clean":"2024-09-10T20:39:24.814173+0000","last_became_active":"2024-09-10T20:39:22.469218+0000","last_became_peered":"2024-09-10T20:39:22.469218+0000","last_unstale":"2024-09-10T20:39:24.814173+0000","last_undegraded":"2024-09-10T20:39:24.814173+0000","last_fullsized":"2024-09-10T20:39:24.814173+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:42:09.052814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252173+0000","last_change":"2024-09-10T20:39:23.466343+0000","last_active":"2024-09-10T20:39:25.252173+0000","last_peered":"2024-09-10T20:39:25.252173+0000","last_clean":"2024-09-10T20:39:25.252173+0000","last_became_active":"2024-09-10T20:39:23.466061+0000","last_became_peered":"2024-09-10T20:39:23.466061+0000","last_unstale":"2024-09-10T20:39:25.252173+0000","last_undegraded":"2024-09-10T20:39:25.252173+0000","last_fullsized":"2024-09-10T20:39:25.252173+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.847458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252100+0000","last_change":"2024-09-10T20:39:22.464013+0000","last_active":"2024-09-10T20:39:25.252100+0000","last_peered":"2024-09-10T20:39:25.252100+0000","last_clean":"2024-09-10T20:39:25.252100+0000","last_became_active":"2024-09-10T20:39:22.463745+0000","last_became_peered":"2024-09-10T20:39:22.463745+0000","last_unstale":"2024-09-10T20:39:25.252100+0000","last_undegraded":"2024-09-10T20:39:25.252100+0000","last_fullsized":"2024-09-10T20:39:25.252100+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:17.407343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469738+0000","last_change":"2024-09-10T20:39:23.470667+0000","last_active":"2024-09-10T20:39:24.469738+0000","last_peered":"2024-09-10T20:39:24.469738+0000","last_clean":"2024-09-10T20:39:24.469738+0000","last_became_active":"2024-09-10T20:39:23.470425+0000","last_became_peered":"2024-09-10T20:39:23.470425+0000","last_unstale":"2024-09-10T20:39:24.469738+0000","last_undegraded":"2024-09-10T20:39:24.469738+0000","last_fullsized":"2024-09-10T20:39:24.469738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:08:02.455782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469712+0000","last_change":"2024-09-10T20:39:22.466186+0000","last_active":"2024-09-10T20:39:24.469712+0000","last_peered":"2024-09-10T20:39:24.469712+0000","last_clean":"2024-09-10T20:39:24.469712+0000","last_became_active":"2024-09-10T20:39:22.465718+0000","last_became_peered":"2024-09-10T20:39:22.465718+0000","last_unstale":"2024-09-10T20:39:24.469712+0000","last_undegraded":"2024-09-10T20:39:24.469712+0000","last_fullsized":"2024-09-10T20:39:24.469712+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:09.130187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469604+0000","last_change":"2024-09-10T20:39:23.474536+0000","last_active":"2024-09-10T20:39:24.469604+0000","last_peered":"2024-09-10T20:39:24.469604+0000","last_clean":"2024-09-10T20:39:24.469604+0000","last_became_active":"2024-09-10T20:39:23.474234+0000","last_became_peered":"2024-09-10T20:39:23.474234+0000","last_unstale":"2024-09-10T20:39:24.469604+0000","last_undegraded":"2024-09-10T20:39:24.469604+0000","last_fullsized":"2024-09-10T20:39:24.469604+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:38.533241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252447+0000","last_change":"2024-09-10T20:39:22.470606+0000","last_active":"2024-09-10T20:39:25.252447+0000","last_peered":"2024-09-10T20:39:25.252447+0000","last_clean":"2024-09-10T20:39:25.252447+0000","last_became_active":"2024-09-10T20:39:22.470313+0000","last_became_peered":"2024-09-10T20:39:22.470313+0000","last_unstale":"2024-09-10T20:39:25.252447+0000","last_undegraded":"2024-09-10T20:39:25.252447+0000","last_fullsized":"2024-09-10T20:39:25.252447+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:13.562757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471621+0000","last_change":"2024-09-10T20:39:23.478464+0000","last_active":"2024-09-10T20:39:24.471621+0000","last_peered":"2024-09-10T20:39:24.471621+0000","last_clean":"2024-09-10T20:39:24.471621+0000","last_became_active":"2024-09-10T20:39:23.477764+0000","last_became_peered":"2024-09-10T20:39:23.477764+0000","last_unstale":"2024-09-10T20:39:24.471621+0000","last_undegraded":"2024-09-10T20:39:24.471621+0000","last_fullsized":"2024-09-10T20:39:24.471621+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:07.988361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"42'3","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251237+0000","last_change":"2024-09-10T20:39:22.466977+0000","last_active":"2024-09-10T20:39:25.251237+0000","last_peered":"2024-09-10T20:39:25.251237+0000","last_clean":"2024-09-10T20:39:25.251237+0000","last_became_active":"2024-09-10T20:39:22.466685+0000","last_became_peered":"2024-09-10T20:39:22.466685+0000","last_unstale":"2024-09-10T20:39:25.251237+0000","last_undegraded":"2024-09-10T20:39:25.251237+0000","last_fullsized":"2024-09-10T20:39:25.251237+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:44:01.537714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252311+0000","last_change":"2024-09-10T20:39:23.472408+0000","last_active":"2024-09-10T20:39:25.252311+0000","last_peered":"2024-09-10T20:39:25.252311+0000","last_clean":"2024-09-10T20:39:25.252311+0000","last_became_active":"2024-09-10T20:39:23.472123+0000","last_became_peered":"2024-09-10T20:39:23.472123+0000","last_unstale":"2024-09-10T20:39:25.252311+0000","last_undegraded":"2024-09-10T20:39:25.252311+0000","last_fullsized":"2024-09-10T20:39:25.252311+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:58:19.163029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":21,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252741+0000","last_change":"2024-09-10T20:39:22.457688+0000","last_active":"2024-09-10T20:39:25.252741+0000","last_peered":"2024-09-10T20:39:25.252741+0000","last_clean":"2024-09-10T20:39:25.252741+0000","last_became_active":"2024-09-10T20:39:22.457326+0000","last_became_peered":"2024-09-10T20:39:22.457326+0000","last_unstale":"2024-09-10T20:39:25.252741+0000","last_undegraded":"2024-09-10T20:39:25.252741+0000","last_fullsized":"2024-09-10T20:39:25.252741+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:51.098445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469742+0000","last_change":"2024-09-10T20:39:23.478920+0000","last_active":"2024-09-10T20:39:24.469742+0000","last_peered":"2024-09-10T20:39:24.469742+0000","last_clean":"2024-09-10T20:39:24.469742+0000","last_became_active":"2024-09-10T20:39:23.478555+0000","last_became_peered":"2024-09-10T20:39:23.478555+0000","last_unstale":"2024-09-10T20:39:24.469742+0000","last_undegraded":"2024-09-10T20:39:24.469742+0000","last_fullsized":"2024-09-10T20:39:24.469742+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:12:21.695423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251793+0000","last_change":"2024-09-10T20:39:22.463919+0000","last_active":"2024-09-10T20:39:25.251793+0000","last_peered":"2024-09-10T20:39:25.251793+0000","last_clean":"2024-09-10T20:39:25.251793+0000","last_became_active":"2024-09-10T20:39:22.463650+0000","last_became_peered":"2024-09-10T20:39:22.463650+0000","last_unstale":"2024-09-10T20:39:25.251793+0000","last_undegraded":"2024-09-10T20:39:25.251793+0000","last_fullsized":"2024-09-10T20:39:25.251793+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:07:05.483324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251655+0000","last_change":"2024-09-10T20:39:23.482909+0000","last_active":"2024-09-10T20:39:25.251655+0000","last_peered":"2024-09-10T20:39:25.251655+0000","last_clean":"2024-09-10T20:39:25.251655+0000","last_became_active":"2024-09-10T20:39:23.482551+0000","last_became_peered":"2024-09-10T20:39:23.482551+0000","last_unstale":"2024-09-10T20:39:25.251655+0000","last_undegraded":"2024-09-10T20:39:25.251655+0000","last_fullsized":"2024-09-10T20:39:25.251655+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:53:50.370194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814152+0000","last_change":"2024-09-10T20:39:22.471268+0000","last_active":"2024-09-10T20:39:24.814152+0000","last_peered":"2024-09-10T20:39:24.814152+0000","last_clean":"2024-09-10T20:39:24.814152+0000","last_became_active":"2024-09-10T20:39:22.470115+0000","last_became_peered":"2024-09-10T20:39:22.470115+0000","last_unstale":"2024-09-10T20:39:24.814152+0000","last_undegraded":"2024-09-10T20:39:24.814152+0000","last_fullsized":"2024-09-10T20:39:24.814152+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251593+0000","last_change":"2024-09-10T20:39:23.465467+0000","last_active":"2024-09-10T20:39:25.251593+0000","last_peered":"2024-09-10T20:39:25.251593+0000","last_clean":"2024-09-10T20:39:25.251593+0000","last_became_active":"2024-09-10T20:39:23.465165+0000","last_became_peered":"2024-09-10T20:39:23.465165+0000","last_unstale":"2024-09-10T20:39:25.251593+0000","last_undegraded":"2024-09-10T20:39:25.251593+0000","last_fullsized":"2024-09-10T20:39:25.251593+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:31:13.123701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469888+0000","last_change":"2024-09-10T20:39:22.471774+0000","last_active":"2024-09-10T20:39:24.469888+0000","last_peered":"2024-09-10T20:39:24.469888+0000","last_clean":"2024-09-10T20:39:24.469888+0000","last_became_active":"2024-09-10T20:39:22.471455+0000","last_became_peered":"2024-09-10T20:39:22.471455+0000","last_unstale":"2024-09-10T20:39:24.469888+0000","last_undegraded":"2024-09-10T20:39:24.469888+0000","last_fullsized":"2024-09-10T20:39:24.469888+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:31.755937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251215+0000","last_change":"2024-09-10T20:39:23.482713+0000","last_active":"2024-09-10T20:39:25.251215+0000","last_peered":"2024-09-10T20:39:25.251215+0000","last_clean":"2024-09-10T20:39:25.251215+0000","last_became_active":"2024-09-10T20:39:23.482442+0000","last_became_peered":"2024-09-10T20:39:23.482442+0000","last_unstale":"2024-09-10T20:39:25.251215+0000","last_undegraded":"2024-09-10T20:39:25.251215+0000","last_fullsized":"2024-09-10T20:39:25.251215+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":23,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252257+0000","last_change":"2024-09-10T20:39:22.468116+0000","last_active":"2024-09-10T20:39:25.252257+0000","last_peered":"2024-09-10T20:39:25.252257+0000","last_clean":"2024-09-10T20:39:25.252257+0000","last_became_active":"2024-09-10T20:39:22.467712+0000","last_became_peered":"2024-09-10T20:39:22.467712+0000","last_unstale":"2024-09-10T20:39:25.252257+0000","last_undegraded":"2024-09-10T20:39:25.252257+0000","last_fullsized":"2024-09-10T20:39:25.252257+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:18:01.811366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251871+0000","last_change":"2024-09-10T20:39:23.473728+0000","last_active":"2024-09-10T20:39:25.251871+0000","last_peered":"2024-09-10T20:39:25.251871+0000","last_clean":"2024-09-10T20:39:25.251871+0000","last_became_active":"2024-09-10T20:39:23.473482+0000","last_became_peered":"2024-09-10T20:39:23.473482+0000","last_unstale":"2024-09-10T20:39:25.251871+0000","last_undegraded":"2024-09-10T20:39:25.251871+0000","last_fullsized":"2024-09-10T20:39:25.251871+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:39.180573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471265+0000","last_change":"2024-09-10T20:39:22.464648+0000","last_active":"2024-09-10T20:39:24.471265+0000","last_peered":"2024-09-10T20:39:24.471265+0000","last_clean":"2024-09-10T20:39:24.471265+0000","last_became_active":"2024-09-10T20:39:22.464381+0000","last_became_peered":"2024-09-10T20:39:22.464381+0000","last_unstale":"2024-09-10T20:39:24.471265+0000","last_undegraded":"2024-09-10T20:39:24.471265+0000","last_fullsized":"2024-09-10T20:39:24.471265+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:34.596334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252249+0000","last_change":"2024-09-10T20:39:23.466893+0000","last_active":"2024-09-10T20:39:25.252249+0000","last_peered":"2024-09-10T20:39:25.252249+0000","last_clean":"2024-09-10T20:39:25.252249+0000","last_became_active":"2024-09-10T20:39:23.466577+0000","last_became_peered":"2024-09-10T20:39:23.466577+0000","last_unstale":"2024-09-10T20:39:25.252249+0000","last_undegraded":"2024-09-10T20:39:25.252249+0000","last_fullsized":"2024-09-10T20:39:25.252249+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:14.547458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469893+0000","last_change":"2024-09-10T20:39:22.472535+0000","last_active":"2024-09-10T20:39:24.469893+0000","last_peered":"2024-09-10T20:39:24.469893+0000","last_clean":"2024-09-10T20:39:24.469893+0000","last_became_active":"2024-09-10T20:39:22.472220+0000","last_became_peered":"2024-09-10T20:39:22.472220+0000","last_unstale":"2024-09-10T20:39:24.469893+0000","last_undegraded":"2024-09-10T20:39:24.469893+0000","last_fullsized":"2024-09-10T20:39:24.469893+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:37:49.040049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251980+0000","last_change":"2024-09-10T20:39:23.480836+0000","last_active":"2024-09-10T20:39:25.251980+0000","last_peered":"2024-09-10T20:39:25.251980+0000","last_clean":"2024-09-10T20:39:25.251980+0000","last_became_active":"2024-09-10T20:39:23.479966+0000","last_became_peered":"2024-09-10T20:39:23.479966+0000","last_unstale":"2024-09-10T20:39:25.251980+0000","last_undegraded":"2024-09-10T20:39:25.251980+0000","last_fullsized":"2024-09-10T20:39:25.251980+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:35.149497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"42'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469794+0000","last_change":"2024-09-10T20:39:22.468576+0000","last_active":"2024-09-10T20:39:24.469794+0000","last_peered":"2024-09-10T20:39:24.469794+0000","last_clean":"2024-09-10T20:39:24.469794+0000","last_became_active":"2024-09-10T20:39:22.468301+0000","last_became_peered":"2024-09-10T20:39:22.468301+0000","last_unstale":"2024-09-10T20:39:24.469794+0000","last_undegraded":"2024-09-10T20:39:24.469794+0000","last_fullsized":"2024-09-10T20:39:24.469794+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:38.187418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469705+0000","last_change":"2024-09-10T20:39:23.480119+0000","last_active":"2024-09-10T20:39:24.469705+0000","last_peered":"2024-09-10T20:39:24.469705+0000","last_clean":"2024-09-10T20:39:24.469705+0000","last_became_active":"2024-09-10T20:39:23.479859+0000","last_became_peered":"2024-09-10T20:39:23.479859+0000","last_unstale":"2024-09-10T20:39:24.469705+0000","last_undegraded":"2024-09-10T20:39:24.469705+0000","last_fullsized":"2024-09-10T20:39:24.469705+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:45.569351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470125+0000","last_change":"2024-09-10T20:39:22.471454+0000","last_active":"2024-09-10T20:39:24.470125+0000","last_peered":"2024-09-10T20:39:24.470125+0000","last_clean":"2024-09-10T20:39:24.470125+0000","last_became_active":"2024-09-10T20:39:22.471186+0000","last_became_peered":"2024-09-10T20:39:22.471186+0000","last_unstale":"2024-09-10T20:39:24.470125+0000","last_undegraded":"2024-09-10T20:39:24.470125+0000","last_fullsized":"2024-09-10T20:39:24.470125+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:11.967787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251934+0000","last_change":"2024-09-10T20:39:23.474293+0000","last_active":"2024-09-10T20:39:25.251934+0000","last_peered":"2024-09-10T20:39:25.251934+0000","last_clean":"2024-09-10T20:39:25.251934+0000","last_became_active":"2024-09-10T20:39:23.474013+0000","last_became_peered":"2024-09-10T20:39:23.474013+0000","last_unstale":"2024-09-10T20:39:25.251934+0000","last_undegraded":"2024-09-10T20:39:25.251934+0000","last_fullsized":"2024-09-10T20:39:25.251934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:41.562077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252571+0000","last_change":"2024-09-10T20:39:22.467010+0000","last_active":"2024-09-10T20:39:25.252571+0000","last_peered":"2024-09-10T20:39:25.252571+0000","last_clean":"2024-09-10T20:39:25.252571+0000","last_became_active":"2024-09-10T20:39:22.466604+0000","last_became_peered":"2024-09-10T20:39:22.466604+0000","last_unstale":"2024-09-10T20:39:25.252571+0000","last_undegraded":"2024-09-10T20:39:25.252571+0000","last_fullsized":"2024-09-10T20:39:25.252571+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:33:38.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.470167+0000","last_change":"2024-09-10T20:39:23.483419+0000","last_active":"2024-09-10T20:39:24.470167+0000","last_peered":"2024-09-10T20:39:24.470167+0000","last_clean":"2024-09-10T20:39:24.470167+0000","last_became_active":"2024-09-10T20:39:23.482573+0000","last_became_peered":"2024-09-10T20:39:23.482573+0000","last_unstale":"2024-09-10T20:39:24.470167+0000","last_undegraded":"2024-09-10T20:39:24.470167+0000","last_fullsized":"2024-09-10T20:39:24.470167+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:34.077485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814400+0000","last_change":"2024-09-10T20:39:22.472184+0000","last_active":"2024-09-10T20:39:24.814400+0000","last_peered":"2024-09-10T20:39:24.814400+0000","last_clean":"2024-09-10T20:39:24.814400+0000","last_became_active":"2024-09-10T20:39:22.471851+0000","last_became_peered":"2024-09-10T20:39:22.471851+0000","last_unstale":"2024-09-10T20:39:24.814400+0000","last_undegraded":"2024-09-10T20:39:24.814400+0000","last_fullsized":"2024-09-10T20:39:24.814400+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:06.279859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471576+0000","last_change":"2024-09-10T20:39:23.470437+0000","last_active":"2024-09-10T20:39:24.471576+0000","last_peered":"2024-09-10T20:39:24.471576+0000","last_clean":"2024-09-10T20:39:24.471576+0000","last_became_active":"2024-09-10T20:39:23.470018+0000","last_became_peered":"2024-09-10T20:39:23.470018+0000","last_unstale":"2024-09-10T20:39:24.471576+0000","last_undegraded":"2024-09-10T20:39:24.471576+0000","last_fullsized":"2024-09-10T20:39:24.471576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:19.864809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252324+0000","last_change":"2024-09-10T20:39:22.470294+0000","last_active":"2024-09-10T20:39:25.252324+0000","last_peered":"2024-09-10T20:39:25.252324+0000","last_clean":"2024-09-10T20:39:25.252324+0000","last_became_active":"2024-09-10T20:39:22.469974+0000","last_became_peered":"2024-09-10T20:39:22.469974+0000","last_unstale":"2024-09-10T20:39:25.252324+0000","last_undegraded":"2024-09-10T20:39:25.252324+0000","last_fullsized":"2024-09-10T20:39:25.252324+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:14.321360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471788+0000","last_change":"2024-09-10T20:39:23.467237+0000","last_active":"2024-09-10T20:39:24.471788+0000","last_peered":"2024-09-10T20:39:24.471788+0000","last_clean":"2024-09-10T20:39:24.471788+0000","last_became_active":"2024-09-10T20:39:23.466955+0000","last_became_peered":"2024-09-10T20:39:23.466955+0000","last_unstale":"2024-09-10T20:39:24.471788+0000","last_undegraded":"2024-09-10T20:39:24.471788+0000","last_fullsized":"2024-09-10T20:39:24.471788+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:17:53.617323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251301+0000","last_change":"2024-09-10T20:39:22.463388+0000","last_active":"2024-09-10T20:39:25.251301+0000","last_peered":"2024-09-10T20:39:25.251301+0000","last_clean":"2024-09-10T20:39:25.251301+0000","last_became_active":"2024-09-10T20:39:22.463050+0000","last_became_peered":"2024-09-10T20:39:22.463050+0000","last_unstale":"2024-09-10T20:39:25.251301+0000","last_undegraded":"2024-09-10T20:39:25.251301+0000","last_fullsized":"2024-09-10T20:39:25.251301+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:49.479271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252422+0000","last_change":"2024-09-10T20:39:23.466822+0000","last_active":"2024-09-10T20:39:25.252422+0000","last_peered":"2024-09-10T20:39:25.252422+0000","last_clean":"2024-09-10T20:39:25.252422+0000","last_became_active":"2024-09-10T20:39:23.466056+0000","last_became_peered":"2024-09-10T20:39:23.466056+0000","last_unstale":"2024-09-10T20:39:25.252422+0000","last_undegraded":"2024-09-10T20:39:25.252422+0000","last_fullsized":"2024-09-10T20:39:25.252422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:03.707588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"43'1","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.007418+0000","last_change":"2024-09-10T20:39:22.466270+0000","last_active":"2024-09-10T20:39:25.007418+0000","last_peered":"2024-09-10T20:39:25.007418+0000","last_clean":"2024-09-10T20:39:25.007418+0000","last_became_active":"2024-09-10T20:39:22.465951+0000","last_became_peered":"2024-09-10T20:39:22.465951+0000","last_unstale":"2024-09-10T20:39:25.007418+0000","last_undegraded":"2024-09-10T20:39:25.007418+0000","last_fullsized":"2024-09-10T20:39:25.007418+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:42.740101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252741+0000","last_change":"2024-09-10T20:39:23.468716+0000","last_active":"2024-09-10T20:39:25.252741+0000","last_peered":"2024-09-10T20:39:25.252741+0000","last_clean":"2024-09-10T20:39:25.252741+0000","last_became_active":"2024-09-10T20:39:23.468450+0000","last_became_peered":"2024-09-10T20:39:23.468450+0000","last_unstale":"2024-09-10T20:39:25.252741+0000","last_undegraded":"2024-09-10T20:39:25.252741+0000","last_fullsized":"2024-09-10T20:39:25.252741+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:27:46.315297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814372+0000","last_change":"2024-09-10T20:39:22.470970+0000","last_active":"2024-09-10T20:39:24.814372+0000","last_peered":"2024-09-10T20:39:24.814372+0000","last_clean":"2024-09-10T20:39:24.814372+0000","last_became_active":"2024-09-10T20:39:22.470662+0000","last_became_peered":"2024-09-10T20:39:22.470662+0000","last_unstale":"2024-09-10T20:39:24.814372+0000","last_undegraded":"2024-09-10T20:39:24.814372+0000","last_fullsized":"2024-09-10T20:39:24.814372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:34:54.628648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252557+0000","last_change":"2024-09-10T20:39:23.476329+0000","last_active":"2024-09-10T20:39:25.252557+0000","last_peered":"2024-09-10T20:39:25.252557+0000","last_clean":"2024-09-10T20:39:25.252557+0000","last_became_active":"2024-09-10T20:39:23.476062+0000","last_became_peered":"2024-09-10T20:39:23.476062+0000","last_unstale":"2024-09-10T20:39:25.252557+0000","last_undegraded":"2024-09-10T20:39:25.252557+0000","last_fullsized":"2024-09-10T20:39:25.252557+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:16:42.873397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251920+0000","last_change":"2024-09-10T20:39:22.467181+0000","last_active":"2024-09-10T20:39:25.251920+0000","last_peered":"2024-09-10T20:39:25.251920+0000","last_clean":"2024-09-10T20:39:25.251920+0000","last_became_active":"2024-09-10T20:39:22.466859+0000","last_became_peered":"2024-09-10T20:39:22.466859+0000","last_unstale":"2024-09-10T20:39:25.251920+0000","last_undegraded":"2024-09-10T20:39:25.251920+0000","last_fullsized":"2024-09-10T20:39:25.251920+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:54.024056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252437+0000","last_change":"2024-09-10T20:39:23.473568+0000","last_active":"2024-09-10T20:39:25.252437+0000","last_peered":"2024-09-10T20:39:25.252437+0000","last_clean":"2024-09-10T20:39:25.252437+0000","last_became_active":"2024-09-10T20:39:23.473257+0000","last_became_peered":"2024-09-10T20:39:23.473257+0000","last_unstale":"2024-09-10T20:39:25.252437+0000","last_undegraded":"2024-09-10T20:39:25.252437+0000","last_fullsized":"2024-09-10T20:39:25.252437+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:00.266585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.814407+0000","last_change":"2024-09-10T20:39:22.467492+0000","last_active":"2024-09-10T20:39:24.814407+0000","last_peered":"2024-09-10T20:39:24.814407+0000","last_clean":"2024-09-10T20:39:24.814407+0000","last_became_active":"2024-09-10T20:39:22.467217+0000","last_became_peered":"2024-09-10T20:39:22.467217+0000","last_unstale":"2024-09-10T20:39:24.814407+0000","last_undegraded":"2024-09-10T20:39:24.814407+0000","last_fullsized":"2024-09-10T20:39:24.814407+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:32.173179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471424+0000","last_change":"2024-09-10T20:39:23.482739+0000","last_active":"2024-09-10T20:39:24.471424+0000","last_peered":"2024-09-10T20:39:24.471424+0000","last_clean":"2024-09-10T20:39:24.471424+0000","last_became_active":"2024-09-10T20:39:23.482483+0000","last_became_peered":"2024-09-10T20:39:23.482483+0000","last_unstale":"2024-09-10T20:39:24.471424+0000","last_undegraded":"2024-09-10T20:39:24.471424+0000","last_fullsized":"2024-09-10T20:39:24.471424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:53.453242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251921+0000","last_change":"2024-09-10T20:39:22.466109+0000","last_active":"2024-09-10T20:39:25.251921+0000","last_peered":"2024-09-10T20:39:25.251921+0000","last_clean":"2024-09-10T20:39:25.251921+0000","last_became_active":"2024-09-10T20:39:22.463916+0000","last_became_peered":"2024-09-10T20:39:22.463916+0000","last_unstale":"2024-09-10T20:39:25.251921+0000","last_undegraded":"2024-09-10T20:39:25.251921+0000","last_fullsized":"2024-09-10T20:39:25.251921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:19.681143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469354+0000","last_change":"2024-09-10T20:39:23.479485+0000","last_active":"2024-09-10T20:39:24.469354+0000","last_peered":"2024-09-10T20:39:24.469354+0000","last_clean":"2024-09-10T20:39:24.469354+0000","last_became_active":"2024-09-10T20:39:23.479219+0000","last_became_peered":"2024-09-10T20:39:23.479219+0000","last_unstale":"2024-09-10T20:39:24.469354+0000","last_undegraded":"2024-09-10T20:39:24.469354+0000","last_fullsized":"2024-09-10T20:39:24.469354+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:22.900530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469982+0000","last_change":"2024-09-10T20:39:22.464748+0000","last_active":"2024-09-10T20:39:24.469982+0000","last_peered":"2024-09-10T20:39:24.469982+0000","last_clean":"2024-09-10T20:39:24.469982+0000","last_became_active":"2024-09-10T20:39:22.464248+0000","last_became_peered":"2024-09-10T20:39:22.464248+0000","last_unstale":"2024-09-10T20:39:24.469982+0000","last_undegraded":"2024-09-10T20:39:24.469982+0000","last_fullsized":"2024-09-10T20:39:24.469982+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:34:33.491750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251582+0000","last_change":"2024-09-10T20:39:23.482269+0000","last_active":"2024-09-10T20:39:25.251582+0000","last_peered":"2024-09-10T20:39:25.251582+0000","last_clean":"2024-09-10T20:39:25.251582+0000","last_became_active":"2024-09-10T20:39:23.481990+0000","last_became_peered":"2024-09-10T20:39:23.481990+0000","last_unstale":"2024-09-10T20:39:25.251582+0000","last_undegraded":"2024-09-10T20:39:25.251582+0000","last_fullsized":"2024-09-10T20:39:25.251582+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:40.821209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.251602+0000","last_change":"2024-09-10T20:39:22.465000+0000","last_active":"2024-09-10T20:39:25.251602+0000","last_peered":"2024-09-10T20:39:25.251602+0000","last_clean":"2024-09-10T20:39:25.251602+0000","last_became_active":"2024-09-10T20:39:22.464669+0000","last_became_peered":"2024-09-10T20:39:22.464669+0000","last_unstale":"2024-09-10T20:39:25.251602+0000","last_undegraded":"2024-09-10T20:39:25.251602+0000","last_fullsized":"2024-09-10T20:39:25.251602+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:43:06.255942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.471722+0000","last_change":"2024-09-10T20:39:23.476662+0000","last_active":"2024-09-10T20:39:24.471722+0000","last_peered":"2024-09-10T20:39:24.471722+0000","last_clean":"2024-09-10T20:39:24.471722+0000","last_became_active":"2024-09-10T20:39:23.476375+0000","last_became_peered":"2024-09-10T20:39:23.476375+0000","last_unstale":"2024-09-10T20:39:24.471722+0000","last_undegraded":"2024-09-10T20:39:24.471722+0000","last_fullsized":"2024-09-10T20:39:24.471722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:22.441589+0000","last_clean_scrub_stamp":"2024-09-10T20:39:22.441589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:46.142798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"42'2","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:25.252456+0000","last_change":"2024-09-10T20:39:22.464857+0000","last_active":"2024-09-10T20:39:25.252456+0000","last_peered":"2024-09-10T20:39:25.252456+0000","last_clean":"2024-09-10T20:39:25.252456+0000","last_became_active":"2024-09-10T20:39:22.464583+0000","last_became_peered":"2024-09-10T20:39:22.464583+0000","last_unstale":"2024-09-10T20:39:25.252456+0000","last_undegraded":"2024-09-10T20:39:25.252456+0000","last_fullsized":"2024-09-10T20:39:25.252456+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:39:21.439516+0000","last_clean_scrub_stamp":"2024-09-10T20:39:21.439516+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:38:15.049485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":149,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-10T20:39:24.469244+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:39:24.469244+0000","last_peered":"2024-09-10T20:39:24.469244+0000","last_clean":"2024-09-10T20:39:24.469244+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:39:24.469244+0000","last_undegraded":"2024-09-10T20:39:24.469244+0000","last_fullsized":"2024-09-10T20:39:24.469244+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509462,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":507904,"data_stored":200539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509461,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":499712,"data_stored":199017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509461,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1093632,"data_stored":789365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509461,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":503808,"data_stored":199599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":499712,"data_stored":198961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542166,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":503808,"data_stored":199543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1093632,"data_stored":789317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542164,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":516096,"data_stored":200631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:39:32.321 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:39:32.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 -- 172.21.15.28:0/601118622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf815f4a0 msgr2=0x7f4cf8070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/601118622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf815f4a0 0x7f4cf8070880 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005fd0 tx=0x7f4ce802a530 comp rx=0 tx=0).stop 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 -- 172.21.15.28:0/601118622 shutdown_connections 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/601118622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cf80b1140 0x7f4cf8070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/601118622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf815f4a0 0x7f4cf8070880 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 -- 172.21.15.28:0/601118622 >> 172.21.15.28:0/601118622 conn(0x7f4cf80d1da0 msgr2=0x7f4cf80d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 -- 172.21.15.28:0/601118622 shutdown_connections 2024-09-10T20:39:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.467+0000 7f4cfed7d640 1 -- 172.21.15.28:0/601118622 wait complete. 2024-09-10T20:39:32.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.468+0000 7f4cfed7d640 1 Processor -- start 2024-09-10T20:39:32.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.468+0000 7f4cfed7d640 1 -- start start 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfed7d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cf8071300 0x7f4cf8155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfed7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf80b1140 0x7f4cf8158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfed7d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf815f4a0 0x7f4cf80b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfed7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf80cc900 con 0x7f4cf80b1140 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfed7d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf80cc780 con 0x7f4cf815f4a0 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfed7d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf80d0420 con 0x7f4cf8071300 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf80b1140 0x7f4cf8158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf80b1140 0x7f4cf8158c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46362/0 (socket says 172.21.15.28:46362) 2024-09-10T20:39:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 -- 172.21.15.28:0/3264416650 learned_addr learned my addr 172.21.15.28:0/3264416650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfcb19640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cf8071300 0x7f4cf8155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cfd31a640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf815f4a0 0x7f4cf80b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 -- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cf8071300 msgr2=0x7f4cf8155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cf8071300 0x7f4cf8155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 -- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf815f4a0 msgr2=0x7f4cf80b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf815f4a0 0x7f4cf80b6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 -- 172.21.15.28:0/3264416650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf8157e50 con 0x7f4cf80b1140 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.469+0000 7f4cf7fff640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf80b1140 0x7f4cf8158c10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00c8c0 tx=0x7f4cec00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.470+0000 7f4cf57fa640 1 -- 172.21.15.28:0/3264416650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cec004220 con 0x7f4cf80b1140 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.470+0000 7f4cf57fa640 1 -- 172.21.15.28:0/3264416650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cec0079b0 con 0x7f4cf80b1140 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.470+0000 7f4cf57fa640 1 -- 172.21.15.28:0/3264416650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cec007de0 con 0x7f4cf80b1140 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.470+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf8158130 con 0x7f4cf80b1140 2024-09-10T20:39:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.470+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf81586f0 con 0x7f4cf80b1140 2024-09-10T20:39:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.471+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf80d1960 con 0x7f4cf80b1140 2024-09-10T20:39:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.472+0000 7f4cf57fa640 1 -- 172.21.15.28:0/3264416650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f4cec005020 con 0x7f4cf80b1140 2024-09-10T20:39:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.472+0000 7f4cf57fa640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4ce4078120 0x7f4ce407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.472+0000 7f4cf57fa640 1 -- 172.21.15.28:0/3264416650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f4cec09a4f0 con 0x7f4cf80b1140 2024-09-10T20:39:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.473+0000 7f4cfcb19640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4ce4078120 0x7f4ce407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.473+0000 7f4cfcb19640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4ce4078120 0x7f4ce407a5e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8033000 tx=0x7f4ce8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.477+0000 7f4cf57fa640 1 -- 172.21.15.28:0/3264416650 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4cf80d1960 con 0x7f4cf80b1140 2024-09-10T20:39:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.733+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f4cf80d0920 con 0x7f4cf80b1140 2024-09-10T20:39:32.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.735+0000 7f4cf57fa640 1 -- 172.21.15.28:0/3264416650 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) ==== 190+0+147 (secure 0 0 0) 0x7f4cec014030 con 0x7f4cf80b1140 2024-09-10T20:39:32.739 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:39:32.739 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:39:32.739 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:39:32.739 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:39:32.739 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:39:32.740 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4ce4078120 msgr2=0x7f4ce407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4ce4078120 0x7f4ce407a5e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8033000 tx=0x7f4ce8004730 comp rx=0 tx=0).stop 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf80b1140 msgr2=0x7f4cf8158c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf80b1140 0x7f4cf8158c10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00c8c0 tx=0x7f4cec00cd80 comp rx=0 tx=0).stop 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 shutdown_connections 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4ce4078120 0x7f4ce407a5e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf815f4a0 0x7f4cf80b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf80b1140 0x7f4cf8158c10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 --2- 172.21.15.28:0/3264416650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cf8071300 0x7f4cf8155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 >> 172.21.15.28:0/3264416650 conn(0x7f4cf80d1da0 msgr2=0x7f4cf80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:32.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.737+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 shutdown_connections 2024-09-10T20:39:32.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.738+0000 7f4cfed7d640 1 -- 172.21.15.28:0/3264416650 wait complete. 2024-09-10T20:39:32.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:32.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 -- 172.21.15.28:0/2848205487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402006d1f0 msgr2=0x7f40200afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 --2- 172.21.15.28:0/2848205487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402006d1f0 0x7f40200afba0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4018005fd0 tx=0x7f401802aa00 comp rx=0 tx=0).stop 2024-09-10T20:39:32.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 -- 172.21.15.28:0/2848205487 shutdown_connections 2024-09-10T20:39:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 --2- 172.21.15.28:0/2848205487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402006d1f0 0x7f40200afba0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 --2- 172.21.15.28:0/2848205487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402015bb40 0x7f40200af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 -- 172.21.15.28:0/2848205487 >> 172.21.15.28:0/2848205487 conn(0x7f402006dbf0 msgr2=0x7f402006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 -- 172.21.15.28:0/2848205487 shutdown_connections 2024-09-10T20:39:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.853+0000 7f4027640640 1 -- 172.21.15.28:0/2848205487 wait complete. 2024-09-10T20:39:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.854+0000 7f4027640640 1 Processor -- start 2024-09-10T20:39:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.854+0000 7f4027640640 1 -- start start 2024-09-10T20:39:32.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4027640640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402006d1f0 0x7f40201f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4027640640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402015bb40 0x7f40201f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4027640640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402015bf20 0x7f402009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4027640640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f402015d520 con 0x7f402015bb40 2024-09-10T20:39:32.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4027640640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f402015d3a0 con 0x7f402006d1f0 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4027640640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f402015d6a0 con 0x7f402015bf20 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4024bdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402015bb40 0x7f40201f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4024bdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402015bb40 0x7f40201f2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46374/0 (socket says 172.21.15.28:46374) 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4024bdb640 1 -- 172.21.15.28:0/3039721361 learned_addr learned my addr 172.21.15.28:0/3039721361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f4025bdd640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402015bf20 0x7f402009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.855+0000 7f40253dc640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402006d1f0 0x7f40201f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4024bdb640 1 -- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402015bf20 msgr2=0x7f402009b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4024bdb640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402015bf20 0x7f402009b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4024bdb640 1 -- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402006d1f0 msgr2=0x7f40201f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4024bdb640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402006d1f0 0x7f40201f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4024bdb640 1 -- 172.21.15.28:0/3039721361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4020156ce0 con 0x7f402015bb40 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4024bdb640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402015bb40 0x7f40201f2fb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f401000b3c0 tx=0x7f401000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4015ffb640 1 -- 172.21.15.28:0/3039721361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010015020 con 0x7f402015bb40 2024-09-10T20:39:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4015ffb640 1 -- 172.21.15.28:0/3039721361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4010011920 con 0x7f402015bb40 2024-09-10T20:39:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4020156fc0 con 0x7f402015bb40 2024-09-10T20:39:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4015ffb640 1 -- 172.21.15.28:0/3039721361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010011db0 con 0x7f402015bb40 2024-09-10T20:39:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.856+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40201f27b0 con 0x7f402015bb40 2024-09-10T20:39:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.857+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40201f2ca0 con 0x7f402015bb40 2024-09-10T20:39:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.858+0000 7f4015ffb640 1 -- 172.21.15.28:0/3039721361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f401001a400 con 0x7f402015bb40 2024-09-10T20:39:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.858+0000 7f4015ffb640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078120 0x7f400407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.858+0000 7f4015ffb640 1 -- 172.21.15.28:0/3039721361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f401009a080 con 0x7f402015bb40 2024-09-10T20:39:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.859+0000 7f40253dc640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078120 0x7f400407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.859+0000 7f40253dc640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078120 0x7f400407a5e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f401c00abe0 tx=0x7f401c00a2d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:32.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:32.863+0000 7f4015ffb640 1 -- 172.21.15.28:0/3039721361 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f401001f030 con 0x7f402015bb40 2024-09-10T20:39:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.083+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f402015dbe0 con 0x7f402015bb40 2024-09-10T20:39:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.084+0000 7f4015ffb640 1 -- 172.21.15.28:0/3039721361 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f4010069e20 con 0x7f402015bb40 2024-09-10T20:39:33.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:33.088 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-10T20:39:26:475950+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-10T20:39:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078120 msgr2=0x7f400407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078120 0x7f400407a5e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f401c00abe0 tx=0x7f401c00a2d0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402015bb40 msgr2=0x7f40201f2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402015bb40 0x7f40201f2fb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f401000b3c0 tx=0x7f401000b7c0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 shutdown_connections 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078120 0x7f400407a5e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402015bf20 0x7f402009b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402015bb40 0x7f40201f2fb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 --2- 172.21.15.28:0/3039721361 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402006d1f0 0x7f40201f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 >> 172.21.15.28:0/3039721361 conn(0x7f402006dbf0 msgr2=0x7f40200af620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 shutdown_connections 2024-09-10T20:39:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.086+0000 7f4027640640 1 -- 172.21.15.28:0/3039721361 wait complete. 2024-09-10T20:39:33.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:39:22.829916+0000', 'modified': '2024-09-10T20:39:25.841932+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5286}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5286': {'gid': 5286, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3839556955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3839556955}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3839556955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5286, 'qdb_cluster': [5286]} 2024-09-10T20:39:33.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:39:33.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-10T20:39:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.611+0000 7f0edbeb6640 1 -- 172.21.15.28:0/3669718378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed415de70 msgr2=0x7f0ed40d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.611+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/3669718378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed415de70 0x7f0ed40d2b90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40d4d80 tx=0x7f0ed002a560 comp rx=0 tx=0).stop 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.611+0000 7f0edbeb6640 1 -- 172.21.15.28:0/3669718378 shutdown_connections 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.611+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/3669718378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ed4161b50 0x7f0ed4161f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.611+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/3669718378 >> v2:172.21.15.159:3300/0 conn(0x7f0ed40cf780 0x7f0ed40cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.611+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/3669718378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed415de70 0x7f0ed40d2b90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.611+0000 7f0edbeb6640 1 -- 172.21.15.28:0/3669718378 >> 172.21.15.28:0/3669718378 conn(0x7f0ed40d3fb0 msgr2=0x7f0ed40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.612+0000 7f0edbeb6640 1 -- 172.21.15.28:0/3669718378 shutdown_connections 2024-09-10T20:39:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.612+0000 7f0edbeb6640 1 -- 172.21.15.28:0/3669718378 wait complete. 2024-09-10T20:39:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.613+0000 7f0edbeb6640 1 Processor -- start 2024-09-10T20:39:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.613+0000 7f0edbeb6640 1 -- start start 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0edbeb6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ed40cf780 0x7f0ed40b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0edbeb6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ed415de70 0x7f0ed40b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0edbeb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4161b50 0x7f0ed4152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0edbeb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed40b6e00 con 0x7f0ed4161b50 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0edbeb6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed415e460 con 0x7f0ed40cf780 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0edbeb6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed40b6f80 con 0x7f0ed415de70 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4161b50 0x7f0ed4152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4161b50 0x7f0ed4152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46394/0 (socket says 172.21.15.28:46394) 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 -- 172.21.15.28:0/85136791 learned_addr learned my addr 172.21.15.28:0/85136791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0ed9c52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ed40cf780 0x7f0ed40b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0ed9451640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ed415de70 0x7f0ed40b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 -- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ed415de70 msgr2=0x7f0ed40b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ed415de70 0x7f0ed40b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 -- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ed40cf780 msgr2=0x7f0ed40b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ed40cf780 0x7f0ed40b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 -- 172.21.15.28:0/85136791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed40b46d0 con 0x7f0ed4161b50 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.614+0000 7f0eda453640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4161b50 0x7f0ed4152540 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00b970 tx=0x7f0ecc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.615+0000 7f0eca7fc640 1 -- 172.21.15.28:0/85136791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc019070 con 0x7f0ed4161b50 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.615+0000 7f0eca7fc640 1 -- 172.21.15.28:0/85136791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ecc00cbb0 con 0x7f0ed4161b50 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.615+0000 7f0eca7fc640 1 -- 172.21.15.28:0/85136791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc014440 con 0x7f0ed4161b50 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.615+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed40b0c80 con 0x7f0ed4161b50 2024-09-10T20:39:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.615+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed40b1240 con 0x7f0ed4161b50 2024-09-10T20:39:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.617+0000 7f0eca7fc640 1 -- 172.21.15.28:0/85136791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f0ecc01d3b0 con 0x7f0ed4161b50 2024-09-10T20:39:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.617+0000 7f0eca7fc640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0eb8078120 0x7f0eb807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.617+0000 7f0eca7fc640 1 -- 172.21.15.28:0/85136791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0ecc09a110 con 0x7f0ed4161b50 2024-09-10T20:39:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.617+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0eb8078120 0x7f0eb807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.618+0000 7f0edec83740 1 Processor -- start 2024-09-10T20:39:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.618+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0eb8078120 0x7f0eb807a5e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0033000 tx=0x7f0ed0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.618+0000 7f0edec83740 1 -- start start 2024-09-10T20:39:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6181f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0edec83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d5cd5f0 0x56324d5cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0edec83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56324d544820 con 0x56324d5cd5f0 2024-09-10T20:39:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56324d533c50 con 0x56324d5f81b0 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56324d537a60 con 0x56324d617df0 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d5cd5f0 0x56324d5cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d5cd5f0 0x56324d5cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46406/0 (socket says 172.21.15.28:46406) 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 -- 172.21.15.28:0/2816927873 learned_addr learned my addr 172.21.15.28:0/2816927873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0ed9451640 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 -- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 msgr2=0x56324d6181f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6181f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 -- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 msgr2=0x56324d5cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 -- 172.21.15.28:0/2816927873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56324d295850 con 0x56324d5cd5f0 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eda453640 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d5cd5f0 0x56324d5cda40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc062140 tx=0x7f0ecc0620b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9ce17fb6c0d0c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eb77fe640 1 -- 172.21.15.28:0/2816927873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc066be0 con 0x56324d5cd5f0 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eb77fe640 1 -- 172.21.15.28:0/2816927873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ecc06b600 con 0x56324d5cd5f0 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.619+0000 7f0eb77fe640 1 -- 172.21.15.28:0/2816927873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0987c0 con 0x56324d5cd5f0 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 -- 172.21.15.28:0/2816927873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d5cd5f0 msgr2=0x56324d5cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d5cd5f0 0x56324d5cda40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc062140 tx=0x7f0ecc0620b0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 -- 172.21.15.28:0/2816927873 shutdown_connections 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d5cd5f0 0x56324d5cda40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6181f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 --2- 172.21.15.28:0/2816927873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 -- 172.21.15.28:0/2816927873 >> 172.21.15.28:0/2816927873 conn(0x56324d5ce6a0 msgr2=0x56324d5d20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 -- 172.21.15.28:0/2816927873 shutdown_connections 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.620+0000 7f0edec83740 1 -- 172.21.15.28:0/2816927873 wait complete. 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.621+0000 7f0edec83740 1 -- start start 2024-09-10T20:39:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.621+0000 7f0edec83740 1 Processor -- start 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d62a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6128f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d5cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56324d2a4620 con 0x56324d612e30 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56324d4d95f0 con 0x56324d5f81b0 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56324d5cc890 con 0x56324d617df0 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 10 client.0 _ll_register_callbacks cb 0x56324d4dd0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d5cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d5cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46416/0 (socket says 172.21.15.28:46416) 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 -- 172.21.15.28:0/1598275033 learned_addr learned my addr 172.21.15.28:0/1598275033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d62a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0ed9451640 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 -- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 msgr2=0x56324d6128f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6128f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 -- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 msgr2=0x56324d62a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d62a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 -- 172.21.15.28:0/1598275033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56324d30c6b0 con 0x56324d612e30 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eda453640 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d5cb5d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc062140 tx=0x7f0ecc098850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebb58a0e083bb5e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eb4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/1598275033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc066be0 con 0x56324d612e30 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/1598275033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0ecc061410 con 0x56324d612e30 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ecc06b600 con 0x56324d612e30 2024-09-10T20:39:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.622+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/1598275033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0ecc061610 con 0x56324d612e30 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 10 client.5439 fetch_fsmap learned FSMap version 26 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x56324d5fb5b0 con 0x56324d612e30 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/1598275033 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ecc06b600 con 0x56324d612e30 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/1598275033 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 3502+0+0 (secure 0 0 0) 0x7f0ecc05db00 con 0x56324d612e30 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 10 client.5439 fetch_fsmap finished waiting for FSMap version 26 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 10 client.5439 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 20 client.5439 populate_metadata read hostname 'smithi028' 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d5fbad0 0x56324d5fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 4 client.5439 mds_command: new command op to 5286 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 --> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56324d546770 con 0x56324d5fbad0 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d5fbad0 0x56324d5fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0eb4ff9640 10 client.5439 ms_handle_connect on v2:172.21.15.28:6833/3839556955 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.623+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d5fbad0 0x56324d5fbeb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da6377f78727c48b server_cookie=4080a9f0fc2879c7 in_seq=0 out_seq=0 2024-09-10T20:39:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/1598275033 <== mds.0 v2:172.21.15.28:6833/3839556955 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56324d546770 con 0x56324d5fbad0 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0eb4ff9640 10 client.5439 handle_command_reply: tid=0 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 client.5439 shutdown 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 msgr2=0x56324d5cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d5cb5d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc062140 tx=0x7f0ecc098850 comp rx=0 tx=0).stop 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 shutdown_connections 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d5fbad0 0x56324d5fbeb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d5cb5d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d6128f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0eb4ff9640 0 client.5439 ms_handle_reset on v2:172.21.15.28:6833/3839556955 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 --2- 172.21.15.28:0/1598275033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d62a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 >> 172.21.15.28:0/1598275033 conn(0x56324d5cd5f0 msgr2=0x56324d5d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 shutdown_connections 2024-09-10T20:39:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 1 -- 172.21.15.28:0/1598275033 wait complete. 2024-09-10T20:39:33.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.624+0000 7f0edec83740 20 client.5439 trim_cache size 0 max 16384 2024-09-10T20:39:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 Processor -- start 2024-09-10T20:39:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 -- start start 2024-09-10T20:39:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d5f81b0 0x56324d5fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d612e30 0x56324d5fc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d617df0 0x56324d5f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41f8ff0 con 0x56324d617df0 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41f8350 con 0x56324d612e30 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed41f89a0 con 0x56324d5f81b0 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.643+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d617df0 0x56324d5f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d617df0 0x56324d5f8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46428/0 (socket says 172.21.15.28:46428) 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 -- 172.21.15.28:0/3720752576 learned_addr learned my addr 172.21.15.28:0/3720752576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d5f81b0 0x56324d5fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0ed9451640 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d612e30 0x56324d5fc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 -- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d5f81b0 msgr2=0x56324d5fbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d5f81b0 0x56324d5fbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 -- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d612e30 msgr2=0x56324d5fc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d612e30 0x56324d5fc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 -- 172.21.15.28:0/3720752576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56324d5cb940 con 0x56324d617df0 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0ed9451640 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d612e30 0x56324d5fc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eda453640 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d617df0 0x56324d5f8d80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc05e730 tx=0x7f0ecc0620b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ec642b75979351 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eb6ffd640 1 -- 172.21.15.28:0/3720752576 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc06b600 con 0x56324d617df0 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eb6ffd640 1 -- 172.21.15.28:0/3720752576 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ecc06b380 con 0x56324d617df0 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.644+0000 7f0eb6ffd640 1 -- 172.21.15.28:0/3720752576 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0090a0 con 0x56324d617df0 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 -- 172.21.15.28:0/3720752576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d617df0 msgr2=0x56324d5f8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d617df0 0x56324d5f8d80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc05e730 tx=0x7f0ecc0620b0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 -- 172.21.15.28:0/3720752576 shutdown_connections 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d617df0 0x56324d5f8d80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d612e30 0x56324d5fc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 --2- 172.21.15.28:0/3720752576 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d5f81b0 0x56324d5fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 -- 172.21.15.28:0/3720752576 >> 172.21.15.28:0/3720752576 conn(0x56324d5cd5f0 msgr2=0x56324d629d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 -- 172.21.15.28:0/3720752576 shutdown_connections 2024-09-10T20:39:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 -- 172.21.15.28:0/3720752576 wait complete. 2024-09-10T20:39:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 -- start start 2024-09-10T20:39:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.645+0000 7f0edec83740 1 Processor -- start 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5f9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0edec83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d68e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0edec83740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d69ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0edec83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56324d2a4620 con 0x56324d612e30 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56324d4d95f0 con 0x56324d5f81b0 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0edec83740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56324d626ad0 con 0x56324d617df0 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0edec83740 10 client.0 _ll_register_callbacks cb 0x7f0ecc061b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0ed9451640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d68e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0ed9451640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d68e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46442/0 (socket says 172.21.15.28:46442) 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0ed9451640 1 -- 172.21.15.28:0/3093292649 learned_addr learned my addr 172.21.15.28:0/3093292649 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5f9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.646+0000 7f0eda453640 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d69ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0ed9451640 1 -- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 msgr2=0x56324d69ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0ed9451640 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d69ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0ed9451640 1 -- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 msgr2=0x56324d5f9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0ed9451640 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5f9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0ed9451640 1 -- 172.21.15.28:0/3093292649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56324d5d0fb0 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0ed9451640 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d68e020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0ed41f9410 tx=0x7f0ec4015730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5b83ad46d699dd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0eb4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/3093292649 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec401b070 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/3093292649 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0ec4020660 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ecc06b380 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/3093292649 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0ec4028050 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 10 client.5445 fetch_fsmap learned FSMap version 26 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x56324d5cc170 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/3093292649 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ecc06b380 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/3093292649 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 3502+0+0 (secure 0 0 0) 0x7f0ec402b020 con 0x56324d612e30 2024-09-10T20:39:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 10 client.5445 fetch_fsmap finished waiting for FSMap version 26 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 10 client.5445 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 20 client.5445 populate_metadata read hostname 'smithi028' 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d63aa00 0x56324d610d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 4 client.5445 mds_command: new command op to 5286 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.647+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 --> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] -- command(tid 0: {"prefix": "status"}) -- 0x56324d542220 con 0x56324d63aa00 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d63aa00 0x56324d610d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0eb4ff9640 10 client.5445 ms_handle_connect on v2:172.21.15.28:6833/3839556955 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0ed9c52640 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d63aa00 0x56324d610d30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbe345f99c84fcc7 server_cookie=bc558b0c36e4657d in_seq=0 out_seq=0 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0eb4ff9640 1 -- 172.21.15.28:0/3093292649 <== mds.0 v2:172.21.15.28:6833/3839556955 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x56324d542220 con 0x56324d63aa00 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0eb4ff9640 10 client.5445 handle_command_reply: tid=0 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 client.5445 shutdown 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 msgr2=0x56324d68e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d68e020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0ed41f9410 tx=0x7f0ec4015730 comp rx=0 tx=0).stop 2024-09-10T20:39:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 shutdown_connections 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:6833/3839556955,v1:172.21.15.28:6835/3839556955] conn(0x56324d63aa00 0x56324d610d30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56324d617df0 0x56324d69ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56324d612e30 0x56324d68e020 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 --2- 172.21.15.28:0/3093292649 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56324d5f81b0 0x56324d5f9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 >> 172.21.15.28:0/3093292649 conn(0x56324d5cd5f0 msgr2=0x56324d617020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0eb4ff9640 0 client.5445 ms_handle_reset on v2:172.21.15.28:6833/3839556955 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 shutdown_connections 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.648+0000 7f0edec83740 1 -- 172.21.15.28:0/3093292649 wait complete. 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edec83740 20 client.5445 trim_cache size 0 max 16384 2024-09-10T20:39:33.658 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5286, 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 10.19653684, 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 43, 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 43, 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.889996708 2024-09-10T20:39:33.659 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:39:33.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0eb8078120 msgr2=0x7f0eb807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0eb8078120 0x7f0eb807a5e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0033000 tx=0x7f0ed0004730 comp rx=0 tx=0).stop 2024-09-10T20:39:33.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4161b50 msgr2=0x7f0ed4152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4161b50 0x7f0ed4152540 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00b970 tx=0x7f0ecc00bd70 comp rx=0 tx=0).stop 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 shutdown_connections 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0eb8078120 0x7f0eb807a5e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4161b50 0x7f0ed4152540 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ed415de70 0x7f0ed40b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 --2- 172.21.15.28:0/85136791 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ed40cf780 0x7f0ed40b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 >> 172.21.15.28:0/85136791 conn(0x7f0ed40d3fb0 msgr2=0x7f0ed40cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 shutdown_connections 2024-09-10T20:39:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.649+0000 7f0edbeb6640 1 -- 172.21.15.28:0/85136791 wait complete. 2024-09-10T20:39:33.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:39:33.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 -- 172.21.15.28:0/2271353824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78161b50 msgr2=0x7efd78161fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/2271353824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78161b50 0x7efd78161fc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7efd7808c020 tx=0x7efd7002aa30 comp rx=0 tx=0).stop 2024-09-10T20:39:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 -- 172.21.15.28:0/2271353824 shutdown_connections 2024-09-10T20:39:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/2271353824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78161b50 0x7efd78161fc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/2271353824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd780cf780 0x7efd780cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/2271353824 >> v2:172.21.15.159:3300/0 conn(0x7efd7815de70 0x7efd780d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 -- 172.21.15.28:0/2271353824 >> 172.21.15.28:0/2271353824 conn(0x7efd780d3fb0 msgr2=0x7efd780d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 -- 172.21.15.28:0/2271353824 shutdown_connections 2024-09-10T20:39:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.764+0000 7efd7ddd3640 1 -- 172.21.15.28:0/2271353824 wait complete. 2024-09-10T20:39:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 Processor -- start 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 -- start start 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd780cf780 0x7efd78160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd7815de70 0x7efd781f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd78161b50 0x7efd781f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd780b6e00 con 0x7efd7815de70 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efd7815e460 con 0x7efd780cf780 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.766+0000 7efd7ddd3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efd780b6f80 con 0x7efd78161b50 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd7815de70 0x7efd781f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd7815de70 0x7efd781f7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46466/0 (socket says 172.21.15.28:46466) 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 -- 172.21.15.28:0/3154243016 learned_addr learned my addr 172.21.15.28:0/3154243016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd777fe640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd780cf780 0x7efd78160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd77fff640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd78161b50 0x7efd781f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 -- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd78161b50 msgr2=0x7efd781f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd78161b50 0x7efd781f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 -- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd780cf780 msgr2=0x7efd78160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd780cf780 0x7efd78160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 -- 172.21.15.28:0/3154243016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd78160b10 con 0x7efd7815de70 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd76ffd640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd7815de70 0x7efd781f7c60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00d720 tx=0x7efd6c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd5ffff640 1 -- 172.21.15.28:0/3154243016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c017070 con 0x7efd7815de70 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd78160df0 con 0x7efd7815de70 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd5ffff640 1 -- 172.21.15.28:0/3154243016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd6c009d20 con 0x7efd7815de70 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd5ffff640 1 -- 172.21.15.28:0/3154243016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0106c0 con 0x7efd7815de70 2024-09-10T20:39:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.767+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd781f89c0 con 0x7efd7815de70 2024-09-10T20:39:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.768+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd781f5830 con 0x7efd7815de70 2024-09-10T20:39:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.769+0000 7efd5ffff640 1 -- 172.21.15.28:0/3154243016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7efd6c010860 con 0x7efd7815de70 2024-09-10T20:39:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.770+0000 7efd5ffff640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd64078120 0x7efd6407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.770+0000 7efd5ffff640 1 -- 172.21.15.28:0/3154243016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7efd6c09a5b0 con 0x7efd7815de70 2024-09-10T20:39:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.770+0000 7efd777fe640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd64078120 0x7efd6407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.770+0000 7efd777fe640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd64078120 0x7efd6407a5e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efd6800bbe0 tx=0x7efd6800b390 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.774+0000 7efd5ffff640 1 -- 172.21.15.28:0/3154243016 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efd6c0626e0 con 0x7efd7815de70 2024-09-10T20:39:34.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.998+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efd780d2b90 con 0x7efd7815de70 2024-09-10T20:39:34.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:33.999+0000 7efd5ffff640 1 -- 172.21.15.28:0/3154243016 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7efd6c067110 con 0x7efd7815de70 2024-09-10T20:39:34.003 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd64078120 msgr2=0x7efd6407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd64078120 0x7efd6407a5e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efd6800bbe0 tx=0x7efd6800b390 comp rx=0 tx=0).stop 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd7815de70 msgr2=0x7efd781f7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd7815de70 0x7efd781f7c60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00d720 tx=0x7efd6c00dbe0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 shutdown_connections 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efd64078120 0x7efd6407a5e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efd78161b50 0x7efd781f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd7815de70 0x7efd781f7c60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 --2- 172.21.15.28:0/3154243016 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efd780cf780 0x7efd78160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 >> 172.21.15.28:0/3154243016 conn(0x7efd780d3fb0 msgr2=0x7efd78075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 shutdown_connections 2024-09-10T20:39:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.001+0000 7efd7ddd3640 1 -- 172.21.15.28:0/3154243016 wait complete. 2024-09-10T20:39:34.015 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:39:34.016 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:39:34.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:34.016 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42657sec preferred_lft 42657sec 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:39:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:39:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:34.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:34.036 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:39:34.097 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:39:34.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:34.098 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:39:34.156 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:39:34.157 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:39:34.157 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:39:34.157 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:39:34.157 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:39:34.157 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:39:34.157 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:39:34.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:39:34.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:39:34.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:39:34.157 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:39:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:39:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:34.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:34.285 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:39:34.302 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:39:34.302 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:39:34.302 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:39:34.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:34.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:39:34.368 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:34.369 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:34.424 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:39:34.425 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:34.425 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:34.480 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:39:34.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:34.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:34.566 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:34.566 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:39:34.566 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:39:34.575 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:34.575 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:39:34.575 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:39:34.575 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:34.575 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:34.580 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:39:34.580 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:39:34.580 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:39:34.580 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:34.580 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:39:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:39:34.583 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:39:34.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:34.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:34.609 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:39:34.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:39:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:34.710 INFO:teuthology.orchestra.run.smithi028.stdout:client5454 2024-09-10T20:39:34.711 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:34.711 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:34.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.868+0000 7fea671c2640 1 -- 172.21.15.28:0/3211697370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea60073820 msgr2=0x7fea60073cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.868+0000 7fea671c2640 1 --2- 172.21.15.28:0/3211697370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea60073820 0x7fea60073cd0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fea58005fd0 tx=0x7fea5802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.869+0000 7fea671c2640 1 -- 172.21.15.28:0/3211697370 shutdown_connections 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.869+0000 7fea671c2640 1 --2- 172.21.15.28:0/3211697370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea60073820 0x7fea60073cd0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.869+0000 7fea671c2640 1 --2- 172.21.15.28:0/3211697370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea601418c0 0x7fea6015f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.869+0000 7fea671c2640 1 -- 172.21.15.28:0/3211697370 >> 172.21.15.28:0/3211697370 conn(0x7fea600b2990 msgr2=0x7fea600b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.869+0000 7fea671c2640 1 -- 172.21.15.28:0/3211697370 shutdown_connections 2024-09-10T20:39:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.869+0000 7fea671c2640 1 -- 172.21.15.28:0/3211697370 wait complete. 2024-09-10T20:39:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.870+0000 7fea671c2640 1 Processor -- start 2024-09-10T20:39:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.870+0000 7fea671c2640 1 -- start start 2024-09-10T20:39:34.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea60073820 0x7fea600b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:34.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea601418c0 0x7fea60130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:34.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea60141ca0 0x7fea60132c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea6006cc60 con 0x7fea601418c0 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fea6006cae0 con 0x7fea60073820 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fea6006cde0 con 0x7fea60141ca0 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea601418c0 0x7fea60130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea601418c0 0x7fea60130dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46496/0 (socket says 172.21.15.28:46496) 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 -- 172.21.15.28:0/769636417 learned_addr learned my addr 172.21.15.28:0/769636417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 -- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea60141ca0 msgr2=0x7fea60132c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea60141ca0 0x7fea60132c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 -- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea60073820 msgr2=0x7fea600b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea60073820 0x7fea600b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 -- 172.21.15.28:0/769636417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6012fd20 con 0x7fea601418c0 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5ffff640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea601418c0 0x7fea60130dc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00c430 tx=0x7fea4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5d7fa640 1 -- 172.21.15.28:0/769636417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c011790 con 0x7fea601418c0 2024-09-10T20:39:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5d7fa640 1 -- 172.21.15.28:0/769636417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea4c017070 con 0x7fea601418c0 2024-09-10T20:39:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea60130000 con 0x7fea601418c0 2024-09-10T20:39:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea5d7fa640 1 -- 172.21.15.28:0/769636417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c011cc0 con 0x7fea601418c0 2024-09-10T20:39:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.871+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea601305c0 con 0x7fea601418c0 2024-09-10T20:39:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.872+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea6006d2a0 con 0x7fea601418c0 2024-09-10T20:39:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.873+0000 7fea5d7fa640 1 -- 172.21.15.28:0/769636417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fea4c01c460 con 0x7fea601418c0 2024-09-10T20:39:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.874+0000 7fea5d7fa640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fea44078120 0x7fea4407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.874+0000 7fea5d7fa640 1 -- 172.21.15.28:0/769636417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fea4c099ce0 con 0x7fea601418c0 2024-09-10T20:39:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.874+0000 7fea64f5e640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fea44078120 0x7fea4407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.875+0000 7fea64f5e640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fea44078120 0x7fea4407a5e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fea54005fd0 tx=0x7fea54005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:34.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:34.878+0000 7fea5d7fa640 1 -- 172.21.15.28:0/769636417 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fea4c021020 con 0x7fea601418c0 2024-09-10T20:39:35.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.099+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fea60073ff0 con 0x7fea601418c0 2024-09-10T20:39:35.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.099+0000 7fea5d7fa640 1 -- 172.21.15.28:0/769636417 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7fea4c062510 con 0x7fea601418c0 2024-09-10T20:39:35.104 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:39:35.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.101+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fea44078120 msgr2=0x7fea4407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:35.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.101+0000 7fea671c2640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fea44078120 0x7fea4407a5e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fea54005fd0 tx=0x7fea54005850 comp rx=0 tx=0).stop 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.101+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea601418c0 msgr2=0x7fea60130dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.101+0000 7fea671c2640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea601418c0 0x7fea60130dc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00c430 tx=0x7fea4c00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.101+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 shutdown_connections 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.102+0000 7fea671c2640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fea44078120 0x7fea4407a5e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.102+0000 7fea671c2640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea60141ca0 0x7fea60132c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.102+0000 7fea671c2640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea601418c0 0x7fea60130dc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.102+0000 7fea671c2640 1 --2- 172.21.15.28:0/769636417 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea60073820 0x7fea600b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.102+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 >> 172.21.15.28:0/769636417 conn(0x7fea600b2990 msgr2=0x7fea600ae9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.102+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 shutdown_connections 2024-09-10T20:39:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.102+0000 7fea671c2640 1 -- 172.21.15.28:0/769636417 wait complete. 2024-09-10T20:39:35.116 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:39:35.116 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:39:35.116 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:39:35.116 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:39:35.116 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:35.117 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:39:35.117 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:39:35.117 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:39:35.117 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:39:35.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:35.117 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:39:35.137 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:39:35.137 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:39:35.138 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42674sec preferred_lft 42674sec 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:39:35.139 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:35.140 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:39:35.140 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:39:35.140 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:39:35.140 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:39:35.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:35.140 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:39:35.205 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:39:35.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:35.207 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:39:35.265 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:39:35.265 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:39:35.266 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:39:35.266 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:39:35.266 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:39:35.266 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:39:35.267 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:39:35.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:39:35.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:39:35.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:39:35.267 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:39:35.358 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:39:35.391 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:35.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:35.398 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:39:35.414 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:39:35.414 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:39:35.414 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:39:35.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:35.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:39:35.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:35.480 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:35.535 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:39:35.536 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:39:35.536 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:35.591 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:39:35.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:35.591 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:35.675 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:35.675 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:39:35.675 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:39:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:39:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:39:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:39:35.689 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:39:35.691 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:39:35.691 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:39:35.691 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:39:35.691 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:39:35.691 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:39:35.692 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:39:35.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:35.692 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:35.757 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:39:35.846 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:39:35.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:39:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:39:35.853 INFO:teuthology.orchestra.run.smithi159.stdout:client4988 2024-09-10T20:39:35.854 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:35.854 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:35.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 -- 172.21.15.28:0/757971550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748162000 msgr2=0x7fc7480d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/757971550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748162000 0x7fc7480d4a80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009830 tx=0x7fc73c02a5a0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 -- 172.21.15.28:0/757971550 shutdown_connections 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/757971550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc748161850 0x7fc7480d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/757971550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748162000 0x7fc7480d4a80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 -- 172.21.15.28:0/757971550 >> 172.21.15.28:0/757971550 conn(0x7fc7480d3f90 msgr2=0x7fc7480d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 -- 172.21.15.28:0/757971550 shutdown_connections 2024-09-10T20:39:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.989+0000 7fc74ef3d640 1 -- 172.21.15.28:0/757971550 wait complete. 2024-09-10T20:39:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.990+0000 7fc74ef3d640 1 Processor -- start 2024-09-10T20:39:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.990+0000 7fc74ef3d640 1 -- start start 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74ef3d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7480b4c20 0x7fc7481f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74ef3d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748161850 0x7fc7481564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74ef3d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc748162000 0x7fc7480b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74ef3d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7480d2320 con 0x7fc748161850 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74ef3d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc7480d21a0 con 0x7fc7480b4c20 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74ef3d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc7480d24a0 con 0x7fc748162000 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748161850 0x7fc7481564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748161850 0x7fc7481564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46520/0 (socket says 172.21.15.28:46520) 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 -- 172.21.15.28:0/1066898691 learned_addr learned my addr 172.21.15.28:0/1066898691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74ccd9640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7480b4c20 0x7fc7481f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc74d4da640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc748162000 0x7fc7480b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 -- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc748162000 msgr2=0x7fc7480b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc748162000 0x7fc7480b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 -- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7480b4c20 msgr2=0x7fc7481f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7480b4c20 0x7fc7481f5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.991+0000 7fc747fff640 1 -- 172.21.15.28:0/1066898691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7480b47d0 con 0x7fc748161850 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.992+0000 7fc747fff640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748161850 0x7fc7481564c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c500 tx=0x7fc73400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.992+0000 7fc7457fa640 1 -- 172.21.15.28:0/1066898691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734017070 con 0x7fc748161850 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.992+0000 7fc7457fa640 1 -- 172.21.15.28:0/1066898691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7340035f0 con 0x7fc748161850 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.992+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7480b0cd0 con 0x7fc748161850 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.992+0000 7fc7457fa640 1 -- 172.21.15.28:0/1066898691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734003a40 con 0x7fc748161850 2024-09-10T20:39:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.992+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc748156a00 con 0x7fc748161850 2024-09-10T20:39:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.993+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7480d1c60 con 0x7fc748161850 2024-09-10T20:39:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.994+0000 7fc7457fa640 1 -- 172.21.15.28:0/1066898691 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fc734003be0 con 0x7fc748161850 2024-09-10T20:39:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.994+0000 7fc7457fa640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc72c078120 0x7fc72c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.994+0000 7fc7457fa640 1 -- 172.21.15.28:0/1066898691 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fc734099740 con 0x7fc748161850 2024-09-10T20:39:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.994+0000 7fc74ccd9640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc72c078120 0x7fc72c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.995+0000 7fc74ccd9640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc72c078120 0x7fc72c07a5e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc73c034000 tx=0x7fc73c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:35.999+0000 7fc7457fa640 1 -- 172.21.15.28:0/1066898691 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc7340619a0 con 0x7fc748161850 2024-09-10T20:39:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.220+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fc74806e2f0 con 0x7fc748161850 2024-09-10T20:39:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.505+0000 7fc7457fa640 1 -- 172.21.15.28:0/1066898691 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v27) ==== 136+0+0 (secure 0 0 0) 0x7fc7340663d0 con 0x7fc748161850 2024-09-10T20:39:36.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.508+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc72c078120 msgr2=0x7fc72c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.508+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc72c078120 0x7fc72c07a5e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc73c034000 tx=0x7fc73c004730 comp rx=0 tx=0).stop 2024-09-10T20:39:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748161850 msgr2=0x7fc7481564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748161850 0x7fc7481564c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c500 tx=0x7fc73400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 shutdown_connections 2024-09-10T20:39:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc72c078120 0x7fc72c07a5e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc748162000 0x7fc7480b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc748161850 0x7fc7481564c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 --2- 172.21.15.28:0/1066898691 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7480b4c20 0x7fc7481f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 >> 172.21.15.28:0/1066898691 conn(0x7fc7480d3f90 msgr2=0x7fc7480ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 shutdown_connections 2024-09-10T20:39:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.509+0000 7fc74ef3d640 1 -- 172.21.15.28:0/1066898691 wait complete. 2024-09-10T20:39:36.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:39:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 -- 172.21.15.28:0/1949863478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8415ee70 msgr2=0x7fde840d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 --2- 172.21.15.28:0/1949863478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8415ee70 0x7fde840d0140 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fde8409e0b0 tx=0x7fde7c02a820 comp rx=0 tx=0).stop 2024-09-10T20:39:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 -- 172.21.15.28:0/1949863478 shutdown_connections 2024-09-10T20:39:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 --2- 172.21.15.28:0/1949863478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8415ee70 0x7fde840d0140 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 --2- 172.21.15.28:0/1949863478 >> v2:172.21.15.159:3300/0 conn(0x7fde841554e0 0x7fde8415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 -- 172.21.15.28:0/1949863478 >> 172.21.15.28:0/1949863478 conn(0x7fde8406cf00 msgr2=0x7fde8406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 -- 172.21.15.28:0/1949863478 shutdown_connections 2024-09-10T20:39:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.627+0000 7fde89124640 1 -- 172.21.15.28:0/1949863478 wait complete. 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.628+0000 7fde89124640 1 Processor -- start 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.628+0000 7fde89124640 1 -- start start 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde89124640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde84155100 0x7fde84159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde89124640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde841554e0 0x7fde84157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde89124640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde8415ee70 0x7fde840700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde89124640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde84072d50 con 0x7fde84155100 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde89124640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fde84072bd0 con 0x7fde841554e0 2024-09-10T20:39:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde89124640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fde84072ed0 con 0x7fde8415ee70 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde84155100 0x7fde84159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde84155100 0x7fde84159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46546/0 (socket says 172.21.15.28:46546) 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.629+0000 7fde82d9d640 1 -- 172.21.15.28:0/2665630288 learned_addr learned my addr 172.21.15.28:0/2665630288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde82d9d640 1 -- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde8415ee70 msgr2=0x7fde840700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde82d9d640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde8415ee70 0x7fde840700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde82d9d640 1 -- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde841554e0 msgr2=0x7fde84157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde82d9d640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde841554e0 0x7fde84157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde82d9d640 1 -- 172.21.15.28:0/2665630288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8415a290 con 0x7fde84155100 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde82d9d640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde84155100 0x7fde84159d50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fde7400c430 tx=0x7fde7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde6b7fe640 1 -- 172.21.15.28:0/2665630288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74011790 con 0x7fde84155100 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde6b7fe640 1 -- 172.21.15.28:0/2665630288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde74017070 con 0x7fde84155100 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde84158040 con 0x7fde84155100 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde6b7fe640 1 -- 172.21.15.28:0/2665630288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74011ce0 con 0x7fde84155100 2024-09-10T20:39:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.630+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde841f82d0 con 0x7fde84155100 2024-09-10T20:39:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.631+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8415e930 con 0x7fde84155100 2024-09-10T20:39:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.632+0000 7fde6b7fe640 1 -- 172.21.15.28:0/2665630288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fde7401c460 con 0x7fde84155100 2024-09-10T20:39:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.632+0000 7fde6b7fe640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fde70078120 0x7fde7007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.632+0000 7fde6b7fe640 1 -- 172.21.15.28:0/2665630288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fde74099ee0 con 0x7fde84155100 2024-09-10T20:39:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.633+0000 7fde8259c640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fde70078120 0x7fde7007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.633+0000 7fde8259c640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fde70078120 0x7fde7007a5e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fde78005fd0 tx=0x7fde78005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.637+0000 7fde6b7fe640 1 -- 172.21.15.28:0/2665630288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fde740626e0 con 0x7fde84155100 2024-09-10T20:39:36.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.846+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fde840741c0 con 0x7fde70078120 2024-09-10T20:39:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.848+0000 7fde6b7fe640 1 -- 172.21.15.28:0/2665630288 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fde840741c0 con 0x7fde70078120 2024-09-10T20:39:36.851 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:39:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:39:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:39:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:39:36.852 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fde70078120 msgr2=0x7fde7007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fde70078120 0x7fde7007a5e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fde78005fd0 tx=0x7fde78005850 comp rx=0 tx=0).stop 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde84155100 msgr2=0x7fde84159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde84155100 0x7fde84159d50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fde7400c430 tx=0x7fde7400c8f0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 shutdown_connections 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fde70078120 0x7fde7007a5e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde8415ee70 0x7fde840700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde841554e0 0x7fde84157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 --2- 172.21.15.28:0/2665630288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde84155100 0x7fde84159d50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 >> 172.21.15.28:0/2665630288 conn(0x7fde8406cf00 msgr2=0x7fde8415df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 shutdown_connections 2024-09-10T20:39:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.850+0000 7fde89124640 1 -- 172.21.15.28:0/2665630288 wait complete. 2024-09-10T20:39:36.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 -- 172.21.15.28:0/552246142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815adf0 msgr2=0x7f32680cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 --2- 172.21.15.28:0/552246142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815adf0 0x7f32680cb230 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f325c009830 tx=0x7f325c02a3f0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 -- 172.21.15.28:0/552246142 shutdown_connections 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 --2- 172.21.15.28:0/552246142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f326815e7a0 0x7f32680d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 --2- 172.21.15.28:0/552246142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815adf0 0x7f32680cb230 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 -- 172.21.15.28:0/552246142 >> 172.21.15.28:0/552246142 conn(0x7f326806cf00 msgr2=0x7f326806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 -- 172.21.15.28:0/552246142 shutdown_connections 2024-09-10T20:39:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.970+0000 7f326e040640 1 -- 172.21.15.28:0/552246142 wait complete. 2024-09-10T20:39:36.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.971+0000 7f326e040640 1 Processor -- start 2024-09-10T20:39:36.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.971+0000 7f326e040640 1 -- start start 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f326e040640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815aa10 0x7f32680cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f326e040640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f326815adf0 0x7f3268150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f326e040640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326815e7a0 0x7f3268151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f326e040640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3268070310 con 0x7f326815e7a0 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f326e040640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3268070190 con 0x7f326815aa10 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f326e040640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3268070490 con 0x7f326815adf0 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326815e7a0 0x7f3268151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326815e7a0 0x7f3268151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46554/0 (socket says 172.21.15.28:46554) 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 -- 172.21.15.28:0/1123653949 learned_addr learned my addr 172.21.15.28:0/1123653949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f32677fe640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815aa10 0x7f32680cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3266ffd640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f326815adf0 0x7f3268150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 -- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f326815adf0 msgr2=0x7f3268150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f326815adf0 0x7f3268150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 -- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815aa10 msgr2=0x7f32680cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815aa10 0x7f32680cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 -- 172.21.15.28:0/1123653949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32681f7a40 con 0x7f326815e7a0 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.972+0000 7f3267fff640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326815e7a0 0x7f3268151010 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f326000ec20 tx=0x7f326000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.973+0000 7f324ffff640 1 -- 172.21.15.28:0/1123653949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260003b70 con 0x7f326815e7a0 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.973+0000 7f324ffff640 1 -- 172.21.15.28:0/1123653949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3260003e00 con 0x7f326815e7a0 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.973+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326815e050 con 0x7f326815e7a0 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.973+0000 7f324ffff640 1 -- 172.21.15.28:0/1123653949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260004280 con 0x7f326815e7a0 2024-09-10T20:39:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.973+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32681f82a0 con 0x7f326815e7a0 2024-09-10T20:39:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.974+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3268070940 con 0x7f326815e7a0 2024-09-10T20:39:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.974+0000 7f324ffff640 1 -- 172.21.15.28:0/1123653949 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f3260004420 con 0x7f326815e7a0 2024-09-10T20:39:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.975+0000 7f324ffff640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3254078120 0x7f325407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.975+0000 7f324ffff640 1 -- 172.21.15.28:0/1123653949 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f326009af10 con 0x7f326815e7a0 2024-09-10T20:39:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.975+0000 7f32677fe640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3254078120 0x7f325407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.976+0000 7f32677fe640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3254078120 0x7f325407a5e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3258007fd0 tx=0x7f3258001410 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:36.979+0000 7f324ffff640 1 -- 172.21.15.28:0/1123653949 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3260063040 con 0x7f326815e7a0 2024-09-10T20:39:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.164+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f32680b2140 con 0x7f326815e7a0 2024-09-10T20:39:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.164+0000 7f324ffff640 1 -- 172.21.15.28:0/1123653949 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7f3260067a70 con 0x7f326815e7a0 2024-09-10T20:39:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3254078120 msgr2=0x7f325407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3254078120 0x7f325407a5e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3258007fd0 tx=0x7f3258001410 comp rx=0 tx=0).stop 2024-09-10T20:39:37.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326815e7a0 msgr2=0x7f3268151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326815e7a0 0x7f3268151010 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f326000ec20 tx=0x7f326000c6a0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 shutdown_connections 2024-09-10T20:39:37.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3254078120 0x7f325407a5e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326815e7a0 0x7f3268151010 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f326815adf0 0x7f3268150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 --2- 172.21.15.28:0/1123653949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f326815aa10 0x7f32680cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.166+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 >> 172.21.15.28:0/1123653949 conn(0x7f326806cf00 msgr2=0x7f32680b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.167+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 shutdown_connections 2024-09-10T20:39:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.167+0000 7f326e040640 1 -- 172.21.15.28:0/1123653949 wait complete. 2024-09-10T20:39:37.182 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:37.182 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:37.198 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:39:37.198 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:37.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:39:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.397+0000 7fcf97487640 1 -- 172.21.15.28:0/399026998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 msgr2=0x7fcf900cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.397+0000 7fcf97487640 1 --2- 172.21.15.28:0/399026998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf900cfbf0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcf9008c020 tx=0x7fcf8002aac0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.397+0000 7fcf97487640 1 -- 172.21.15.28:0/399026998 shutdown_connections 2024-09-10T20:39:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.397+0000 7fcf97487640 1 --2- 172.21.15.28:0/399026998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf900cfbf0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.397+0000 7fcf97487640 1 --2- 172.21.15.28:0/399026998 >> v2:172.21.15.159:3300/0 conn(0x7fcf9015de70 0x7fcf900d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.397+0000 7fcf97487640 1 -- 172.21.15.28:0/399026998 >> 172.21.15.28:0/399026998 conn(0x7fcf900d3fb0 msgr2=0x7fcf900d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.398+0000 7fcf97487640 1 -- 172.21.15.28:0/399026998 shutdown_connections 2024-09-10T20:39:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.398+0000 7fcf97487640 1 -- 172.21.15.28:0/399026998 wait complete. 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 Processor -- start 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 -- start start 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf900cf780 0x7fcf901f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf9015de70 0x7fcf901f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf90152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf900b6f80 con 0x7fcf90162260 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcf900b6e00 con 0x7fcf900cf780 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.399+0000 7fcf97487640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcf900b7100 con 0x7fcf9015de70 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf90152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf90152270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46568/0 (socket says 172.21.15.28:46568) 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 -- 172.21.15.28:0/1250750389 learned_addr learned my addr 172.21.15.28:0/1250750389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95223640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf900cf780 0x7fcf901f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf94a22640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf9015de70 0x7fcf901f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 -- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf9015de70 msgr2=0x7fcf901f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf9015de70 0x7fcf901f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 -- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf900cf780 msgr2=0x7fcf901f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf900cf780 0x7fcf901f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 -- 172.21.15.28:0/1250750389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf901f4a90 con 0x7fcf90162260 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf95a24640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf90152270 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009d60 tx=0x7fcf80003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf8dffb640 1 -- 172.21.15.28:0/1250750389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80044070 con 0x7fcf90162260 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf8dffb640 1 -- 172.21.15.28:0/1250750389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf80004250 con 0x7fcf90162260 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf900b44c0 con 0x7fcf90162260 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.400+0000 7fcf8dffb640 1 -- 172.21.15.28:0/1250750389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf800046a0 con 0x7fcf90162260 2024-09-10T20:39:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.401+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf901f4d50 con 0x7fcf90162260 2024-09-10T20:39:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.402+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf900d2b90 con 0x7fcf90162260 2024-09-10T20:39:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.402+0000 7fcf8dffb640 1 -- 172.21.15.28:0/1250750389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fcf8003f410 con 0x7fcf90162260 2024-09-10T20:39:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.403+0000 7fcf8dffb640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcf7c078120 0x7fcf7c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.403+0000 7fcf8dffb640 1 -- 172.21.15.28:0/1250750389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fcf800ba670 con 0x7fcf90162260 2024-09-10T20:39:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.403+0000 7fcf95223640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcf7c078120 0x7fcf7c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.404+0000 7fcf95223640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcf7c078120 0x7fcf7c07a5e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcf88005fd0 tx=0x7fcf8800b390 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.407+0000 7fcf8dffb640 1 -- 172.21.15.28:0/1250750389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcf80032030 con 0x7fcf90162260 2024-09-10T20:39:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.604+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcf9008dfd0 con 0x7fcf90162260 2024-09-10T20:39:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.604+0000 7fcf8dffb640 1 -- 172.21.15.28:0/1250750389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7fcf800828b0 con 0x7fcf90162260 2024-09-10T20:39:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:39:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.606+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcf7c078120 msgr2=0x7fcf7c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.606+0000 7fcf97487640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcf7c078120 0x7fcf7c07a5e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcf88005fd0 tx=0x7fcf8800b390 comp rx=0 tx=0).stop 2024-09-10T20:39:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 msgr2=0x7fcf90152270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf90152270 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009d60 tx=0x7fcf80003ce0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 shutdown_connections 2024-09-10T20:39:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcf7c078120 0x7fcf7c07a5e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf90162260 0x7fcf90152270 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcf9015de70 0x7fcf901f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 --2- 172.21.15.28:0/1250750389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcf900cf780 0x7fcf901f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 >> 172.21.15.28:0/1250750389 conn(0x7fcf900d3fb0 msgr2=0x7fcf900d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 shutdown_connections 2024-09-10T20:39:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.607+0000 7fcf97487640 1 -- 172.21.15.28:0/1250750389 wait complete. 2024-09-10T20:39:37.621 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:39:37.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:37.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:39:37.651 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:39:37.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:37.651 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:39:37.707 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:37.707 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:37.728 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:39:37.728 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:39:37.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:39:37.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 -- 172.21.15.28:0/3515935096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 msgr2=0x7f6bd00b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/3515935096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 0x7f6bd00b47a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009830 tx=0x7f6bb802a510 comp rx=0 tx=0).stop 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 -- 172.21.15.28:0/3515935096 shutdown_connections 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/3515935096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 0x7f6bd00b47a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/3515935096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd00cd3d0 0x7f6bd00cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 -- 172.21.15.28:0/3515935096 >> 172.21.15.28:0/3515935096 conn(0x7f6bd006cf00 msgr2=0x7f6bd006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 -- 172.21.15.28:0/3515935096 shutdown_connections 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.889+0000 7f6bd58fe640 1 -- 172.21.15.28:0/3515935096 wait complete. 2024-09-10T20:39:37.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 Processor -- start 2024-09-10T20:39:37.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 -- start start 2024-09-10T20:39:37.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 0x7f6bd01f5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00cd3d0 0x7f6bd01f6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd00d0090 0x7f6bd015e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd00d09a0 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd00d0820 con 0x7f6bd00b4340 2024-09-10T20:39:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.891+0000 7f6bd58fe640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd00b3590 con 0x7f6bd00d0090 2024-09-10T20:39:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00cd3d0 0x7f6bd01f6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00cd3d0 0x7f6bd01f6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46576/0 (socket says 172.21.15.28:46576) 2024-09-10T20:39:37.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 -- 172.21.15.28:0/2163554974 learned_addr learned my addr 172.21.15.28:0/2163554974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:37.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bcf7fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd00d0090 0x7f6bd015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bceffd640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 0x7f6bd01f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 -- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd00d0090 msgr2=0x7f6bd015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd00d0090 0x7f6bd015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 -- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 msgr2=0x7f6bd01f5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:37.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 0x7f6bd01f5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 -- 172.21.15.28:0/2163554974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd01f67c0 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bcf7fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd00d0090 0x7f6bd015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bceffd640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 0x7f6bd01f5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bce7fc640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00cd3d0 0x7f6bd01f6280 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009950 tx=0x7f6bb8032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bc7fff640 1 -- 172.21.15.28:0/2163554974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8040070 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bc7fff640 1 -- 172.21.15.28:0/2163554974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bb8005310 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd00ad100 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bc7fff640 1 -- 172.21.15.28:0/2163554974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8005c90 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.892+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd00ad6c0 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.893+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd00b3ab0 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.894+0000 7f6bc7fff640 1 -- 172.21.15.28:0/2163554974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f6bb80047a0 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.894+0000 7f6bc7fff640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6bb0078120 0x7f6bb007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:37.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.895+0000 7f6bc7fff640 1 -- 172.21.15.28:0/2163554974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f6bb80bb540 con 0x7f6bd00cd3d0 2024-09-10T20:39:37.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.895+0000 7f6bceffd640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6bb0078120 0x7f6bb007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:37.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.895+0000 7f6bceffd640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6bb0078120 0x7f6bb007a5e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8005fd0 tx=0x7f6bc8009500 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:37.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:37.899+0000 7f6bc7fff640 1 -- 172.21.15.28:0/2163554974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6bd01f67c0 con 0x7f6bd00cd3d0 2024-09-10T20:39:38.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.094+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6bd00b2070 con 0x7f6bd00cd3d0 2024-09-10T20:39:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.095+0000 7f6bc7fff640 1 -- 172.21.15.28:0/2163554974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7f6bb8083c30 con 0x7f6bd00cd3d0 2024-09-10T20:39:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:39:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6bb0078120 msgr2=0x7f6bb007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6bb0078120 0x7f6bb007a5e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8005fd0 tx=0x7f6bc8009500 comp rx=0 tx=0).stop 2024-09-10T20:39:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00cd3d0 msgr2=0x7f6bd01f6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00cd3d0 0x7f6bd01f6280 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009950 tx=0x7f6bb8032c10 comp rx=0 tx=0).stop 2024-09-10T20:39:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 shutdown_connections 2024-09-10T20:39:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6bb0078120 0x7f6bb007a5e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bd00d0090 0x7f6bd015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00cd3d0 0x7f6bd01f6280 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 --2- 172.21.15.28:0/2163554974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bd00b4340 0x7f6bd01f5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 >> 172.21.15.28:0/2163554974 conn(0x7f6bd006cf00 msgr2=0x7f6bd006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 shutdown_connections 2024-09-10T20:39:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.097+0000 7f6bd58fe640 1 -- 172.21.15.28:0/2163554974 wait complete. 2024-09-10T20:39:38.111 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:39:38.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:38.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:39:38.146 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:39:38.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:39:38.146 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:39:38.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.307+0000 7fad5686a640 1 -- 172.21.15.28:0/2967374528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad38004800 msgr2=0x7fad38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.307+0000 7fad5686a640 1 --2- 172.21.15.28:0/2967374528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad38004800 0x7fad38004c80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fad44009b80 tx=0x7fad38005830 comp rx=0 tx=0).stop 2024-09-10T20:39:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.307+0000 7fad54e07640 1 -- 172.21.15.28:0/2967374528 reap_dead start 2024-09-10T20:39:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.307+0000 7fad5686a640 1 -- 172.21.15.28:0/2967374528 shutdown_connections 2024-09-10T20:39:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.307+0000 7fad5686a640 1 -- 172.21.15.28:0/2967374528 >> 172.21.15.28:0/2967374528 conn(0x7fad500d1e40 msgr2=0x7fad500d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.307+0000 7fad5686a640 1 -- 172.21.15.28:0/2967374528 shutdown_connections 2024-09-10T20:39:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.307+0000 7fad5686a640 1 -- 172.21.15.28:0/2967374528 wait complete. 2024-09-10T20:39:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.308+0000 7fad5686a640 1 Processor -- start 2024-09-10T20:39:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.308+0000 7fad5686a640 1 -- start start 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad5686a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad38004800 0x7fad500af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad5686a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad501f4320 0x7fad501f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad5686a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5015e5b0 0x7fad5015ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad5686a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad500cd5f0 con 0x7fad5015e5b0 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad5686a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fad500d0d50 con 0x7fad501f4320 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad5686a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fad500b5a80 con 0x7fad38004800 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad54e07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5015e5b0 0x7fad5015ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad54e07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5015e5b0 0x7fad5015ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46586/0 (socket says 172.21.15.28:46586) 2024-09-10T20:39:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad54e07640 1 -- 172.21.15.28:0/2749537904 learned_addr learned my addr 172.21.15.28:0/2749537904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad4ffff640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad38004800 0x7fad500af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.309+0000 7fad4f7fe640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad501f4320 0x7fad501f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad54e07640 1 -- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad38004800 msgr2=0x7fad500af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad54e07640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad38004800 0x7fad500af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad54e07640 1 -- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad501f4320 msgr2=0x7fad501f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad54e07640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad501f4320 0x7fad501f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad54e07640 1 -- 172.21.15.28:0/2749537904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad501f7df0 con 0x7fad5015e5b0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad4f7fe640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad501f4320 0x7fad501f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad54e07640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5015e5b0 0x7fad5015ea30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fad48005fd0 tx=0x7fad48011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad4cff9640 1 -- 172.21.15.28:0/2749537904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48004b20 con 0x7fad5015e5b0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad4cff9640 1 -- 172.21.15.28:0/2749537904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad48004cc0 con 0x7fad5015e5b0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad4cff9640 1 -- 172.21.15.28:0/2749537904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48003bd0 con 0x7fad5015e5b0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad501f80d0 con 0x7fad5015e5b0 2024-09-10T20:39:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.310+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad501f8690 con 0x7fad5015e5b0 2024-09-10T20:39:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.311+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad501f8890 con 0x7fad5015e5b0 2024-09-10T20:39:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.312+0000 7fad4cff9640 1 -- 172.21.15.28:0/2749537904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7fad48018070 con 0x7fad5015e5b0 2024-09-10T20:39:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.312+0000 7fad4cff9640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fad3c078150 0x7fad3c07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.312+0000 7fad4cff9640 1 -- 172.21.15.28:0/2749537904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fad4809a8f0 con 0x7fad5015e5b0 2024-09-10T20:39:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.312+0000 7fad4ffff640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fad3c078150 0x7fad3c07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.313+0000 7fad4ffff640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fad3c078150 0x7fad3c07a610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fad500aec40 tx=0x7fad400067a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.316+0000 7fad4cff9640 1 -- 172.21.15.28:0/2749537904 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fad501f8890 con 0x7fad5015e5b0 2024-09-10T20:39:38.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.541+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad500b2940 con 0x7fad5015e5b0 2024-09-10T20:39:38.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.541+0000 7fad4cff9640 1 -- 172.21.15.28:0/2749537904 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7fad4801f060 con 0x7fad5015e5b0 2024-09-10T20:39:38.545 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:38.546 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:39:38.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.543+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fad3c078150 msgr2=0x7fad3c07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.543+0000 7fad5686a640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fad3c078150 0x7fad3c07a610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fad500aec40 tx=0x7fad400067a0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.543+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5015e5b0 msgr2=0x7fad5015ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.543+0000 7fad5686a640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5015e5b0 0x7fad5015ea30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fad48005fd0 tx=0x7fad48011ef0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 shutdown_connections 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fad3c078150 0x7fad3c07a610 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5015e5b0 0x7fad5015ea30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad501f4320 0x7fad501f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 --2- 172.21.15.28:0/2749537904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad38004800 0x7fad500af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 >> 172.21.15.28:0/2749537904 conn(0x7fad500d1e40 msgr2=0x7fad500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 shutdown_connections 2024-09-10T20:39:38.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.544+0000 7fad5686a640 1 -- 172.21.15.28:0/2749537904 wait complete. 2024-09-10T20:39:38.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:39:38.680 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-10T20:39:38.680 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-10T20:39:38.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol26360 2024-09-10T20:39:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 -- 172.21.15.28:0/728568084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 msgr2=0x7f55700ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 --2- 172.21.15.28:0/728568084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f55700ced20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5568005fd0 tx=0x7f556802a970 comp rx=0 tx=0).stop 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 -- 172.21.15.28:0/728568084 shutdown_connections 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 --2- 172.21.15.28:0/728568084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f55700ced20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 --2- 172.21.15.28:0/728568084 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55700b3580 0x7f557006e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 -- 172.21.15.28:0/728568084 >> 172.21.15.28:0/728568084 conn(0x7f557006d110 msgr2=0x7f55700b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 -- 172.21.15.28:0/728568084 shutdown_connections 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.785+0000 7f55777e2640 1 -- 172.21.15.28:0/728568084 wait complete. 2024-09-10T20:39:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 Processor -- start 2024-09-10T20:39:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 -- start start 2024-09-10T20:39:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f5570155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5570152e00 0x7f557015c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55701f60d0 0x7f55701f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f557015f820 con 0x7f557006e940 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f55700cf630 con 0x7f55701f60d0 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f55777e2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f557015f9a0 con 0x7f5570152e00 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.787+0000 7f557557e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f5570155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f5570155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46604/0 (socket says 172.21.15.28:46604) 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 -- 172.21.15.28:0/2305992482 learned_addr learned my addr 172.21.15.28:0/2305992482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f5575d7f640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55701f60d0 0x7f55701f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f5574d7d640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5570152e00 0x7f557015c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 -- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5570152e00 msgr2=0x7f557015c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5570152e00 0x7f557015c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 -- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55701f60d0 msgr2=0x7f55701f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55701f60d0 0x7f55701f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 -- 172.21.15.28:0/2305992482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557015a500 con 0x7f557006e940 2024-09-10T20:39:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f557557e640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f5570155d70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f556c00b3f0 tx=0x7f556c00b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f5565ffb640 1 -- 172.21.15.28:0/2305992482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c015020 con 0x7f557006e940 2024-09-10T20:39:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f5565ffb640 1 -- 172.21.15.28:0/2305992482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f556c011920 con 0x7f557006e940 2024-09-10T20:39:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557015a7e0 con 0x7f557006e940 2024-09-10T20:39:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.788+0000 7f5565ffb640 1 -- 172.21.15.28:0/2305992482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c011db0 con 0x7f557006e940 2024-09-10T20:39:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.789+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557015ada0 con 0x7f557006e940 2024-09-10T20:39:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.790+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f557015ff50 con 0x7f557006e940 2024-09-10T20:39:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.790+0000 7f5565ffb640 1 -- 172.21.15.28:0/2305992482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f556c01a400 con 0x7f557006e940 2024-09-10T20:39:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.791+0000 7f5565ffb640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5554078120 0x7f555407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.791+0000 7f5565ffb640 1 -- 172.21.15.28:0/2305992482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f556c099f60 con 0x7f557006e940 2024-09-10T20:39:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.791+0000 7f5574d7d640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5554078120 0x7f555407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.791+0000 7f5574d7d640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5554078120 0x7f555407a5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5570159d60 tx=0x7f5560009110 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:38.795+0000 7f5565ffb640 1 -- 172.21.15.28:0/2305992482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f556c01f030 con 0x7f557006e940 2024-09-10T20:39:39.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.009+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol26360", "target": ["mon-mgr", ""]}) -- 0x7f557015be30 con 0x7f5554078120 2024-09-10T20:39:39.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:39.016+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:39.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:39.016+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:39.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:39.016+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:39.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:39.016+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:39.020 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:39:39.016+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:39:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.047+0000 7f5565ffb640 1 -- 172.21.15.28:0/2305992482 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f557015be30 con 0x7f5554078120 2024-09-10T20:39:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5554078120 msgr2=0x7f555407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5554078120 0x7f555407a5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5570159d60 tx=0x7f5560009110 comp rx=0 tx=0).stop 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 msgr2=0x7f5570155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f5570155d70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f556c00b3f0 tx=0x7f556c00b7f0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 shutdown_connections 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5554078120 0x7f555407a5e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55701f60d0 0x7f55701f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5570152e00 0x7f557015c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 --2- 172.21.15.28:0/2305992482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557006e940 0x7f5570155d70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 >> 172.21.15.28:0/2305992482 conn(0x7f557006d110 msgr2=0x7f5570151da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 shutdown_connections 2024-09-10T20:39:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.049+0000 7f55777e2640 1 -- 172.21.15.28:0/2305992482 wait complete. 2024-09-10T20:39:39.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.164+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2958221464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980cd3d0 msgr2=0x7f8d980cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.164+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2958221464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980cd3d0 0x7f8d980cc570 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8d980b3df0 tx=0x7f8d8802a5f0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.165+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2958221464 shutdown_connections 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.165+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2958221464 >> v2:172.21.15.159:3300/0 conn(0x7f8d980b4340 0x7f8d980d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.165+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2958221464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980cd3d0 0x7f8d980cc570 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.165+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2958221464 >> 172.21.15.28:0/2958221464 conn(0x7f8d9806cf00 msgr2=0x7f8d9806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.165+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2958221464 shutdown_connections 2024-09-10T20:39:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.165+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2958221464 wait complete. 2024-09-10T20:39:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.166+0000 7f8d9cb84640 1 Processor -- start 2024-09-10T20:39:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.166+0000 7f8d9cb84640 1 -- start start 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d9cb84640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d980b4340 0x7f8d98154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d9cb84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b47d0 0x7f8d98144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d9cb84640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980cd3d0 0x7f8d98145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d9cb84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d980d0960 con 0x7f8d980b47d0 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d9cb84640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8d980d07e0 con 0x7f8d980b4340 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d9cb84640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8d980b3590 con 0x7f8d980cd3d0 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b47d0 0x7f8d98144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b47d0 0x7f8d98144dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46626/0 (socket says 172.21.15.28:46626) 2024-09-10T20:39:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 -- 172.21.15.28:0/2353433897 learned_addr learned my addr 172.21.15.28:0/2353433897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d9659c640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d980b4340 0x7f8d98154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d96d9d640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980cd3d0 0x7f8d98145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 -- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980cd3d0 msgr2=0x7f8d98145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980cd3d0 0x7f8d98145310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 -- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d980b4340 msgr2=0x7f8d98154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d980b4340 0x7f8d98154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 -- 172.21.15.28:0/2353433897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9809c6a0 con 0x7f8d980b47d0 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d96d9d640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980cd3d0 0x7f8d98145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.167+0000 7f8d95d9b640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b47d0 0x7f8d98144dd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00d770 tx=0x7f8d8c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.168+0000 7f8d86ffd640 1 -- 172.21.15.28:0/2353433897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c017070 con 0x7f8d980b47d0 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.168+0000 7f8d86ffd640 1 -- 172.21.15.28:0/2353433897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d8c009d20 con 0x7f8d980b47d0 2024-09-10T20:39:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.168+0000 7f8d86ffd640 1 -- 172.21.15.28:0/2353433897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c005280 con 0x7f8d980b47d0 2024-09-10T20:39:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.168+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9809c980 con 0x7f8d980b47d0 2024-09-10T20:39:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.168+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9809cf40 con 0x7f8d980b47d0 2024-09-10T20:39:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.169+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d980b3a60 con 0x7f8d980b47d0 2024-09-10T20:39:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.169+0000 7f8d86ffd640 1 -- 172.21.15.28:0/2353433897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f8d8c005420 con 0x7f8d980b47d0 2024-09-10T20:39:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.170+0000 7f8d86ffd640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8d78078120 0x7f8d7807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.170+0000 7f8d86ffd640 1 -- 172.21.15.28:0/2353433897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f8d8c09a2b0 con 0x7f8d980b47d0 2024-09-10T20:39:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.170+0000 7f8d9659c640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8d78078120 0x7f8d7807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.171+0000 7f8d9659c640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8d78078120 0x7f8d7807a5e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d88033000 tx=0x7f8d88004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.174+0000 7f8d86ffd640 1 -- 172.21.15.28:0/2353433897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8d8c0623d0 con 0x7f8d980b47d0 2024-09-10T20:39:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.393+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d980008d0 con 0x7f8d980b47d0 2024-09-10T20:39:39.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.393+0000 7f8d86ffd640 1 -- 172.21.15.28:0/2353433897 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f8d8c06b140 con 0x7f8d980b47d0 2024-09-10T20:39:39.397 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:39.397 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:39:39.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8d78078120 msgr2=0x7f8d7807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8d78078120 0x7f8d7807a5e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d88033000 tx=0x7f8d88004730 comp rx=0 tx=0).stop 2024-09-10T20:39:39.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b47d0 msgr2=0x7f8d98144dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b47d0 0x7f8d98144dd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00d770 tx=0x7f8d8c00dc30 comp rx=0 tx=0).stop 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 shutdown_connections 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8d78078120 0x7f8d7807a5e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980cd3d0 0x7f8d98145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b47d0 0x7f8d98144dd0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 --2- 172.21.15.28:0/2353433897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d980b4340 0x7f8d98154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 >> 172.21.15.28:0/2353433897 conn(0x7f8d9806cf00 msgr2=0x7f8d98073300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 shutdown_connections 2024-09-10T20:39:39.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.395+0000 7f8d9cb84640 1 -- 172.21.15.28:0/2353433897 wait complete. 2024-09-10T20:39:39.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:39:39.534 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.50198562300000005,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.975061564538237,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:39:39.534 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.975061564538237 }, "state": "open", "uptime": 0.501985623 } ] 2024-09-10T20:39:39.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 -- 172.21.15.28:0/108892139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec15a190 msgr2=0x7f0cec0b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/108892139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec15a190 0x7f0cec0b2ba0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc005fd0 tx=0x7f0cdc02a4a0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 -- 172.21.15.28:0/108892139 shutdown_connections 2024-09-10T20:39:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/108892139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cec0b4d10 0x7f0cec0ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/108892139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec15a190 0x7f0cec0b2ba0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 -- 172.21.15.28:0/108892139 >> 172.21.15.28:0/108892139 conn(0x7f0cec06cf00 msgr2=0x7f0cec06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 -- 172.21.15.28:0/108892139 shutdown_connections 2024-09-10T20:39:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.635+0000 7f0cf0df3640 1 -- 172.21.15.28:0/108892139 wait complete. 2024-09-10T20:39:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.637+0000 7f0cf0df3640 1 Processor -- start 2024-09-10T20:39:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.637+0000 7f0cf0df3640 1 -- start start 2024-09-10T20:39:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cf0df3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec0b4d10 0x7f0cec0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cf0df3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cec15a190 0x7f0cec073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cf0df3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cec15df70 0x7f0cec157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cf0df3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec0d0270 con 0x7f0cec0b4d10 2024-09-10T20:39:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cf0df3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec0d00f0 con 0x7f0cec15a190 2024-09-10T20:39:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cf0df3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0cec0d03f0 con 0x7f0cec15df70 2024-09-10T20:39:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec0b4d10 0x7f0cec0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec0b4d10 0x7f0cec0cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46640/0 (socket says 172.21.15.28:46640) 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 -- 172.21.15.28:0/2710963397 learned_addr learned my addr 172.21.15.28:0/2710963397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cead9d640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cec15df70 0x7f0cec157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0ce1d9b640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cec15a190 0x7f0cec073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 -- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cec15df70 msgr2=0x7f0cec157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cec15df70 0x7f0cec157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 -- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cec15a190 msgr2=0x7f0cec073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cec15a190 0x7f0cec073ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 -- 172.21.15.28:0/2710963397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cec151c50 con 0x7f0cec0b4d10 2024-09-10T20:39:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0ce1d9b640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cec15a190 0x7f0cec073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0cea59c640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec0b4d10 0x7f0cec0cb470 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009850 tx=0x7f0cdc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.638+0000 7f0ce37fe640 1 -- 172.21.15.28:0/2710963397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc03f070 con 0x7f0cec0b4d10 2024-09-10T20:39:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.639+0000 7f0ce37fe640 1 -- 172.21.15.28:0/2710963397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cdc004890 con 0x7f0cec0b4d10 2024-09-10T20:39:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.639+0000 7f0ce37fe640 1 -- 172.21.15.28:0/2710963397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc035030 con 0x7f0cec0b4d10 2024-09-10T20:39:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.639+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cec151f30 con 0x7f0cec0b4d10 2024-09-10T20:39:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.639+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cec157780 con 0x7f0cec0b4d10 2024-09-10T20:39:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.640+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cec0b2ca0 con 0x7f0cec0b4d10 2024-09-10T20:39:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.640+0000 7f0ce37fe640 1 -- 172.21.15.28:0/2710963397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 100754+0+0 (secure 0 0 0) 0x7f0cdc004c60 con 0x7f0cec0b4d10 2024-09-10T20:39:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.641+0000 7f0ce37fe640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ccc078150 0x7f0ccc07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.641+0000 7f0ce37fe640 1 -- 172.21.15.28:0/2710963397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0cdc0bb390 con 0x7f0cec0b4d10 2024-09-10T20:39:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.641+0000 7f0ce1d9b640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ccc078150 0x7f0ccc07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.642+0000 7f0ce1d9b640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ccc078150 0x7f0ccc07a610 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0cec150510 tx=0x7f0cd4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.645+0000 7f0ce37fe640 1 -- 172.21.15.28:0/2710963397 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0cec0b2ca0 con 0x7f0cec0b4d10 2024-09-10T20:39:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.865+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cec157590 con 0x7f0cec0b4d10 2024-09-10T20:39:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.866+0000 7f0ce37fe640 1 -- 172.21.15.28:0/2710963397 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f0cdc083b00 con 0x7f0cec0b4d10 2024-09-10T20:39:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ccc078150 msgr2=0x7f0ccc07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ccc078150 0x7f0ccc07a610 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0cec150510 tx=0x7f0cd4008040 comp rx=0 tx=0).stop 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec0b4d10 msgr2=0x7f0cec0cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec0b4d10 0x7f0cec0cb470 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009850 tx=0x7f0cdc0039e0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 shutdown_connections 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ccc078150 0x7f0ccc07a610 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cec15df70 0x7f0cec157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cec15a190 0x7f0cec073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 --2- 172.21.15.28:0/2710963397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cec0b4d10 0x7f0cec0cb470 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 >> 172.21.15.28:0/2710963397 conn(0x7f0cec06cf00 msgr2=0x7f0cec15b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 shutdown_connections 2024-09-10T20:39:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:39.868+0000 7f0cf0df3640 1 -- 172.21.15.28:0/2710963397 wait complete. 2024-09-10T20:39:39.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:39:40.004 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.971972163,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.951211808094303,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:39:40.004 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.951211808094303 }, "state": "open", "uptime": 0.971972163 } ] 2024-09-10T20:39:40.005 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:39:45.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 -- 172.21.15.28:0/575346996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 msgr2=0x7f1c9815e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/575346996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c9815e7e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1c98074360 tx=0x7f1c8c02a4d0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 -- 172.21.15.28:0/575346996 shutdown_connections 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/575346996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c98073cf0 0x7f1c9815ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/575346996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c9815e7e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/575346996 >> v2:172.21.15.159:3300/0 conn(0x7f1c98151cc0 0x7f1c980cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 -- 172.21.15.28:0/575346996 >> 172.21.15.28:0/575346996 conn(0x7f1c980d0420 msgr2=0x7f1c980d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 -- 172.21.15.28:0/575346996 shutdown_connections 2024-09-10T20:39:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.121+0000 7f1c9e06a640 1 -- 172.21.15.28:0/575346996 wait complete. 2024-09-10T20:39:45.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.122+0000 7f1c9e06a640 1 Processor -- start 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.123+0000 7f1c9e06a640 1 -- start start 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.123+0000 7f1c9e06a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c980b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.123+0000 7f1c9e06a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c98073cf0 0x7f1c981e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.123+0000 7f1c9e06a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c98151cc0 0x7f1c981e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.123+0000 7f1c9e06a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c980cb210 con 0x7f1c980738b0 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.123+0000 7f1c9e06a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9815f640 con 0x7f1c98151cc0 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.123+0000 7f1c9e06a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c980cb390 con 0x7f1c98073cf0 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c980b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c980b3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43850/0 (socket says 172.21.15.28:43850) 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 -- 172.21.15.28:0/3812457466 learned_addr learned my addr 172.21.15.28:0/3812457466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c96ffd640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c98073cf0 0x7f1c981e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c97fff640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c98151cc0 0x7f1c981e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 -- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c98073cf0 msgr2=0x7f1c981e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c98073cf0 0x7f1c981e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 -- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c98151cc0 msgr2=0x7f1c981e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c98151cc0 0x7f1c981e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 -- 172.21.15.28:0/3812457466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c980b4d10 con 0x7f1c980738b0 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c96ffd640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c98073cf0 0x7f1c981e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c97fff640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c98151cc0 0x7f1c981e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c977fe640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c980b3a20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800cce0 tx=0x7f1c8800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c7ffff640 1 -- 172.21.15.28:0/3812457466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800f450 con 0x7f1c980738b0 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c7ffff640 1 -- 172.21.15.28:0/3812457466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c8800f6e0 con 0x7f1c980738b0 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c981f8bc0 con 0x7f1c980738b0 2024-09-10T20:39:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c7ffff640 1 -- 172.21.15.28:0/3812457466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800fb30 con 0x7f1c980738b0 2024-09-10T20:39:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.124+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c981f9180 con 0x7f1c980738b0 2024-09-10T20:39:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.125+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c980cab10 con 0x7f1c980738b0 2024-09-10T20:39:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.126+0000 7f1c7ffff640 1 -- 172.21.15.28:0/3812457466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f1c88020030 con 0x7f1c980738b0 2024-09-10T20:39:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.127+0000 7f1c7ffff640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c840781b0 0x7f1c8407a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.127+0000 7f1c7ffff640 1 -- 172.21.15.28:0/3812457466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f1c8809ac00 con 0x7f1c980738b0 2024-09-10T20:39:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.127+0000 7f1c96ffd640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c840781b0 0x7f1c8407a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.127+0000 7f1c96ffd640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c840781b0 0x7f1c8407a670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c033000 tx=0x7f1c8c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.131+0000 7f1c7ffff640 1 -- 172.21.15.28:0/3812457466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c980cab10 con 0x7f1c980738b0 2024-09-10T20:39:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.356+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c98150be0 con 0x7f1c980738b0 2024-09-10T20:39:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.357+0000 7f1c7ffff640 1 -- 172.21.15.28:0/3812457466 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f1c88063e20 con 0x7f1c980738b0 2024-09-10T20:39:45.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:39:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c840781b0 msgr2=0x7f1c8407a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c840781b0 0x7f1c8407a670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c033000 tx=0x7f1c8c004730 comp rx=0 tx=0).stop 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 msgr2=0x7f1c980b3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c980b3a20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800cce0 tx=0x7f1c8800a620 comp rx=0 tx=0).stop 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 shutdown_connections 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1c840781b0 0x7f1c8407a670 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c98151cc0 0x7f1c981e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c98073cf0 0x7f1c981e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 --2- 172.21.15.28:0/3812457466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c980738b0 0x7f1c980b3a20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 >> 172.21.15.28:0/3812457466 conn(0x7f1c980d0420 msgr2=0x7f1c980b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 shutdown_connections 2024-09-10T20:39:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:45.359+0000 7f1c9e06a640 1 -- 172.21.15.28:0/3812457466 wait complete. 2024-09-10T20:39:45.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:39:45.495 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":6.463814878,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.674435850632751,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:39:45.496 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.67443585063275 }, "state": "open", "uptime": 6.463814878 } ] 2024-09-10T20:39:45.496 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:39:50.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.614+0000 7fcb91c47640 1 -- 172.21.15.28:0/1441977655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb74004d40 msgr2=0x7fcb740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.614+0000 7fcb91c47640 1 --2- 172.21.15.28:0/1441977655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb74004d40 0x7fcb740051c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400e420 tx=0x7fcb74005830 comp rx=0 tx=0).stop 2024-09-10T20:39:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.614+0000 7fcb8bfff640 1 -- 172.21.15.28:0/1441977655 reap_dead start 2024-09-10T20:39:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.614+0000 7fcb91c47640 1 -- 172.21.15.28:0/1441977655 shutdown_connections 2024-09-10T20:39:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.614+0000 7fcb91c47640 1 -- 172.21.15.28:0/1441977655 >> 172.21.15.28:0/1441977655 conn(0x7fcb8c06dbf0 msgr2=0x7fcb8c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.614+0000 7fcb91c47640 1 -- 172.21.15.28:0/1441977655 shutdown_connections 2024-09-10T20:39:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.614+0000 7fcb91c47640 1 -- 172.21.15.28:0/1441977655 wait complete. 2024-09-10T20:39:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.615+0000 7fcb91c47640 1 Processor -- start 2024-09-10T20:39:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.615+0000 7fcb91c47640 1 -- start start 2024-09-10T20:39:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb91c47640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb74004d40 0x7fcb8c150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb91c47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c09b2b0 0x7fcb8c09b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb91c47640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb8c09e780 0x7fcb8c09ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb91c47640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c15edf0 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb91c47640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c0b54a0 con 0x7fcb8c09e780 2024-09-10T20:39:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb91c47640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcb8c15ef70 con 0x7fcb74004d40 2024-09-10T20:39:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb8affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c09b2b0 0x7fcb8c09b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb8affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c09b2b0 0x7fcb8c09b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54316/0 (socket says 172.21.15.28:54316) 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb8affd640 1 -- 172.21.15.28:0/151787445 learned_addr learned my addr 172.21.15.28:0/151787445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb8bfff640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb8c09e780 0x7fcb8c09ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.616+0000 7fcb8b7fe640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb74004d40 0x7fcb8c150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb8affd640 1 -- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb74004d40 msgr2=0x7fcb8c150f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb8affd640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb74004d40 0x7fcb8c150f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb8affd640 1 -- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb8c09e780 msgr2=0x7fcb8c09ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb8affd640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb8c09e780 0x7fcb8c09ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb8affd640 1 -- 172.21.15.28:0/151787445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8c1f3fb0 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb8b7fe640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb74004d40 0x7fcb8c150f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb8affd640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c09b2b0 0x7fcb8c09b730 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcb800066b0 tx=0x7fcb800039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb73fff640 1 -- 172.21.15.28:0/151787445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80015070 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb73fff640 1 -- 172.21.15.28:0/151787445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb80011920 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb8c1f4290 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb73fff640 1 -- 172.21.15.28:0/151787445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80011d70 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.617+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb8c1f4850 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.618+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb8c070f20 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.619+0000 7fcb73fff640 1 -- 172.21.15.28:0/151787445 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7fcb80010420 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.619+0000 7fcb73fff640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcb780781b0 0x7fcb7807a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.619+0000 7fcb73fff640 1 -- 172.21.15.28:0/151787445 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fcb8009afb0 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.620+0000 7fcb8b7fe640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcb780781b0 0x7fcb7807a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.620+0000 7fcb8b7fe640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcb780781b0 0x7fcb7807a670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c004630 tx=0x7fcb7c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.624+0000 7fcb73fff640 1 -- 172.21.15.28:0/151787445 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcb80062f60 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.844+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb8c0d0a60 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.845+0000 7fcb73fff640 1 -- 172.21.15.28:0/151787445 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7fcb80067990 con 0x7fcb8c09b2b0 2024-09-10T20:39:50.849 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:50.849 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:50.849 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcb780781b0 msgr2=0x7fcb7807a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcb780781b0 0x7fcb7807a670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c004630 tx=0x7fcb7c005f90 comp rx=0 tx=0).stop 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c09b2b0 msgr2=0x7fcb8c09b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c09b2b0 0x7fcb8c09b730 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcb800066b0 tx=0x7fcb800039c0 comp rx=0 tx=0).stop 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 shutdown_connections 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fcb780781b0 0x7fcb7807a670 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb8c09e780 0x7fcb8c09ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c09b2b0 0x7fcb8c09b730 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 --2- 172.21.15.28:0/151787445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb74004d40 0x7fcb8c150f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 >> 172.21.15.28:0/151787445 conn(0x7fcb8c06dbf0 msgr2=0x7fcb8c0aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 shutdown_connections 2024-09-10T20:39:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:50.847+0000 7fcb91c47640 1 -- 172.21.15.28:0/151787445 wait complete. 2024-09-10T20:39:50.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:39:50.982 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":11.949657763999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.401446351101104,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:39:50.983 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.401446351101104 }, "state": "open", "uptime": 11.949657764 } ] 2024-09-10T20:39:50.983 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:39:55.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:39:56.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.098+0000 7f4ff68b0640 1 -- 172.21.15.28:0/3644633876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 msgr2=0x7f4ff00d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.098+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/3644633876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff00d0540 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8005fd0 tx=0x7f4fe802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.098+0000 7f4ff68b0640 1 -- 172.21.15.28:0/3644633876 shutdown_connections 2024-09-10T20:39:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.098+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/3644633876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff00d0540 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.098+0000 7f4ff68b0640 1 -- 172.21.15.28:0/3644633876 >> 172.21.15.28:0/3644633876 conn(0x7f4ff006cf00 msgr2=0x7f4ff006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.098+0000 7f4ff68b0640 1 -- 172.21.15.28:0/3644633876 shutdown_connections 2024-09-10T20:39:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.098+0000 7f4ff68b0640 1 -- 172.21.15.28:0/3644633876 wait complete. 2024-09-10T20:39:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.100+0000 7f4ff68b0640 1 Processor -- start 2024-09-10T20:39:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.100+0000 7f4ff68b0640 1 -- start start 2024-09-10T20:39:56.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff68b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff0145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:56.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff68b0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ff0155100 0x7f4ff00b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff68b0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ff01554e0 0x7f4ff0146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff68b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff0072f60 con 0x7f4ff00d0140 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff68b0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff0072de0 con 0x7f4ff0155100 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff68b0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff00730e0 con 0x7f4ff01554e0 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff0145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff0145850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54328/0 (socket says 172.21.15.28:54328) 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 -- 172.21.15.28:0/2648252399 learned_addr learned my addr 172.21.15.28:0/2648252399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4fef7fe640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ff0155100 0x7f4ff00b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff4e4d640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ff01554e0 0x7f4ff0146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 -- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ff01554e0 msgr2=0x7f4ff0146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ff01554e0 0x7f4ff0146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 -- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ff0155100 msgr2=0x7f4ff00b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ff0155100 0x7f4ff00b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 -- 172.21.15.28:0/2648252399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff0158370 con 0x7f4ff00d0140 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4ff4e4d640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ff01554e0 0x7f4ff0146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4feffff640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff0145850 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000c440 tx=0x7f4fe000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4fecff9640 1 -- 172.21.15.28:0/2648252399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0005480 con 0x7f4ff00d0140 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.101+0000 7f4fecff9640 1 -- 172.21.15.28:0/2648252399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fe00035f0 con 0x7f4ff00d0140 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.102+0000 7f4fecff9640 1 -- 172.21.15.28:0/2648252399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0003a20 con 0x7f4ff00d0140 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.102+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff0158650 con 0x7f4ff00d0140 2024-09-10T20:39:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.102+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff0158c10 con 0x7f4ff00d0140 2024-09-10T20:39:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.103+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff0073600 con 0x7f4ff00d0140 2024-09-10T20:39:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.103+0000 7f4fecff9640 1 -- 172.21.15.28:0/2648252399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f4fe0003bc0 con 0x7f4ff00d0140 2024-09-10T20:39:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.104+0000 7f4fecff9640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4fdc078350 0x7f4fdc07a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:39:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.104+0000 7f4fecff9640 1 -- 172.21.15.28:0/2648252399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f4fe0099e30 con 0x7f4ff00d0140 2024-09-10T20:39:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.104+0000 7f4fef7fe640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4fdc078350 0x7f4fdc07a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:39:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.105+0000 7f4fef7fe640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4fdc078350 0x7f4fdc07a810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4ff015abf0 tx=0x7f4fe4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:39:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.108+0000 7f4fecff9640 1 -- 172.21.15.28:0/2648252399 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4fe0062430 con 0x7f4ff00d0140 2024-09-10T20:39:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.336+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ff009dca0 con 0x7f4ff00d0140 2024-09-10T20:39:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.336+0000 7f4fecff9640 1 -- 172.21.15.28:0/2648252399 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f4fe0061d30 con 0x7f4ff00d0140 2024-09-10T20:39:56.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:39:56.342 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:39:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4fdc078350 msgr2=0x7f4fdc07a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4fdc078350 0x7f4fdc07a810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4ff015abf0 tx=0x7f4fe4009040 comp rx=0 tx=0).stop 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 msgr2=0x7f4ff0145850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff0145850 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000c440 tx=0x7f4fe000c900 comp rx=0 tx=0).stop 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 shutdown_connections 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4fdc078350 0x7f4fdc07a810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ff01554e0 0x7f4ff0146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ff0155100 0x7f4ff00b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 --2- 172.21.15.28:0/2648252399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff00d0140 0x7f4ff0145850 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:39:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 >> 172.21.15.28:0/2648252399 conn(0x7f4ff006cf00 msgr2=0x7f4ff009aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:39:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 shutdown_connections 2024-09-10T20:39:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:39:56.339+0000 7f4ff68b0640 1 -- 172.21.15.28:0/2648252399 wait complete. 2024-09-10T20:39:56.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:39:56.475 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":17.443500421,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.131504568493472,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:39:56.476 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.131504568493472 }, "state": "open", "uptime": 17.443500421 } ] 2024-09-10T20:39:56.476 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:01.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:40:01.056 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:40:01.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 -- 172.21.15.28:0/2432387353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118015de70 msgr2=0x7f11800d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 --2- 172.21.15.28:0/2432387353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118015de70 0x7f11800d2b90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f11800d4d80 tx=0x7f117002a5f0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 -- 172.21.15.28:0/2432387353 shutdown_connections 2024-09-10T20:40:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 --2- 172.21.15.28:0/2432387353 >> v2:172.21.15.159:3300/0 conn(0x7f11800cf780 0x7f11800cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 --2- 172.21.15.28:0/2432387353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118015de70 0x7f11800d2b90 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 -- 172.21.15.28:0/2432387353 >> 172.21.15.28:0/2432387353 conn(0x7f11800d3fb0 msgr2=0x7f11800d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 -- 172.21.15.28:0/2432387353 shutdown_connections 2024-09-10T20:40:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.581+0000 7f118482a640 1 -- 172.21.15.28:0/2432387353 wait complete. 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 Processor -- start 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 -- start start 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11800cf780 0x7f11801609a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f118015de70 0x7f1180152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1180161b50 0x7f11801f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11800b7690 con 0x7f11800cf780 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f11800b7510 con 0x7f118015de70 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f118482a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f11800b7810 con 0x7f1180161b50 2024-09-10T20:40:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.583+0000 7f117e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11800cf780 0x7f11801609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11800cf780 0x7f11801609a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40856/0 (socket says 172.21.15.28:40856) 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 -- 172.21.15.28:0/2122174507 learned_addr learned my addr 172.21.15.28:0/2122174507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f1177fff640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f118015de70 0x7f1180152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117ed9d640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1180161b50 0x7f11801f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 -- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1180161b50 msgr2=0x7f11801f7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1180161b50 0x7f11801f7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 -- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f118015de70 msgr2=0x7f1180152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f118015de70 0x7f1180152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 -- 172.21.15.28:0/2122174507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1180160ee0 con 0x7f11800cf780 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117ed9d640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1180161b50 0x7f11801f7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:40:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f117e59c640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11800cf780 0x7f11801609a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1170000c00 tx=0x7f1170003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f1176ffd640 1 -- 172.21.15.28:0/2122174507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117003f070 con 0x7f11800cf780 2024-09-10T20:40:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f1176ffd640 1 -- 172.21.15.28:0/2122174507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11700047a0 con 0x7f11800cf780 2024-09-10T20:40:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118015fc00 con 0x7f11800cf780 2024-09-10T20:40:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f1176ffd640 1 -- 172.21.15.28:0/2122174507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170035030 con 0x7f11800cf780 2024-09-10T20:40:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.584+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f118009de40 con 0x7f11800cf780 2024-09-10T20:40:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.585+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f118009dc50 con 0x7f11800cf780 2024-09-10T20:40:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.586+0000 7f1176ffd640 1 -- 172.21.15.28:0/2122174507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f1170004bb0 con 0x7f11800cf780 2024-09-10T20:40:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.586+0000 7f1176ffd640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f11600781b0 0x7f116007a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.587+0000 7f1176ffd640 1 -- 172.21.15.28:0/2122174507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f11700bb2c0 con 0x7f11800cf780 2024-09-10T20:40:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.587+0000 7f1177fff640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f11600781b0 0x7f116007a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.587+0000 7f1177fff640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f11600781b0 0x7f116007a670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f11800d17b0 tx=0x7f116800b390 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.591+0000 7f1176ffd640 1 -- 172.21.15.28:0/2122174507 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1170083330 con 0x7f11800cf780 2024-09-10T20:40:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.813+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11800b9f30 con 0x7f11800cf780 2024-09-10T20:40:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.814+0000 7f1176ffd640 1 -- 172.21.15.28:0/2122174507 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f117008c0a0 con 0x7f11800cf780 2024-09-10T20:40:01.819 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:01.819 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:01.819 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f11600781b0 msgr2=0x7f116007a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f11600781b0 0x7f116007a670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f11800d17b0 tx=0x7f116800b390 comp rx=0 tx=0).stop 2024-09-10T20:40:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11800cf780 msgr2=0x7f11801609a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11800cf780 0x7f11801609a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1170000c00 tx=0x7f1170003910 comp rx=0 tx=0).stop 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 shutdown_connections 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f11600781b0 0x7f116007a670 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1180161b50 0x7f11801f7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f118015de70 0x7f1180152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 --2- 172.21.15.28:0/2122174507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11800cf780 0x7f11801609a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 >> 172.21.15.28:0/2122174507 conn(0x7f11800d3fb0 msgr2=0x7f11800cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 shutdown_connections 2024-09-10T20:40:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:01.816+0000 7f118482a640 1 -- 172.21.15.28:0/2122174507 wait complete. 2024-09-10T20:40:01.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:01.955 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":11,"uptime":22.923343477,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.865642536984566,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:01.956 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 11, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.865642536984566 }, "state": "open", "uptime": 22.923343477 } ] 2024-09-10T20:40:01.956 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:06.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:07.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.072+0000 7f3d897ad640 1 -- 172.21.15.28:0/653793094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840cd3d0 msgr2=0x7f3d840cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.072+0000 7f3d897ad640 1 --2- 172.21.15.28:0/653793094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840cd3d0 0x7f3d840cc570 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3d840b3df0 tx=0x7f3d7402a8b0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.072+0000 7f3d897ad640 1 -- 172.21.15.28:0/653793094 shutdown_connections 2024-09-10T20:40:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.072+0000 7f3d897ad640 1 --2- 172.21.15.28:0/653793094 >> v2:172.21.15.159:3300/0 conn(0x7f3d840b4340 0x7f3d840d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.072+0000 7f3d897ad640 1 --2- 172.21.15.28:0/653793094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840cd3d0 0x7f3d840cc570 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.072+0000 7f3d897ad640 1 -- 172.21.15.28:0/653793094 >> 172.21.15.28:0/653793094 conn(0x7f3d8406cf00 msgr2=0x7f3d8406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.073+0000 7f3d897ad640 1 -- 172.21.15.28:0/653793094 shutdown_connections 2024-09-10T20:40:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.073+0000 7f3d897ad640 1 -- 172.21.15.28:0/653793094 wait complete. 2024-09-10T20:40:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.074+0000 7f3d897ad640 1 Processor -- start 2024-09-10T20:40:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.074+0000 7f3d897ad640 1 -- start start 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d897ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840b4340 0x7f3d8409cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d897ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d840b47d0 0x7f3d841e58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d897ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d840cd3d0 0x7f3d841e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d897ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d840d0960 con 0x7f3d840b4340 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d897ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3d840d07e0 con 0x7f3d840b47d0 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d897ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3d840b3590 con 0x7f3d840cd3d0 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840b4340 0x7f3d8409cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840b4340 0x7f3d8409cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40864/0 (socket says 172.21.15.28:40864) 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 -- 172.21.15.28:0/4047371943 learned_addr learned my addr 172.21.15.28:0/4047371943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d827fc640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d840b47d0 0x7f3d841e58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d837fe640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d840cd3d0 0x7f3d841e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 -- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d840cd3d0 msgr2=0x7f3d841e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d840cd3d0 0x7f3d841e5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 -- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d840b47d0 msgr2=0x7f3d841e58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d840b47d0 0x7f3d841e58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 -- 172.21.15.28:0/4047371943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d841f35a0 con 0x7f3d840b4340 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d837fe640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d840cd3d0 0x7f3d841e5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.075+0000 7f3d82ffd640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840b4340 0x7f3d8409cf50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009b30 tx=0x7f3d74003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.076+0000 7f3d6b7fe640 1 -- 172.21.15.28:0/4047371943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74031da0 con 0x7f3d840b4340 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.076+0000 7f3d6b7fe640 1 -- 172.21.15.28:0/4047371943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d74003e10 con 0x7f3d840b4340 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.076+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d841561b0 con 0x7f3d840b4340 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.076+0000 7f3d6b7fe640 1 -- 172.21.15.28:0/4047371943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74033330 con 0x7f3d840b4340 2024-09-10T20:40:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.076+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d84156770 con 0x7f3d840b4340 2024-09-10T20:40:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.077+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d840b3a60 con 0x7f3d840b4340 2024-09-10T20:40:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.078+0000 7f3d6b7fe640 1 -- 172.21.15.28:0/4047371943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f3d7404b030 con 0x7f3d840b4340 2024-09-10T20:40:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.078+0000 7f3d6b7fe640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3d700781b0 0x7f3d7007a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.078+0000 7f3d6b7fe640 1 -- 172.21.15.28:0/4047371943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f3d740bb490 con 0x7f3d840b4340 2024-09-10T20:40:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.078+0000 7f3d827fc640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3d700781b0 0x7f3d7007a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.079+0000 7f3d827fc640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3d700781b0 0x7f3d7007a670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d8415ed60 tx=0x7f3d7800a2d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.082+0000 7f3d6b7fe640 1 -- 172.21.15.28:0/4047371943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3d7403f030 con 0x7f3d840b4340 2024-09-10T20:40:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.307+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d840cfa30 con 0x7f3d840b4340 2024-09-10T20:40:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.308+0000 7f3d6b7fe640 1 -- 172.21.15.28:0/4047371943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f3d74083b20 con 0x7f3d840b4340 2024-09-10T20:40:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:07.312 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:07.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3d700781b0 msgr2=0x7f3d7007a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:07.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3d700781b0 0x7f3d7007a670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d8415ed60 tx=0x7f3d7800a2d0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840b4340 msgr2=0x7f3d8409cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:07.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840b4340 0x7f3d8409cf50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009b30 tx=0x7f3d74003c20 comp rx=0 tx=0).stop 2024-09-10T20:40:07.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 shutdown_connections 2024-09-10T20:40:07.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3d700781b0 0x7f3d7007a670 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d840cd3d0 0x7f3d841e5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d840b47d0 0x7f3d841e58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 --2- 172.21.15.28:0/4047371943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d840b4340 0x7f3d8409cf50 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 >> 172.21.15.28:0/4047371943 conn(0x7f3d8406cf00 msgr2=0x7f3d8415ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 shutdown_connections 2024-09-10T20:40:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:07.310+0000 7f3d897ad640 1 -- 172.21.15.28:0/4047371943 wait complete. 2024-09-10T20:40:07.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:07.447 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":10,"uptime":28.415186194,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.602554182509998,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:07.447 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.602554182509998 }, "state": "open", "uptime": 28.415186194 } ] 2024-09-10T20:40:07.447 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:12.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.559+0000 7faa52133640 1 -- 172.21.15.28:0/3483129205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa34004d40 msgr2=0x7faa340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.559+0000 7faa52133640 1 --2- 172.21.15.28:0/3483129205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa34004d40 0x7faa340051c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7faa44009f00 tx=0x7faa34005830 comp rx=0 tx=0).stop 2024-09-10T20:40:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.559+0000 7faa4bfff640 1 -- 172.21.15.28:0/3483129205 reap_dead start 2024-09-10T20:40:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.559+0000 7faa52133640 1 -- 172.21.15.28:0/3483129205 shutdown_connections 2024-09-10T20:40:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.559+0000 7faa52133640 1 -- 172.21.15.28:0/3483129205 >> 172.21.15.28:0/3483129205 conn(0x7faa4c06cf00 msgr2=0x7faa4c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.559+0000 7faa52133640 1 -- 172.21.15.28:0/3483129205 shutdown_connections 2024-09-10T20:40:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.559+0000 7faa52133640 1 -- 172.21.15.28:0/3483129205 wait complete. 2024-09-10T20:40:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.560+0000 7faa52133640 1 Processor -- start 2024-09-10T20:40:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.560+0000 7faa52133640 1 -- start start 2024-09-10T20:40:12.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa52133640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa34004d40 0x7faa4c14dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:12.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa52133640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c140d40 0x7faa4c14c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:12.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa52133640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa4c09dbc0 0x7faa4c14cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:12.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa52133640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0d0710 con 0x7faa4c140d40 2024-09-10T20:40:12.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa52133640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0d0590 con 0x7faa4c09dbc0 2024-09-10T20:40:12.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa52133640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c0d0890 con 0x7faa34004d40 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c140d40 0x7faa4c14c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c140d40 0x7faa4c14c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59078/0 (socket says 172.21.15.28:59078) 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 -- 172.21.15.28:0/1936423996 learned_addr learned my addr 172.21.15.28:0/1936423996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4b7fe640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa34004d40 0x7faa4c14dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4bfff640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa4c09dbc0 0x7faa4c14cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 -- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa34004d40 msgr2=0x7faa4c14dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa34004d40 0x7faa4c14dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 -- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa4c09dbc0 msgr2=0x7faa4c14cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa4c09dbc0 0x7faa4c14cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.561+0000 7faa4affd640 1 -- 172.21.15.28:0/1936423996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c09d140 con 0x7faa4c140d40 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.562+0000 7faa4affd640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c140d40 0x7faa4c14c720 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faa40003820 tx=0x7faa40003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.562+0000 7faa33fff640 1 -- 172.21.15.28:0/1936423996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa400118f0 con 0x7faa4c140d40 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.562+0000 7faa33fff640 1 -- 172.21.15.28:0/1936423996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa40018020 con 0x7faa4c140d40 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.562+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c09aa30 con 0x7faa4c140d40 2024-09-10T20:40:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.562+0000 7faa33fff640 1 -- 172.21.15.28:0/1936423996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40011d30 con 0x7faa4c140d40 2024-09-10T20:40:12.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.562+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c09aff0 con 0x7faa4c140d40 2024-09-10T20:40:12.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.563+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c15bd50 con 0x7faa4c140d40 2024-09-10T20:40:12.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.564+0000 7faa33fff640 1 -- 172.21.15.28:0/1936423996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7faa40010040 con 0x7faa4c140d40 2024-09-10T20:40:12.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.564+0000 7faa33fff640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa380781e0 0x7faa3807a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:12.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.564+0000 7faa33fff640 1 -- 172.21.15.28:0/1936423996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7faa4009d250 con 0x7faa4c140d40 2024-09-10T20:40:12.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.564+0000 7faa4b7fe640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa380781e0 0x7faa3807a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:12.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.565+0000 7faa4b7fe640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa380781e0 0x7faa3807a6a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009cd0 tx=0x7faa3c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.568+0000 7faa33fff640 1 -- 172.21.15.28:0/1936423996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faa40022020 con 0x7faa4c140d40 2024-09-10T20:40:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.788+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa4c0cf6e0 con 0x7faa4c140d40 2024-09-10T20:40:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.789+0000 7faa33fff640 1 -- 172.21.15.28:0/1936423996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7faa4c09d140 con 0x7faa4c140d40 2024-09-10T20:40:12.793 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:12.793 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa380781e0 msgr2=0x7faa3807a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa380781e0 0x7faa3807a6a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009cd0 tx=0x7faa3c005f90 comp rx=0 tx=0).stop 2024-09-10T20:40:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c140d40 msgr2=0x7faa4c14c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c140d40 0x7faa4c14c720 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faa40003820 tx=0x7faa40003c20 comp rx=0 tx=0).stop 2024-09-10T20:40:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 shutdown_connections 2024-09-10T20:40:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7faa380781e0 0x7faa3807a6a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa4c09dbc0 0x7faa4c14cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c140d40 0x7faa4c14c720 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 --2- 172.21.15.28:0/1936423996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa34004d40 0x7faa4c14dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 >> 172.21.15.28:0/1936423996 conn(0x7faa4c06cf00 msgr2=0x7faa4c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 shutdown_connections 2024-09-10T20:40:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:12.791+0000 7faa52133640 1 -- 172.21.15.28:0/1936423996 wait complete. 2024-09-10T20:40:12.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:12.928 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":10,"uptime":33.897029195999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.343253036857671,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:12.929 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.34325303685767 }, "state": "open", "uptime": 33.897029196 } ] 2024-09-10T20:40:12.929 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:17.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:18.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.046+0000 7ff7fd22a640 1 -- 172.21.15.28:0/4102942706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e0003a00 msgr2=0x7ff7e0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:18.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.046+0000 7ff7fd22a640 1 --2- 172.21.15.28:0/4102942706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e0003a00 0x7ff7e0003e20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009b50 tx=0x7ff7e0005830 comp rx=0 tx=0).stop 2024-09-10T20:40:18.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.046+0000 7ff7f759e640 1 -- 172.21.15.28:0/4102942706 reap_dead start 2024-09-10T20:40:18.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.046+0000 7ff7fd22a640 1 -- 172.21.15.28:0/4102942706 shutdown_connections 2024-09-10T20:40:18.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.046+0000 7ff7fd22a640 1 -- 172.21.15.28:0/4102942706 >> 172.21.15.28:0/4102942706 conn(0x7ff7f80d3fb0 msgr2=0x7ff7f80d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.046+0000 7ff7fd22a640 1 -- 172.21.15.28:0/4102942706 shutdown_connections 2024-09-10T20:40:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.046+0000 7ff7fd22a640 1 -- 172.21.15.28:0/4102942706 wait complete. 2024-09-10T20:40:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 Processor -- start 2024-09-10T20:40:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 -- start start 2024-09-10T20:40:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7e0003a00 0x7ff7f81f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f809e6f0 0x7ff7f809eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7f80cc390 0x7ff7f80cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f80d2d10 con 0x7ff7f809e6f0 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f80d2b90 con 0x7ff7e0003a00 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.048+0000 7ff7fd22a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f80d2e90 con 0x7ff7f80cc390 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f809e6f0 0x7ff7f809eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f809e6f0 0x7ff7f809eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59100/0 (socket says 172.21.15.28:59100) 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 -- 172.21.15.28:0/6942702 learned_addr learned my addr 172.21.15.28:0/6942702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f6d9d640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7e0003a00 0x7ff7f81f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f759e640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7f80cc390 0x7ff7f80cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 -- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7f80cc390 msgr2=0x7ff7f80cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7f80cc390 0x7ff7f80cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 -- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7e0003a00 msgr2=0x7ff7f81f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7e0003a00 0x7ff7f81f5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 -- 172.21.15.28:0/6942702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f815e670 con 0x7ff7f809e6f0 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7f659c640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f809e6f0 0x7ff7f809eb70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec011ab0 tx=0x7ff7ec011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7df7fe640 1 -- 172.21.15.28:0/6942702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec01e070 con 0x7ff7f809e6f0 2024-09-10T20:40:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7df7fe640 1 -- 172.21.15.28:0/6942702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7ec00fb50 con 0x7ff7f809e6f0 2024-09-10T20:40:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f815e950 con 0x7ff7f809e6f0 2024-09-10T20:40:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7df7fe640 1 -- 172.21.15.28:0/6942702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec019410 con 0x7ff7f809e6f0 2024-09-10T20:40:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.049+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f815ef10 con 0x7ff7f809e6f0 2024-09-10T20:40:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.050+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f8162340 con 0x7ff7f809e6f0 2024-09-10T20:40:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.051+0000 7ff7df7fe640 1 -- 172.21.15.28:0/6942702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7ff7ec0195b0 con 0x7ff7f809e6f0 2024-09-10T20:40:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.052+0000 7ff7df7fe640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff7e40781e0 0x7ff7e407a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.052+0000 7ff7df7fe640 1 -- 172.21.15.28:0/6942702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7ff7ec09ac40 con 0x7ff7f809e6f0 2024-09-10T20:40:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.052+0000 7ff7f6d9d640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff7e40781e0 0x7ff7e407a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.053+0000 7ff7f6d9d640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff7e40781e0 0x7ff7e407a6a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8035000 tx=0x7ff7e802d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:18.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.056+0000 7ff7df7fe640 1 -- 172.21.15.28:0/6942702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff7ec062bf0 con 0x7ff7f809e6f0 2024-09-10T20:40:18.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.279+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7f80b9f30 con 0x7ff7f809e6f0 2024-09-10T20:40:18.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.280+0000 7ff7df7fe640 1 -- 172.21.15.28:0/6942702 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7ff7ec067620 con 0x7ff7f809e6f0 2024-09-10T20:40:18.284 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:18.284 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:18.284 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff7e40781e0 msgr2=0x7ff7e407a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff7e40781e0 0x7ff7e407a6a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8035000 tx=0x7ff7e802d040 comp rx=0 tx=0).stop 2024-09-10T20:40:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f809e6f0 msgr2=0x7ff7f809eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f809e6f0 0x7ff7f809eb70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec011ab0 tx=0x7ff7ec011eb0 comp rx=0 tx=0).stop 2024-09-10T20:40:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 shutdown_connections 2024-09-10T20:40:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff7e40781e0 0x7ff7e407a6a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7f80cc390 0x7ff7f80cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f809e6f0 0x7ff7f809eb70 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 --2- 172.21.15.28:0/6942702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7e0003a00 0x7ff7f81f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 >> 172.21.15.28:0/6942702 conn(0x7ff7f80d3fb0 msgr2=0x7ff7f80b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 shutdown_connections 2024-09-10T20:40:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:18.282+0000 7ff7fd22a640 1 -- 172.21.15.28:0/6942702 wait complete. 2024-09-10T20:40:18.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:18.419 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":39.386871910000004,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.086844148903058,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:18.420 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.086844148903058 }, "state": "open", "uptime": 39.38687191 } ] 2024-09-10T20:40:18.420 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:23.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 -- 172.21.15.28:0/1391983392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c15eef0 msgr2=0x7f5a1c0cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 --2- 172.21.15.28:0/1391983392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c15eef0 0x7f5a1c0cb210 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5a14005fd0 tx=0x7f5a1402aa90 comp rx=0 tx=0).stop 2024-09-10T20:40:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 -- 172.21.15.28:0/1391983392 shutdown_connections 2024-09-10T20:40:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 --2- 172.21.15.28:0/1391983392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c15eef0 0x7f5a1c0cb210 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 --2- 172.21.15.28:0/1391983392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a1c151cc0 0x7f5a1c0cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 -- 172.21.15.28:0/1391983392 >> 172.21.15.28:0/1391983392 conn(0x7f5a1c0d0420 msgr2=0x7f5a1c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 -- 172.21.15.28:0/1391983392 shutdown_connections 2024-09-10T20:40:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.536+0000 7f5a2259b640 1 -- 172.21.15.28:0/1391983392 wait complete. 2024-09-10T20:40:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.537+0000 7f5a2259b640 1 Processor -- start 2024-09-10T20:40:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a2259b640 1 -- start start 2024-09-10T20:40:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a2259b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a1c0738b0 0x7f5a1c14d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a2259b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c151cc0 0x7f5a1c06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a2259b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a1c15eef0 0x7f5a1c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a2259b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c073e70 con 0x7f5a1c151cc0 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a2259b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c073cf0 con 0x7f5a1c0738b0 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a2259b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1c073ff0 con 0x7f5a1c15eef0 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c151cc0 0x7f5a1c06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c151cc0 0x7f5a1c06f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49000/0 (socket says 172.21.15.28:49000) 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.538+0000 7f5a1b7fe640 1 -- 172.21.15.28:0/2390273818 learned_addr learned my addr 172.21.15.28:0/2390273818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a1bfff640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a1c0738b0 0x7f5a1c14d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a20b38640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a1c15eef0 0x7f5a1c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a1b7fe640 1 -- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a1c15eef0 msgr2=0x7f5a1c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a1b7fe640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a1c15eef0 0x7f5a1c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a1b7fe640 1 -- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a1c0738b0 msgr2=0x7f5a1c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a1b7fe640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a1c0738b0 0x7f5a1c14d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a1b7fe640 1 -- 172.21.15.28:0/2390273818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c14dde0 con 0x7f5a1c151cc0 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a1b7fe640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c151cc0 0x7f5a1c06f6e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c430 tx=0x7f5a1000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a18ff9640 1 -- 172.21.15.28:0/2390273818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10017070 con 0x7f5a1c151cc0 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a18ff9640 1 -- 172.21.15.28:0/2390273818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a100035f0 con 0x7f5a1c151cc0 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1c0b1ec0 con 0x7f5a1c151cc0 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a18ff9640 1 -- 172.21.15.28:0/2390273818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10003a80 con 0x7f5a1c151cc0 2024-09-10T20:40:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.539+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1c0b1900 con 0x7f5a1c151cc0 2024-09-10T20:40:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.540+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a1c0cc1b0 con 0x7f5a1c151cc0 2024-09-10T20:40:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.541+0000 7f5a18ff9640 1 -- 172.21.15.28:0/2390273818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f5a10003c20 con 0x7f5a1c151cc0 2024-09-10T20:40:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.542+0000 7f5a18ff9640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a080781b0 0x7f5a0807a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.542+0000 7f5a18ff9640 1 -- 172.21.15.28:0/2390273818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f5a10099a20 con 0x7f5a1c151cc0 2024-09-10T20:40:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.542+0000 7f5a1bfff640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a080781b0 0x7f5a0807a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.542+0000 7f5a1bfff640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a080781b0 0x7f5a0807a670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c005fd0 tx=0x7f5a0c005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.546+0000 7f5a18ff9640 1 -- 172.21.15.28:0/2390273818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5a10062020 con 0x7f5a1c151cc0 2024-09-10T20:40:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.769+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a1c068d10 con 0x7f5a1c151cc0 2024-09-10T20:40:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.770+0000 7f5a18ff9640 1 -- 172.21.15.28:0/2390273818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f5a10061920 con 0x7f5a1c151cc0 2024-09-10T20:40:23.774 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:23.774 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a080781b0 msgr2=0x7f5a0807a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a080781b0 0x7f5a0807a670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c005fd0 tx=0x7f5a0c005850 comp rx=0 tx=0).stop 2024-09-10T20:40:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c151cc0 msgr2=0x7f5a1c06f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c151cc0 0x7f5a1c06f6e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c430 tx=0x7f5a1000c8f0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 shutdown_connections 2024-09-10T20:40:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5a080781b0 0x7f5a0807a670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a1c15eef0 0x7f5a1c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c151cc0 0x7f5a1c06f6e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 --2- 172.21.15.28:0/2390273818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a1c0738b0 0x7f5a1c14d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 >> 172.21.15.28:0/2390273818 conn(0x7f5a1c0d0420 msgr2=0x7f5a1c150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 shutdown_connections 2024-09-10T20:40:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:23.772+0000 7f5a2259b640 1 -- 172.21.15.28:0/2390273818 wait complete. 2024-09-10T20:40:23.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:23.908 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":44.875714574,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.83371289826438,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:23.909 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.83371289826438 }, "state": "open", "uptime": 44.875714574 } ] 2024-09-10T20:40:23.909 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:28.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.023+0000 7f475f975640 1 -- 172.21.15.28:0/1423174054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d0790 msgr2=0x7f4758154400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.023+0000 7f475f975640 1 --2- 172.21.15.28:0/1423174054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d0790 0x7f4758154400 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f47580cbc80 tx=0x7f475002a980 comp rx=0 tx=0).stop 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.023+0000 7f475f975640 1 -- 172.21.15.28:0/1423174054 shutdown_connections 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.023+0000 7f475f975640 1 --2- 172.21.15.28:0/1423174054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d0790 0x7f4758154400 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.023+0000 7f475f975640 1 --2- 172.21.15.28:0/1423174054 >> v2:172.21.15.159:3300/0 conn(0x7f47580707b0 0x7f4758153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.023+0000 7f475f975640 1 -- 172.21.15.28:0/1423174054 >> 172.21.15.28:0/1423174054 conn(0x7f475806cf00 msgr2=0x7f475806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.024+0000 7f475f975640 1 -- 172.21.15.28:0/1423174054 shutdown_connections 2024-09-10T20:40:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.024+0000 7f475f975640 1 -- 172.21.15.28:0/1423174054 wait complete. 2024-09-10T20:40:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.025+0000 7f475f975640 1 Processor -- start 2024-09-10T20:40:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.025+0000 7f475f975640 1 -- start start 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475f975640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47580707b0 0x7f47581500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475f975640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d03b0 0x7f4758156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475f975640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47580d0790 0x7f47580b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475f975640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f475815a830 con 0x7f47580d03b0 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475f975640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4758073b50 con 0x7f47580707b0 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475f975640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f475815a9b0 con 0x7f47580d0790 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d03b0 0x7f4758156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d03b0 0x7f4758156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49014/0 (socket says 172.21.15.28:49014) 2024-09-10T20:40:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 -- 172.21.15.28:0/3842733941 learned_addr learned my addr 172.21.15.28:0/3842733941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475d711640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47580707b0 0x7f47581500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475df12640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47580d0790 0x7f47580b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 -- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47580d0790 msgr2=0x7f47580b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47580d0790 0x7f47580b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 -- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47580707b0 msgr2=0x7f47581500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47580707b0 0x7f47581500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 -- 172.21.15.28:0/3842733941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4758155560 con 0x7f47580d03b0 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475df12640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47580d0790 0x7f47580b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475d711640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47580707b0 0x7f47581500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.026+0000 7f475cf10640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d03b0 0x7f4758156000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f474800c920 tx=0x7f474800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.027+0000 7f474dffb640 1 -- 172.21.15.28:0/3842733941 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748007c10 con 0x7f47580d03b0 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.027+0000 7f474dffb640 1 -- 172.21.15.28:0/3842733941 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f474800f450 con 0x7f47580d03b0 2024-09-10T20:40:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.027+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4758156d60 con 0x7f47580d03b0 2024-09-10T20:40:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.027+0000 7f474dffb640 1 -- 172.21.15.28:0/3842733941 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474800f8a0 con 0x7f47580d03b0 2024-09-10T20:40:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.027+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4758157300 con 0x7f47580d03b0 2024-09-10T20:40:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.028+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475815e520 con 0x7f47580d03b0 2024-09-10T20:40:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.028+0000 7f474dffb640 1 -- 172.21.15.28:0/3842733941 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f4748020030 con 0x7f47580d03b0 2024-09-10T20:40:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.029+0000 7f474dffb640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f473c0781b0 0x7f473c07a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.029+0000 7f474dffb640 1 -- 172.21.15.28:0/3842733941 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f474809a800 con 0x7f47580d03b0 2024-09-10T20:40:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.030+0000 7f475d711640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f473c0781b0 0x7f473c07a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.030+0000 7f475d711640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f473c0781b0 0x7f473c07a670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4754003eb0 tx=0x7f4754009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.034+0000 7f474dffb640 1 -- 172.21.15.28:0/3842733941 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4748014030 con 0x7f47580d03b0 2024-09-10T20:40:29.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.255+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47580b3110 con 0x7f47580d03b0 2024-09-10T20:40:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.255+0000 7f474dffb640 1 -- 172.21.15.28:0/3842733941 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f4748062e90 con 0x7f47580d03b0 2024-09-10T20:40:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:29.260 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:29.261 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f473c0781b0 msgr2=0x7f473c07a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f473c0781b0 0x7f473c07a670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4754003eb0 tx=0x7f4754009040 comp rx=0 tx=0).stop 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d03b0 msgr2=0x7f4758156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d03b0 0x7f4758156000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f474800c920 tx=0x7f474800cde0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 shutdown_connections 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f473c0781b0 0x7f473c07a670 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47580d0790 0x7f47580b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47580d03b0 0x7f4758156000 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 --2- 172.21.15.28:0/3842733941 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47580707b0 0x7f47581500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 >> 172.21.15.28:0/3842733941 conn(0x7f475806cf00 msgr2=0x7f47580b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 shutdown_connections 2024-09-10T20:40:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:29.258+0000 7f475f975640 1 -- 172.21.15.28:0/3842733941 wait complete. 2024-09-10T20:40:29.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:29.395 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":50.362557293000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.583862061793667,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:29.395 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.583862061793667 }, "state": "open", "uptime": 50.362557293 } ] 2024-09-10T20:40:29.395 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:31.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:40:31.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:40:34.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.512+0000 7f3704ebd640 1 -- 172.21.15.28:0/3424476440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 msgr2=0x7f37000d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.512+0000 7f3704ebd640 1 --2- 172.21.15.28:0/3424476440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f37000d0140 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f36f8005fd0 tx=0x7f36f802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.512+0000 7f3704ebd640 1 -- 172.21.15.28:0/3424476440 shutdown_connections 2024-09-10T20:40:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.512+0000 7f3704ebd640 1 --2- 172.21.15.28:0/3424476440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f37000d0140 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.512+0000 7f3704ebd640 1 --2- 172.21.15.28:0/3424476440 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37001554e0 0x7f370015e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.512+0000 7f3704ebd640 1 -- 172.21.15.28:0/3424476440 >> 172.21.15.28:0/3424476440 conn(0x7f370006cf00 msgr2=0x7f370006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.513+0000 7f3704ebd640 1 -- 172.21.15.28:0/3424476440 shutdown_connections 2024-09-10T20:40:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.513+0000 7f3704ebd640 1 -- 172.21.15.28:0/3424476440 wait complete. 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.514+0000 7f3704ebd640 1 Processor -- start 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.514+0000 7f3704ebd640 1 -- start start 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f3704ebd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3700155100 0x7f37000b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f3704ebd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37001554e0 0x7f3700152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f3704ebd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f370015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f3704ebd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3700072f60 con 0x7f370015ee70 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f3704ebd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3700072de0 con 0x7f3700155100 2024-09-10T20:40:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f3704ebd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f37000730e0 con 0x7f37001554e0 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f370015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f370015bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43084/0 (socket says 172.21.15.28:43084) 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 -- 172.21.15.28:0/514208849 learned_addr learned my addr 172.21.15.28:0/514208849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fdd9b640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37001554e0 0x7f3700152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fe59c640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3700155100 0x7f37000b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 -- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37001554e0 msgr2=0x7f3700152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37001554e0 0x7f3700152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 -- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3700155100 msgr2=0x7f37000b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3700155100 0x7f37000b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 -- 172.21.15.28:0/514208849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37001534b0 con 0x7f370015ee70 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.515+0000 7f36fed9d640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f370015bd50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009cd0 tx=0x7f36f8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.516+0000 7f36eeffd640 1 -- 172.21.15.28:0/514208849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8040070 con 0x7f370015ee70 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.516+0000 7f36eeffd640 1 -- 172.21.15.28:0/514208849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36f8004130 con 0x7f370015ee70 2024-09-10T20:40:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.516+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37000affe0 con 0x7f370015ee70 2024-09-10T20:40:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.516+0000 7f36eeffd640 1 -- 172.21.15.28:0/514208849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8004540 con 0x7f370015ee70 2024-09-10T20:40:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.516+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37000b05a0 con 0x7f370015ee70 2024-09-10T20:40:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.517+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3700073590 con 0x7f370015ee70 2024-09-10T20:40:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.517+0000 7f36eeffd640 1 -- 172.21.15.28:0/514208849 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f36f804d030 con 0x7f370015ee70 2024-09-10T20:40:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.518+0000 7f36eeffd640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f36e00781b0 0x7f36e007a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.518+0000 7f36eeffd640 1 -- 172.21.15.28:0/514208849 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f36f80bba50 con 0x7f370015ee70 2024-09-10T20:40:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.518+0000 7f36fe59c640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f36e00781b0 0x7f36e007a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.519+0000 7f36fe59c640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f36e00781b0 0x7f36e007a670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f36f0007fd0 tx=0x7f36f0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.522+0000 7f36eeffd640 1 -- 172.21.15.28:0/514208849 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f37001534b0 con 0x7f370015ee70 2024-09-10T20:40:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.747+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37000008d0 con 0x7f370015ee70 2024-09-10T20:40:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.748+0000 7f36eeffd640 1 -- 172.21.15.28:0/514208849 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f36f8083fb0 con 0x7f370015ee70 2024-09-10T20:40:34.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:34.753 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:34.753 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f36e00781b0 msgr2=0x7f36e007a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f36e00781b0 0x7f36e007a670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f36f0007fd0 tx=0x7f36f0009040 comp rx=0 tx=0).stop 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 msgr2=0x7f370015bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f370015bd50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009cd0 tx=0x7f36f8003970 comp rx=0 tx=0).stop 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 shutdown_connections 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f36e00781b0 0x7f36e007a670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370015ee70 0x7f370015bd50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37001554e0 0x7f3700152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 --2- 172.21.15.28:0/514208849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3700155100 0x7f37000b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 >> 172.21.15.28:0/514208849 conn(0x7f370006cf00 msgr2=0x7f370006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 shutdown_connections 2024-09-10T20:40:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:34.750+0000 7f3704ebd640 1 -- 172.21.15.28:0/514208849 wait complete. 2024-09-10T20:40:34.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:34.886 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":55.853399899999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.33697995882531,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:34.887 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.33697995882531 }, "state": "open", "uptime": 55.8533999 } ] 2024-09-10T20:40:34.887 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:39.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 -- 172.21.15.28:0/4178873589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80b4340 msgr2=0x7f81e80d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 --2- 172.21.15.28:0/4178873589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80b4340 0x7f81e80d0090 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f81e80d0c80 tx=0x7f81dc02a560 comp rx=0 tx=0).stop 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 -- 172.21.15.28:0/4178873589 shutdown_connections 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 --2- 172.21.15.28:0/4178873589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e80b47d0 0x7f81e80d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 --2- 172.21.15.28:0/4178873589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80b4340 0x7f81e80d0090 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 --2- 172.21.15.28:0/4178873589 >> v2:172.21.15.159:3300/0 conn(0x7f81e80cd3d0 0x7f81e80cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 -- 172.21.15.28:0/4178873589 >> 172.21.15.28:0/4178873589 conn(0x7f81e806cf00 msgr2=0x7f81e806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 -- 172.21.15.28:0/4178873589 shutdown_connections 2024-09-10T20:40:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.003+0000 7f81ec857640 1 -- 172.21.15.28:0/4178873589 wait complete. 2024-09-10T20:40:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 Processor -- start 2024-09-10T20:40:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 -- start start 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e80b4340 0x7f81e81f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e80b47d0 0x7f81e8146df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80cd3d0 0x7f81e809bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81e80b3210 con 0x7f81e80cd3d0 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f81e80b3090 con 0x7f81e80b4340 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.005+0000 7f81ec857640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f81e80b3390 con 0x7f81e80b47d0 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80cd3d0 0x7f81e809bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80cd3d0 0x7f81e809bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43088/0 (socket says 172.21.15.28:43088) 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 -- 172.21.15.28:0/728904506 learned_addr learned my addr 172.21.15.28:0/728904506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e659c640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e80b4340 0x7f81e81f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e5d9b640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e80b47d0 0x7f81e8146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 -- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e80b47d0 msgr2=0x7f81e8146df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e80b47d0 0x7f81e8146df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 -- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e80b4340 msgr2=0x7f81e81f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e80b4340 0x7f81e81f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 -- 172.21.15.28:0/728904506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e80b10b0 con 0x7f81e80cd3d0 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81e6d9d640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80cd3d0 0x7f81e809bc70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f81e000daa0 tx=0x7f81e000df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81d6ffd640 1 -- 172.21.15.28:0/728904506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e0018070 con 0x7f81e80cd3d0 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81d6ffd640 1 -- 172.21.15.28:0/728904506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81e0004b20 con 0x7f81e80cd3d0 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e81f6560 con 0x7f81e80cd3d0 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81d6ffd640 1 -- 172.21.15.28:0/728904506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e0003970 con 0x7f81e80cd3d0 2024-09-10T20:40:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.006+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e81f6ac0 con 0x7f81e80cd3d0 2024-09-10T20:40:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.008+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e815ec10 con 0x7f81e80cd3d0 2024-09-10T20:40:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.008+0000 7f81d6ffd640 1 -- 172.21.15.28:0/728904506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f81e0012760 con 0x7f81e80cd3d0 2024-09-10T20:40:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.009+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f81c80781e0 0x7f81c807a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.009+0000 7f81d6ffd640 1 -- 172.21.15.28:0/728904506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f81e009e5e0 con 0x7f81e80cd3d0 2024-09-10T20:40:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.009+0000 7f81e659c640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f81c80781e0 0x7f81c807a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.009+0000 7f81e659c640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f81c80781e0 0x7f81c807a6a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f81d8005fd0 tx=0x7f81d8005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.013+0000 7f81d6ffd640 1 -- 172.21.15.28:0/728904506 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f81e81f6560 con 0x7f81e80cd3d0 2024-09-10T20:40:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.233+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81e8147330 con 0x7f81e80cd3d0 2024-09-10T20:40:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.234+0000 7f81d6ffd640 1 -- 172.21.15.28:0/728904506 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f81e0066be0 con 0x7f81e80cd3d0 2024-09-10T20:40:40.238 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:40.238 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f81c80781e0 msgr2=0x7f81c807a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f81c80781e0 0x7f81c807a6a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f81d8005fd0 tx=0x7f81d8005850 comp rx=0 tx=0).stop 2024-09-10T20:40:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80cd3d0 msgr2=0x7f81e809bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80cd3d0 0x7f81e809bc70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f81e000daa0 tx=0x7f81e000df60 comp rx=0 tx=0).stop 2024-09-10T20:40:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 shutdown_connections 2024-09-10T20:40:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f81c80781e0 0x7f81c807a6a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e80cd3d0 0x7f81e809bc70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f81e80b47d0 0x7f81e8146df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 --2- 172.21.15.28:0/728904506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81e80b4340 0x7f81e81f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 >> 172.21.15.28:0/728904506 conn(0x7f81e806cf00 msgr2=0x7f81e815ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 shutdown_connections 2024-09-10T20:40:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:40.236+0000 7f81ec857640 1 -- 172.21.15.28:0/728904506 wait complete. 2024-09-10T20:40:40.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:40.373 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":61.341242592999997,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.093342492426942,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:40.373 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.09334249242694 }, "state": "open", "uptime": 61.341242593 } ] 2024-09-10T20:40:40.373 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:45.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.490+0000 7f8700d5d640 1 -- 172.21.15.28:0/2244037675 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0aefe0 msgr2=0x7f86fc0b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.490+0000 7f8700d5d640 1 --2- 172.21.15.28:0/2244037675 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0aefe0 0x7f86fc0b2f40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009830 tx=0x7f86f002a510 comp rx=0 tx=0).stop 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.490+0000 7f8700d5d640 1 -- 172.21.15.28:0/2244037675 shutdown_connections 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.490+0000 7f8700d5d640 1 --2- 172.21.15.28:0/2244037675 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0aefe0 0x7f86fc0b2f40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.490+0000 7f8700d5d640 1 --2- 172.21.15.28:0/2244037675 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86fc15e160 0x7f86fc0704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.490+0000 7f8700d5d640 1 -- 172.21.15.28:0/2244037675 >> 172.21.15.28:0/2244037675 conn(0x7f86fc0b1c40 msgr2=0x7f86fc0b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.490+0000 7f8700d5d640 1 -- 172.21.15.28:0/2244037675 shutdown_connections 2024-09-10T20:40:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.491+0000 7f8700d5d640 1 -- 172.21.15.28:0/2244037675 wait complete. 2024-09-10T20:40:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.492+0000 7f8700d5d640 1 Processor -- start 2024-09-10T20:40:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.492+0000 7f8700d5d640 1 -- start start 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f8700d5d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0735f0 0x7f86fc14da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f8700d5d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86fc0aefe0 0x7f86fc158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f8700d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc15e160 0x7f86fc158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f8700d5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc0aebb0 con 0x7f86fc15e160 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f8700d5d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc0aea30 con 0x7f86fc0735f0 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f8700d5d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f86fc0aed30 con 0x7f86fc0aefe0 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc15e160 0x7f86fc158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc15e160 0x7f86fc158ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34622/0 (socket says 172.21.15.28:34622) 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 -- 172.21.15.28:0/1206829578 learned_addr learned my addr 172.21.15.28:0/1206829578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fa59c640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0735f0 0x7f86fc14da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86f9d9b640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86fc0aefe0 0x7f86fc158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 -- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86fc0aefe0 msgr2=0x7f86fc158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86fc0aefe0 0x7f86fc158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 -- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0735f0 msgr2=0x7f86fc14da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0735f0 0x7f86fc14da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.493+0000 7f86fad9d640 1 -- 172.21.15.28:0/1206829578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc0c5d70 con 0x7f86fc15e160 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.494+0000 7f86fad9d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc15e160 0x7f86fc158ea0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f86f400c400 tx=0x7f86f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.494+0000 7f86eaffd640 1 -- 172.21.15.28:0/1206829578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4011790 con 0x7f86fc15e160 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.494+0000 7f86eaffd640 1 -- 172.21.15.28:0/1206829578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86f4017070 con 0x7f86fc15e160 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.494+0000 7f86eaffd640 1 -- 172.21.15.28:0/1206829578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4011ce0 con 0x7f86fc15e160 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.494+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86fc09bc60 con 0x7f86fc15e160 2024-09-10T20:40:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.494+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86fc09c220 con 0x7f86fc15e160 2024-09-10T20:40:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.495+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86fc074af0 con 0x7f86fc15e160 2024-09-10T20:40:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.495+0000 7f86eaffd640 1 -- 172.21.15.28:0/1206829578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7f86f401c460 con 0x7f86fc15e160 2024-09-10T20:40:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.496+0000 7f86eaffd640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f86dc0781e0 0x7f86dc07a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.496+0000 7f86eaffd640 1 -- 172.21.15.28:0/1206829578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f86f4099e50 con 0x7f86fc15e160 2024-09-10T20:40:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.496+0000 7f86fa59c640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f86dc0781e0 0x7f86dc07a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:45.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.497+0000 7f86fa59c640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f86dc0781e0 0x7f86dc07a6a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f86fc06fea0 tx=0x7f86ec009500 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.500+0000 7f86eaffd640 1 -- 172.21.15.28:0/1206829578 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f86fc074af0 con 0x7f86fc15e160 2024-09-10T20:40:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.723+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86fc073d40 con 0x7f86fc15e160 2024-09-10T20:40:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.723+0000 7f86eaffd640 1 -- 172.21.15.28:0/1206829578 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f86f4061f60 con 0x7f86fc15e160 2024-09-10T20:40:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:45.727 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:45.728 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f86dc0781e0 msgr2=0x7f86dc07a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f86dc0781e0 0x7f86dc07a6a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f86fc06fea0 tx=0x7f86ec009500 comp rx=0 tx=0).stop 2024-09-10T20:40:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc15e160 msgr2=0x7f86fc158ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc15e160 0x7f86fc158ea0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f86f400c400 tx=0x7f86f400c8c0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 shutdown_connections 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f86dc0781e0 0x7f86dc07a6a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc15e160 0x7f86fc158ea0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86fc0aefe0 0x7f86fc158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 --2- 172.21.15.28:0/1206829578 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86fc0735f0 0x7f86fc14da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 >> 172.21.15.28:0/1206829578 conn(0x7f86fc0b1c40 msgr2=0x7f86fc0afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 shutdown_connections 2024-09-10T20:40:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:45.725+0000 7f8700d5d640 1 -- 172.21.15.28:0/1206829578 wait complete. 2024-09-10T20:40:45.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:45.862 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":66.830085253999997,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.852731193532598,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:45.863 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.8527311935326 }, "state": "open", "uptime": 66.830085254 } ] 2024-09-10T20:40:45.863 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:50.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 -- 172.21.15.28:0/1605097693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c146a80 msgr2=0x7ff12c0d16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 --2- 172.21.15.28:0/1605097693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c146a80 0x7ff12c0d16e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff128005fd0 tx=0x7ff12802a530 comp rx=0 tx=0).stop 2024-09-10T20:40:50.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 -- 172.21.15.28:0/1605097693 shutdown_connections 2024-09-10T20:40:50.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 --2- 172.21.15.28:0/1605097693 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff12c162060 0x7ff12c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:50.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 --2- 172.21.15.28:0/1605097693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c146a80 0x7ff12c0d16e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:50.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 -- 172.21.15.28:0/1605097693 >> 172.21.15.28:0/1605097693 conn(0x7ff12c06dbf0 msgr2=0x7ff12c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:50.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 -- 172.21.15.28:0/1605097693 shutdown_connections 2024-09-10T20:40:50.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.980+0000 7ff13352b640 1 -- 172.21.15.28:0/1605097693 wait complete. 2024-09-10T20:40:50.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.981+0000 7ff13352b640 1 Processor -- start 2024-09-10T20:40:50.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.981+0000 7ff13352b640 1 -- start start 2024-09-10T20:40:50.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff13352b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff12c146a80 0x7ff12c14e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:50.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff13352b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff12c146e60 0x7ff12c1f8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff13352b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c162060 0x7ff12c1455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff13352b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c06f760 con 0x7ff12c162060 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff13352b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c06d490 con 0x7ff12c146a80 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff13352b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c06f8e0 con 0x7ff12c146e60 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff131ac8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c162060 0x7ff12c1455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff131ac8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c162060 0x7ff12c1455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33454/0 (socket says 172.21.15.28:33454) 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff131ac8640 1 -- 172.21.15.28:0/3545281498 learned_addr learned my addr 172.21.15.28:0/3545281498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff1312c7640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff12c146a80 0x7ff12c14e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.982+0000 7ff130ac6640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff12c146e60 0x7ff12c1f8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff131ac8640 1 -- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff12c146e60 msgr2=0x7ff12c1f8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff131ac8640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff12c146e60 0x7ff12c1f8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff131ac8640 1 -- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff12c146a80 msgr2=0x7ff12c14e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff131ac8640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff12c146a80 0x7ff12c14e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff131ac8640 1 -- 172.21.15.28:0/3545281498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c1f5870 con 0x7ff12c162060 2024-09-10T20:40:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff131ac8640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c162060 0x7ff12c1455e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff12400cae0 tx=0x7ff124007220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff121ffb640 1 -- 172.21.15.28:0/3545281498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124007da0 con 0x7ff12c162060 2024-09-10T20:40:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff121ffb640 1 -- 172.21.15.28:0/3545281498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff124004c30 con 0x7ff12c162060 2024-09-10T20:40:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff121ffb640 1 -- 172.21.15.28:0/3545281498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124005280 con 0x7ff12c162060 2024-09-10T20:40:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.983+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12c0b19d0 con 0x7ff12c162060 2024-09-10T20:40:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.984+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff12c0b1f90 con 0x7ff12c162060 2024-09-10T20:40:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.985+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff12c0d16c0 con 0x7ff12c162060 2024-09-10T20:40:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.985+0000 7ff121ffb640 1 -- 172.21.15.28:0/3545281498 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 100805+0+0 (secure 0 0 0) 0x7ff124005420 con 0x7ff12c162060 2024-09-10T20:40:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.986+0000 7ff121ffb640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff1100781b0 0x7ff11007a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.986+0000 7ff121ffb640 1 -- 172.21.15.28:0/3545281498 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7ff12409ae70 con 0x7ff12c162060 2024-09-10T20:40:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.986+0000 7ff1312c7640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff1100781b0 0x7ff11007a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.987+0000 7ff1312c7640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff1100781b0 0x7ff11007a670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff128033000 tx=0x7ff128004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:50.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:50.990+0000 7ff121ffb640 1 -- 172.21.15.28:0/3545281498 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff124014070 con 0x7ff12c162060 2024-09-10T20:40:51.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.213+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff12c0ae9d0 con 0x7ff12c162060 2024-09-10T20:40:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.213+0000 7ff121ffb640 1 -- 172.21.15.28:0/3545281498 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7ff124062d00 con 0x7ff12c162060 2024-09-10T20:40:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:51.218 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff1100781b0 msgr2=0x7ff11007a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff1100781b0 0x7ff11007a670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff128033000 tx=0x7ff128004730 comp rx=0 tx=0).stop 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c162060 msgr2=0x7ff12c1455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c162060 0x7ff12c1455e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff12400cae0 tx=0x7ff124007220 comp rx=0 tx=0).stop 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 shutdown_connections 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff1100781b0 0x7ff11007a670 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff12c162060 0x7ff12c1455e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff12c146e60 0x7ff12c1f8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 --2- 172.21.15.28:0/3545281498 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff12c146a80 0x7ff12c14e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 >> 172.21.15.28:0/3545281498 conn(0x7ff12c06dbf0 msgr2=0x7ff12c1614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 shutdown_connections 2024-09-10T20:40:51.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:51.215+0000 7ff13352b640 1 -- 172.21.15.28:0/3545281498 wait complete. 2024-09-10T20:40:51.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:51.352 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5021,"entity":{"name":{"type":"client","num":5021},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":2005231082}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":72.319927890000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.615109031372224,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5021 172.21.15.159:0/2005231082","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:40:51.352 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 2005231082, "type": "any" }, "name": { "num": 5021, "type": "client" } }, "id": 5021, "inst": "client.5021 172.21.15.159:0/2005231082", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.615109031372224 }, "state": "open", "uptime": 72.31992789 } ] 2024-09-10T20:40:51.352 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-10T20:40:56.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:56.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 -- 172.21.15.28:0/2419685065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c415dec0 msgr2=0x7f43c40d48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:56.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/2419685065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c415dec0 0x7f43c40d48f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f43b8005fd0 tx=0x7f43b802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 -- 172.21.15.28:0/2419685065 shutdown_connections 2024-09-10T20:40:56.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/2419685065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43c40cf3f0 0x7f43c40d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/2419685065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c415dec0 0x7f43c40d48f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 -- 172.21.15.28:0/2419685065 >> 172.21.15.28:0/2419685065 conn(0x7f43c40d3dd0 msgr2=0x7f43c40d41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:56.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 -- 172.21.15.28:0/2419685065 shutdown_connections 2024-09-10T20:40:56.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.472+0000 7f43c9f9d640 1 -- 172.21.15.28:0/2419685065 wait complete. 2024-09-10T20:40:56.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.473+0000 7f43c9f9d640 1 Processor -- start 2024-09-10T20:40:56.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.473+0000 7f43c9f9d640 1 -- start start 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c9f9d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43c40cf3f0 0x7f43c409f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c9f9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c40d2390 0x7f43c40a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c9f9d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43c415dec0 0x7f43c4154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c9f9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43c40d0ef0 con 0x7f43c40d2390 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c9f9d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f43c40b8910 con 0x7f43c415dec0 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c9f9d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f43c40d1070 con 0x7f43c40cf3f0 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c40d2390 0x7f43c40a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c40d2390 0x7f43c40a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33470/0 (socket says 172.21.15.28:33470) 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c3fff640 1 -- 172.21.15.28:0/3899304366 learned_addr learned my addr 172.21.15.28:0/3899304366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c8f9b640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43c40cf3f0 0x7f43c409f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.474+0000 7f43c979c640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43c415dec0 0x7f43c4154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c3fff640 1 -- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43c40cf3f0 msgr2=0x7f43c409f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c3fff640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43c40cf3f0 0x7f43c409f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c3fff640 1 -- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43c415dec0 msgr2=0x7f43c4154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c3fff640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43c415dec0 0x7f43c4154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c3fff640 1 -- 172.21.15.28:0/3899304366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c409dd00 con 0x7f43c40d2390 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c8f9b640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43c40cf3f0 0x7f43c409f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c3fff640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c40d2390 0x7f43c40a11c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f43b000e9f0 tx=0x7f43b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c17fa640 1 -- 172.21.15.28:0/3899304366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b000cd20 con 0x7f43c40d2390 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c17fa640 1 -- 172.21.15.28:0/3899304366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43b0010430 con 0x7f43c40d2390 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c409dfe0 con 0x7f43c40d2390 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c17fa640 1 -- 172.21.15.28:0/3899304366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0010840 con 0x7f43c40d2390 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.475+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c409e5a0 con 0x7f43c40d2390 2024-09-10T20:40:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.476+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c409ea80 con 0x7f43c40d2390 2024-09-10T20:40:56.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.477+0000 7f43c17fa640 1 -- 172.21.15.28:0/3899304366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f43b00109e0 con 0x7f43c40d2390 2024-09-10T20:40:56.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.477+0000 7f43c17fa640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43a8078150 0x7f43a807a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:56.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.478+0000 7f43c17fa640 1 -- 172.21.15.28:0/3899304366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f43b009ad50 con 0x7f43c40d2390 2024-09-10T20:40:56.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.478+0000 7f43c8f9b640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43a8078150 0x7f43a807a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:56.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.478+0000 7f43c8f9b640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43a8078150 0x7f43a807a610 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f43b8033000 tx=0x7f43b8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:56.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.482+0000 7f43c17fa640 1 -- 172.21.15.28:0/3899304366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f43c409ea80 con 0x7f43c40d2390 2024-09-10T20:40:56.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.703+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43c40d1c30 con 0x7f43c40d2390 2024-09-10T20:40:56.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.703+0000 7f43c17fa640 1 -- 172.21.15.28:0/3899304366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f43b00634e0 con 0x7f43c40d2390 2024-09-10T20:40:56.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:56.707 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43a8078150 msgr2=0x7f43a807a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43a8078150 0x7f43a807a610 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f43b8033000 tx=0x7f43b8004730 comp rx=0 tx=0).stop 2024-09-10T20:40:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c40d2390 msgr2=0x7f43c40a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c40d2390 0x7f43c40a11c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f43b000e9f0 tx=0x7f43b000eeb0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 shutdown_connections 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43a8078150 0x7f43a807a610 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43c415dec0 0x7f43c4154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c40d2390 0x7f43c40a11c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 --2- 172.21.15.28:0/3899304366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f43c40cf3f0 0x7f43c409f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 >> 172.21.15.28:0/3899304366 conn(0x7f43c40d3dd0 msgr2=0x7f43c40cfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 shutdown_connections 2024-09-10T20:40:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:56.705+0000 7f43c9f9d640 1 -- 172.21.15.28:0/3899304366 wait complete. 2024-09-10T20:40:56.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2024-09-10T20:40:56.841 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-10T20:40:56.842 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-10T20:40:56.842 DEBUG:tasks.ceph_test_case:wait_until_true: success in 70s and 0 retries 2024-09-10T20:40:56.842 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:40:56.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:40:56.883 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:40:56.935 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:40:56.935 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:40:56.936 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:40:56.936 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:40:56.952 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:40:56.952 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:40:56.952 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:40:56.952 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:40:56.972 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:40:56.972 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:40:56.972 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:40:56.972 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:40:57.009 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:40:57.009 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:40:57.009 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:40:57.010 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:40:57.029 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:40:57.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 -- 172.21.15.28:0/3079433774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 msgr2=0x7f119c15e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 --2- 172.21.15.28:0/3079433774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c15e520 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1198005fd0 tx=0x7f119802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 -- 172.21.15.28:0/3079433774 shutdown_connections 2024-09-10T20:40:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 --2- 172.21.15.28:0/3079433774 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f119c1545d0 0x7f119c15a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 --2- 172.21.15.28:0/3079433774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c15e520 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 -- 172.21.15.28:0/3079433774 >> 172.21.15.28:0/3079433774 conn(0x7f119c06cf00 msgr2=0x7f119c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:57.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 -- 172.21.15.28:0/3079433774 shutdown_connections 2024-09-10T20:40:57.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.151+0000 7f11a3b52640 1 -- 172.21.15.28:0/3079433774 wait complete. 2024-09-10T20:40:57.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.152+0000 7f11a3b52640 1 Processor -- start 2024-09-10T20:40:57.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.152+0000 7f11a3b52640 1 -- start start 2024-09-10T20:40:57.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a3b52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c0cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a3b52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f119c0d0790 0x7f119c0ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a3b52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f119c1545d0 0x7f119c0ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a3b52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f119c0707b0 con 0x7f119c0d03b0 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a3b52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f119c073b50 con 0x7f119c1545d0 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a3b52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f119c070930 con 0x7f119c0d0790 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c0cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c0cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33492/0 (socket says 172.21.15.28:33492) 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 -- 172.21.15.28:0/2381492656 learned_addr learned my addr 172.21.15.28:0/2381492656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a20ef640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f119c1545d0 0x7f119c0ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a10ed640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f119c0d0790 0x7f119c0ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 -- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f119c0d0790 msgr2=0x7f119c0ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f119c0d0790 0x7f119c0ad550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 -- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f119c1545d0 msgr2=0x7f119c0ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f119c1545d0 0x7f119c0ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.153+0000 7f11a18ee640 1 -- 172.21.15.28:0/2381492656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c0ae340 con 0x7f119c0d03b0 2024-09-10T20:40:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.154+0000 7f11a18ee640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c0cb700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1198006010 tx=0x7f11980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.154+0000 7f11927fc640 1 -- 172.21.15.28:0/2381492656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119803f070 con 0x7f119c0d03b0 2024-09-10T20:40:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.154+0000 7f11927fc640 1 -- 172.21.15.28:0/2381492656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1198005600 con 0x7f119c0d03b0 2024-09-10T20:40:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.154+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c153230 con 0x7f119c0d03b0 2024-09-10T20:40:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.154+0000 7f11927fc640 1 -- 172.21.15.28:0/2381492656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198035030 con 0x7f119c0d03b0 2024-09-10T20:40:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.154+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c1537f0 con 0x7f119c0d03b0 2024-09-10T20:40:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.155+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119c0cf2b0 con 0x7f119c0d03b0 2024-09-10T20:40:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.155+0000 7f11927fc640 1 -- 172.21.15.28:0/2381492656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f11980049e0 con 0x7f119c0d03b0 2024-09-10T20:40:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.156+0000 7f11927fc640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1180078120 0x7f118007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.156+0000 7f11927fc640 1 -- 172.21.15.28:0/2381492656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f11980bafb0 con 0x7f119c0d03b0 2024-09-10T20:40:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.156+0000 7f11a10ed640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1180078120 0x7f118007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.157+0000 7f11a10ed640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1180078120 0x7f118007a5e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f119c150e90 tx=0x7f118c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.160+0000 7f11927fc640 1 -- 172.21.15.28:0/2381492656 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f119c0cf2b0 con 0x7f119c0d03b0 2024-09-10T20:40:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.382+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f119c15e520 con 0x7f119c0d03b0 2024-09-10T20:40:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.383+0000 7f11927fc640 1 -- 172.21.15.28:0/2381492656 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f11980830f0 con 0x7f119c0d03b0 2024-09-10T20:40:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:57.387 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1180078120 msgr2=0x7f118007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1180078120 0x7f118007a5e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f119c150e90 tx=0x7f118c009040 comp rx=0 tx=0).stop 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 msgr2=0x7f119c0cb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c0cb700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1198006010 tx=0x7f11980038a0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 shutdown_connections 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1180078120 0x7f118007a5e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f119c1545d0 0x7f119c0ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f119c0d0790 0x7f119c0ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 --2- 172.21.15.28:0/2381492656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f119c0d03b0 0x7f119c0cb700 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 >> 172.21.15.28:0/2381492656 conn(0x7f119c06cf00 msgr2=0x7f119c0ccf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 shutdown_connections 2024-09-10T20:40:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.385+0000 7f11a3b52640 1 -- 172.21.15.28:0/2381492656 wait complete. 2024-09-10T20:40:57.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.501+0000 7fec2b133640 1 -- 172.21.15.28:0/1331605771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec10003fd0 msgr2=0x7fec10004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.501+0000 7fec2b133640 1 --2- 172.21.15.28:0/1331605771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec10003fd0 0x7fec10004800 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009ed0 tx=0x7fec10005830 comp rx=0 tx=0).stop 2024-09-10T20:40:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.501+0000 7fec296d0640 1 -- 172.21.15.28:0/1331605771 reap_dead start 2024-09-10T20:40:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.501+0000 7fec2b133640 1 -- 172.21.15.28:0/1331605771 shutdown_connections 2024-09-10T20:40:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.501+0000 7fec2b133640 1 -- 172.21.15.28:0/1331605771 >> 172.21.15.28:0/1331605771 conn(0x7fec240d3fb0 msgr2=0x7fec240d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.501+0000 7fec2b133640 1 -- 172.21.15.28:0/1331605771 shutdown_connections 2024-09-10T20:40:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.501+0000 7fec2b133640 1 -- 172.21.15.28:0/1331605771 wait complete. 2024-09-10T20:40:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.502+0000 7fec2b133640 1 Processor -- start 2024-09-10T20:40:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.502+0000 7fec2b133640 1 -- start start 2024-09-10T20:40:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec2b133640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec10003fd0 0x7fec241f8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec2b133640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240b0ae0 0x7fec240b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec2b133640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec240b3310 0x7fec241f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec2b133640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec240d2d10 con 0x7fec240b3310 2024-09-10T20:40:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec2b133640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fec240d2b90 con 0x7fec240b0ae0 2024-09-10T20:40:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec2b133640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fec240d2e90 con 0x7fec10003fd0 2024-09-10T20:40:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec240b3310 0x7fec241f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec240b3310 0x7fec241f4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33512/0 (socket says 172.21.15.28:33512) 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 -- 172.21.15.28:0/3935849480 learned_addr learned my addr 172.21.15.28:0/3935849480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec28ecf640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec10003fd0 0x7fec241f8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec23fff640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240b0ae0 0x7fec240b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 -- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec10003fd0 msgr2=0x7fec241f8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec10003fd0 0x7fec241f8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 -- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240b0ae0 msgr2=0x7fec240b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240b0ae0 0x7fec240b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec296d0640 1 -- 172.21.15.28:0/3935849480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec240b0f70 con 0x7fec240b3310 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.503+0000 7fec23fff640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240b0ae0 0x7fec240b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.504+0000 7fec296d0640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec240b3310 0x7fec241f4460 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fec240d4db0 tx=0x7fec1c00e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.504+0000 7fec217fa640 1 -- 172.21.15.28:0/3935849480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c015020 con 0x7fec240b3310 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.504+0000 7fec217fa640 1 -- 172.21.15.28:0/3935849480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec1c010980 con 0x7fec240b3310 2024-09-10T20:40:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.504+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec240b1250 con 0x7fec240b3310 2024-09-10T20:40:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.504+0000 7fec217fa640 1 -- 172.21.15.28:0/3935849480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c010dd0 con 0x7fec240b3310 2024-09-10T20:40:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.504+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec240b1810 con 0x7fec240b3310 2024-09-10T20:40:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.505+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec241622b0 con 0x7fec240b3310 2024-09-10T20:40:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.505+0000 7fec217fa640 1 -- 172.21.15.28:0/3935849480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fec1c0193b0 con 0x7fec240b3310 2024-09-10T20:40:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.506+0000 7fec217fa640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fec04078120 0x7fec0407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.506+0000 7fec217fa640 1 -- 172.21.15.28:0/3935849480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7fec1c09a8e0 con 0x7fec240b3310 2024-09-10T20:40:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.506+0000 7fec28ecf640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fec04078120 0x7fec0407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.507+0000 7fec28ecf640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fec04078120 0x7fec0407a5e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fec14009a40 tx=0x7fec14005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:57.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.511+0000 7fec217fa640 1 -- 172.21.15.28:0/3935849480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fec1c01f030 con 0x7fec240b3310 2024-09-10T20:40:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.733+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec240b9f30 con 0x7fec240b3310 2024-09-10T20:40:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.734+0000 7fec217fa640 1 -- 172.21.15.28:0/3935849480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7fec1c062fb0 con 0x7fec240b3310 2024-09-10T20:40:57.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:57.738 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fec04078120 msgr2=0x7fec0407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fec04078120 0x7fec0407a5e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fec14009a40 tx=0x7fec14005f90 comp rx=0 tx=0).stop 2024-09-10T20:40:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec240b3310 msgr2=0x7fec241f4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec240b3310 0x7fec241f4460 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fec240d4db0 tx=0x7fec1c00e9a0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 shutdown_connections 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fec04078120 0x7fec0407a5e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec240b3310 0x7fec241f4460 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240b0ae0 0x7fec240b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 --2- 172.21.15.28:0/3935849480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec10003fd0 0x7fec241f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 >> 172.21.15.28:0/3935849480 conn(0x7fec240d3fb0 msgr2=0x7fec240cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 shutdown_connections 2024-09-10T20:40:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.736+0000 7fec2b133640 1 -- 172.21.15.28:0/3935849480 wait complete. 2024-09-10T20:40:57.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:40:57.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2877975158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf0162090 msgr2=0x7f6cf00b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2877975158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf0162090 0x7f6cf00b6e00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00d4d80 tx=0x7f6ce802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2877975158 shutdown_connections 2024-09-10T20:40:57.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2877975158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf0162090 0x7f6cf00b6e00 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2877975158 >> v2:172.21.15.159:3300/0 conn(0x7f6cf00d2b90 0x7f6cf0161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2877975158 >> 172.21.15.28:0/2877975158 conn(0x7f6cf00d3fb0 msgr2=0x7f6cf00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:57.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2877975158 shutdown_connections 2024-09-10T20:40:57.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.852+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2877975158 wait complete. 2024-09-10T20:40:57.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 Processor -- start 2024-09-10T20:40:57.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 -- start start 2024-09-10T20:40:57.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2b90 0x7f6cf015b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf015de70 0x7f6cf00a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf0162090 0x7f6cf00a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf015e3d0 con 0x7f6cf015de70 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf015e250 con 0x7f6cf00d2b90 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.854+0000 7f6cf4e40640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf00b7340 con 0x7f6cf0162090 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf015de70 0x7f6cf00a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf015de70 0x7f6cf00a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33536/0 (socket says 172.21.15.28:33536) 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 -- 172.21.15.28:0/2712434404 learned_addr learned my addr 172.21.15.28:0/2712434404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6cee59c640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2b90 0x7f6cf015b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ceed9d640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf0162090 0x7f6cf00a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 -- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf0162090 msgr2=0x7f6cf00a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf0162090 0x7f6cf00a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 -- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2b90 msgr2=0x7f6cf015b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2b90 0x7f6cf015b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 -- 172.21.15.28:0/2712434404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf01f7cd0 con 0x7f6cf015de70 2024-09-10T20:40:57.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ceed9d640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf0162090 0x7f6cf00a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:40:57.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce7fff640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf015de70 0x7f6cf00a1310 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800c890 tx=0x7f6cd800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:57.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce6ffd640 1 -- 172.21.15.28:0/2712434404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8004220 con 0x7f6cf015de70 2024-09-10T20:40:57.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce6ffd640 1 -- 172.21.15.28:0/2712434404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cd80079b0 con 0x7f6cf015de70 2024-09-10T20:40:57.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6ce6ffd640 1 -- 172.21.15.28:0/2712434404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8007de0 con 0x7f6cf015de70 2024-09-10T20:40:57.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf01f7fb0 con 0x7f6cf015de70 2024-09-10T20:40:57.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.855+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf01f8570 con 0x7f6cf015de70 2024-09-10T20:40:57.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.856+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf00b7810 con 0x7f6cf015de70 2024-09-10T20:40:57.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.857+0000 7f6ce6ffd640 1 -- 172.21.15.28:0/2712434404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f6cd8005020 con 0x7f6cf015de70 2024-09-10T20:40:57.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.857+0000 7f6ce6ffd640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6cd0078120 0x7f6cd007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:57.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.857+0000 7f6ce6ffd640 1 -- 172.21.15.28:0/2712434404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f6cd809a6b0 con 0x7f6cf015de70 2024-09-10T20:40:57.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.858+0000 7f6cee59c640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6cd0078120 0x7f6cd007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:57.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.858+0000 7f6cee59c640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6cd0078120 0x7f6cd007a5e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6cf01f2230 tx=0x7f6ce0007e90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:57.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:57.862+0000 7f6ce6ffd640 1 -- 172.21.15.28:0/2712434404 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6cd8014030 con 0x7f6cf015de70 2024-09-10T20:40:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.049+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cf00b9f30 con 0x7f6cf015de70 2024-09-10T20:40:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.050+0000 7f6ce6ffd640 1 -- 172.21.15.28:0/2712434404 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16901 (secure 0 0 0) 0x7f6cd8062740 con 0x7f6cf015de70 2024-09-10T20:40:58.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:58.054 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:39:53.544595+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:39:20.458884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-10T20:39:21.778513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.0","mappings":[{"from":5,"to":7}]},{"pgid":"7.14","mappings":[{"from":5,"to":7}]},{"pgid":"7.2e","mappings":[{"from":5,"to":0}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:40:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6cd0078120 msgr2=0x7f6cd007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6cd0078120 0x7f6cd007a5e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6cf01f2230 tx=0x7f6ce0007e90 comp rx=0 tx=0).stop 2024-09-10T20:40:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf015de70 msgr2=0x7f6cf00a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf015de70 0x7f6cf00a1310 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800c890 tx=0x7f6cd800cd50 comp rx=0 tx=0).stop 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 shutdown_connections 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6cd0078120 0x7f6cd007a5e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cf0162090 0x7f6cf00a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cf015de70 0x7f6cf00a1310 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 --2- 172.21.15.28:0/2712434404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cf00d2b90 0x7f6cf015b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 >> 172.21.15.28:0/2712434404 conn(0x7f6cf00d3fb0 msgr2=0x7f6cf00cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 shutdown_connections 2024-09-10T20:40:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.052+0000 7f6cf4e40640 1 -- 172.21.15.28:0/2712434404 wait complete. 2024-09-10T20:40:58.067 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:40:58.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:58.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.169+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3030002383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dac0041c0 msgr2=0x7f7dac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.169+0000 7f7dc81ec640 1 --2- 172.21.15.28:0/3030002383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dac0041c0 0x7f7dac004800 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7db0006f90 tx=0x7f7dac005830 comp rx=0 tx=0).stop 2024-09-10T20:40:58.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.169+0000 7f7dc6789640 1 -- 172.21.15.28:0/3030002383 reap_dead start 2024-09-10T20:40:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.169+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3030002383 shutdown_connections 2024-09-10T20:40:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.169+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3030002383 >> 172.21.15.28:0/3030002383 conn(0x7f7dc006cf00 msgr2=0x7f7dc006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.169+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3030002383 shutdown_connections 2024-09-10T20:40:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.169+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3030002383 wait complete. 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.170+0000 7f7dc81ec640 1 Processor -- start 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.170+0000 7f7dc81ec640 1 -- start start 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc81ec640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7dac0041c0 0x7f7dc0152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc81ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc01f24a0 0x7f7dc00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc81ec640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7dc00afed0 0x7f7dc00b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc81ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc00b3210 con 0x7f7dc01f24a0 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc81ec640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc00b3090 con 0x7f7dac0041c0 2024-09-10T20:40:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc81ec640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7dc00b3390 con 0x7f7dc00afed0 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc5787640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc01f24a0 0x7f7dc00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc5787640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc01f24a0 0x7f7dc00b2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33556/0 (socket says 172.21.15.28:33556) 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc5787640 1 -- 172.21.15.28:0/3942806145 learned_addr learned my addr 172.21.15.28:0/3942806145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc6789640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7dc00afed0 0x7f7dc00b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.171+0000 7f7dc5f88640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7dac0041c0 0x7f7dc0152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc5787640 1 -- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7dc00afed0 msgr2=0x7f7dc00b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc5787640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7dc00afed0 0x7f7dc00b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc5787640 1 -- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7dac0041c0 msgr2=0x7f7dc0152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc5787640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7dac0041c0 0x7f7dc0152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc5787640 1 -- 172.21.15.28:0/3942806145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc01f28c0 con 0x7f7dc01f24a0 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc5787640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc01f24a0 0x7f7dc00b2000 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7db0031040 tx=0x7f7db0006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7db67fc640 1 -- 172.21.15.28:0/3942806145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0032030 con 0x7f7dc01f24a0 2024-09-10T20:40:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7db67fc640 1 -- 172.21.15.28:0/3942806145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7db0003af0 con 0x7f7dc01f24a0 2024-09-10T20:40:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc01f2b70 con 0x7f7dc01f24a0 2024-09-10T20:40:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7db67fc640 1 -- 172.21.15.28:0/3942806145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db002e460 con 0x7f7dc01f24a0 2024-09-10T20:40:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.172+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc01f3130 con 0x7f7dc01f24a0 2024-09-10T20:40:58.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.173+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dc006e4d0 con 0x7f7dc01f24a0 2024-09-10T20:40:58.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.174+0000 7f7db67fc640 1 -- 172.21.15.28:0/3942806145 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f7db002e600 con 0x7f7dc01f24a0 2024-09-10T20:40:58.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.174+0000 7f7db67fc640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7da4078120 0x7f7da407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.174+0000 7f7db67fc640 1 -- 172.21.15.28:0/3942806145 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f7db00ba6c0 con 0x7f7dc01f24a0 2024-09-10T20:40:58.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.175+0000 7f7dc5f88640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7da4078120 0x7f7da407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.175+0000 7f7dc5f88640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7da4078120 0x7f7da407a5e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc006560 tx=0x7f7dbc006b10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.178+0000 7f7db67fc640 1 -- 172.21.15.28:0/3942806145 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7db0047030 con 0x7f7dc01f24a0 2024-09-10T20:40:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.405+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dc00b38b0 con 0x7f7dc01f24a0 2024-09-10T20:40:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.405+0000 7f7db67fc640 1 -- 172.21.15.28:0/3942806145 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f7db0082e30 con 0x7f7dc01f24a0 2024-09-10T20:40:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:58.410 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7da4078120 msgr2=0x7f7da407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7da4078120 0x7f7da407a5e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc006560 tx=0x7f7dbc006b10 comp rx=0 tx=0).stop 2024-09-10T20:40:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc01f24a0 msgr2=0x7f7dc00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc01f24a0 0x7f7dc00b2000 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7db0031040 tx=0x7f7db0006ca0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 shutdown_connections 2024-09-10T20:40:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7da4078120 0x7f7da407a5e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7dc00afed0 0x7f7dc00b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc01f24a0 0x7f7dc00b2000 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 --2- 172.21.15.28:0/3942806145 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7dac0041c0 0x7f7dc0152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 >> 172.21.15.28:0/3942806145 conn(0x7f7dc006cf00 msgr2=0x7f7dc00758b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:58.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 shutdown_connections 2024-09-10T20:40:58.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.408+0000 7f7dc81ec640 1 -- 172.21.15.28:0/3942806145 wait complete. 2024-09-10T20:40:58.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:40:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.525+0000 7f7456d16640 1 -- 172.21.15.28:0/227890451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bf20 msgr2=0x7f745006cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.525+0000 7f7456d16640 1 --2- 172.21.15.28:0/227890451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bf20 0x7f745006cae0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f743c009830 tx=0x7f743c02a5c0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.525+0000 7f7456d16640 1 -- 172.21.15.28:0/227890451 shutdown_connections 2024-09-10T20:40:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.525+0000 7f7456d16640 1 --2- 172.21.15.28:0/227890451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bf20 0x7f745006cae0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.525+0000 7f7456d16640 1 -- 172.21.15.28:0/227890451 >> 172.21.15.28:0/227890451 conn(0x7f745006dbf0 msgr2=0x7f745006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.525+0000 7f7456d16640 1 -- 172.21.15.28:0/227890451 shutdown_connections 2024-09-10T20:40:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.525+0000 7f7456d16640 1 -- 172.21.15.28:0/227890451 wait complete. 2024-09-10T20:40:58.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.526+0000 7f7456d16640 1 Processor -- start 2024-09-10T20:40:58.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.526+0000 7f7456d16640 1 -- start start 2024-09-10T20:40:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f7456d16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745006d020 0x7f74501557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f7456d16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bb40 0x7f74501f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f7456d16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745015bf20 0x7f74501f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f7456d16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f745015d680 con 0x7f745015bb40 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f7456d16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f745015d500 con 0x7f745015bf20 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f7456d16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f745015d800 con 0x7f745006d020 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bb40 0x7f74501f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bb40 0x7f74501f5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33570/0 (socket says 172.21.15.28:33570) 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 -- 172.21.15.28:0/3930232760 learned_addr learned my addr 172.21.15.28:0/3930232760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f7454ab2640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745006d020 0x7f74501557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f74552b3640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745015bf20 0x7f74501f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 -- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745006d020 msgr2=0x7f74501557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745006d020 0x7f74501557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 -- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745015bf20 msgr2=0x7f74501f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745015bf20 0x7f74501f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f744ffff640 1 -- 172.21.15.28:0/3930232760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74501571a0 con 0x7f745015bb40 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.527+0000 7f74552b3640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745015bf20 0x7f74501f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.528+0000 7f744ffff640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bb40 0x7f74501f5780 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f743c009950 tx=0x7f743c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.528+0000 7f744d7fa640 1 -- 172.21.15.28:0/3930232760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c031d60 con 0x7f745015bb40 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.528+0000 7f744d7fa640 1 -- 172.21.15.28:0/3930232760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f743c033e10 con 0x7f745015bb40 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.528+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7450156100 con 0x7f745015bb40 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.528+0000 7f744d7fa640 1 -- 172.21.15.28:0/3930232760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c033310 con 0x7f745015bb40 2024-09-10T20:40:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.528+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74501566c0 con 0x7f745015bb40 2024-09-10T20:40:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.529+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f745015dcb0 con 0x7f745015bb40 2024-09-10T20:40:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.530+0000 7f744d7fa640 1 -- 172.21.15.28:0/3930232760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f743c04b030 con 0x7f745015bb40 2024-09-10T20:40:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.530+0000 7f744d7fa640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7434078150 0x7f743407a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.530+0000 7f744d7fa640 1 -- 172.21.15.28:0/3930232760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f743c0bb320 con 0x7f745015bb40 2024-09-10T20:40:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.531+0000 7f7454ab2640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7434078150 0x7f743407a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.531+0000 7f7454ab2640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7434078150 0x7f743407a610 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7444001680 tx=0x7f7444009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:58.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.534+0000 7f744d7fa640 1 -- 172.21.15.28:0/3930232760 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f743c03f030 con 0x7f745015bb40 2024-09-10T20:40:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.754+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f745015cf60 con 0x7f745015bb40 2024-09-10T20:40:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.754+0000 7f744d7fa640 1 -- 172.21.15.28:0/3930232760 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5229 (secure 0 0 0) 0x7f743c083960 con 0x7f745015bb40 2024-09-10T20:40:58.759 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:58.759 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-10T20:39:36:502610+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:39:22.829916+0000","modified":"2024-09-10T20:39:25.841932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3839556955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3839556955},{"type":"v1","addr":"172.21.15.28:6835","nonce":3839556955}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":3}]} 2024-09-10T20:40:58.759 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-10T20:40:58.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.756+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7434078150 msgr2=0x7f743407a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.756+0000 7f7456d16640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7434078150 0x7f743407a610 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7444001680 tx=0x7f7444009040 comp rx=0 tx=0).stop 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.756+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bb40 msgr2=0x7f74501f5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.756+0000 7f7456d16640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bb40 0x7f74501f5780 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f743c009950 tx=0x7f743c033c20 comp rx=0 tx=0).stop 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 shutdown_connections 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7434078150 0x7f743407a610 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f745015bf20 0x7f74501f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f745015bb40 0x7f74501f5780 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 --2- 172.21.15.28:0/3930232760 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f745006d020 0x7f74501557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 >> 172.21.15.28:0/3930232760 conn(0x7f745006dbf0 msgr2=0x7f7450159b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 shutdown_connections 2024-09-10T20:40:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.757+0000 7f7456d16640 1 -- 172.21.15.28:0/3930232760 wait complete. 2024-09-10T20:40:58.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.873+0000 7f0d024e6640 1 -- 172.21.15.28:0/3203087338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc1554e0 msgr2=0x7f0cfc15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.873+0000 7f0d024e6640 1 --2- 172.21.15.28:0/3203087338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc1554e0 0x7f0cfc15e930 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009830 tx=0x7f0cf002a510 comp rx=0 tx=0).stop 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.874+0000 7f0d024e6640 1 -- 172.21.15.28:0/3203087338 shutdown_connections 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.874+0000 7f0d024e6640 1 --2- 172.21.15.28:0/3203087338 >> v2:172.21.15.159:3300/0 conn(0x7f0cfc15ee70 0x7f0cfc0d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.874+0000 7f0d024e6640 1 --2- 172.21.15.28:0/3203087338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc1554e0 0x7f0cfc15e930 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.874+0000 7f0d024e6640 1 -- 172.21.15.28:0/3203087338 >> 172.21.15.28:0/3203087338 conn(0x7f0cfc06cf00 msgr2=0x7f0cfc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.874+0000 7f0d024e6640 1 -- 172.21.15.28:0/3203087338 shutdown_connections 2024-09-10T20:40:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.874+0000 7f0d024e6640 1 -- 172.21.15.28:0/3203087338 wait complete. 2024-09-10T20:40:58.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.875+0000 7f0d024e6640 1 Processor -- start 2024-09-10T20:40:58.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.875+0000 7f0d024e6640 1 -- start start 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0d024e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cfc155100 0x7f0cfc09dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0d024e6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cfc1554e0 0x7f0cfc14b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0d024e6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc15ee70 0x7f0cfc06e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0d024e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc072f60 con 0x7f0cfc155100 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0d024e6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc072de0 con 0x7f0cfc1554e0 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0d024e6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0cfc0730e0 con 0x7f0cfc15ee70 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cfc155100 0x7f0cfc09dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cfc155100 0x7f0cfc09dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33586/0 (socket says 172.21.15.28:33586) 2024-09-10T20:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 -- 172.21.15.28:0/1057137655 learned_addr learned my addr 172.21.15.28:0/1057137655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0d00a83640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc15ee70 0x7f0cfc06e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfb7fe640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cfc1554e0 0x7f0cfc14b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 -- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc15ee70 msgr2=0x7f0cfc06e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc15ee70 0x7f0cfc06e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 -- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cfc1554e0 msgr2=0x7f0cfc14b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cfc1554e0 0x7f0cfc14b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.876+0000 7f0cfbfff640 1 -- 172.21.15.28:0/1057137655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc0aeb80 con 0x7f0cfc155100 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.877+0000 7f0cfbfff640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cfc155100 0x7f0cfc09dc30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00c430 tx=0x7f0cec00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.877+0000 7f0cf8ff9640 1 -- 172.21.15.28:0/1057137655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec011790 con 0x7f0cfc155100 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.877+0000 7f0cf8ff9640 1 -- 172.21.15.28:0/1057137655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cec017070 con 0x7f0cfc155100 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.877+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cfc0affd0 con 0x7f0cfc155100 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.877+0000 7f0cf8ff9640 1 -- 172.21.15.28:0/1057137655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec011ce0 con 0x7f0cfc155100 2024-09-10T20:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.877+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cfc0b0830 con 0x7f0cfc155100 2024-09-10T20:40:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.878+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cfc073590 con 0x7f0cfc155100 2024-09-10T20:40:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.879+0000 7f0cf8ff9640 1 -- 172.21.15.28:0/1057137655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f0cec01c460 con 0x7f0cfc155100 2024-09-10T20:40:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.879+0000 7f0cf8ff9640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ce8078120 0x7f0ce807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.879+0000 7f0cf8ff9640 1 -- 172.21.15.28:0/1057137655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f0cec099e20 con 0x7f0cfc155100 2024-09-10T20:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.880+0000 7f0cfb7fe640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ce8078120 0x7f0ce807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.880+0000 7f0cfb7fe640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ce8078120 0x7f0ce807a5e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0034000 tx=0x7f0cf0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:58.883+0000 7f0cf8ff9640 1 -- 172.21.15.28:0/1057137655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0cec062590 con 0x7f0cfc155100 2024-09-10T20:40:59.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.069+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cfc0b0fd0 con 0x7f0cfc155100 2024-09-10T20:40:59.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.070+0000 7f0cf8ff9640 1 -- 172.21.15.28:0/1057137655 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16901 (secure 0 0 0) 0x7f0cec061e90 con 0x7f0cfc155100 2024-09-10T20:40:59.074 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:40:59.074 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:39:53.544595+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:39:20.458884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-10T20:39:21.778513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.0","mappings":[{"from":5,"to":7}]},{"pgid":"7.14","mappings":[{"from":5,"to":7}]},{"pgid":"7.2e","mappings":[{"from":5,"to":0}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:40:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ce8078120 msgr2=0x7f0ce807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ce8078120 0x7f0ce807a5e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0034000 tx=0x7f0cf0004730 comp rx=0 tx=0).stop 2024-09-10T20:40:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cfc155100 msgr2=0x7f0cfc09dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cfc155100 0x7f0cfc09dc30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00c430 tx=0x7f0cec00c830 comp rx=0 tx=0).stop 2024-09-10T20:40:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 shutdown_connections 2024-09-10T20:40:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ce8078120 0x7f0ce807a5e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cfc15ee70 0x7f0cfc06e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cfc1554e0 0x7f0cfc14b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 --2- 172.21.15.28:0/1057137655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cfc155100 0x7f0cfc09dc30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 >> 172.21.15.28:0/1057137655 conn(0x7f0cfc06cf00 msgr2=0x7f0cfc0cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 shutdown_connections 2024-09-10T20:40:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.072+0000 7f0d024e6640 1 -- 172.21.15.28:0/1057137655 wait complete. 2024-09-10T20:40:59.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:40:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.190+0000 7f1633953640 1 -- 172.21.15.28:0/3404525941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 msgr2=0x7f162c0caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.190+0000 7f1633953640 1 --2- 172.21.15.28:0/3404525941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0caa70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f161c005fd0 tx=0x7f161c02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.190+0000 7f1633953640 1 -- 172.21.15.28:0/3404525941 shutdown_connections 2024-09-10T20:40:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.190+0000 7f1633953640 1 --2- 172.21.15.28:0/3404525941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0caa70 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.190+0000 7f1633953640 1 -- 172.21.15.28:0/3404525941 >> 172.21.15.28:0/3404525941 conn(0x7f162c0d0420 msgr2=0x7f162c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.190+0000 7f1633953640 1 -- 172.21.15.28:0/3404525941 shutdown_connections 2024-09-10T20:40:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.190+0000 7f1633953640 1 -- 172.21.15.28:0/3404525941 wait complete. 2024-09-10T20:40:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.191+0000 7f1633953640 1 Processor -- start 2024-09-10T20:40:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.191+0000 7f1633953640 1 -- start start 2024-09-10T20:40:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1633953640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f162c06d1f0 0x7f162c141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1633953640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1633953640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f162c156510 0x7f162c0c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1633953640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f162c06f2c0 con 0x7f162c156130 2024-09-10T20:40:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1633953640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f162c06f140 con 0x7f162c156510 2024-09-10T20:40:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1633953640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f162c073900 con 0x7f162c06d1f0 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0c9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33618/0 (socket says 172.21.15.28:33618) 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 -- 172.21.15.28:0/2130688558 learned_addr learned my addr 172.21.15.28:0/2130688558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f16316ef640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f162c06d1f0 0x7f162c141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1631ef0640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f162c156510 0x7f162c0c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 -- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f162c06d1f0 msgr2=0x7f162c141ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f162c06d1f0 0x7f162c141ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 -- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f162c156510 msgr2=0x7f162c0c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f162c156510 0x7f162c0c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1630eee640 1 -- 172.21.15.28:0/2130688558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162c0c8a70 con 0x7f162c156130 2024-09-10T20:40:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.192+0000 7f1631ef0640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f162c156510 0x7f162c0c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.193+0000 7f1630eee640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0c9b10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f162000c370 tx=0x7f162000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.193+0000 7f1629ffb640 1 -- 172.21.15.28:0/2130688558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620017070 con 0x7f162c156130 2024-09-10T20:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.193+0000 7f1629ffb640 1 -- 172.21.15.28:0/2130688558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16200053c0 con 0x7f162c156130 2024-09-10T20:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.193+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f162c0c8d50 con 0x7f162c156130 2024-09-10T20:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.193+0000 7f1629ffb640 1 -- 172.21.15.28:0/2130688558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620003850 con 0x7f162c156130 2024-09-10T20:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.193+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f162c0c9310 con 0x7f162c156130 2024-09-10T20:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.194+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f162c0c9800 con 0x7f162c156130 2024-09-10T20:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.194+0000 7f1629ffb640 1 -- 172.21.15.28:0/2130688558 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f16200039f0 con 0x7f162c156130 2024-09-10T20:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.195+0000 7f1629ffb640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1618078120 0x7f161807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.195+0000 7f1629ffb640 1 -- 172.21.15.28:0/2130688558 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5854+0+0 (secure 0 0 0) 0x7f1620099730 con 0x7f162c156130 2024-09-10T20:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.195+0000 7f16316ef640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1618078120 0x7f161807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.196+0000 7f16316ef640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1618078120 0x7f161807a5e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f161c034000 tx=0x7f161c004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.199+0000 7f1629ffb640 1 -- 172.21.15.28:0/2130688558 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1620061850 con 0x7f162c156130 2024-09-10T20:40:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.420+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f162c073db0 con 0x7f162c156130 2024-09-10T20:40:59.424 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:40:59.421+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:40:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.422+0000 7f1629ffb640 1 -- 172.21.15.28:0/2130688558 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) ==== 161+0+0 (secure 0 0 0) 0x7f1620066280 con 0x7f162c156130 2024-09-10T20:40:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:40:59.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1618078120 msgr2=0x7f161807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1618078120 0x7f161807a5e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f161c034000 tx=0x7f161c004ab0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 msgr2=0x7f162c0c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0c9b10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f162000c370 tx=0x7f162000c830 comp rx=0 tx=0).stop 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 shutdown_connections 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1618078120 0x7f161807a5e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f162c156510 0x7f162c0c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f162c156130 0x7f162c0c9b10 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 --2- 172.21.15.28:0/2130688558 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f162c06d1f0 0x7f162c141ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.425+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 >> 172.21.15.28:0/2130688558 conn(0x7f162c0d0420 msgr2=0x7f162c0b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.426+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 shutdown_connections 2024-09-10T20:40:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.426+0000 7f1633953640 1 -- 172.21.15.28:0/2130688558 wait complete. 2024-09-10T20:40:59.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:40:59.468 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:40:59.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 -- 172.21.15.28:0/4138725537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe68072d70 msgr2=0x7efe680ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/4138725537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe68072d70 0x7efe680ce1c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7efe60009830 tx=0x7efe6002a5a0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 -- 172.21.15.28:0/4138725537 shutdown_connections 2024-09-10T20:40:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/4138725537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe68072d70 0x7efe680ce1c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/4138725537 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6815bbc0 0x7efe68070020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 -- 172.21.15.28:0/4138725537 >> 172.21.15.28:0/4138725537 conn(0x7efe680b1800 msgr2=0x7efe680b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:40:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 -- 172.21.15.28:0/4138725537 shutdown_connections 2024-09-10T20:40:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.549+0000 7efe6ecfe640 1 -- 172.21.15.28:0/4138725537 wait complete. 2024-09-10T20:40:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.550+0000 7efe6ecfe640 1 Processor -- start 2024-09-10T20:40:59.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.550+0000 7efe6ecfe640 1 -- start start 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ecfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6806f820 0x7efe6814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ecfe640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe68072d70 0x7efe68157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ecfe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6815bbc0 0x7efe68157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ecfe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe680ce880 con 0x7efe6806f820 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ecfe640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe680ce700 con 0x7efe68072d70 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ecfe640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe680cea00 con 0x7efe6815bbc0 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ca9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6806f820 0x7efe6814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ca9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6806f820 0x7efe6814dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33654/0 (socket says 172.21.15.28:33654) 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.551+0000 7efe6ca9a640 1 -- 172.21.15.28:0/3424980753 learned_addr learned my addr 172.21.15.28:0/3424980753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6d29b640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6815bbc0 0x7efe68157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe67fff640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe68072d70 0x7efe68157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ca9a640 1 -- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6815bbc0 msgr2=0x7efe68157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ca9a640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6815bbc0 0x7efe68157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ca9a640 1 -- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe68072d70 msgr2=0x7efe68157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ca9a640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe68072d70 0x7efe68157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ca9a640 1 -- 172.21.15.28:0/3424980753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6814fcd0 con 0x7efe6806f820 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6d29b640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6815bbc0 0x7efe68157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe67fff640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe68072d70 0x7efe68157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ca9a640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6806f820 0x7efe6814dc40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7efe5000d7a0 tx=0x7efe5000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe657fa640 1 -- 172.21.15.28:0/3424980753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50017070 con 0x7efe6806f820 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe657fa640 1 -- 172.21.15.28:0/3424980753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe50009d20 con 0x7efe6806f820 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6814ffb0 con 0x7efe6806f820 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe657fa640 1 -- 172.21.15.28:0/3424980753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50004280 con 0x7efe6806f820 2024-09-10T20:40:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.552+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe68150570 con 0x7efe6806f820 2024-09-10T20:40:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.553+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe680cef20 con 0x7efe6806f820 2024-09-10T20:40:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.554+0000 7efe657fa640 1 -- 172.21.15.28:0/3424980753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7efe50005020 con 0x7efe6806f820 2024-09-10T20:40:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.554+0000 7efe657fa640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe48078120 0x7efe4807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:40:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.555+0000 7efe657fa640 1 -- 172.21.15.28:0/3424980753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5940+0+0 (secure 0 0 0) 0x7efe5009a6a0 con 0x7efe6806f820 2024-09-10T20:40:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.555+0000 7efe67fff640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe48078120 0x7efe4807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:40:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.555+0000 7efe67fff640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe48078120 0x7efe4807a5e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe580015a0 tx=0x7efe58009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:40:59.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.559+0000 7efe657fa640 1 -- 172.21.15.28:0/3424980753 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efe50020020 con 0x7efe6806f820 2024-09-10T20:40:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:40:59.784+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7efe68074570 con 0x7efe6806f820 2024-09-10T20:41:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.424+0000 7efe657fa640 1 -- 172.21.15.28:0/3424980753 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7efe50062d20 con 0x7efe6806f820 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe48078120 msgr2=0x7efe4807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe48078120 0x7efe4807a5e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe580015a0 tx=0x7efe58009040 comp rx=0 tx=0).stop 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6806f820 msgr2=0x7efe6814dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6806f820 0x7efe6814dc40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7efe5000d7a0 tx=0x7efe5000dc60 comp rx=0 tx=0).stop 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 shutdown_connections 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7efe48078120 0x7efe4807a5e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe6815bbc0 0x7efe68157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe68072d70 0x7efe68157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 --2- 172.21.15.28:0/3424980753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6806f820 0x7efe6814dc40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 >> 172.21.15.28:0/3424980753 conn(0x7efe680b1800 msgr2=0x7efe6815a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:00.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 shutdown_connections 2024-09-10T20:41:00.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.428+0000 7efe6ecfe640 1 -- 172.21.15.28:0/3424980753 wait complete. 2024-09-10T20:41:00.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:41:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 -- 172.21.15.28:0/2018040499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9674161850 msgr2=0x7f96740d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 --2- 172.21.15.28:0/2018040499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9674161850 0x7f96740d4a80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9668005fd0 tx=0x7f966802a670 comp rx=0 tx=0).stop 2024-09-10T20:41:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 -- 172.21.15.28:0/2018040499 shutdown_connections 2024-09-10T20:41:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 --2- 172.21.15.28:0/2018040499 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9674161850 0x7f96740d4a80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 --2- 172.21.15.28:0/2018040499 >> v2:172.21.15.159:3300/0 conn(0x7f96740b4c20 0x7f96740d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 -- 172.21.15.28:0/2018040499 >> 172.21.15.28:0/2018040499 conn(0x7f96740d3f90 msgr2=0x7f96740d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 -- 172.21.15.28:0/2018040499 shutdown_connections 2024-09-10T20:41:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.597+0000 7f967a1da640 1 -- 172.21.15.28:0/2018040499 wait complete. 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.598+0000 7f967a1da640 1 Processor -- start 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.598+0000 7f967a1da640 1 -- start start 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f967a1da640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96740b4c20 0x7f96740cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f967a1da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9674161850 0x7f967414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f967a1da640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9674162000 0x7f96740b90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f967a1da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96740d2320 con 0x7f9674161850 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f967a1da640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f96740d21a0 con 0x7f9674162000 2024-09-10T20:41:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f967a1da640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f96740d24a0 con 0x7f96740b4c20 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9674161850 0x7f967414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9674161850 0x7f967414f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44332/0 (socket says 172.21.15.28:44332) 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 -- 172.21.15.28:0/4066190309 learned_addr learned my addr 172.21.15.28:0/4066190309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9673fff640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9674162000 0x7f96740b90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f96737fe640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96740b4c20 0x7f96740cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 -- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96740b4c20 msgr2=0x7f96740cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96740b4c20 0x7f96740cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 -- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9674162000 msgr2=0x7f96740b90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9674162000 0x7f96740b90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.599+0000 7f9672ffd640 1 -- 172.21.15.28:0/4066190309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96740cb520 con 0x7f9674161850 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.600+0000 7f9672ffd640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9674161850 0x7f967414f600 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9668000c00 tx=0x7f96680038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.600+0000 7f965bfff640 1 -- 172.21.15.28:0/4066190309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966803f070 con 0x7f9674161850 2024-09-10T20:41:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.600+0000 7f965bfff640 1 -- 172.21.15.28:0/4066190309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96680047a0 con 0x7f9674161850 2024-09-10T20:41:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.600+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96740cb800 con 0x7f9674161850 2024-09-10T20:41:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.600+0000 7f965bfff640 1 -- 172.21.15.28:0/4066190309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668035030 con 0x7f9674161850 2024-09-10T20:41:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.600+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96740cbdc0 con 0x7f9674161850 2024-09-10T20:41:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.601+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96740b9f10 con 0x7f9674161850 2024-09-10T20:41:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.602+0000 7f965bfff640 1 -- 172.21.15.28:0/4066190309 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f9668004bf0 con 0x7f9674161850 2024-09-10T20:41:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.602+0000 7f965bfff640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9660078120 0x7f966007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.602+0000 7f965bfff640 1 -- 172.21.15.28:0/4066190309 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5940+0+0 (secure 0 0 0) 0x7f96680bb510 con 0x7f9674161850 2024-09-10T20:41:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.602+0000 7f96737fe640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9660078120 0x7f966007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.603+0000 7f96737fe640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9660078120 0x7f966007a5e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9664001720 tx=0x7f9664009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.606+0000 7f965bfff640 1 -- 172.21.15.28:0/4066190309 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9668083540 con 0x7f9674161850 2024-09-10T20:41:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:00.797+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9674156f40 con 0x7f9674161850 2024-09-10T20:41:01.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:41:01.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:41:01.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.704+0000 7f965bfff640 1 -- 172.21.15.28:0/4066190309 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) ==== 181+0+0 (secure 0 0 0) 0x7f9668087f70 con 0x7f9674161850 2024-09-10T20:41:01.708 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9660078120 msgr2=0x7f966007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9660078120 0x7f966007a5e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9664001720 tx=0x7f9664009040 comp rx=0 tx=0).stop 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9674161850 msgr2=0x7f967414f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9674161850 0x7f967414f600 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9668000c00 tx=0x7f96680038a0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 shutdown_connections 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9660078120 0x7f966007a5e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9674162000 0x7f96740b90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9674161850 0x7f967414f600 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 --2- 172.21.15.28:0/4066190309 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96740b4c20 0x7f96740cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 >> 172.21.15.28:0/4066190309 conn(0x7f96740d3f90 msgr2=0x7f967406d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:01.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 shutdown_connections 2024-09-10T20:41:01.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.708+0000 7f967a1da640 1 -- 172.21.15.28:0/4066190309 wait complete. 2024-09-10T20:41:01.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:41:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 -- 172.21.15.28:0/170921856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 msgr2=0x7fbae80741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 --2- 172.21.15.28:0/170921856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80741b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbae815ff60 tx=0x7fbae002a5f0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 -- 172.21.15.28:0/170921856 shutdown_connections 2024-09-10T20:41:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 --2- 172.21.15.28:0/170921856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80741b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 --2- 172.21.15.28:0/170921856 >> v2:172.21.15.159:3300/0 conn(0x7fbae806e390 0x7fbae815cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 -- 172.21.15.28:0/170921856 >> 172.21.15.28:0/170921856 conn(0x7fbae80cfc80 msgr2=0x7fbae80d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 -- 172.21.15.28:0/170921856 shutdown_connections 2024-09-10T20:41:01.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.839+0000 7fbaeceef640 1 -- 172.21.15.28:0/170921856 wait complete. 2024-09-10T20:41:01.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.840+0000 7fbaeceef640 1 Processor -- start 2024-09-10T20:41:01.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.840+0000 7fbaeceef640 1 -- start start 2024-09-10T20:41:01.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbaeceef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbae806e390 0x7fbae80a9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:01.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbaeceef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:01.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbaeceef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbae80b5030 0x7fbae80c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbaeceef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80ca2d0 con 0x7fbae8073d40 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbaeceef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80ca450 con 0x7fbae80b5030 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbaeceef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbae80ca5d0 con 0x7fbae806e390 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbae5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbae5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44348/0 (socket says 172.21.15.28:44348) 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.841+0000 7fbae5d9b640 1 -- 172.21.15.28:0/795952826 learned_addr learned my addr 172.21.15.28:0/795952826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae6d9d640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbae80b5030 0x7fbae80c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae659c640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbae806e390 0x7fbae80a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae5d9b640 1 -- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbae806e390 msgr2=0x7fbae80a9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae5d9b640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbae806e390 0x7fbae80a9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae5d9b640 1 -- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbae80b5030 msgr2=0x7fbae80c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae5d9b640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbae80b5030 0x7fbae80c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae5d9b640 1 -- 172.21.15.28:0/795952826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae80c8f20 con 0x7fbae8073d40 2024-09-10T20:41:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae6d9d640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbae80b5030 0x7fbae80c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbae5d9b640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80c83b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00db00 tx=0x7fbadc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbad6ffd640 1 -- 172.21.15.28:0/795952826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc017070 con 0x7fbae8073d40 2024-09-10T20:41:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbad6ffd640 1 -- 172.21.15.28:0/795952826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbadc003970 con 0x7fbae8073d40 2024-09-10T20:41:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae80c9200 con 0x7fbae8073d40 2024-09-10T20:41:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbad6ffd640 1 -- 172.21.15.28:0/795952826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc003dc0 con 0x7fbae8073d40 2024-09-10T20:41:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.842+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae81f3850 con 0x7fbae8073d40 2024-09-10T20:41:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.843+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae806eff0 con 0x7fbae8073d40 2024-09-10T20:41:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.844+0000 7fbad6ffd640 1 -- 172.21.15.28:0/795952826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fbadc004020 con 0x7fbae8073d40 2024-09-10T20:41:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.844+0000 7fbad6ffd640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbac8078120 0x7fbac807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.844+0000 7fbad6ffd640 1 -- 172.21.15.28:0/795952826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5502+0+0 (secure 0 0 0) 0x7fbadc09a890 con 0x7fbae8073d40 2024-09-10T20:41:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.844+0000 7fbae659c640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbac8078120 0x7fbac807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.845+0000 7fbae659c640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbac8078120 0x7fbac807a5e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbad80015a0 tx=0x7fbad8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:01.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:01.849+0000 7fbad6ffd640 1 -- 172.21.15.28:0/795952826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbae806eff0 con 0x7fbae8073d40 2024-09-10T20:41:02.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.033+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbae80caaf0 con 0x7fbae8073d40 2024-09-10T20:41:02.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.708+0000 7fbad6ffd640 1 -- 172.21.15.28:0/795952826 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) ==== 169+0+0 (secure 0 0 0) 0x7fbadc020030 con 0x7fbae8073d40 2024-09-10T20:41:02.713 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:41:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbac8078120 msgr2=0x7fbac807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbac8078120 0x7fbac807a5e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbad80015a0 tx=0x7fbad8009040 comp rx=0 tx=0).stop 2024-09-10T20:41:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 msgr2=0x7fbae80c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80c83b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00db00 tx=0x7fbadc00b260 comp rx=0 tx=0).stop 2024-09-10T20:41:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 shutdown_connections 2024-09-10T20:41:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbac8078120 0x7fbac807a5e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbae80b5030 0x7fbae80c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8073d40 0x7fbae80c83b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 --2- 172.21.15.28:0/795952826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbae806e390 0x7fbae80a9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 >> 172.21.15.28:0/795952826 conn(0x7fbae80cfc80 msgr2=0x7fbae80cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 shutdown_connections 2024-09-10T20:41:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.712+0000 7fbaeceef640 1 -- 172.21.15.28:0/795952826 wait complete. 2024-09-10T20:41:02.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.839+0000 7fd134a08640 1 -- 172.21.15.28:0/338939469 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 msgr2=0x7fd1300cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.839+0000 7fd134a08640 1 --2- 172.21.15.28:0/338939469 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 0x7fd1300cc570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd120009830 tx=0x7fd12002a510 comp rx=0 tx=0).stop 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.840+0000 7fd134a08640 1 -- 172.21.15.28:0/338939469 shutdown_connections 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.840+0000 7fd134a08640 1 --2- 172.21.15.28:0/338939469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1300b4340 0x7fd1300b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.840+0000 7fd134a08640 1 --2- 172.21.15.28:0/338939469 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 0x7fd1300cc570 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.840+0000 7fd134a08640 1 -- 172.21.15.28:0/338939469 >> 172.21.15.28:0/338939469 conn(0x7fd13006cf00 msgr2=0x7fd13006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.840+0000 7fd134a08640 1 -- 172.21.15.28:0/338939469 shutdown_connections 2024-09-10T20:41:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.840+0000 7fd134a08640 1 -- 172.21.15.28:0/338939469 wait complete. 2024-09-10T20:41:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.841+0000 7fd134a08640 1 Processor -- start 2024-09-10T20:41:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.841+0000 7fd134a08640 1 -- start start 2024-09-10T20:41:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd134a08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1300b4340 0x7fd130150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd134a08640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 0x7fd1300af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd134a08640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1300d0090 0x7fd1300b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd134a08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1300d0980 con 0x7fd1300b4340 2024-09-10T20:41:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd134a08640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd1300d0800 con 0x7fd1300cd3d0 2024-09-10T20:41:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd134a08640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd1300b3590 con 0x7fd1300d0090 2024-09-10T20:41:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1300b4340 0x7fd130150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1300b4340 0x7fd130150770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44354/0 (socket says 172.21.15.28:44354) 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 -- 172.21.15.28:0/2458281534 learned_addr learned my addr 172.21.15.28:0/2458281534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12ed9d640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1300d0090 0x7fd1300b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12dd9b640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 0x7fd1300af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 -- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1300d0090 msgr2=0x7fd1300b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1300d0090 0x7fd1300b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 -- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 msgr2=0x7fd1300af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 0x7fd1300af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 -- 172.21.15.28:0/2458281534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1300b01b0 con 0x7fd1300b4340 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12dd9b640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 0x7fd1300af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.842+0000 7fd12e59c640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1300b4340 0x7fd130150770 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd120009950 tx=0x7fd120032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.843+0000 7fd11effd640 1 -- 172.21.15.28:0/2458281534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd120040070 con 0x7fd1300b4340 2024-09-10T20:41:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.843+0000 7fd11effd640 1 -- 172.21.15.28:0/2458281534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd120005310 con 0x7fd1300b4340 2024-09-10T20:41:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.843+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1300b0490 con 0x7fd1300b4340 2024-09-10T20:41:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.843+0000 7fd11effd640 1 -- 172.21.15.28:0/2458281534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1200047a0 con 0x7fd1300b4340 2024-09-10T20:41:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.843+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1300c4f20 con 0x7fd1300b4340 2024-09-10T20:41:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.844+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1300b3ab0 con 0x7fd1300b4340 2024-09-10T20:41:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.844+0000 7fd11effd640 1 -- 172.21.15.28:0/2458281534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fd120004940 con 0x7fd1300b4340 2024-09-10T20:41:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.845+0000 7fd11effd640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd110078150 0x7fd11007a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.845+0000 7fd11effd640 1 -- 172.21.15.28:0/2458281534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7fd1200bb290 con 0x7fd1300b4340 2024-09-10T20:41:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.845+0000 7fd12dd9b640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd110078150 0x7fd11007a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.846+0000 7fd12dd9b640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd110078150 0x7fd11007a610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b1250 tx=0x7fd124009260 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:02.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:02.849+0000 7fd11effd640 1 -- 172.21.15.28:0/2458281534 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd120083d10 con 0x7fd1300b4340 2024-09-10T20:41:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.034+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fd13015f210 con 0x7fd1300b4340 2024-09-10T20:41:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.037+0000 7fd11effd640 1 -- 172.21.15.28:0/2458281534 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v31) ==== 71+0+0 (secure 0 0 0) 0x7fd120083610 con 0x7fd1300b4340 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd110078150 msgr2=0x7fd11007a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd110078150 0x7fd11007a610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b1250 tx=0x7fd124009260 comp rx=0 tx=0).stop 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1300b4340 msgr2=0x7fd130150770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1300b4340 0x7fd130150770 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd120009950 tx=0x7fd120032c10 comp rx=0 tx=0).stop 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 shutdown_connections 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd110078150 0x7fd11007a610 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1300d0090 0x7fd1300b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1300cd3d0 0x7fd1300af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 --2- 172.21.15.28:0/2458281534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1300b4340 0x7fd130150770 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 >> 172.21.15.28:0/2458281534 conn(0x7fd13006cf00 msgr2=0x7fd130070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 shutdown_connections 2024-09-10T20:41:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.040+0000 7fd134a08640 1 -- 172.21.15.28:0/2458281534 wait complete. 2024-09-10T20:41:03.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-09-10T20:41:03.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 -- 172.21.15.28:0/1572458728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 msgr2=0x7f78ac15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 --2- 172.21.15.28:0/1572458728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac15e930 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f78a0005fd0 tx=0x7f78a002a530 comp rx=0 tx=0).stop 2024-09-10T20:41:03.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 -- 172.21.15.28:0/1572458728 shutdown_connections 2024-09-10T20:41:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 --2- 172.21.15.28:0/1572458728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac15e930 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 --2- 172.21.15.28:0/1572458728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78ac155100 0x7f78ac15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 -- 172.21.15.28:0/1572458728 >> 172.21.15.28:0/1572458728 conn(0x7f78ac06cf00 msgr2=0x7f78ac06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 -- 172.21.15.28:0/1572458728 shutdown_connections 2024-09-10T20:41:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.156+0000 7f78b2833640 1 -- 172.21.15.28:0/1572458728 wait complete. 2024-09-10T20:41:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.157+0000 7f78b2833640 1 Processor -- start 2024-09-10T20:41:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78b2833640 1 -- start start 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78b2833640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78ac155100 0x7f78ac06f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78b2833640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78b2833640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78ac15ee70 0x7f78ac158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78b2833640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac072f60 con 0x7f78ac1554e0 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78b2833640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac072de0 con 0x7f78ac155100 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78b2833640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f78ac0730e0 con 0x7f78ac15ee70 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.158+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44370/0 (socket says 172.21.15.28:44370) 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78ab7fe640 1 -- 172.21.15.28:0/542838555 learned_addr learned my addr 172.21.15.28:0/542838555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78abfff640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78ac155100 0x7f78ac06f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78b0dd0640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78ac15ee70 0x7f78ac158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78ab7fe640 1 -- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78ac15ee70 msgr2=0x7f78ac158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78ab7fe640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78ac15ee70 0x7f78ac158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78ab7fe640 1 -- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78ac155100 msgr2=0x7f78ac06f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78ab7fe640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78ac155100 0x7f78ac06f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78ab7fe640 1 -- 172.21.15.28:0/542838555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac06e5e0 con 0x7f78ac1554e0 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78abfff640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78ac155100 0x7f78ac06f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78ab7fe640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac0b20e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f78a0000c00 tx=0x7f78a0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78a8ff9640 1 -- 172.21.15.28:0/542838555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a003f070 con 0x7f78ac1554e0 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78a8ff9640 1 -- 172.21.15.28:0/542838555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78a00047a0 con 0x7f78ac1554e0 2024-09-10T20:41:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac06e880 con 0x7f78ac1554e0 2024-09-10T20:41:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78a8ff9640 1 -- 172.21.15.28:0/542838555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0035030 con 0x7f78ac1554e0 2024-09-10T20:41:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.159+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac06ee40 con 0x7f78ac1554e0 2024-09-10T20:41:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.160+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac0735d0 con 0x7f78ac1554e0 2024-09-10T20:41:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.161+0000 7f78a8ff9640 1 -- 172.21.15.28:0/542838555 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f78a0004c10 con 0x7f78ac1554e0 2024-09-10T20:41:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.161+0000 7f78a8ff9640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7898078120 0x7f789807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.162+0000 7f78a8ff9640 1 -- 172.21.15.28:0/542838555 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f78a00bb2e0 con 0x7f78ac1554e0 2024-09-10T20:41:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.162+0000 7f78abfff640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7898078120 0x7f789807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.162+0000 7f78abfff640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7898078120 0x7f789807a5e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f789c005fd0 tx=0x7f789c00f360 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.168+0000 7f78a8ff9640 1 -- 172.21.15.28:0/542838555 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f78ac15c290 con 0x7f78ac1554e0 2024-09-10T20:41:03.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.424+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7f78ac0b2620 con 0x7f78ac1554e0 2024-09-10T20:41:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.709+0000 7f78a8ff9640 1 -- 172.21.15.28:0/542838555 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v159) ==== 140+0+0 (secure 0 0 0) 0x7f78a0083670 con 0x7f78ac1554e0 2024-09-10T20:41:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7898078120 msgr2=0x7f789807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7898078120 0x7f789807a5e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f789c005fd0 tx=0x7f789c00f360 comp rx=0 tx=0).stop 2024-09-10T20:41:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 msgr2=0x7f78ac0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac0b20e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f78a0000c00 tx=0x7f78a0003910 comp rx=0 tx=0).stop 2024-09-10T20:41:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 shutdown_connections 2024-09-10T20:41:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7898078120 0x7f789807a5e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78ac15ee70 0x7f78ac158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac1554e0 0x7f78ac0b20e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 --2- 172.21.15.28:0/542838555 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78ac155100 0x7f78ac06f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 >> 172.21.15.28:0/542838555 conn(0x7f78ac06cf00 msgr2=0x7f78ac09d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 shutdown_connections 2024-09-10T20:41:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.712+0000 7f78b2833640 1 -- 172.21.15.28:0/542838555 wait complete. 2024-09-10T20:41:03.728 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-10T20:41:03.729 INFO:tasks.cephfs_test_runner:Starting test: test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate handcrafted .meta file on legacy subvol root doesn't break the system 2024-09-10T20:41:03.729 INFO:tasks.cephfs_test_runner:test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:41:03.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.831+0000 7f9797460640 1 -- 172.21.15.28:0/500182651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f977c003a80 msgr2=0x7f977c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.831+0000 7f9797460640 1 --2- 172.21.15.28:0/500182651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f977c003a80 0x7f977c003ea0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f978c0016c0 tx=0x7f977c005830 comp rx=0 tx=0).stop 2024-09-10T20:41:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.831+0000 7f97959fd640 1 -- 172.21.15.28:0/500182651 reap_dead start 2024-09-10T20:41:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.831+0000 7f9797460640 1 -- 172.21.15.28:0/500182651 shutdown_connections 2024-09-10T20:41:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.831+0000 7f9797460640 1 -- 172.21.15.28:0/500182651 >> 172.21.15.28:0/500182651 conn(0x7f97900d3fb0 msgr2=0x7f97900d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.831+0000 7f9797460640 1 -- 172.21.15.28:0/500182651 shutdown_connections 2024-09-10T20:41:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.831+0000 7f9797460640 1 -- 172.21.15.28:0/500182651 wait complete. 2024-09-10T20:41:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.832+0000 7f9797460640 1 Processor -- start 2024-09-10T20:41:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.832+0000 7f9797460640 1 -- start start 2024-09-10T20:41:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f9797460640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f977c003a80 0x7f9790161340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f9797460640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97901f80b0 0x7f97900b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f9797460640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97900b5d40 0x7f979015c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f9797460640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f979015e3d0 con 0x7f97901f80b0 2024-09-10T20:41:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f9797460640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f979015e250 con 0x7f97900b5d40 2024-09-10T20:41:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f9797460640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9790161b50 con 0x7f977c003a80 2024-09-10T20:41:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f97949fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97901f80b0 0x7f97900b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f97949fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97901f80b0 0x7f97900b0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44376/0 (socket says 172.21.15.28:44376) 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f97949fb640 1 -- 172.21.15.28:0/3503810551 learned_addr learned my addr 172.21.15.28:0/3503810551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.833+0000 7f97951fc640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f977c003a80 0x7f9790161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97959fd640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97900b5d40 0x7f979015c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97949fb640 1 -- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f977c003a80 msgr2=0x7f9790161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97949fb640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f977c003a80 0x7f9790161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97949fb640 1 -- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97900b5d40 msgr2=0x7f979015c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97949fb640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97900b5d40 0x7f979015c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97949fb640 1 -- 172.21.15.28:0/3503810551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97901f4b30 con 0x7f97901f80b0 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97951fc640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f977c003a80 0x7f9790161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f97949fb640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97901f80b0 0x7f97900b0ae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9780011b10 tx=0x7f9780009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f9785ffb640 1 -- 172.21.15.28:0/3503810551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978001e070 con 0x7f97901f80b0 2024-09-10T20:41:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f9785ffb640 1 -- 172.21.15.28:0/3503810551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9780004050 con 0x7f97901f80b0 2024-09-10T20:41:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97901f8620 con 0x7f97901f80b0 2024-09-10T20:41:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f9785ffb640 1 -- 172.21.15.28:0/3503810551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978000fea0 con 0x7f97901f80b0 2024-09-10T20:41:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.834+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97901f8be0 con 0x7f97901f80b0 2024-09-10T20:41:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.835+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97900b6890 con 0x7f97901f80b0 2024-09-10T20:41:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.836+0000 7f9785ffb640 1 -- 172.21.15.28:0/3503810551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f97800035f0 con 0x7f97901f80b0 2024-09-10T20:41:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.836+0000 7f9785ffb640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9774078120 0x7f977407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.836+0000 7f9785ffb640 1 -- 172.21.15.28:0/3503810551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f978009ab50 con 0x7f97901f80b0 2024-09-10T20:41:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.837+0000 7f97951fc640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9774078120 0x7f977407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.837+0000 7f97951fc640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9774078120 0x7f977407a5e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f978c032000 tx=0x7f978c001190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:03.841+0000 7f9785ffb640 1 -- 172.21.15.28:0/3503810551 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f97900b6890 con 0x7f97901f80b0 2024-09-10T20:41:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.031+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97900cf780 con 0x7f97901f80b0 2024-09-10T20:41:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.032+0000 7f9785ffb640 1 -- 172.21.15.28:0/3503810551 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+13032 (secure 0 0 0) 0x7f97800044a0 con 0x7f97901f80b0 2024-09-10T20:41:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:04.037 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:02.705185+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3839556955":"2024-09-11T20:40:59.421826+0000","172.21.15.28:6833/3839556955":"2024-09-11T20:40:59.421826+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9774078120 msgr2=0x7f977407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9774078120 0x7f977407a5e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f978c032000 tx=0x7f978c001190 comp rx=0 tx=0).stop 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97901f80b0 msgr2=0x7f97900b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97901f80b0 0x7f97900b0ae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9780011b10 tx=0x7f9780009e90 comp rx=0 tx=0).stop 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 shutdown_connections 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9774078120 0x7f977407a5e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f97900b5d40 0x7f979015c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97901f80b0 0x7f97900b0ae0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 --2- 172.21.15.28:0/3503810551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f977c003a80 0x7f9790161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 >> 172.21.15.28:0/3503810551 conn(0x7f97900d3fb0 msgr2=0x7f97900b83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 shutdown_connections 2024-09-10T20:41:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.034+0000 7f9797460640 1 -- 172.21.15.28:0/3503810551 wait complete. 2024-09-10T20:41:04.050 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:41:04.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:41:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 -- 172.21.15.28:0/4243442410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b080ced20 msgr2=0x7f2b080cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/4243442410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b080ced20 0x7f2b080cf150 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2b080cd440 tx=0x7f2af802a980 comp rx=0 tx=0).stop 2024-09-10T20:41:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 -- 172.21.15.28:0/4243442410 shutdown_connections 2024-09-10T20:41:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/4243442410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b080ced20 0x7f2b080cf150 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/4243442410 >> v2:172.21.15.159:3300/0 conn(0x7f2b08152e00 0x7f2b080b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 -- 172.21.15.28:0/4243442410 >> 172.21.15.28:0/4243442410 conn(0x7f2b0806d110 msgr2=0x7f2b080b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 -- 172.21.15.28:0/4243442410 shutdown_connections 2024-09-10T20:41:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.153+0000 7f2b0d02a640 1 -- 172.21.15.28:0/4243442410 wait complete. 2024-09-10T20:41:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.154+0000 7f2b0d02a640 1 Processor -- start 2024-09-10T20:41:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.154+0000 7f2b0d02a640 1 -- start start 2024-09-10T20:41:04.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0d02a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b080ced20 0x7f2b0813e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0d02a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b08152e00 0x7f2b08155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0d02a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b08155ca0 0x7f2b08072680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0d02a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0815f9a0 con 0x7f2b08152e00 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0d02a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0815f820 con 0x7f2b08155ca0 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0d02a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0815fb20 con 0x7f2b080ced20 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b08152e00 0x7f2b08155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b08152e00 0x7f2b08155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44394/0 (socket says 172.21.15.28:44394) 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0659c640 1 -- 172.21.15.28:0/1636959298 learned_addr learned my addr 172.21.15.28:0/1636959298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b06d9d640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b080ced20 0x7f2b0813e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.155+0000 7f2b0759e640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b08155ca0 0x7f2b08072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0659c640 1 -- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b080ced20 msgr2=0x7f2b0813e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0659c640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b080ced20 0x7f2b0813e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0659c640 1 -- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b08155ca0 msgr2=0x7f2b08072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0659c640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b08155ca0 0x7f2b08072680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0659c640 1 -- 172.21.15.28:0/1636959298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b080733b0 con 0x7f2b08152e00 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0759e640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b08155ca0 0x7f2b08072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0659c640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b08152e00 0x7f2b08155760 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2af000c440 tx=0x7f2af000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2aff7fe640 1 -- 172.21.15.28:0/1636959298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0017070 con 0x7f2b08152e00 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2aff7fe640 1 -- 172.21.15.28:0/1636959298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2af00035f0 con 0x7f2b08152e00 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b08073690 con 0x7f2b08152e00 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2aff7fe640 1 -- 172.21.15.28:0/1636959298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0003a40 con 0x7f2b08152e00 2024-09-10T20:41:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.156+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b08073c50 con 0x7f2b08152e00 2024-09-10T20:41:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.157+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b081560b0 con 0x7f2b08152e00 2024-09-10T20:41:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.158+0000 7f2aff7fe640 1 -- 172.21.15.28:0/1636959298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f2af0003be0 con 0x7f2b08152e00 2024-09-10T20:41:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.158+0000 7f2aff7fe640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2ae8078120 0x7f2ae807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.159+0000 7f2aff7fe640 1 -- 172.21.15.28:0/1636959298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f2af0099400 con 0x7f2b08152e00 2024-09-10T20:41:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.159+0000 7f2b06d9d640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2ae8078120 0x7f2ae807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.159+0000 7f2b06d9d640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2ae8078120 0x7f2ae807a5e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2b000098e0 tx=0x7f2b00009060 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.163+0000 7f2aff7fe640 1 -- 172.21.15.28:0/1636959298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2b081560b0 con 0x7f2b08152e00 2024-09-10T20:41:04.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.349+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2b081600c0 con 0x7f2b08152e00 2024-09-10T20:41:04.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.349+0000 7f2aff7fe640 1 -- 172.21.15.28:0/1636959298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f2af00010c0 con 0x7f2b08152e00 2024-09-10T20:41:04.354 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:41:04.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2ae8078120 msgr2=0x7f2ae807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2ae8078120 0x7f2ae807a5e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2b000098e0 tx=0x7f2b00009060 comp rx=0 tx=0).stop 2024-09-10T20:41:04.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b08152e00 msgr2=0x7f2b08155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b08152e00 0x7f2b08155760 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2af000c440 tx=0x7f2af000c900 comp rx=0 tx=0).stop 2024-09-10T20:41:04.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 shutdown_connections 2024-09-10T20:41:04.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2ae8078120 0x7f2ae807a5e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b08155ca0 0x7f2b08072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b08152e00 0x7f2b08155760 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 --2- 172.21.15.28:0/1636959298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b080ced20 0x7f2b0813e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 >> 172.21.15.28:0/1636959298 conn(0x7f2b0806d110 msgr2=0x7f2b08151bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:04.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 shutdown_connections 2024-09-10T20:41:04.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.352+0000 7f2b0d02a640 1 -- 172.21.15.28:0/1636959298 wait complete. 2024-09-10T20:41:04.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 -- 172.21.15.28:0/2710286516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280d2b90 msgr2=0x7fa128161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/2710286516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280d2b90 0x7fa128161b50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa110005fd0 tx=0x7fa11002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 -- 172.21.15.28:0/2710286516 shutdown_connections 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/2710286516 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280d2b90 0x7fa128161b50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/2710286516 >> v2:172.21.15.159:3300/0 conn(0x7fa128162090 0x7fa1280b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 -- 172.21.15.28:0/2710286516 >> 172.21.15.28:0/2710286516 conn(0x7fa1280d3fb0 msgr2=0x7fa1280d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 -- 172.21.15.28:0/2710286516 shutdown_connections 2024-09-10T20:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.469+0000 7fa12cb6b640 1 -- 172.21.15.28:0/2710286516 wait complete. 2024-09-10T20:41:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.470+0000 7fa12cb6b640 1 Processor -- start 2024-09-10T20:41:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.470+0000 7fa12cb6b640 1 -- start start 2024-09-10T20:41:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa12cb6b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1280d2b90 0x7fa12809e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa12cb6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12815de70 0x7fa1281f5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa12cb6b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa128162090 0x7fa12809f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa12cb6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa12815e3d0 con 0x7fa12815de70 2024-09-10T20:41:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa12cb6b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa12815e250 con 0x7fa1280d2b90 2024-09-10T20:41:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa12cb6b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa1280b7340 con 0x7fa128162090 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa125d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12815de70 0x7fa1281f5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa125d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12815de70 0x7fa1281f5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44402/0 (socket says 172.21.15.28:44402) 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa125d9b640 1 -- 172.21.15.28:0/3450965120 learned_addr learned my addr 172.21.15.28:0/3450965120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa12659c640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1280d2b90 0x7fa12809e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa126d9d640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa128162090 0x7fa12809f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa125d9b640 1 -- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa128162090 msgr2=0x7fa12809f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa125d9b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa128162090 0x7fa12809f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa125d9b640 1 -- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1280d2b90 msgr2=0x7fa12809e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.471+0000 7fa125d9b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1280d2b90 0x7fa12809e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.472+0000 7fa125d9b640 1 -- 172.21.15.28:0/3450965120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1281f59c0 con 0x7fa12815de70 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.472+0000 7fa125d9b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12815de70 0x7fa1281f5480 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa110000c00 tx=0x7fa110003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.472+0000 7fa11effd640 1 -- 172.21.15.28:0/3450965120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11003f070 con 0x7fa12815de70 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.472+0000 7fa11effd640 1 -- 172.21.15.28:0/3450965120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa110005600 con 0x7fa12815de70 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.472+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1280a1810 con 0x7fa12815de70 2024-09-10T20:41:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.472+0000 7fa11effd640 1 -- 172.21.15.28:0/3450965120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110035030 con 0x7fa12815de70 2024-09-10T20:41:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.472+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12809dbf0 con 0x7fa12815de70 2024-09-10T20:41:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.473+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1280b7860 con 0x7fa12815de70 2024-09-10T20:41:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.474+0000 7fa11effd640 1 -- 172.21.15.28:0/3450965120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fa110004a00 con 0x7fa12815de70 2024-09-10T20:41:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.474+0000 7fa11effd640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa108078120 0x7fa10807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.474+0000 7fa11effd640 1 -- 172.21.15.28:0/3450965120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7fa1100bab90 con 0x7fa12815de70 2024-09-10T20:41:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.474+0000 7fa12659c640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa108078120 0x7fa10807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.475+0000 7fa12659c640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa108078120 0x7fa10807a5e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa1281f25f0 tx=0x7fa120009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.478+0000 7fa11effd640 1 -- 172.21.15.28:0/3450965120 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa110083070 con 0x7fa12815de70 2024-09-10T20:41:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:04.735+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fa1280b9f30 con 0x7fa12815de70 2024-09-10T20:41:05.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.715+0000 7fa11effd640 1 -- 172.21.15.28:0/3450965120 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v161) ==== 164+0+0 (secure 0 0 0) 0x7fa110087aa0 con 0x7fa12815de70 2024-09-10T20:41:05.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa108078120 msgr2=0x7fa10807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa108078120 0x7fa10807a5e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa1281f25f0 tx=0x7fa120009040 comp rx=0 tx=0).stop 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12815de70 msgr2=0x7fa1281f5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12815de70 0x7fa1281f5480 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa110000c00 tx=0x7fa110003820 comp rx=0 tx=0).stop 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 shutdown_connections 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa108078120 0x7fa10807a5e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa128162090 0x7fa12809f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12815de70 0x7fa1281f5480 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 --2- 172.21.15.28:0/3450965120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1280d2b90 0x7fa12809e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:05.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 >> 172.21.15.28:0/3450965120 conn(0x7fa1280d3fb0 msgr2=0x7fa12806ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:05.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 shutdown_connections 2024-09-10T20:41:05.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.718+0000 7fa12cb6b640 1 -- 172.21.15.28:0/3450965120 wait complete. 2024-09-10T20:41:05.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:41:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.843+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/725540180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780b21b0 msgr2=0x7f4a78162110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.844+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/725540180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780b21b0 0x7f4a78162110 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009830 tx=0x7f4a6802a510 comp rx=0 tx=0).stop 2024-09-10T20:41:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.844+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/725540180 shutdown_connections 2024-09-10T20:41:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.844+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/725540180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780b21b0 0x7f4a78162110 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.844+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/725540180 >> 172.21.15.28:0/725540180 conn(0x7f4a780d1e40 msgr2=0x7f4a780d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.844+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/725540180 shutdown_connections 2024-09-10T20:41:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.844+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/725540180 wait complete. 2024-09-10T20:41:05.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.845+0000 7f4a7fe6a640 1 Processor -- start 2024-09-10T20:41:05.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.845+0000 7f4a7fe6a640 1 -- start start 2024-09-10T20:41:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7fe6a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a780b21b0 0x7f4a7815c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7fe6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780d03f0 0x7f4a78150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7fe6a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a780d07d0 0x7f4a781f8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7fe6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a78169550 con 0x7f4a780d03f0 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7fe6a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4a781693d0 con 0x7f4a780d07d0 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7fe6a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4a781696d0 con 0x7f4a780b21b0 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780d03f0 0x7f4a78150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780d03f0 0x7f4a78150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44410/0 (socket says 172.21.15.28:44410) 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 -- 172.21.15.28:0/4071345302 learned_addr learned my addr 172.21.15.28:0/4071345302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7e407640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a780d07d0 0x7f4a781f8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7dc06640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a780b21b0 0x7f4a7815c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 -- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a780b21b0 msgr2=0x7f4a7815c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a780b21b0 0x7f4a7815c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 -- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a780d07d0 msgr2=0x7f4a781f8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a780d07d0 0x7f4a781f8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 -- 172.21.15.28:0/4071345302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a781555e0 con 0x7f4a780d03f0 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a7d405640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780d03f0 0x7f4a78150800 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009950 tx=0x7f4a68032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.846+0000 7f4a6e7fc640 1 -- 172.21.15.28:0/4071345302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68040070 con 0x7f4a780d03f0 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.847+0000 7f4a6e7fc640 1 -- 172.21.15.28:0/4071345302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a68005310 con 0x7f4a780d03f0 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.847+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7815cb80 con 0x7f4a780d03f0 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.847+0000 7f4a6e7fc640 1 -- 172.21.15.28:0/4071345302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a680047a0 con 0x7f4a780d03f0 2024-09-10T20:41:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.847+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a7815d140 con 0x7f4a780d03f0 2024-09-10T20:41:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.847+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a780d0d90 con 0x7f4a780d03f0 2024-09-10T20:41:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.848+0000 7f4a6e7fc640 1 -- 172.21.15.28:0/4071345302 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f4a6804c030 con 0x7f4a780d03f0 2024-09-10T20:41:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.849+0000 7f4a6e7fc640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4a5c078120 0x7f4a5c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.849+0000 7f4a6e7fc640 1 -- 172.21.15.28:0/4071345302 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f4a680bb010 con 0x7f4a780d03f0 2024-09-10T20:41:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.849+0000 7f4a7dc06640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4a5c078120 0x7f4a5c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.850+0000 7f4a7dc06640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4a5c078120 0x7f4a5c07a5e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4a74004ea0 tx=0x7f4a74009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:05.853+0000 7f4a6e7fc640 1 -- 172.21.15.28:0/4071345302 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4a68083b10 con 0x7f4a780d03f0 2024-09-10T20:41:06.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.040+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f4a7814ed20 con 0x7f4a780d03f0 2024-09-10T20:41:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.040+0000 7f4a6e7fc640 1 -- 172.21.15.28:0/4071345302 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v31) ==== 59+0+1998 (secure 0 0 0) 0x7f4a68083410 con 0x7f4a780d03f0 2024-09-10T20:41:06.045 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:41:06.045 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:41:06.045 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:41:06.045 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:41:06.045 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:41:06.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:41:06.047 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:41:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4a5c078120 msgr2=0x7f4a5c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4a5c078120 0x7f4a5c07a5e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4a74004ea0 tx=0x7f4a74009040 comp rx=0 tx=0).stop 2024-09-10T20:41:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780d03f0 msgr2=0x7f4a78150800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780d03f0 0x7f4a78150800 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009950 tx=0x7f4a68032c10 comp rx=0 tx=0).stop 2024-09-10T20:41:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 shutdown_connections 2024-09-10T20:41:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4a5c078120 0x7f4a5c07a5e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a780d07d0 0x7f4a781f8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a780d03f0 0x7f4a78150800 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 --2- 172.21.15.28:0/4071345302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a780b21b0 0x7f4a7815c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 >> 172.21.15.28:0/4071345302 conn(0x7f4a780d1e40 msgr2=0x7f4a78161f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 shutdown_connections 2024-09-10T20:41:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.042+0000 7f4a7fe6a640 1 -- 172.21.15.28:0/4071345302 wait complete. 2024-09-10T20:41:06.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 -- 172.21.15.28:0/718074890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc0b3580 msgr2=0x7f4ccc06e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/718074890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc0b3580 0x7f4ccc06e400 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0cd440 tx=0x7f4cb402a4d0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 -- 172.21.15.28:0/718074890 shutdown_connections 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/718074890 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc06e940 0x7f4ccc0ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/718074890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc0b3580 0x7f4ccc06e400 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/718074890 >> v2:172.21.15.159:3300/0 conn(0x7f4ccc152e00 0x7f4ccc0b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 -- 172.21.15.28:0/718074890 >> 172.21.15.28:0/718074890 conn(0x7f4ccc06d110 msgr2=0x7f4ccc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 -- 172.21.15.28:0/718074890 shutdown_connections 2024-09-10T20:41:06.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.162+0000 7f4cd1d94640 1 -- 172.21.15.28:0/718074890 wait complete. 2024-09-10T20:41:06.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.163+0000 7f4cd1d94640 1 Processor -- start 2024-09-10T20:41:06.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.163+0000 7f4cd1d94640 1 -- start start 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4cd1d94640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ccc06e940 0x7f4ccc1f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4cd1d94640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc152e00 0x7f4ccc158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4cd1d94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc142f30 0x7f4ccc159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4cd1d94640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc0cf5f0 con 0x7f4ccc142f30 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4cd1d94640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc0cf470 con 0x7f4ccc06e940 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4cd1d94640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4ccc15f820 con 0x7f4ccc152e00 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc142f30 0x7f4ccc159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc142f30 0x7f4ccc159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44434/0 (socket says 172.21.15.28:44434) 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 -- 172.21.15.28:0/1853294974 learned_addr learned my addr 172.21.15.28:0/1853294974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ccc06e940 0x7f4ccc1f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4cc3fff640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc152e00 0x7f4ccc158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 -- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc152e00 msgr2=0x7f4ccc158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc152e00 0x7f4ccc158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 -- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ccc06e940 msgr2=0x7f4ccc1f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ccc06e940 0x7f4ccc1f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 -- 172.21.15.28:0/1853294974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc1f60e0 con 0x7f4ccc142f30 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.164+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc142f30 0x7f4ccc159450 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400eaa0 tx=0x7f4cc400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.165+0000 7f4cc8ff9640 1 -- 172.21.15.28:0/1853294974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc400cdb0 con 0x7f4ccc142f30 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.165+0000 7f4cc8ff9640 1 -- 172.21.15.28:0/1853294974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cc4004c10 con 0x7f4ccc142f30 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.165+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ccc1f6380 con 0x7f4ccc142f30 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.165+0000 7f4cc8ff9640 1 -- 172.21.15.28:0/1853294974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4005020 con 0x7f4ccc142f30 2024-09-10T20:41:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.165+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ccc1f6940 con 0x7f4ccc142f30 2024-09-10T20:41:06.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.166+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ccc150720 con 0x7f4ccc142f30 2024-09-10T20:41:06.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.166+0000 7f4cc8ff9640 1 -- 172.21.15.28:0/1853294974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f4cc400f040 con 0x7f4ccc142f30 2024-09-10T20:41:06.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.167+0000 7f4cc8ff9640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4cac078120 0x7f4cac07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.167+0000 7f4cc8ff9640 1 -- 172.21.15.28:0/1853294974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f4cc409a370 con 0x7f4ccc142f30 2024-09-10T20:41:06.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.167+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4cac078120 0x7f4cac07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.168+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4cac078120 0x7f4cac07a5e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00bad0 tx=0x7f4cbc00b2e0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:06.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.171+0000 7f4cc8ff9640 1 -- 172.21.15.28:0/1853294974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4cc4016030 con 0x7f4ccc142f30 2024-09-10T20:41:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.354+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f4ccc06e400 con 0x7f4ccc142f30 2024-09-10T20:41:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.357+0000 7f4cc8ff9640 1 -- 172.21.15.28:0/1853294974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f4cc4062840 con 0x7f4ccc142f30 2024-09-10T20:41:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4cac078120 msgr2=0x7f4cac07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4cac078120 0x7f4cac07a5e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00bad0 tx=0x7f4cbc00b2e0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc142f30 msgr2=0x7f4ccc159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc142f30 0x7f4ccc159450 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400eaa0 tx=0x7f4cc400ef60 comp rx=0 tx=0).stop 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 shutdown_connections 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4cac078120 0x7f4cac07a5e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc142f30 0x7f4ccc159450 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc152e00 0x7f4ccc158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 --2- 172.21.15.28:0/1853294974 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ccc06e940 0x7f4ccc1f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 >> 172.21.15.28:0/1853294974 conn(0x7f4ccc06d110 msgr2=0x7f4ccc15c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.359+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 shutdown_connections 2024-09-10T20:41:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.360+0000 7f4cd1d94640 1 -- 172.21.15.28:0/1853294974 wait complete. 2024-09-10T20:41:06.375 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:41:06.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:41:06.418 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:41:06.465 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:41:06.465 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:06.465 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:06.481 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:41:06.481 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:06.481 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:06.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:41:06.500 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:06.500 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:06.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:41:06.538 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:06.538 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:06.558 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:41:06.558 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:41:06.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.681+0000 7f117ac7f640 1 -- 172.21.15.28:0/2477457182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f115c004800 msgr2=0x7f115c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.681+0000 7f117ac7f640 1 --2- 172.21.15.28:0/2477457182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f115c004800 0x7f115c004c80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f116800c2c0 tx=0x7f115c005830 comp rx=0 tx=0).stop 2024-09-10T20:41:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.681+0000 7f117921c640 1 -- 172.21.15.28:0/2477457182 reap_dead start 2024-09-10T20:41:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.681+0000 7f117ac7f640 1 -- 172.21.15.28:0/2477457182 shutdown_connections 2024-09-10T20:41:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.681+0000 7f117ac7f640 1 -- 172.21.15.28:0/2477457182 >> 172.21.15.28:0/2477457182 conn(0x7f11740d3fb0 msgr2=0x7f11740d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.681+0000 7f117ac7f640 1 -- 172.21.15.28:0/2477457182 shutdown_connections 2024-09-10T20:41:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.681+0000 7f117ac7f640 1 -- 172.21.15.28:0/2477457182 wait complete. 2024-09-10T20:41:06.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.682+0000 7f117ac7f640 1 Processor -- start 2024-09-10T20:41:06.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.682+0000 7f117ac7f640 1 -- start start 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f117ac7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f115c004800 0x7f11740cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f117ac7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1174160850 0x7f11741f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f117ac7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1174160cd0 0x7f117409f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f117ac7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11740d2d10 con 0x7f1174160850 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f117ac7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f11740d2b90 con 0x7f1174160cd0 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f117ac7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f11740d2e90 con 0x7f115c004800 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1174160850 0x7f11741f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1174160850 0x7f11741f4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44462/0 (socket says 172.21.15.28:44462) 2024-09-10T20:41:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 -- 172.21.15.28:0/2495902591 learned_addr learned my addr 172.21.15.28:0/2495902591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f117921c640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1174160cd0 0x7f117409f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1178a1b640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f115c004800 0x7f11740cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 -- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f115c004800 msgr2=0x7f11740cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f115c004800 0x7f11740cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 -- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1174160cd0 msgr2=0x7f117409f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1174160cd0 0x7f117409f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1173fff640 1 -- 172.21.15.28:0/2495902591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11741f54e0 con 0x7f1174160850 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.683+0000 7f1178a1b640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f115c004800 0x7f11740cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.684+0000 7f1173fff640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1174160850 0x7f11741f4a60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f116800c260 tx=0x7f1168006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.684+0000 7f11717fa640 1 -- 172.21.15.28:0/2495902591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168031040 con 0x7f1174160850 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.684+0000 7f11717fa640 1 -- 172.21.15.28:0/2495902591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11680035f0 con 0x7f1174160850 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.684+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11741f57c0 con 0x7f1174160850 2024-09-10T20:41:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.684+0000 7f11717fa640 1 -- 172.21.15.28:0/2495902591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168003a00 con 0x7f1174160850 2024-09-10T20:41:06.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.684+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11740cd7a0 con 0x7f1174160850 2024-09-10T20:41:06.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.685+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11740cfbc0 con 0x7f1174160850 2024-09-10T20:41:06.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.686+0000 7f11717fa640 1 -- 172.21.15.28:0/2495902591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f1168003ba0 con 0x7f1174160850 2024-09-10T20:41:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.686+0000 7f11717fa640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1160078120 0x7f116007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.686+0000 7f11717fa640 1 -- 172.21.15.28:0/2495902591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f11680ba020 con 0x7f1174160850 2024-09-10T20:41:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.686+0000 7f1178a1b640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1160078120 0x7f116007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.687+0000 7f1178a1b640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1160078120 0x7f116007a5e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f116c005fd0 tx=0x7f116c009500 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.690+0000 7f11717fa640 1 -- 172.21.15.28:0/2495902591 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1168082a10 con 0x7f1174160850 2024-09-10T20:41:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.912+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11740b9f30 con 0x7f1174160850 2024-09-10T20:41:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.912+0000 7f11717fa640 1 -- 172.21.15.28:0/2495902591 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+3914 (secure 0 0 0) 0x7f1168082310 con 0x7f1174160850 2024-09-10T20:41:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:06.917 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-10T20:41:00:423696+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1605164441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1605164441},{"type":"v1","addr":"172.21.15.159:6837","nonce":1605164441}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/1120592134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":1120592134},{"type":"v1","addr":"172.21.15.159:6836","nonce":1120592134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3062696739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3062696739},{"type":"v1","addr":"172.21.15.28:6834","nonce":3062696739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5673,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/333350527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":333350527},{"type":"v1","addr":"172.21.15.28:6835","nonce":333350527}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[]} 2024-09-10T20:41:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-10T20:41:06.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1160078120 msgr2=0x7f116007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1160078120 0x7f116007a5e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f116c005fd0 tx=0x7f116c009500 comp rx=0 tx=0).stop 2024-09-10T20:41:06.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1174160850 msgr2=0x7f11741f4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:06.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1174160850 0x7f11741f4a60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f116800c260 tx=0x7f1168006d70 comp rx=0 tx=0).stop 2024-09-10T20:41:06.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 shutdown_connections 2024-09-10T20:41:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1160078120 0x7f116007a5e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1174160cd0 0x7f117409f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1174160850 0x7f11741f4a60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 --2- 172.21.15.28:0/2495902591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f115c004800 0x7f11740cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 >> 172.21.15.28:0/2495902591 conn(0x7f11740d3fb0 msgr2=0x7f11740b8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 shutdown_connections 2024-09-10T20:41:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:06.914+0000 7f117ac7f640 1 -- 172.21.15.28:0/2495902591 wait complete. 2024-09-10T20:41:06.931 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:41:06.931 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:41:06.931 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:41:06.931 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:06.931 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:41:06.931 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:41:06.932 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:41:06.932 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:06.932 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:41:06.932 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:41:06.932 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:41:06.932 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:06.932 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:41:06.932 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:41:06.933 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:41:06.933 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:06.941 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:41:06.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:41:06.942 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:41:06.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:41:06.944 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:41:06.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:41:06.945 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:41:06.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:41:06.984 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:41:06.984 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:06.985 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:41:06.985 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:06.986 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:41:06.987 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:06.987 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:41:06.987 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:06.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:41:07.061 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:41:07.062 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:41:07.063 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-10T20:41:07.058+0000 7f72bca0d640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6833/0: (98) Address already in use 2024-09-10T20:41:07.070 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:41:07.070 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 -- 172.21.15.28:0/826228586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc15de70 msgr2=0x7fb3bc0d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 --2- 172.21.15.28:0/826228586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc15de70 0x7fb3bc0d2b90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8005fd0 tx=0x7fb3b802a530 comp rx=0 tx=0).stop 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 -- 172.21.15.28:0/826228586 shutdown_connections 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 --2- 172.21.15.28:0/826228586 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3bc162260 0x7fb3bc0cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 --2- 172.21.15.28:0/826228586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc15de70 0x7fb3bc0d2b90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 -- 172.21.15.28:0/826228586 >> 172.21.15.28:0/826228586 conn(0x7fb3bc0d3fb0 msgr2=0x7fb3bc0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 -- 172.21.15.28:0/826228586 shutdown_connections 2024-09-10T20:41:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.093+0000 7fb3c4085640 1 -- 172.21.15.28:0/826228586 wait complete. 2024-09-10T20:41:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.094+0000 7fb3c4085640 1 Processor -- start 2024-09-10T20:41:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.094+0000 7fb3c4085640 1 -- start start 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c4085640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc0cf780 0x7fb3bc14d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c4085640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3bc15de70 0x7fb3bc1f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c4085640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3bc162260 0x7fb3bc1f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c4085640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc0b7190 con 0x7fb3bc0cf780 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c4085640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc0b7010 con 0x7fb3bc15de70 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c4085640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb3bc0b7310 con 0x7fb3bc162260 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc0cf780 0x7fb3bc14d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc0cf780 0x7fb3bc14d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44504/0 (socket says 172.21.15.28:44504) 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 -- 172.21.15.28:0/1426918611 learned_addr learned my addr 172.21.15.28:0/1426918611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1620640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3bc15de70 0x7fb3bc1f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c2622640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3bc162260 0x7fb3bc1f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 -- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3bc162260 msgr2=0x7fb3bc1f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3bc162260 0x7fb3bc1f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 -- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3bc15de70 msgr2=0x7fb3bc1f8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3bc15de70 0x7fb3bc1f8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 -- 172.21.15.28:0/1426918611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3bc1f5680 con 0x7fb3bc0cf780 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c1e21640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc0cf780 0x7fb3bc14d4e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8000c00 tx=0x7fb3b80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3b27fc640 1 -- 172.21.15.28:0/1426918611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b803f070 con 0x7fb3bc0cf780 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3b27fc640 1 -- 172.21.15.28:0/1426918611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3b8004890 con 0x7fb3bc0cf780 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3bc1f5960 con 0x7fb3bc0cf780 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.095+0000 7fb3b27fc640 1 -- 172.21.15.28:0/1426918611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8035030 con 0x7fb3bc0cf780 2024-09-10T20:41:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.096+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3bc1f83a0 con 0x7fb3bc0cf780 2024-09-10T20:41:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.096+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3bc0b78b0 con 0x7fb3bc0cf780 2024-09-10T20:41:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.097+0000 7fb3b27fc640 1 -- 172.21.15.28:0/1426918611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fb3b8004ce0 con 0x7fb3bc0cf780 2024-09-10T20:41:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.098+0000 7fb3b27fc640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb3a0078120 0x7fb3a007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.098+0000 7fb3b27fc640 1 -- 172.21.15.28:0/1426918611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7fb3b80bb2e0 con 0x7fb3bc0cf780 2024-09-10T20:41:07.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.098+0000 7fb3c1620640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb3a0078120 0x7fb3a007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.099+0000 7fb3c1620640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb3a0078120 0x7fb3a007a5e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0cdb20 tx=0x7fb3ac007e30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:07.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.102+0000 7fb3b27fc640 1 -- 172.21.15.28:0/1426918611 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb3b8083890 con 0x7fb3bc0cf780 2024-09-10T20:41:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.298+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb3bc0b4f70 con 0x7fb3bc0cf780 2024-09-10T20:41:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.737+0000 7fb3b27fc640 1 -- 172.21.15.28:0/1426918611 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v50) ==== 97+0+0 (secure 0 0 0) 0x7fb3b80882c0 con 0x7fb3bc0cf780 2024-09-10T20:41:07.743 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:41:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb3a0078120 msgr2=0x7fb3a007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb3a0078120 0x7fb3a007a5e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0cdb20 tx=0x7fb3ac007e30 comp rx=0 tx=0).stop 2024-09-10T20:41:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc0cf780 msgr2=0x7fb3bc14d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc0cf780 0x7fb3bc14d4e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8000c00 tx=0x7fb3b80039e0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 shutdown_connections 2024-09-10T20:41:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb3a0078120 0x7fb3a007a5e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3bc162260 0x7fb3bc1f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb3bc15de70 0x7fb3bc1f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 --2- 172.21.15.28:0/1426918611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc0cf780 0x7fb3bc14d4e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 >> 172.21.15.28:0/1426918611 conn(0x7fb3bc0d3fb0 msgr2=0x7fb3bc0b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 shutdown_connections 2024-09-10T20:41:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.740+0000 7fb3c4085640 1 -- 172.21.15.28:0/1426918611 wait complete. 2024-09-10T20:41:07.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:41:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 -- 172.21.15.28:0/3363584672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e015de70 msgr2=0x7f80e00d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3363584672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e015de70 0x7f80e00d2b90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009830 tx=0x7f80d802a510 comp rx=0 tx=0).stop 2024-09-10T20:41:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 -- 172.21.15.28:0/3363584672 shutdown_connections 2024-09-10T20:41:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3363584672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80e0162260 0x7f80e00cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3363584672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e015de70 0x7f80e00d2b90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 -- 172.21.15.28:0/3363584672 >> 172.21.15.28:0/3363584672 conn(0x7f80e00d3fb0 msgr2=0x7f80e00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 -- 172.21.15.28:0/3363584672 shutdown_connections 2024-09-10T20:41:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.859+0000 7f80e81f3640 1 -- 172.21.15.28:0/3363584672 wait complete. 2024-09-10T20:41:07.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.860+0000 7f80e81f3640 1 Processor -- start 2024-09-10T20:41:07.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e81f3640 1 -- start start 2024-09-10T20:41:07.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e81f3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80e00cf780 0x7f80e00cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e81f3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80e015de70 0x7f80e015a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e81f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e0162260 0x7f80e00cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e81f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80e00b7190 con 0x7f80e0162260 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e81f3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f80e00b7010 con 0x7f80e00cf780 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e81f3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f80e00b7310 con 0x7f80e015de70 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e6790640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e0162260 0x7f80e00cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e6790640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e0162260 0x7f80e00cb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44524/0 (socket says 172.21.15.28:44524) 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.861+0000 7f80e6790640 1 -- 172.21.15.28:0/3709026401 learned_addr learned my addr 172.21.15.28:0/3709026401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e578e640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80e015de70 0x7f80e015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e5f8f640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80e00cf780 0x7f80e00cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e6790640 1 -- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80e015de70 msgr2=0x7f80e015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e6790640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80e015de70 0x7f80e015a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e6790640 1 -- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80e00cf780 msgr2=0x7f80e00cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:07.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e6790640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80e00cf780 0x7f80e00cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e6790640 1 -- 172.21.15.28:0/3709026401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e00a0530 con 0x7f80e0162260 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e5f8f640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80e00cf780 0x7f80e00cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e6790640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e0162260 0x7f80e00cb080 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f80d000b970 tx=0x7f80d000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80de7fc640 1 -- 172.21.15.28:0/3709026401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0019070 con 0x7f80e0162260 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80de7fc640 1 -- 172.21.15.28:0/3709026401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80d000cc50 con 0x7f80e0162260 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e00cb9d0 con 0x7f80e0162260 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80de7fc640 1 -- 172.21.15.28:0/3709026401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d00146a0 con 0x7f80e0162260 2024-09-10T20:41:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.862+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e015a9d0 con 0x7f80e0162260 2024-09-10T20:41:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.863+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e00b7870 con 0x7f80e0162260 2024-09-10T20:41:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.864+0000 7f80de7fc640 1 -- 172.21.15.28:0/3709026401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f80d001d3b0 con 0x7f80e0162260 2024-09-10T20:41:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.864+0000 7f80de7fc640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f80cc078120 0x7f80cc07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.864+0000 7f80de7fc640 1 -- 172.21.15.28:0/3709026401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f80d0099de0 con 0x7f80e0162260 2024-09-10T20:41:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.865+0000 7f80e5f8f640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f80cc078120 0x7f80cc07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.865+0000 7f80e5f8f640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f80cc078120 0x7f80cc07a5e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f80d8034000 tx=0x7f80d8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:07.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:07.869+0000 7f80de7fc640 1 -- 172.21.15.28:0/3709026401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f80d0062930 con 0x7f80e0162260 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.122+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f80e00b9f30 con 0x7f80e0162260 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.123+0000 7f80de7fc640 1 -- 172.21.15.28:0/3709026401 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v26) ==== 80+0+5304 (secure 0 0 0) 0x7f80d0062230 con 0x7f80e0162260 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:41:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:41:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:41:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:41:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:41:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:41:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:41:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:41:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:41:08.142 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:41:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f80cc078120 msgr2=0x7f80cc07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f80cc078120 0x7f80cc07a5e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f80d8034000 tx=0x7f80d8004730 comp rx=0 tx=0).stop 2024-09-10T20:41:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e0162260 msgr2=0x7f80e00cb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e0162260 0x7f80e00cb080 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f80d000b970 tx=0x7f80d000be30 comp rx=0 tx=0).stop 2024-09-10T20:41:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 shutdown_connections 2024-09-10T20:41:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f80cc078120 0x7f80cc07a5e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e0162260 0x7f80e00cb080 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80e015de70 0x7f80e015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 --2- 172.21.15.28:0/3709026401 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80e00cf780 0x7f80e00cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 >> 172.21.15.28:0/3709026401 conn(0x7f80e00d3fb0 msgr2=0x7f80e0075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 shutdown_connections 2024-09-10T20:41:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.125+0000 7f80e81f3640 1 -- 172.21.15.28:0/3709026401 wait complete. 2024-09-10T20:41:08.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:41:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.245+0000 7f8e97685640 1 -- 172.21.15.28:0/2686112983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e9015de70 msgr2=0x7f8e900d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.245+0000 7f8e97685640 1 --2- 172.21.15.28:0/2686112983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e9015de70 0x7f8e900d2b90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8e900d4d80 tx=0x7f8e8c02a5f0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.245+0000 7f8e97685640 1 -- 172.21.15.28:0/2686112983 shutdown_connections 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.245+0000 7f8e97685640 1 --2- 172.21.15.28:0/2686112983 >> v2:172.21.15.159:3300/0 conn(0x7f8e900cf780 0x7f8e900cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.245+0000 7f8e97685640 1 --2- 172.21.15.28:0/2686112983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e9015de70 0x7f8e900d2b90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.245+0000 7f8e97685640 1 -- 172.21.15.28:0/2686112983 >> 172.21.15.28:0/2686112983 conn(0x7f8e900d3fb0 msgr2=0x7f8e900d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.246+0000 7f8e97685640 1 -- 172.21.15.28:0/2686112983 shutdown_connections 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.246+0000 7f8e97685640 1 -- 172.21.15.28:0/2686112983 wait complete. 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.247+0000 7f8e97685640 1 Processor -- start 2024-09-10T20:41:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.247+0000 7f8e97685640 1 -- start start 2024-09-10T20:41:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.247+0000 7f8e97685640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e900cf780 0x7f8e9009f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.247+0000 7f8e97685640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e9015de70 0x7f8e900a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.247+0000 7f8e97685640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e90161b50 0x7f8e9015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e97685640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e900b7690 con 0x7f8e90161b50 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e97685640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8e900b7510 con 0x7f8e900cf780 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e97685640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8e900b7810 con 0x7f8e9015de70 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e90161b50 0x7f8e9015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e90161b50 0x7f8e9015b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44552/0 (socket says 172.21.15.28:44552) 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 -- 172.21.15.28:0/3517356977 learned_addr learned my addr 172.21.15.28:0/3517356977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95421640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e900cf780 0x7f8e9009f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e94c20640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e9015de70 0x7f8e900a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 -- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e9015de70 msgr2=0x7f8e900a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e9015de70 0x7f8e900a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 -- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e900cf780 msgr2=0x7f8e9009f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e900cf780 0x7f8e9009f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 -- 172.21.15.28:0/3517356977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e90160510 con 0x7f8e90161b50 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e94c20640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e9015de70 0x7f8e900a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e95c22640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e90161b50 0x7f8e9015b470 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c700 tx=0x7f8e8800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e85ffb640 1 -- 172.21.15.28:0/3517356977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88003b70 con 0x7f8e90161b50 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e85ffb640 1 -- 172.21.15.28:0/3517356977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e88003d10 con 0x7f8e90161b50 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e901607f0 con 0x7f8e90161b50 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e85ffb640 1 -- 172.21.15.28:0/3517356977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8800aab0 con 0x7f8e90161b50 2024-09-10T20:41:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.248+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e90160db0 con 0x7f8e90161b50 2024-09-10T20:41:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.249+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e900cfbf0 con 0x7f8e90161b50 2024-09-10T20:41:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.250+0000 7f8e85ffb640 1 -- 172.21.15.28:0/3517356977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f8e8800ac50 con 0x7f8e90161b50 2024-09-10T20:41:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.251+0000 7f8e85ffb640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8e74078120 0x7f8e7407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.251+0000 7f8e85ffb640 1 -- 172.21.15.28:0/3517356977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f8e88099bc0 con 0x7f8e90161b50 2024-09-10T20:41:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.251+0000 7f8e95421640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8e74078120 0x7f8e7407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.251+0000 7f8e95421640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8e74078120 0x7f8e7407a5e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c033000 tx=0x7f8e8c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:08.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.255+0000 7f8e85ffb640 1 -- 172.21.15.28:0/3517356977 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8e88062600 con 0x7f8e90161b50 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.443+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f8e900b9f30 con 0x7f8e90161b50 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.444+0000 7f8e85ffb640 1 -- 172.21.15.28:0/3517356977 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) ==== 84+0+65 (secure 0 0 0) 0x7f8e88061f00 con 0x7f8e90161b50 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:41:08.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:08.450 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:41:08.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8e74078120 msgr2=0x7f8e7407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8e74078120 0x7f8e7407a5e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c033000 tx=0x7f8e8c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e90161b50 msgr2=0x7f8e9015b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e90161b50 0x7f8e9015b470 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c700 tx=0x7f8e8800cbc0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 shutdown_connections 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8e74078120 0x7f8e7407a5e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e90161b50 0x7f8e9015b470 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e9015de70 0x7f8e900a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 --2- 172.21.15.28:0/3517356977 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e900cf780 0x7f8e9009f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.446+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 >> 172.21.15.28:0/3517356977 conn(0x7f8e900d3fb0 msgr2=0x7f8e900b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.447+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 shutdown_connections 2024-09-10T20:41:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.447+0000 7f8e97685640 1 -- 172.21.15.28:0/3517356977 wait complete. 2024-09-10T20:41:08.462 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:41:08.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:41:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 -- 172.21.15.28:0/1114295023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84161850 msgr2=0x7f9e840d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 --2- 172.21.15.28:0/1114295023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84161850 0x7f9e840d1c60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9e840b6ad0 tx=0x7f9e7c02a940 comp rx=0 tx=0).stop 2024-09-10T20:41:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 -- 172.21.15.28:0/1114295023 shutdown_connections 2024-09-10T20:41:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 --2- 172.21.15.28:0/1114295023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84161850 0x7f9e840d1c60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 --2- 172.21.15.28:0/1114295023 >> v2:172.21.15.159:3300/0 conn(0x7f9e84162000 0x7f9e840d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 -- 172.21.15.28:0/1114295023 >> 172.21.15.28:0/1114295023 conn(0x7f9e840d3f90 msgr2=0x7f9e840d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 -- 172.21.15.28:0/1114295023 shutdown_connections 2024-09-10T20:41:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.566+0000 7f9e8a952640 1 -- 172.21.15.28:0/1114295023 wait complete. 2024-09-10T20:41:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.568+0000 7f9e8a952640 1 Processor -- start 2024-09-10T20:41:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.568+0000 7f9e8a952640 1 -- start start 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.568+0000 7f9e8a952640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e840b4c20 0x7f9e840b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.568+0000 7f9e8a952640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e84161850 0x7f9e84152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e8a952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84162000 0x7f9e840cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e8a952640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e840d21a0 con 0x7f9e84162000 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e8a952640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84162700 con 0x7f9e840b4c20 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e8a952640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9e840d2320 con 0x7f9e84161850 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84162000 0x7f9e840cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84162000 0x7f9e840cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44570/0 (socket says 172.21.15.28:44570) 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 -- 172.21.15.28:0/2830778030 learned_addr learned my addr 172.21.15.28:0/2830778030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e83fff640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e840b4c20 0x7f9e840b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e837fe640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e84161850 0x7f9e84152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 -- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e84161850 msgr2=0x7f9e84152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e84161850 0x7f9e84152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 -- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e840b4c20 msgr2=0x7f9e840b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e840b4c20 0x7f9e840b96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 -- 172.21.15.28:0/2830778030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e840cb5a0 con 0x7f9e84162000 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e83fff640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e840b4c20 0x7f9e840b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e837fe640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e84161850 0x7f9e84152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e88eef640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84162000 0x7f9e840cb060 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009ca0 tx=0x7f9e7c031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e80ff9640 1 -- 172.21.15.28:0/2830778030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c030430 con 0x7f9e84162000 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e80ff9640 1 -- 172.21.15.28:0/2830778030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e7c004240 con 0x7f9e84162000 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.569+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e84152bc0 con 0x7f9e84162000 2024-09-10T20:41:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.570+0000 7f9e80ff9640 1 -- 172.21.15.28:0/2830778030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c004690 con 0x7f9e84162000 2024-09-10T20:41:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.570+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e84153460 con 0x7f9e84162000 2024-09-10T20:41:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.570+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e840d1c60 con 0x7f9e84162000 2024-09-10T20:41:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.571+0000 7f9e80ff9640 1 -- 172.21.15.28:0/2830778030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f9e7c02f070 con 0x7f9e84162000 2024-09-10T20:41:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.572+0000 7f9e80ff9640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e70078120 0x7f9e7007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.572+0000 7f9e80ff9640 1 -- 172.21.15.28:0/2830778030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f9e7c0bb540 con 0x7f9e84162000 2024-09-10T20:41:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.572+0000 7f9e83fff640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e70078120 0x7f9e7007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.573+0000 7f9e83fff640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e70078120 0x7f9e7007a5e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9e74005fd0 tx=0x7f9e7400f360 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:08.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.576+0000 7f9e80ff9640 1 -- 172.21.15.28:0/2830778030 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9e7c041e40 con 0x7f9e84162000 2024-09-10T20:41:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:08.764+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9e840b9f10 con 0x7f9e84162000 2024-09-10T20:41:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.752+0000 7f9e80ff9640 1 -- 172.21.15.28:0/2830778030 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) ==== 138+0+0 (secure 0 0 0) 0x7f9e7c083ff0 con 0x7f9e84162000 2024-09-10T20:41:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e70078120 msgr2=0x7f9e7007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e70078120 0x7f9e7007a5e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9e74005fd0 tx=0x7f9e7400f360 comp rx=0 tx=0).stop 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84162000 msgr2=0x7f9e840cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84162000 0x7f9e840cb060 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009ca0 tx=0x7f9e7c031e20 comp rx=0 tx=0).stop 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 shutdown_connections 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9e70078120 0x7f9e7007a5e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e84162000 0x7f9e840cb060 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e84161850 0x7f9e84152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 --2- 172.21.15.28:0/2830778030 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e840b4c20 0x7f9e840b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 >> 172.21.15.28:0/2830778030 conn(0x7f9e840d3f90 msgr2=0x7f9e8415d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 shutdown_connections 2024-09-10T20:41:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.756+0000 7f9e8a952640 1 -- 172.21.15.28:0/2830778030 wait complete. 2024-09-10T20:41:09.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:41:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.893+0000 7f4e85047640 1 -- 172.21.15.28:0/3336401469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 msgr2=0x7f4e800b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.893+0000 7f4e85047640 1 --2- 172.21.15.28:0/3336401469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 0x7f4e800b23d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009be0 tx=0x7f4e7802a920 comp rx=0 tx=0).stop 2024-09-10T20:41:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.893+0000 7f4e85047640 1 -- 172.21.15.28:0/3336401469 shutdown_connections 2024-09-10T20:41:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.893+0000 7f4e85047640 1 --2- 172.21.15.28:0/3336401469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 0x7f4e800b23d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.893+0000 7f4e85047640 1 -- 172.21.15.28:0/3336401469 >> 172.21.15.28:0/3336401469 conn(0x7f4e800d1da0 msgr2=0x7f4e800d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.893+0000 7f4e85047640 1 -- 172.21.15.28:0/3336401469 shutdown_connections 2024-09-10T20:41:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.893+0000 7f4e85047640 1 -- 172.21.15.28:0/3336401469 wait complete. 2024-09-10T20:41:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 Processor -- start 2024-09-10T20:41:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 -- start start 2024-09-10T20:41:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e800b1000 0x7f4e800c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e800ce7b0 0x7f4e8015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 0x7f4e80157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e800b2a90 con 0x7f4e800ce7b0 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4e800b2910 con 0x7f4e800b1000 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.895+0000 7f4e85047640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4e800b2c10 con 0x7f4e800cf2a0 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e800ce7b0 0x7f4e8015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e800ce7b0 0x7f4e8015a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44580/0 (socket says 172.21.15.28:44580) 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 -- 172.21.15.28:0/308864627 learned_addr learned my addr 172.21.15.28:0/308864627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7f59e640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 0x7f4e80157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7ed9d640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e800b1000 0x7f4e800c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 -- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 msgr2=0x7f4e80157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 0x7f4e80157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 -- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e800b1000 msgr2=0x7f4e800c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e800b1000 0x7f4e800c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 -- 172.21.15.28:0/308864627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e80158160 con 0x7f4e800ce7b0 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e7e59c640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e800ce7b0 0x7f4e8015a9c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c460 tx=0x7f4e7400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e677fe640 1 -- 172.21.15.28:0/308864627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74016020 con 0x7f4e800ce7b0 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e677fe640 1 -- 172.21.15.28:0/308864627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e74012940 con 0x7f4e800ce7b0 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8006f7d0 con 0x7f4e800ce7b0 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e677fe640 1 -- 172.21.15.28:0/308864627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74012d90 con 0x7f4e800ce7b0 2024-09-10T20:41:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.896+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e800c7fb0 con 0x7f4e800ce7b0 2024-09-10T20:41:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.897+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e800ceb90 con 0x7f4e800ce7b0 2024-09-10T20:41:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.898+0000 7f4e677fe640 1 -- 172.21.15.28:0/308864627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f4e7401a3b0 con 0x7f4e800ce7b0 2024-09-10T20:41:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.899+0000 7f4e677fe640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4e6c078150 0x7f4e6c07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.899+0000 7f4e677fe640 1 -- 172.21.15.28:0/308864627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5227+0+0 (secure 0 0 0) 0x7f4e74099dc0 con 0x7f4e800ce7b0 2024-09-10T20:41:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.899+0000 7f4e7ed9d640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4e6c078150 0x7f4e6c07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.900+0000 7f4e7ed9d640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4e6c078150 0x7f4e6c07a610 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4e70000c00 tx=0x7f4e70009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:09.904+0000 7f4e677fe640 1 -- 172.21.15.28:0/308864627 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4e74020030 con 0x7f4e800ce7b0 2024-09-10T20:41:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.091+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4e801e8a30 con 0x7f4e800ce7b0 2024-09-10T20:41:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.757+0000 7f4e677fe640 1 -- 172.21.15.28:0/308864627 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) ==== 170+0+0 (secure 0 0 0) 0x7f4e74062040 con 0x7f4e800ce7b0 2024-09-10T20:41:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.761+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4e6c078150 msgr2=0x7f4e6c07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.761+0000 7f4e85047640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4e6c078150 0x7f4e6c07a610 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4e70000c00 tx=0x7f4e70009040 comp rx=0 tx=0).stop 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.761+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e800ce7b0 msgr2=0x7f4e8015a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.761+0000 7f4e85047640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e800ce7b0 0x7f4e8015a9c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c460 tx=0x7f4e7400c920 comp rx=0 tx=0).stop 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.762+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 shutdown_connections 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.762+0000 7f4e85047640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4e6c078150 0x7f4e6c07a610 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.762+0000 7f4e85047640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e800cf2a0 0x7f4e80157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.762+0000 7f4e85047640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e800ce7b0 0x7f4e8015a9c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.762+0000 7f4e85047640 1 --2- 172.21.15.28:0/308864627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e800b1000 0x7f4e800c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.762+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 >> 172.21.15.28:0/308864627 conn(0x7f4e800d1da0 msgr2=0x7f4e800b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.762+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 shutdown_connections 2024-09-10T20:41:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.763+0000 7f4e85047640 1 -- 172.21.15.28:0/308864627 wait complete. 2024-09-10T20:41:10.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:41:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 -- 172.21.15.28:0/3099944352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe950158a10 msgr2=0x7fe95015b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 --2- 172.21.15.28:0/3099944352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe950158a10 0x7fe95015b470 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe9500d0030 tx=0x7fe94c02a4a0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 -- 172.21.15.28:0/3099944352 shutdown_connections 2024-09-10T20:41:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 --2- 172.21.15.28:0/3099944352 >> v2:172.21.15.159:3300/0 conn(0x7fe9500cb2e0 0x7fe9500ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 --2- 172.21.15.28:0/3099944352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe950158a10 0x7fe95015b470 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 -- 172.21.15.28:0/3099944352 >> 172.21.15.28:0/3099944352 conn(0x7fe9500d0090 msgr2=0x7fe9500d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 -- 172.21.15.28:0/3099944352 shutdown_connections 2024-09-10T20:41:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.900+0000 7fe956e39640 1 -- 172.21.15.28:0/3099944352 wait complete. 2024-09-10T20:41:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.901+0000 7fe956e39640 1 Processor -- start 2024-09-10T20:41:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.901+0000 7fe956e39640 1 -- start start 2024-09-10T20:41:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe956e39640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe950070f90 0x7fe9501f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe956e39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9500cb2e0 0x7fe9500b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe956e39640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe950158a10 0x7fe9500b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe956e39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9500cee20 con 0x7fe9500cb2e0 2024-09-10T20:41:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe956e39640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe9500ceca0 con 0x7fe950158a10 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe956e39640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe9500cefa0 con 0x7fe950070f90 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9500cb2e0 0x7fe9500b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9500cb2e0 0x7fe9500b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44238/0 (socket says 172.21.15.28:44238) 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 -- 172.21.15.28:0/1696401152 learned_addr learned my addr 172.21.15.28:0/1696401152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe9553d6640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe950158a10 0x7fe9500b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe954bd5640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe950070f90 0x7fe9501f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 -- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe950070f90 msgr2=0x7fe9501f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe950070f90 0x7fe9501f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 -- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe950158a10 msgr2=0x7fe9500b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe950158a10 0x7fe9500b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.902+0000 7fe947fff640 1 -- 172.21.15.28:0/1696401152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9501f35d0 con 0x7fe9500cb2e0 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.903+0000 7fe947fff640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9500cb2e0 0x7fe9500b3020 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c530 tx=0x7fe94000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.903+0000 7fe9457fa640 1 -- 172.21.15.28:0/1696401152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940016020 con 0x7fe9500cb2e0 2024-09-10T20:41:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.903+0000 7fe9457fa640 1 -- 172.21.15.28:0/1696401152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe940012920 con 0x7fe9500cb2e0 2024-09-10T20:41:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.903+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95015a390 con 0x7fe9500cb2e0 2024-09-10T20:41:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.903+0000 7fe9457fa640 1 -- 172.21.15.28:0/1696401152 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940012d50 con 0x7fe9500cb2e0 2024-09-10T20:41:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.903+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95015a950 con 0x7fe9500cb2e0 2024-09-10T20:41:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.904+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe95015ab90 con 0x7fe9500cb2e0 2024-09-10T20:41:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.905+0000 7fe9457fa640 1 -- 172.21.15.28:0/1696401152 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fe94001a3b0 con 0x7fe9500cb2e0 2024-09-10T20:41:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.905+0000 7fe9457fa640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe93c078150 0x7fe93c07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.905+0000 7fe9457fa640 1 -- 172.21.15.28:0/1696401152 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5609+0+0 (secure 0 0 0) 0x7fe940099f10 con 0x7fe9500cb2e0 2024-09-10T20:41:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.906+0000 7fe954bd5640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe93c078150 0x7fe93c07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.906+0000 7fe954bd5640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe93c078150 0x7fe93c07a610 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe94c034000 tx=0x7fe94c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:10.910+0000 7fe9457fa640 1 -- 172.21.15.28:0/1696401152 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe95015ab90 con 0x7fe9500cb2e0 2024-09-10T20:41:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.135+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe9500ce550 con 0x7fe9500cb2e0 2024-09-10T20:41:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.760+0000 7fe9457fa640 1 -- 172.21.15.28:0/1696401152 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v32) ==== 172+0+0 (secure 0 0 0) 0x7fe940020030 con 0x7fe9500cb2e0 2024-09-10T20:41:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 8 and data pool 9 2024-09-10T20:41:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.764+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe93c078150 msgr2=0x7fe93c07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.764+0000 7fe956e39640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe93c078150 0x7fe93c07a610 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe94c034000 tx=0x7fe94c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.764+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9500cb2e0 msgr2=0x7fe9500b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.764+0000 7fe956e39640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9500cb2e0 0x7fe9500b3020 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c530 tx=0x7fe94000c9f0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.765+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 shutdown_connections 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.765+0000 7fe956e39640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe93c078150 0x7fe93c07a610 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.765+0000 7fe956e39640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe950158a10 0x7fe9500b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.765+0000 7fe956e39640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9500cb2e0 0x7fe9500b3020 secure :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c530 tx=0x7fe94000c9f0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.765+0000 7fe956e39640 1 --2- 172.21.15.28:0/1696401152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe950070f90 0x7fe9501f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.765+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 >> 172.21.15.28:0/1696401152 conn(0x7fe9500d0090 msgr2=0x7fe9500b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.766+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 shutdown_connections 2024-09-10T20:41:11.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.766+0000 7fe956e39640 1 -- 172.21.15.28:0/1696401152 wait complete. 2024-09-10T20:41:11.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 -- 172.21.15.28:0/3858335572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 msgr2=0x7f2bd40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3858335572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd40706d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc005fd0 tx=0x7f2bcc02aa00 comp rx=0 tx=0).stop 2024-09-10T20:41:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 -- 172.21.15.28:0/3858335572 shutdown_connections 2024-09-10T20:41:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3858335572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd40706d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3858335572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40d0c80 0x7f2bd415bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 -- 172.21.15.28:0/3858335572 >> 172.21.15.28:0/3858335572 conn(0x7f2bd406dbf0 msgr2=0x7f2bd406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:11.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 -- 172.21.15.28:0/3858335572 shutdown_connections 2024-09-10T20:41:11.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.911+0000 7f2bdc640640 1 -- 172.21.15.28:0/3858335572 wait complete. 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.912+0000 7f2bdc640640 1 Processor -- start 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.912+0000 7f2bdc640640 1 -- start start 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bdc640640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd4145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bdc640640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40d0c80 0x7f2bd414bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bdc640640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd40d1060 0x7f2bd4159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bdc640640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd415edf0 con 0x7f2bd4070280 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bdc640640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd40b54a0 con 0x7f2bd40d1060 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bdc640640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd415ef70 con 0x7f2bd40d0c80 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd4145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd4145390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44258/0 (socket says 172.21.15.28:44258) 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 -- 172.21.15.28:0/3759818366 learned_addr learned my addr 172.21.15.28:0/3759818366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bdabdd640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd40d1060 0x7f2bd4159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bd9bdb640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40d0c80 0x7f2bd414bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 -- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40d0c80 msgr2=0x7f2bd414bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40d0c80 0x7f2bd414bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 -- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd40d1060 msgr2=0x7f2bd4159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd40d1060 0x7f2bd4159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.913+0000 7f2bda3dc640 1 -- 172.21.15.28:0/3759818366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd40d0120 con 0x7f2bd4070280 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.914+0000 7f2bda3dc640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd4145390 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800b4d0 tx=0x7f2bc800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.914+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/3759818366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8014070 con 0x7f2bd4070280 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.914+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/3759818366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bc8010960 con 0x7f2bd4070280 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.914+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd40d0400 con 0x7f2bd4070280 2024-09-10T20:41:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.914+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/3759818366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8010db0 con 0x7f2bd4070280 2024-09-10T20:41:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.914+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd41487a0 con 0x7f2bd4070280 2024-09-10T20:41:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.915+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bd41489e0 con 0x7f2bd4070280 2024-09-10T20:41:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.916+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/3759818366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f2bc8023480 con 0x7f2bd4070280 2024-09-10T20:41:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.916+0000 7f2bc6ffd640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2bb8078120 0x7f2bb807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.917+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/3759818366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f2bc809a5a0 con 0x7f2bd4070280 2024-09-10T20:41:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.917+0000 7f2bd9bdb640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2bb8078120 0x7f2bb807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.917+0000 7f2bd9bdb640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2bb8078120 0x7f2bb807a5e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2bd415a8a0 tx=0x7f2bd0009060 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:11.920+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/3759818366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2bc80627c0 con 0x7f2bd4070280 2024-09-10T20:41:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.106+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bd40d05d0 con 0x7f2bd4070280 2024-09-10T20:41:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.107+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/3759818366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16738 (secure 0 0 0) 0x7f2bc80671f0 con 0x7f2bd4070280 2024-09-10T20:41:12.111 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:12.111 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:11.137992+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:08.769779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:10.096656+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2bb8078120 msgr2=0x7f2bb807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2bb8078120 0x7f2bb807a5e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2bd415a8a0 tx=0x7f2bd0009060 comp rx=0 tx=0).stop 2024-09-10T20:41:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 msgr2=0x7f2bd4145390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd4145390 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800b4d0 tx=0x7f2bc800b990 comp rx=0 tx=0).stop 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 shutdown_connections 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2bb8078120 0x7f2bb807a5e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2bd40d1060 0x7f2bd4159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2bd40d0c80 0x7f2bd414bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 --2- 172.21.15.28:0/3759818366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd4070280 0x7f2bd4145390 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 >> 172.21.15.28:0/3759818366 conn(0x7f2bd406dbf0 msgr2=0x7f2bd4152210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 shutdown_connections 2024-09-10T20:41:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.109+0000 7f2bdc640640 1 -- 172.21.15.28:0/3759818366 wait complete. 2024-09-10T20:41:12.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 -- 172.21.15.28:0/1901389975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c156130 msgr2=0x7f2b8c0caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/1901389975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c156130 0x7f2b8c0caa70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005fd0 tx=0x7f2b7c02a530 comp rx=0 tx=0).stop 2024-09-10T20:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 -- 172.21.15.28:0/1901389975 shutdown_connections 2024-09-10T20:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/1901389975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b8c06d1f0 0x7f2b8c06ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/1901389975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c156130 0x7f2b8c0caa70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 -- 172.21.15.28:0/1901389975 >> 172.21.15.28:0/1901389975 conn(0x7f2b8c0d0420 msgr2=0x7f2b8c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 -- 172.21.15.28:0/1901389975 shutdown_connections 2024-09-10T20:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.225+0000 7f2b90fbb640 1 -- 172.21.15.28:0/1901389975 wait complete. 2024-09-10T20:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.226+0000 7f2b90fbb640 1 Processor -- start 2024-09-10T20:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.226+0000 7f2b90fbb640 1 -- start start 2024-09-10T20:41:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b90fbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c06d1f0 0x7f2b8c149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b90fbb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b8c156130 0x7f2b8c1f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b90fbb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b8c156510 0x7f2b8c147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b90fbb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c073400 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b90fbb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c0afa60 con 0x7f2b8c156130 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b90fbb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2b8c073580 con 0x7f2b8c156510 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b8a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c06d1f0 0x7f2b8c149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b8a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c06d1f0 0x7f2b8c149130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44276/0 (socket says 172.21.15.28:44276) 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b8a59c640 1 -- 172.21.15.28:0/883614816 learned_addr learned my addr 172.21.15.28:0/883614816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b83fff640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b8c156130 0x7f2b8c1f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.227+0000 7f2b8ad9d640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b8c156510 0x7f2b8c147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b8a59c640 1 -- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b8c156510 msgr2=0x7f2b8c147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b8a59c640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b8c156510 0x7f2b8c147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b8a59c640 1 -- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b8c156130 msgr2=0x7f2b8c1f68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b8a59c640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b8c156130 0x7f2b8c1f68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b8a59c640 1 -- 172.21.15.28:0/883614816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c149670 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b8ad9d640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b8c156510 0x7f2b8c147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b8a59c640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c06d1f0 0x7f2b8c149130 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c000c00 tx=0x7f2b7c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b82ffd640 1 -- 172.21.15.28:0/883614816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c03f070 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b82ffd640 1 -- 172.21.15.28:0/883614816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b7c004890 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8c1f6de0 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b82ffd640 1 -- 172.21.15.28:0/883614816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c035030 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.228+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8c1441d0 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.229+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8c06ef30 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.230+0000 7f2b82ffd640 1 -- 172.21.15.28:0/883614816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f2b7c004ce0 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.230+0000 7f2b82ffd640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2b6c078120 0x7f2b6c07a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.230+0000 7f2b82ffd640 1 -- 172.21.15.28:0/883614816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f2b7c0bb400 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.230+0000 7f2b83fff640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2b6c078120 0x7f2b6c07a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.231+0000 7f2b83fff640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2b6c078120 0x7f2b6c07a5e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c154e20 tx=0x7f2b74008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.234+0000 7f2b82ffd640 1 -- 172.21.15.28:0/883614816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2b8c1f6de0 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.421+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b8c0d11c0 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.422+0000 7f2b82ffd640 1 -- 172.21.15.28:0/883614816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16738 (secure 0 0 0) 0x7f2b7c083b80 con 0x7f2b8c06d1f0 2024-09-10T20:41:12.426 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:11.137992+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:08.769779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:10.096656+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:12.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2b6c078120 msgr2=0x7f2b6c07a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2b6c078120 0x7f2b6c07a5e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c154e20 tx=0x7f2b74008040 comp rx=0 tx=0).stop 2024-09-10T20:41:12.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c06d1f0 msgr2=0x7f2b8c149130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c06d1f0 0x7f2b8c149130 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c000c00 tx=0x7f2b7c0039e0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 shutdown_connections 2024-09-10T20:41:12.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2b6c078120 0x7f2b6c07a5e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2b8c156510 0x7f2b8c147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2b8c156130 0x7f2b8c1f68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 --2- 172.21.15.28:0/883614816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8c06d1f0 0x7f2b8c149130 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 >> 172.21.15.28:0/883614816 conn(0x7f2b8c0d0420 msgr2=0x7f2b8c15d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:12.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 shutdown_connections 2024-09-10T20:41:12.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.424+0000 7f2b90fbb640 1 -- 172.21.15.28:0/883614816 wait complete. 2024-09-10T20:41:12.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:41:12.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 -- 172.21.15.28:0/3842456656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 msgr2=0x7f29d40706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 --2- 172.21.15.28:0/3842456656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d40706e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009be0 tx=0x7f29cc02a840 comp rx=0 tx=0).stop 2024-09-10T20:41:12.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 -- 172.21.15.28:0/3842456656 shutdown_connections 2024-09-10T20:41:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 --2- 172.21.15.28:0/3842456656 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d4073ea0 0x7f29d40cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 --2- 172.21.15.28:0/3842456656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d40706e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 -- 172.21.15.28:0/3842456656 >> 172.21.15.28:0/3842456656 conn(0x7f29d40d0d50 msgr2=0x7f29d40d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 -- 172.21.15.28:0/3842456656 shutdown_connections 2024-09-10T20:41:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.545+0000 7f29d93c6640 1 -- 172.21.15.28:0/3842456656 wait complete. 2024-09-10T20:41:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.546+0000 7f29d93c6640 1 Processor -- start 2024-09-10T20:41:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.546+0000 7f29d93c6640 1 -- start start 2024-09-10T20:41:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d93c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d4073ea0 0x7f29d415c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d93c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d41511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d93c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d415bd80 0x7f29d40ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d93c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29d40cce80 con 0x7f29d4151db0 2024-09-10T20:41:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d93c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f29d40ccd00 con 0x7f29d4073ea0 2024-09-10T20:41:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d93c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f29d40cd000 con 0x7f29d415bd80 2024-09-10T20:41:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d41511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d41511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44286/0 (socket says 172.21.15.28:44286) 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d27fc640 1 -- 172.21.15.28:0/2814307365 learned_addr learned my addr 172.21.15.28:0/2814307365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d2ffd640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d4073ea0 0x7f29d415c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d37fe640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d415bd80 0x7f29d40ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d27fc640 1 -- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d415bd80 msgr2=0x7f29d40ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.547+0000 7f29d27fc640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d415bd80 0x7f29d40ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29d27fc640 1 -- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d4073ea0 msgr2=0x7f29d415c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29d27fc640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d4073ea0 0x7f29d415c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29d27fc640 1 -- 172.21.15.28:0/2814307365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d4155360 con 0x7f29d4151db0 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29d27fc640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d41511d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c4d0 tx=0x7f29bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2814307365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc016020 con 0x7f29d4151db0 2024-09-10T20:41:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2814307365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29bc012920 con 0x7f29d4151db0 2024-09-10T20:41:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d406d740 con 0x7f29d4151db0 2024-09-10T20:41:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2814307365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc012d30 con 0x7f29d4151db0 2024-09-10T20:41:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.548+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d4154880 con 0x7f29d4151db0 2024-09-10T20:41:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.549+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d40cc3b0 con 0x7f29d4151db0 2024-09-10T20:41:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.549+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2814307365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f29bc01a3b0 con 0x7f29d4151db0 2024-09-10T20:41:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.550+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4078150 0x7f29b407a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.550+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2814307365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f29bc09a000 con 0x7f29d4151db0 2024-09-10T20:41:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.550+0000 7f29d2ffd640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4078150 0x7f29b407a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.551+0000 7f29d2ffd640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4078150 0x7f29b407a610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f29cc034000 tx=0x7f29cc004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.555+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2814307365 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f29bc020030 con 0x7f29d4151db0 2024-09-10T20:41:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:12.781+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f29d40b6cd0 con 0x7f29d4151db0 2024-09-10T20:41:13.747 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-10T20:41:13.743+0000 7f41fcc44640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:41:13.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.776+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2814307365 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) ==== 118+0+0 (secure 0 0 0) 0x7f29bc0628c0 con 0x7f29d4151db0 2024-09-10T20:41:13.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4078150 msgr2=0x7f29b407a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:13.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4078150 0x7f29b407a610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f29cc034000 tx=0x7f29cc004ab0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 msgr2=0x7f29d41511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:13.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d41511d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c4d0 tx=0x7f29bc00c990 comp rx=0 tx=0).stop 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 shutdown_connections 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f29b4078150 0x7f29b407a610 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f29d415bd80 0x7f29d40ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29d4151db0 0x7f29d41511d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 --2- 172.21.15.28:0/2814307365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f29d4073ea0 0x7f29d415c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 >> 172.21.15.28:0/2814307365 conn(0x7f29d40d0d50 msgr2=0x7f29d40b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.779+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 shutdown_connections 2024-09-10T20:41:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.780+0000 7f29d93c6640 1 -- 172.21.15.28:0/2814307365 wait complete. 2024-09-10T20:41:13.797 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:41:13.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:41:13.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 -- 172.21.15.28:0/2276283573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5564070770 msgr2=0x7f5564143830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:13.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 --2- 172.21.15.28:0/2276283573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5564070770 0x7f5564143830 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f55640af750 tx=0x7f556002a4d0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 -- 172.21.15.28:0/2276283573 shutdown_connections 2024-09-10T20:41:13.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 --2- 172.21.15.28:0/2276283573 >> v2:172.21.15.159:3300/0 conn(0x7f556415da80 0x7f556415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 --2- 172.21.15.28:0/2276283573 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5564072d20 0x7f556415a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 --2- 172.21.15.28:0/2276283573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5564070770 0x7f5564143830 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 -- 172.21.15.28:0/2276283573 >> 172.21.15.28:0/2276283573 conn(0x7f556406d110 msgr2=0x7f55640b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:13.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 -- 172.21.15.28:0/2276283573 shutdown_connections 2024-09-10T20:41:13.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.909+0000 7f556c116640 1 -- 172.21.15.28:0/2276283573 wait complete. 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.910+0000 7f556c116640 1 Processor -- start 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.910+0000 7f556c116640 1 -- start start 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556c116640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5564070770 0x7f5564139b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556c116640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5564072d20 0x7f55640cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556c116640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f556415da80 0x7f556406e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556c116640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f556415b090 con 0x7f556415da80 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556c116640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f556415af10 con 0x7f5564070770 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556c116640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f556415b210 con 0x7f5564072d20 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f556415da80 0x7f556406e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f556415da80 0x7f556406e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44314/0 (socket says 172.21.15.28:44314) 2024-09-10T20:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 -- 172.21.15.28:0/3344294339 learned_addr learned my addr 172.21.15.28:0/3344294339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f5569eb2640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5564070770 0x7f5564139b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f55696b1640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5564072d20 0x7f55640cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 -- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5564072d20 msgr2=0x7f55640cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5564072d20 0x7f55640cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 -- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5564070770 msgr2=0x7f5564139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5564070770 0x7f5564139b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.911+0000 7f556a6b3640 1 -- 172.21.15.28:0/3344294339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55640ce790 con 0x7f556415da80 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.912+0000 7f556a6b3640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f556415da80 0x7f556406e400 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f555c00cab0 tx=0x7f555c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.912+0000 7f555a7fc640 1 -- 172.21.15.28:0/3344294339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c007d30 con 0x7f556415da80 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.912+0000 7f555a7fc640 1 -- 172.21.15.28:0/3344294339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f555c003990 con 0x7f556415da80 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.912+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55641f9b20 con 0x7f556415da80 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.912+0000 7f555a7fc640 1 -- 172.21.15.28:0/3344294339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c003de0 con 0x7f556415da80 2024-09-10T20:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.912+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55640ced30 con 0x7f556415da80 2024-09-10T20:41:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.913+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5564143830 con 0x7f556415da80 2024-09-10T20:41:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.914+0000 7f555a7fc640 1 -- 172.21.15.28:0/3344294339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f555c020060 con 0x7f556415da80 2024-09-10T20:41:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.914+0000 7f555a7fc640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5548078120 0x7f554807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.914+0000 7f555a7fc640 1 -- 172.21.15.28:0/3344294339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f555c09a3f0 con 0x7f556415da80 2024-09-10T20:41:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.914+0000 7f5569eb2640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5548078120 0x7f554807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.915+0000 7f5569eb2640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5548078120 0x7f554807a5e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5560033000 tx=0x7f5560004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:13.918+0000 7f555a7fc640 1 -- 172.21.15.28:0/3344294339 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f555c014070 con 0x7f556415da80 2024-09-10T20:41:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.143+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f55640cef00 con 0x7f556415da80 2024-09-10T20:41:14.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.785+0000 7f555a7fc640 1 -- 172.21.15.28:0/3344294339 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f555c062520 con 0x7f556415da80 2024-09-10T20:41:14.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5548078120 msgr2=0x7f554807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5548078120 0x7f554807a5e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5560033000 tx=0x7f5560004730 comp rx=0 tx=0).stop 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f556415da80 msgr2=0x7f556406e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f556415da80 0x7f556406e400 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f555c00cab0 tx=0x7f555c00ceb0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 shutdown_connections 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f5548078120 0x7f554807a5e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f556415da80 0x7f556406e400 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5564072d20 0x7f55640cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 --2- 172.21.15.28:0/3344294339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5564070770 0x7f5564139b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.788+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 >> 172.21.15.28:0/3344294339 conn(0x7f556406d110 msgr2=0x7f5564142a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.789+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 shutdown_connections 2024-09-10T20:41:14.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.789+0000 7f556c116640 1 -- 172.21.15.28:0/3344294339 wait complete. 2024-09-10T20:41:14.806 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:41:14.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.916+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/738400969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 msgr2=0x7ff384003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.916+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/738400969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff384003e70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009b50 tx=0x7ff384005830 comp rx=0 tx=0).stop 2024-09-10T20:41:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.916+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/738400969 shutdown_connections 2024-09-10T20:41:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.916+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/738400969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff384003e70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.916+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/738400969 >> 172.21.15.28:0/738400969 conn(0x7ff39c0d3fb0 msgr2=0x7ff39c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.916+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/738400969 shutdown_connections 2024-09-10T20:41:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.916+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/738400969 wait complete. 2024-09-10T20:41:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.917+0000 7ff3a1a7f640 1 Processor -- start 2024-09-10T20:41:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.917+0000 7ff3a1a7f640 1 -- start start 2024-09-10T20:41:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff3a1a7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff39c157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff3a1a7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff39c15e5f0 0x7ff39c0a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff3a1a7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff39c0a1700 0x7ff39c15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff3a1a7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c0d2f20 con 0x7ff384003a50 2024-09-10T20:41:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff3a1a7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c0d2da0 con 0x7ff39c0a1700 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff3a1a7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff39c15e410 con 0x7ff39c15e5f0 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff39b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff39c157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff39b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff39c157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44336/0 (socket says 172.21.15.28:44336) 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff39b7fe640 1 -- 172.21.15.28:0/3294919648 learned_addr learned my addr 172.21.15.28:0/3294919648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff39bfff640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff39c0a1700 0x7ff39c15f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.918+0000 7ff39affd640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff39c15e5f0 0x7ff39c0a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff39b7fe640 1 -- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff39c15e5f0 msgr2=0x7ff39c0a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff39b7fe640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff39c15e5f0 0x7ff39c0a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff39b7fe640 1 -- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff39c0a1700 msgr2=0x7ff39c15f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff39b7fe640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff39c0a1700 0x7ff39c15f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff39b7fe640 1 -- 172.21.15.28:0/3294919648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c1f5320 con 0x7ff384003a50 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff39bfff640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff39c0a1700 0x7ff39c15f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:14.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff39b7fe640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff39c157030 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009a70 tx=0x7ff38c0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:14.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff383fff640 1 -- 172.21.15.28:0/3294919648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c038040 con 0x7ff384003a50 2024-09-10T20:41:14.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff383fff640 1 -- 172.21.15.28:0/3294919648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff38c0339d0 con 0x7ff384003a50 2024-09-10T20:41:14.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39c1f5600 con 0x7ff384003a50 2024-09-10T20:41:14.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff383fff640 1 -- 172.21.15.28:0/3294919648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c033e20 con 0x7ff384003a50 2024-09-10T20:41:14.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.919+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff39c1f5bc0 con 0x7ff384003a50 2024-09-10T20:41:14.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.920+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff39c0a1b10 con 0x7ff384003a50 2024-09-10T20:41:14.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.921+0000 7ff383fff640 1 -- 172.21.15.28:0/3294919648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7ff38c03b070 con 0x7ff384003a50 2024-09-10T20:41:14.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.921+0000 7ff383fff640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff388078120 0x7ff38807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:14.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.921+0000 7ff383fff640 1 -- 172.21.15.28:0/3294919648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ff38c0ba160 con 0x7ff384003a50 2024-09-10T20:41:14.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.922+0000 7ff39affd640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff388078120 0x7ff38807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:14.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.922+0000 7ff39affd640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff388078120 0x7ff38807a5e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff39c06eb20 tx=0x7ff390001920 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:14.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:14.926+0000 7ff383fff640 1 -- 172.21.15.28:0/3294919648 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff39c0a1b10 con 0x7ff384003a50 2024-09-10T20:41:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.146+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff39c161b50 con 0x7ff384003a50 2024-09-10T20:41:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.147+0000 7ff383fff640 1 -- 172.21.15.28:0/3294919648 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5223 (secure 0 0 0) 0x7ff38c04a020 con 0x7ff384003a50 2024-09-10T20:41:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:15.151 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"btime":"2024-09-10T20:41:14:782123+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/485554639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":485554639},{"type":"v1","addr":"172.21.15.159:6836","nonce":485554639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1807026586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1807026586},{"type":"v1","addr":"172.21.15.159:6837","nonce":1807026586}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/975621681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":975621681},{"type":"v1","addr":"172.21.15.28:6834","nonce":975621681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:11.137947+0000","modified":"2024-09-10T20:41:14.144276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3887400201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3887400201},{"type":"v1","addr":"172.21.15.28:6835","nonce":3887400201}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-10T20:41:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-10T20:41:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff388078120 msgr2=0x7ff38807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff388078120 0x7ff38807a5e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff39c06eb20 tx=0x7ff390001920 comp rx=0 tx=0).stop 2024-09-10T20:41:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 msgr2=0x7ff39c157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff39c157030 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009a70 tx=0x7ff38c0336f0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 shutdown_connections 2024-09-10T20:41:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff388078120 0x7ff38807a5e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff39c0a1700 0x7ff39c15f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff39c15e5f0 0x7ff39c0a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 --2- 172.21.15.28:0/3294919648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384003a50 0x7ff39c157030 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 >> 172.21.15.28:0/3294919648 conn(0x7ff39c0d3fb0 msgr2=0x7ff39c06ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 shutdown_connections 2024-09-10T20:41:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.149+0000 7ff3a1a7f640 1 -- 172.21.15.28:0/3294919648 wait complete. 2024-09-10T20:41:15.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.267+0000 7f6186724640 1 -- 172.21.15.28:0/329078387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6168004d40 msgr2=0x7f61680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.267+0000 7f6186724640 1 --2- 172.21.15.28:0/329078387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6168004d40 0x7f61680051c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f617800f490 tx=0x7f6168005830 comp rx=0 tx=0).stop 2024-09-10T20:41:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.267+0000 7f6184cc1640 1 -- 172.21.15.28:0/329078387 reap_dead start 2024-09-10T20:41:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.267+0000 7f6186724640 1 -- 172.21.15.28:0/329078387 shutdown_connections 2024-09-10T20:41:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.267+0000 7f6186724640 1 -- 172.21.15.28:0/329078387 >> 172.21.15.28:0/329078387 conn(0x7f618006cf00 msgr2=0x7f618006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.267+0000 7f6186724640 1 -- 172.21.15.28:0/329078387 shutdown_connections 2024-09-10T20:41:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.267+0000 7f6186724640 1 -- 172.21.15.28:0/329078387 wait complete. 2024-09-10T20:41:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.268+0000 7f6186724640 1 Processor -- start 2024-09-10T20:41:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.268+0000 7f6186724640 1 -- start start 2024-09-10T20:41:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6186724640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6180149770 0x7f61801f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6186724640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800af1d0 0x7f61801f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6186724640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61801f5d90 0x7f618015d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6186724640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61800ce6f0 con 0x7f6180149770 2024-09-10T20:41:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6186724640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f61800ce570 con 0x7f61800af1d0 2024-09-10T20:41:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6186724640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f61800ce870 con 0x7f61801f5d90 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6180149770 0x7f61801f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6180149770 0x7f61801f8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44346/0 (socket says 172.21.15.28:44346) 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 -- 172.21.15.28:0/4039312615 learned_addr learned my addr 172.21.15.28:0/4039312615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617f7fe640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800af1d0 0x7f61801f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6184cc1640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61801f5d90 0x7f618015d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 -- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61801f5d90 msgr2=0x7f618015d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61801f5d90 0x7f618015d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 -- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800af1d0 msgr2=0x7f61801f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800af1d0 0x7f61801f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f617ffff640 1 -- 172.21.15.28:0/4039312615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618015c360 con 0x7f6180149770 2024-09-10T20:41:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.269+0000 7f6184cc1640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61801f5d90 0x7f618015d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.270+0000 7f617ffff640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6180149770 0x7f61801f8640 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6170006600 tx=0x7f6170003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.270+0000 7f617cff9640 1 -- 172.21.15.28:0/4039312615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170015070 con 0x7f6180149770 2024-09-10T20:41:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.270+0000 7f617cff9640 1 -- 172.21.15.28:0/4039312615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6170011920 con 0x7f6180149770 2024-09-10T20:41:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.270+0000 7f617cff9640 1 -- 172.21.15.28:0/4039312615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170011d70 con 0x7f6180149770 2024-09-10T20:41:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.270+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618015c640 con 0x7f6180149770 2024-09-10T20:41:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.270+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61800af630 con 0x7f6180149770 2024-09-10T20:41:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.271+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61800ce030 con 0x7f6180149770 2024-09-10T20:41:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.271+0000 7f617cff9640 1 -- 172.21.15.28:0/4039312615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f61700103c0 con 0x7f6180149770 2024-09-10T20:41:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.272+0000 7f617cff9640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f616c078150 0x7f616c07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.272+0000 7f617cff9640 1 -- 172.21.15.28:0/4039312615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f617009ae20 con 0x7f6180149770 2024-09-10T20:41:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.272+0000 7f617f7fe640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f616c078150 0x7f616c07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.273+0000 7f617f7fe640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f616c078150 0x7f616c07a610 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6180154fa0 tx=0x7f6174005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.276+0000 7f617cff9640 1 -- 172.21.15.28:0/4039312615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f61700635a0 con 0x7f6180149770 2024-09-10T20:41:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.465+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61800b49d0 con 0x7f6180149770 2024-09-10T20:41:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.466+0000 7f617cff9640 1 -- 172.21.15.28:0/4039312615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16725 (secure 0 0 0) 0x7f6170062ea0 con 0x7f6180149770 2024-09-10T20:41:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:15.470 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":56,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:12.768390+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:08.769779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:10.096656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f616c078150 msgr2=0x7f616c07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f616c078150 0x7f616c07a610 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6180154fa0 tx=0x7f6174005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6180149770 msgr2=0x7f61801f8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6180149770 0x7f61801f8640 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6170006600 tx=0x7f6170003aa0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 shutdown_connections 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f616c078150 0x7f616c07a610 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61801f5d90 0x7f618015d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61800af1d0 0x7f61801f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 --2- 172.21.15.28:0/4039312615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6180149770 0x7f61801f8640 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 >> 172.21.15.28:0/4039312615 conn(0x7f618006cf00 msgr2=0x7f6180158570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 shutdown_connections 2024-09-10T20:41:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.468+0000 7f6186724640 1 -- 172.21.15.28:0/4039312615 wait complete. 2024-09-10T20:41:15.483 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:41:15.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:41:15.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3075756939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa41554e0 msgr2=0x7f3aa415e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3075756939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa41554e0 0x7f3aa415e930 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3a98005fd0 tx=0x7f3a9802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3075756939 shutdown_connections 2024-09-10T20:41:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3075756939 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3aa415ee70 0x7f3aa40d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3075756939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa41554e0 0x7f3aa415e930 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3075756939 >> 172.21.15.28:0/3075756939 conn(0x7f3aa406cf00 msgr2=0x7f3aa406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3075756939 shutdown_connections 2024-09-10T20:41:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.585+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3075756939 wait complete. 2024-09-10T20:41:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.586+0000 7f3aaa2a9640 1 Processor -- start 2024-09-10T20:41:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.586+0000 7f3aaa2a9640 1 -- start start 2024-09-10T20:41:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aaa2a9640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3aa4155100 0x7f3aa409d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aaa2a9640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3aa41554e0 0x7f3aa4140bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aaa2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa415ee70 0x7f3aa41434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aaa2a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa4072de0 con 0x7f3aa415ee70 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aaa2a9640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa40d0af0 con 0x7f3aa4155100 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aaa2a9640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa4072f60 con 0x7f3aa41554e0 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aa8846640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa415ee70 0x7f3aa41434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aa8846640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa415ee70 0x7f3aa41434d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44366/0 (socket says 172.21.15.28:44366) 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aa8846640 1 -- 172.21.15.28:0/3334235366 learned_addr learned my addr 172.21.15.28:0/3334235366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aa3fff640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3aa4155100 0x7f3aa409d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aa37fe640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3aa41554e0 0x7f3aa4140bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aa8846640 1 -- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3aa41554e0 msgr2=0x7f3aa4140bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.587+0000 7f3aa8846640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3aa41554e0 0x7f3aa4140bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aa8846640 1 -- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3aa4155100 msgr2=0x7f3aa409d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aa8846640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3aa4155100 0x7f3aa409d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aa8846640 1 -- 172.21.15.28:0/3334235366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa4143fe0 con 0x7f3aa415ee70 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aa8846640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa415ee70 0x7f3aa41434d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00da70 tx=0x7f3a9c00df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3334235366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c017070 con 0x7f3aa415ee70 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3334235366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a9c003970 con 0x7f3aa415ee70 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa41442c0 con 0x7f3aa415ee70 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3334235366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c003dc0 con 0x7f3aa415ee70 2024-09-10T20:41:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.588+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa409dbd0 con 0x7f3aa415ee70 2024-09-10T20:41:15.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.589+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa415e930 con 0x7f3aa415ee70 2024-09-10T20:41:15.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.590+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3334235366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f3a9c003f60 con 0x7f3aa415ee70 2024-09-10T20:41:15.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.590+0000 7f3aa0ff9640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a90078120 0x7f3a9007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.591+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3334235366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f3a9c09a4d0 con 0x7f3aa415ee70 2024-09-10T20:41:15.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.591+0000 7f3aa3fff640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a90078120 0x7f3a9007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.591+0000 7f3aa3fff640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a90078120 0x7f3a9007a5e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a94001720 tx=0x7f3a94009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:15.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.595+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3334235366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3a9c062730 con 0x7f3aa415ee70 2024-09-10T20:41:15.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.777+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3aa406d380 con 0x7f3a90078120 2024-09-10T20:41:15.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.781+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3334235366 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+243257 (secure 0 0 0) 0x7f3aa406d380 con 0x7f3a90078120 2024-09-10T20:41:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:15.790 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.784+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a90078120 msgr2=0x7f3a9007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.784+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a90078120 0x7f3a9007a5e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a94001720 tx=0x7f3a94009040 comp rx=0 tx=0).stop 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.784+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa415ee70 msgr2=0x7f3aa41434d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.784+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa415ee70 0x7f3aa41434d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00da70 tx=0x7f3a9c00df30 comp rx=0 tx=0).stop 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 shutdown_connections 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3a90078120 0x7f3a9007a5e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa415ee70 0x7f3aa41434d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3aa41554e0 0x7f3aa4140bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 --2- 172.21.15.28:0/3334235366 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3aa4155100 0x7f3aa409d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 >> 172.21.15.28:0/3334235366 conn(0x7f3aa406cf00 msgr2=0x7f3aa4073cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:15.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 shutdown_connections 2024-09-10T20:41:15.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.785+0000 7f3aaa2a9640 1 -- 172.21.15.28:0/3334235366 wait complete. 2024-09-10T20:41:15.801 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-10T20:41:14.019262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":124,"acting":124,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238872,"kb_used_data":5972,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735848,"statfs":{"total":773094113280,"available":772849508352,"internally_reserved":0,"allocated":6115328,"data_stored":3185659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.262692"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.769513+0000","last_change":"2024-09-10T20:41:10.769513+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.769513+0000","last_undegraded":"2024-09-10T20:41:10.769513+0000","last_fullsized":"2024-09-10T20:41:10.769513+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.772037+0000","last_change":"2024-09-10T20:41:10.772037+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.772037+0000","last_undegraded":"2024-09-10T20:41:10.772037+0000","last_fullsized":"2024-09-10T20:41:10.772037+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.771568+0000","last_change":"2024-09-10T20:41:10.771568+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.771568+0000","last_undegraded":"2024-09-10T20:41:10.771568+0000","last_fullsized":"2024-09-10T20:41:10.771568+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.766884+0000","last_change":"2024-09-10T20:41:10.766884+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.766884+0000","last_undegraded":"2024-09-10T20:41:10.766884+0000","last_fullsized":"2024-09-10T20:41:10.766884+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.767166+0000","last_change":"2024-09-10T20:41:10.767166+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.767166+0000","last_undegraded":"2024-09-10T20:41:10.767166+0000","last_fullsized":"2024-09-10T20:41:10.767166+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.769035+0000","last_change":"2024-09-10T20:41:10.769035+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.769035+0000","last_undegraded":"2024-09-10T20:41:10.769035+0000","last_fullsized":"2024-09-10T20:41:10.769035+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.767889+0000","last_change":"2024-09-10T20:41:10.767889+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.767889+0000","last_undegraded":"2024-09-10T20:41:10.767889+0000","last_fullsized":"2024-09-10T20:41:10.767889+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.768889+0000","last_change":"2024-09-10T20:41:10.768889+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.768889+0000","last_undegraded":"2024-09-10T20:41:10.768889+0000","last_fullsized":"2024-09-10T20:41:10.768889+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.767500+0000","last_change":"2024-09-10T20:41:10.767500+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.767500+0000","last_undegraded":"2024-09-10T20:41:10.767500+0000","last_fullsized":"2024-09-10T20:41:10.767500+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.768619+0000","last_change":"2024-09-10T20:41:10.768619+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.768619+0000","last_undegraded":"2024-09-10T20:41:10.768619+0000","last_fullsized":"2024-09-10T20:41:10.768619+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.771158+0000","last_change":"2024-09-10T20:41:10.771158+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.771158+0000","last_undegraded":"2024-09-10T20:41:10.771158+0000","last_fullsized":"2024-09-10T20:41:10.771158+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.766804+0000","last_change":"2024-09-10T20:41:10.766804+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.766804+0000","last_undegraded":"2024-09-10T20:41:10.766804+0000","last_fullsized":"2024-09-10T20:41:10.766804+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.771855+0000","last_change":"2024-09-10T20:41:10.771855+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.771855+0000","last_undegraded":"2024-09-10T20:41:10.771855+0000","last_fullsized":"2024-09-10T20:41:10.771855+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.771470+0000","last_change":"2024-09-10T20:41:10.771470+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.771470+0000","last_undegraded":"2024-09-10T20:41:10.771470+0000","last_fullsized":"2024-09-10T20:41:10.771470+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.758621+0000","last_change":"2024-09-10T20:41:09.758621+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.758621+0000","last_undegraded":"2024-09-10T20:41:09.758621+0000","last_fullsized":"2024-09-10T20:41:09.758621+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.759403+0000","last_change":"2024-09-10T20:41:09.759403+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.759403+0000","last_undegraded":"2024-09-10T20:41:09.759403+0000","last_fullsized":"2024-09-10T20:41:09.759403+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.761202+0000","last_change":"2024-09-10T20:41:09.761202+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.761202+0000","last_undegraded":"2024-09-10T20:41:09.761202+0000","last_fullsized":"2024-09-10T20:41:09.761202+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.781924+0000","last_change":"2024-09-10T20:41:10.779236+0000","last_active":"2024-09-10T20:41:10.781924+0000","last_peered":"2024-09-10T20:41:10.781924+0000","last_clean":"2024-09-10T20:41:10.781924+0000","last_became_active":"2024-09-10T20:41:10.778960+0000","last_became_peered":"2024-09-10T20:41:10.778960+0000","last_unstale":"2024-09-10T20:41:10.781924+0000","last_undegraded":"2024-09-10T20:41:10.781924+0000","last_fullsized":"2024-09-10T20:41:10.781924+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:44.052976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.780424+0000","last_change":"2024-09-10T20:41:10.777277+0000","last_active":"2024-09-10T20:41:10.780424+0000","last_peered":"2024-09-10T20:41:10.780424+0000","last_clean":"2024-09-10T20:41:10.780424+0000","last_became_active":"2024-09-10T20:41:10.776709+0000","last_became_peered":"2024-09-10T20:41:10.776709+0000","last_unstale":"2024-09-10T20:41:10.780424+0000","last_undegraded":"2024-09-10T20:41:10.780424+0000","last_fullsized":"2024-09-10T20:41:10.780424+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:22.515110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.761471+0000","last_change":"2024-09-10T20:41:09.761471+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.761471+0000","last_undegraded":"2024-09-10T20:41:09.761471+0000","last_fullsized":"2024-09-10T20:41:09.761471+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.766539+0000","last_change":"2024-09-10T20:41:10.766539+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.766539+0000","last_undegraded":"2024-09-10T20:41:10.766539+0000","last_fullsized":"2024-09-10T20:41:10.766539+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.781769+0000","last_change":"2024-09-10T20:41:10.780820+0000","last_active":"2024-09-10T20:41:10.781769+0000","last_peered":"2024-09-10T20:41:10.781769+0000","last_clean":"2024-09-10T20:41:10.781769+0000","last_became_active":"2024-09-10T20:41:10.780510+0000","last_became_peered":"2024-09-10T20:41:10.780510+0000","last_unstale":"2024-09-10T20:41:10.781769+0000","last_undegraded":"2024-09-10T20:41:10.781769+0000","last_fullsized":"2024-09-10T20:41:10.781769+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:16:33.962691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.778418+0000","last_change":"2024-09-10T20:41:10.775128+0000","last_active":"2024-09-10T20:41:10.778418+0000","last_peered":"2024-09-10T20:41:10.778418+0000","last_clean":"2024-09-10T20:41:10.778418+0000","last_became_active":"2024-09-10T20:41:10.774832+0000","last_became_peered":"2024-09-10T20:41:10.774832+0000","last_unstale":"2024-09-10T20:41:10.778418+0000","last_undegraded":"2024-09-10T20:41:10.778418+0000","last_fullsized":"2024-09-10T20:41:10.778418+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:33.920836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.769518+0000","last_change":"2024-09-10T20:41:10.769518+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.769518+0000","last_undegraded":"2024-09-10T20:41:10.769518+0000","last_fullsized":"2024-09-10T20:41:10.769518+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.779661+0000","last_change":"2024-09-10T20:41:10.778435+0000","last_active":"2024-09-10T20:41:10.779661+0000","last_peered":"2024-09-10T20:41:10.779661+0000","last_clean":"2024-09-10T20:41:10.779661+0000","last_became_active":"2024-09-10T20:41:10.778171+0000","last_became_peered":"2024-09-10T20:41:10.778171+0000","last_unstale":"2024-09-10T20:41:10.779661+0000","last_undegraded":"2024-09-10T20:41:10.779661+0000","last_fullsized":"2024-09-10T20:41:10.779661+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:05:30.030169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.770065+0000","last_change":"2024-09-10T20:41:10.770065+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.770065+0000","last_undegraded":"2024-09-10T20:41:10.770065+0000","last_fullsized":"2024-09-10T20:41:10.770065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.762096+0000","last_change":"2024-09-10T20:41:09.762096+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.762096+0000","last_undegraded":"2024-09-10T20:41:09.762096+0000","last_fullsized":"2024-09-10T20:41:09.762096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.781888+0000","last_change":"2024-09-10T20:41:10.780029+0000","last_active":"2024-09-10T20:41:10.781888+0000","last_peered":"2024-09-10T20:41:10.781888+0000","last_clean":"2024-09-10T20:41:10.781888+0000","last_became_active":"2024-09-10T20:41:10.779741+0000","last_became_peered":"2024-09-10T20:41:10.779741+0000","last_unstale":"2024-09-10T20:41:10.781888+0000","last_undegraded":"2024-09-10T20:41:10.781888+0000","last_fullsized":"2024-09-10T20:41:10.781888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:02.036963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.771276+0000","last_change":"2024-09-10T20:41:10.771276+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.771276+0000","last_undegraded":"2024-09-10T20:41:10.771276+0000","last_fullsized":"2024-09-10T20:41:10.771276+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.784078+0000","last_change":"2024-09-10T20:41:10.783219+0000","last_active":"2024-09-10T20:41:10.784078+0000","last_peered":"2024-09-10T20:41:10.784078+0000","last_clean":"2024-09-10T20:41:10.784078+0000","last_became_active":"2024-09-10T20:41:10.782877+0000","last_became_peered":"2024-09-10T20:41:10.782877+0000","last_unstale":"2024-09-10T20:41:10.784078+0000","last_undegraded":"2024-09-10T20:41:10.784078+0000","last_fullsized":"2024-09-10T20:41:10.784078+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:47:54.798922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.766459+0000","last_change":"2024-09-10T20:41:10.766459+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.766459+0000","last_undegraded":"2024-09-10T20:41:10.766459+0000","last_fullsized":"2024-09-10T20:41:10.766459+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.759183+0000","last_change":"2024-09-10T20:41:09.759183+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.759183+0000","last_undegraded":"2024-09-10T20:41:09.759183+0000","last_fullsized":"2024-09-10T20:41:09.759183+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.768871+0000","last_change":"2024-09-10T20:41:10.768871+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.768871+0000","last_undegraded":"2024-09-10T20:41:10.768871+0000","last_fullsized":"2024-09-10T20:41:10.768871+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.781845+0000","last_change":"2024-09-10T20:41:10.779762+0000","last_active":"2024-09-10T20:41:10.781845+0000","last_peered":"2024-09-10T20:41:10.781845+0000","last_clean":"2024-09-10T20:41:10.781845+0000","last_became_active":"2024-09-10T20:41:10.779444+0000","last_became_peered":"2024-09-10T20:41:10.779444+0000","last_unstale":"2024-09-10T20:41:10.781845+0000","last_undegraded":"2024-09-10T20:41:10.781845+0000","last_fullsized":"2024-09-10T20:41:10.781845+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:40:00.218718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.766226+0000","last_change":"2024-09-10T20:41:10.766226+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.766226+0000","last_undegraded":"2024-09-10T20:41:10.766226+0000","last_fullsized":"2024-09-10T20:41:10.766226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.778184+0000","last_change":"2024-09-10T20:41:10.776282+0000","last_active":"2024-09-10T20:41:10.778184+0000","last_peered":"2024-09-10T20:41:10.778184+0000","last_clean":"2024-09-10T20:41:10.778184+0000","last_became_active":"2024-09-10T20:41:10.775927+0000","last_became_peered":"2024-09-10T20:41:10.775927+0000","last_unstale":"2024-09-10T20:41:10.778184+0000","last_undegraded":"2024-09-10T20:41:10.778184+0000","last_fullsized":"2024-09-10T20:41:10.778184+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:46.812437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.770631+0000","last_change":"2024-09-10T20:41:10.770631+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.770631+0000","last_undegraded":"2024-09-10T20:41:10.770631+0000","last_fullsized":"2024-09-10T20:41:10.770631+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.781810+0000","last_change":"2024-09-10T20:41:10.780541+0000","last_active":"2024-09-10T20:41:10.781810+0000","last_peered":"2024-09-10T20:41:10.781810+0000","last_clean":"2024-09-10T20:41:10.781810+0000","last_became_active":"2024-09-10T20:41:10.780251+0000","last_became_peered":"2024-09-10T20:41:10.780251+0000","last_unstale":"2024-09-10T20:41:10.781810+0000","last_undegraded":"2024-09-10T20:41:10.781810+0000","last_fullsized":"2024-09-10T20:41:10.781810+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:06.029465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.767076+0000","last_change":"2024-09-10T20:41:10.767076+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.767076+0000","last_undegraded":"2024-09-10T20:41:10.767076+0000","last_fullsized":"2024-09-10T20:41:10.767076+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.760009+0000","last_change":"2024-09-10T20:41:09.760009+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.760009+0000","last_undegraded":"2024-09-10T20:41:09.760009+0000","last_fullsized":"2024-09-10T20:41:09.760009+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.769632+0000","last_change":"2024-09-10T20:41:10.769632+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.769632+0000","last_undegraded":"2024-09-10T20:41:10.769632+0000","last_fullsized":"2024-09-10T20:41:10.769632+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.758944+0000","last_change":"2024-09-10T20:41:09.758944+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.758944+0000","last_undegraded":"2024-09-10T20:41:09.758944+0000","last_fullsized":"2024-09-10T20:41:09.758944+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.783162+0000","last_change":"2024-09-10T20:41:10.781773+0000","last_active":"2024-09-10T20:41:10.783162+0000","last_peered":"2024-09-10T20:41:10.783162+0000","last_clean":"2024-09-10T20:41:10.783162+0000","last_became_active":"2024-09-10T20:41:10.781293+0000","last_became_peered":"2024-09-10T20:41:10.781293+0000","last_unstale":"2024-09-10T20:41:10.783162+0000","last_undegraded":"2024-09-10T20:41:10.783162+0000","last_fullsized":"2024-09-10T20:41:10.783162+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:26.759188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.770810+0000","last_change":"2024-09-10T20:41:10.770810+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.770810+0000","last_undegraded":"2024-09-10T20:41:10.770810+0000","last_fullsized":"2024-09-10T20:41:10.770810+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.762156+0000","last_change":"2024-09-10T20:41:09.762156+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.762156+0000","last_undegraded":"2024-09-10T20:41:09.762156+0000","last_fullsized":"2024-09-10T20:41:09.762156+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.765848+0000","last_change":"2024-09-10T20:41:10.765848+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.765848+0000","last_undegraded":"2024-09-10T20:41:10.765848+0000","last_fullsized":"2024-09-10T20:41:10.765848+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.758613+0000","last_change":"2024-09-10T20:41:09.758613+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.758613+0000","last_undegraded":"2024-09-10T20:41:09.758613+0000","last_fullsized":"2024-09-10T20:41:09.758613+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.761952+0000","last_change":"2024-09-10T20:41:09.761952+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.761952+0000","last_undegraded":"2024-09-10T20:41:09.761952+0000","last_fullsized":"2024-09-10T20:41:09.761952+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.776956+0000","last_change":"2024-09-10T20:41:10.775679+0000","last_active":"2024-09-10T20:41:10.776956+0000","last_peered":"2024-09-10T20:41:10.776956+0000","last_clean":"2024-09-10T20:41:10.776956+0000","last_became_active":"2024-09-10T20:41:10.775233+0000","last_became_peered":"2024-09-10T20:41:10.775233+0000","last_unstale":"2024-09-10T20:41:10.776956+0000","last_undegraded":"2024-09-10T20:41:10.776956+0000","last_fullsized":"2024-09-10T20:41:10.776956+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:18:25.254788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.766639+0000","last_change":"2024-09-10T20:41:10.766639+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.766639+0000","last_undegraded":"2024-09-10T20:41:10.766639+0000","last_fullsized":"2024-09-10T20:41:10.766639+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.759497+0000","last_change":"2024-09-10T20:41:09.759497+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.759497+0000","last_undegraded":"2024-09-10T20:41:09.759497+0000","last_fullsized":"2024-09-10T20:41:09.759497+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.769188+0000","last_change":"2024-09-10T20:41:10.769188+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.769188+0000","last_undegraded":"2024-09-10T20:41:10.769188+0000","last_fullsized":"2024-09-10T20:41:10.769188+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.761202+0000","last_change":"2024-09-10T20:41:09.761202+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.761202+0000","last_undegraded":"2024-09-10T20:41:09.761202+0000","last_fullsized":"2024-09-10T20:41:09.761202+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.776349+0000","last_change":"2024-09-10T20:41:10.774717+0000","last_active":"2024-09-10T20:41:10.776349+0000","last_peered":"2024-09-10T20:41:10.776349+0000","last_clean":"2024-09-10T20:41:10.776349+0000","last_became_active":"2024-09-10T20:41:10.774273+0000","last_became_peered":"2024-09-10T20:41:10.774273+0000","last_unstale":"2024-09-10T20:41:10.776349+0000","last_undegraded":"2024-09-10T20:41:10.776349+0000","last_fullsized":"2024-09-10T20:41:10.776349+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:52:50.855186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.778799+0000","last_change":"2024-09-10T20:41:10.777536+0000","last_active":"2024-09-10T20:41:10.778799+0000","last_peered":"2024-09-10T20:41:10.778799+0000","last_clean":"2024-09-10T20:41:10.778799+0000","last_became_active":"2024-09-10T20:41:10.777247+0000","last_became_peered":"2024-09-10T20:41:10.777247+0000","last_unstale":"2024-09-10T20:41:10.778799+0000","last_undegraded":"2024-09-10T20:41:10.778799+0000","last_fullsized":"2024-09-10T20:41:10.778799+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:00.009980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.759396+0000","last_change":"2024-09-10T20:41:09.759396+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.759396+0000","last_undegraded":"2024-09-10T20:41:09.759396+0000","last_fullsized":"2024-09-10T20:41:09.759396+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-09-10T20:41:10.768122+0000","last_change":"2024-09-10T20:41:10.768122+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.768122+0000","last_undegraded":"2024-09-10T20:41:10.768122+0000","last_fullsized":"2024-09-10T20:41:10.768122+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.781473+0000","last_change":"2024-09-10T20:41:10.779772+0000","last_active":"2024-09-10T20:41:10.781473+0000","last_peered":"2024-09-10T20:41:10.781473+0000","last_clean":"2024-09-10T20:41:10.781473+0000","last_became_active":"2024-09-10T20:41:10.779487+0000","last_became_peered":"2024-09-10T20:41:10.779487+0000","last_unstale":"2024-09-10T20:41:10.781473+0000","last_undegraded":"2024-09-10T20:41:10.781473+0000","last_fullsized":"2024-09-10T20:41:10.781473+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:09:43.606764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.760902+0000","last_change":"2024-09-10T20:41:09.760902+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.760902+0000","last_undegraded":"2024-09-10T20:41:09.760902+0000","last_fullsized":"2024-09-10T20:41:09.760902+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-09-10T20:41:09.758819+0000","last_change":"2024-09-10T20:41:09.758819+0000","last_active":"2024-09-10T20:41:09.752159+0000","last_peered":"2024-09-10T20:41:09.752159+0000","last_clean":"2024-09-10T20:41:09.752159+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:09.758819+0000","last_undegraded":"2024-09-10T20:41:09.758819+0000","last_fullsized":"2024-09-10T20:41:09.758819+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:41:10.754365+0000","last_change":"2024-09-10T20:41:10.754365+0000","last_active":"2024-09-10T20:41:10.754365+0000","last_peered":"2024-09-10T20:41:10.754365+0000","last_clean":"2024-09-10T20:41:10.754365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:10.754365+0000","last_undegraded":"2024-09-10T20:41:10.754365+0000","last_fullsized":"2024-09-10T20:41:10.754365+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.783003+0000","last_change":"2024-09-10T20:41:10.779842+0000","last_active":"2024-09-10T20:41:10.783003+0000","last_peered":"2024-09-10T20:41:10.783003+0000","last_clean":"2024-09-10T20:41:10.783003+0000","last_became_active":"2024-09-10T20:41:10.779559+0000","last_became_peered":"2024-09-10T20:41:10.779559+0000","last_unstale":"2024-09-10T20:41:10.783003+0000","last_undegraded":"2024-09-10T20:41:10.783003+0000","last_fullsized":"2024-09-10T20:41:10.783003+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:43.317737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":171,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-10T20:41:10.761542+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:41:10.761542+0000","last_peered":"2024-09-10T20:41:10.761542+0000","last_clean":"2024-09-10T20:41:10.761542+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:41:10.761542+0000","last_undegraded":"2024-09-10T20:41:10.761542+0000","last_fullsized":"2024-09-10T20:41:10.761542+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":4},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509482,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":626688,"data_stored":255429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509481,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":626688,"data_stored":255429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509481,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1220608,"data_stored":845797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":626688,"data_stored":255429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":614400,"data_stored":249612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":577536,"data_stored":234371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1208320,"data_stored":839980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":614400,"data_stored":249612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:41:15.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:41:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.941+0000 7fd9955f3640 1 -- 172.21.15.28:0/2747728598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156510 msgr2=0x7fd99006ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.941+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2747728598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156510 0x7fd99006ef30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd988005fd0 tx=0x7fd98802aa00 comp rx=0 tx=0).stop 2024-09-10T20:41:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.942+0000 7fd9955f3640 1 -- 172.21.15.28:0/2747728598 shutdown_connections 2024-09-10T20:41:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.942+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2747728598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156510 0x7fd99006ef30 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.942+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2747728598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd99006d1f0 0x7fd9900caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.942+0000 7fd9955f3640 1 -- 172.21.15.28:0/2747728598 >> 172.21.15.28:0/2747728598 conn(0x7fd9900d0420 msgr2=0x7fd9900d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.942+0000 7fd9955f3640 1 -- 172.21.15.28:0/2747728598 shutdown_connections 2024-09-10T20:41:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.942+0000 7fd9955f3640 1 -- 172.21.15.28:0/2747728598 wait complete. 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.943+0000 7fd9955f3640 1 Processor -- start 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.943+0000 7fd9955f3640 1 -- start start 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.943+0000 7fd9955f3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd99006d1f0 0x7fd99014f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.943+0000 7fd9955f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156130 0x7fd9901e56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.943+0000 7fd9955f3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd990156510 0x7fd9900b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9955f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9900733b0 con 0x7fd990156130 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9955f3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd990073230 con 0x7fd99006d1f0 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9955f3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd990073530 con 0x7fd990156510 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156130 0x7fd9901e56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156130 0x7fd9901e56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44388/0 (socket says 172.21.15.28:44388) 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 -- 172.21.15.28:0/2185757020 learned_addr learned my addr 172.21.15.28:0/2185757020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd98f7fe640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd990156510 0x7fd9900b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd98effd640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd99006d1f0 0x7fd99014f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 -- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd990156510 msgr2=0x7fd9900b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd990156510 0x7fd9900b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 -- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd99006d1f0 msgr2=0x7fd99014f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd99006d1f0 0x7fd99014f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 -- 172.21.15.28:0/2185757020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9901f6690 con 0x7fd990156130 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd98effd640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd99006d1f0 0x7fd99014f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd987fff640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156130 0x7fd9901e56e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd97800b4a0 tx=0x7fd97800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9877fe640 1 -- 172.21.15.28:0/2185757020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978014070 con 0x7fd990156130 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9877fe640 1 -- 172.21.15.28:0/2185757020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd978010700 con 0x7fd990156130 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9901f6970 con 0x7fd990156130 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9877fe640 1 -- 172.21.15.28:0/2185757020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978010b50 con 0x7fd990156130 2024-09-10T20:41:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.944+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9901f6f30 con 0x7fd990156130 2024-09-10T20:41:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.945+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9900caa70 con 0x7fd990156130 2024-09-10T20:41:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.946+0000 7fd9877fe640 1 -- 172.21.15.28:0/2185757020 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fd978017360 con 0x7fd990156130 2024-09-10T20:41:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.946+0000 7fd9877fe640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd970078150 0x7fd97007a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.946+0000 7fd9877fe640 1 -- 172.21.15.28:0/2185757020 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fd978099d00 con 0x7fd990156130 2024-09-10T20:41:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.947+0000 7fd98effd640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd970078150 0x7fd97007a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.947+0000 7fd98effd640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd970078150 0x7fd97007a610 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd980003eb0 tx=0x7fd980009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:15.951+0000 7fd9877fe640 1 -- 172.21.15.28:0/2185757020 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd978061eb0 con 0x7fd990156130 2024-09-10T20:41:16.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.135+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9900d11c0 con 0x7fd970078150 2024-09-10T20:41:16.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.139+0000 7fd9877fe640 1 -- 172.21.15.28:0/2185757020 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7fd980056840 con 0x7fd970078150 2024-09-10T20:41:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:41:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd970078150 msgr2=0x7fd97007a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd970078150 0x7fd97007a610 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd980003eb0 tx=0x7fd980009040 comp rx=0 tx=0).stop 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156130 msgr2=0x7fd9901e56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156130 0x7fd9901e56e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd97800b4a0 tx=0x7fd97800b960 comp rx=0 tx=0).stop 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 shutdown_connections 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd970078150 0x7fd97007a610 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd990156510 0x7fd9900b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd990156130 0x7fd9901e56e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 --2- 172.21.15.28:0/2185757020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd99006d1f0 0x7fd99014f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 >> 172.21.15.28:0/2185757020 conn(0x7fd9900d0420 msgr2=0x7fd9900aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.142+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 shutdown_connections 2024-09-10T20:41:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.143+0000 7fd9955f3640 1 -- 172.21.15.28:0/2185757020 wait complete. 2024-09-10T20:41:16.157 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-10T20:41:16.019651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243448,"kb_used_data":6408,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754731272,"statfs":{"total":773094113280,"available":772844822528,"internally_reserved":0,"allocated":6561792,"data_stored":3383788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001139"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069686+0000","last_change":"2024-09-10T20:41:11.780685+0000","last_active":"2024-09-10T20:41:14.069686+0000","last_peered":"2024-09-10T20:41:14.069686+0000","last_clean":"2024-09-10T20:41:14.069686+0000","last_became_active":"2024-09-10T20:41:11.780444+0000","last_became_peered":"2024-09-10T20:41:11.780444+0000","last_unstale":"2024-09-10T20:41:14.069686+0000","last_undegraded":"2024-09-10T20:41:14.069686+0000","last_fullsized":"2024-09-10T20:41:14.069686+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:16.867846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069871+0000","last_change":"2024-09-10T20:41:11.774902+0000","last_active":"2024-09-10T20:41:14.069871+0000","last_peered":"2024-09-10T20:41:14.069871+0000","last_clean":"2024-09-10T20:41:14.069871+0000","last_became_active":"2024-09-10T20:41:11.774625+0000","last_became_peered":"2024-09-10T20:41:11.774625+0000","last_unstale":"2024-09-10T20:41:14.069871+0000","last_undegraded":"2024-09-10T20:41:14.069871+0000","last_fullsized":"2024-09-10T20:41:14.069871+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:52.200417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068706+0000","last_change":"2024-09-10T20:41:11.778488+0000","last_active":"2024-09-10T20:41:14.068706+0000","last_peered":"2024-09-10T20:41:14.068706+0000","last_clean":"2024-09-10T20:41:14.068706+0000","last_became_active":"2024-09-10T20:41:11.778181+0000","last_became_peered":"2024-09-10T20:41:11.778181+0000","last_unstale":"2024-09-10T20:41:14.068706+0000","last_undegraded":"2024-09-10T20:41:14.068706+0000","last_fullsized":"2024-09-10T20:41:14.068706+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:16.867846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777440+0000","last_change":"2024-09-10T20:41:11.786182+0000","last_active":"2024-09-10T20:41:12.777440+0000","last_peered":"2024-09-10T20:41:12.777440+0000","last_clean":"2024-09-10T20:41:12.777440+0000","last_became_active":"2024-09-10T20:41:11.784635+0000","last_became_peered":"2024-09-10T20:41:11.784635+0000","last_unstale":"2024-09-10T20:41:12.777440+0000","last_undegraded":"2024-09-10T20:41:12.777440+0000","last_fullsized":"2024-09-10T20:41:12.777440+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:14.569879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069018+0000","last_change":"2024-09-10T20:41:11.782205+0000","last_active":"2024-09-10T20:41:14.069018+0000","last_peered":"2024-09-10T20:41:14.069018+0000","last_clean":"2024-09-10T20:41:14.069018+0000","last_became_active":"2024-09-10T20:41:11.781918+0000","last_became_peered":"2024-09-10T20:41:11.781918+0000","last_unstale":"2024-09-10T20:41:14.069018+0000","last_undegraded":"2024-09-10T20:41:14.069018+0000","last_fullsized":"2024-09-10T20:41:14.069018+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:28.158543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068724+0000","last_change":"2024-09-10T20:41:11.787483+0000","last_active":"2024-09-10T20:41:14.068724+0000","last_peered":"2024-09-10T20:41:14.068724+0000","last_clean":"2024-09-10T20:41:14.068724+0000","last_became_active":"2024-09-10T20:41:11.786939+0000","last_became_peered":"2024-09-10T20:41:11.786939+0000","last_unstale":"2024-09-10T20:41:14.068724+0000","last_undegraded":"2024-09-10T20:41:14.068724+0000","last_fullsized":"2024-09-10T20:41:14.068724+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:48:49.400460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777185+0000","last_change":"2024-09-10T20:41:11.783397+0000","last_active":"2024-09-10T20:41:12.777185+0000","last_peered":"2024-09-10T20:41:12.777185+0000","last_clean":"2024-09-10T20:41:12.777185+0000","last_became_active":"2024-09-10T20:41:11.783046+0000","last_became_peered":"2024-09-10T20:41:11.783046+0000","last_unstale":"2024-09-10T20:41:12.777185+0000","last_undegraded":"2024-09-10T20:41:12.777185+0000","last_fullsized":"2024-09-10T20:41:12.777185+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:10.741521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.783115+0000","last_change":"2024-09-10T20:41:11.780833+0000","last_active":"2024-09-10T20:41:12.783115+0000","last_peered":"2024-09-10T20:41:12.783115+0000","last_clean":"2024-09-10T20:41:12.783115+0000","last_became_active":"2024-09-10T20:41:11.780560+0000","last_became_peered":"2024-09-10T20:41:11.780560+0000","last_unstale":"2024-09-10T20:41:12.783115+0000","last_undegraded":"2024-09-10T20:41:12.783115+0000","last_fullsized":"2024-09-10T20:41:12.783115+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:56:43.648199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587264+0000","last_change":"2024-09-10T20:41:11.777812+0000","last_active":"2024-09-10T20:41:13.587264+0000","last_peered":"2024-09-10T20:41:13.587264+0000","last_clean":"2024-09-10T20:41:13.587264+0000","last_became_active":"2024-09-10T20:41:11.777514+0000","last_became_peered":"2024-09-10T20:41:11.777514+0000","last_unstale":"2024-09-10T20:41:13.587264+0000","last_undegraded":"2024-09-10T20:41:13.587264+0000","last_fullsized":"2024-09-10T20:41:13.587264+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:23.517317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069472+0000","last_change":"2024-09-10T20:41:11.776865+0000","last_active":"2024-09-10T20:41:14.069472+0000","last_peered":"2024-09-10T20:41:14.069472+0000","last_clean":"2024-09-10T20:41:14.069472+0000","last_became_active":"2024-09-10T20:41:11.776592+0000","last_became_peered":"2024-09-10T20:41:11.776592+0000","last_unstale":"2024-09-10T20:41:14.069472+0000","last_undegraded":"2024-09-10T20:41:14.069472+0000","last_fullsized":"2024-09-10T20:41:14.069472+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:06.499336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782610+0000","last_change":"2024-09-10T20:41:11.784450+0000","last_active":"2024-09-10T20:41:12.782610+0000","last_peered":"2024-09-10T20:41:12.782610+0000","last_clean":"2024-09-10T20:41:12.782610+0000","last_became_active":"2024-09-10T20:41:11.783698+0000","last_became_peered":"2024-09-10T20:41:11.783698+0000","last_unstale":"2024-09-10T20:41:12.782610+0000","last_undegraded":"2024-09-10T20:41:12.782610+0000","last_fullsized":"2024-09-10T20:41:12.782610+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:51:55.064126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587214+0000","last_change":"2024-09-10T20:41:11.777265+0000","last_active":"2024-09-10T20:41:13.587214+0000","last_peered":"2024-09-10T20:41:13.587214+0000","last_clean":"2024-09-10T20:41:13.587214+0000","last_became_active":"2024-09-10T20:41:11.776935+0000","last_became_peered":"2024-09-10T20:41:11.776935+0000","last_unstale":"2024-09-10T20:41:13.587214+0000","last_undegraded":"2024-09-10T20:41:13.587214+0000","last_fullsized":"2024-09-10T20:41:13.587214+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:33:20.401409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068027+0000","last_change":"2024-09-10T20:41:11.778441+0000","last_active":"2024-09-10T20:41:14.068027+0000","last_peered":"2024-09-10T20:41:14.068027+0000","last_clean":"2024-09-10T20:41:14.068027+0000","last_became_active":"2024-09-10T20:41:11.778184+0000","last_became_peered":"2024-09-10T20:41:11.778184+0000","last_unstale":"2024-09-10T20:41:14.068027+0000","last_undegraded":"2024-09-10T20:41:14.068027+0000","last_fullsized":"2024-09-10T20:41:14.068027+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:35:59.079151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068119+0000","last_change":"2024-09-10T20:41:11.784624+0000","last_active":"2024-09-10T20:41:14.068119+0000","last_peered":"2024-09-10T20:41:14.068119+0000","last_clean":"2024-09-10T20:41:14.068119+0000","last_became_active":"2024-09-10T20:41:11.784367+0000","last_became_peered":"2024-09-10T20:41:11.784367+0000","last_unstale":"2024-09-10T20:41:14.068119+0000","last_undegraded":"2024-09-10T20:41:14.068119+0000","last_fullsized":"2024-09-10T20:41:14.068119+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:06.499336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.067752+0000","last_change":"2024-09-10T20:41:11.780803+0000","last_active":"2024-09-10T20:41:14.067752+0000","last_peered":"2024-09-10T20:41:14.067752+0000","last_clean":"2024-09-10T20:41:14.067752+0000","last_became_active":"2024-09-10T20:41:11.780491+0000","last_became_peered":"2024-09-10T20:41:11.780491+0000","last_unstale":"2024-09-10T20:41:14.067752+0000","last_undegraded":"2024-09-10T20:41:14.067752+0000","last_fullsized":"2024-09-10T20:41:14.067752+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:48:49.400460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068543+0000","last_change":"2024-09-10T20:41:11.781545+0000","last_active":"2024-09-10T20:41:14.068543+0000","last_peered":"2024-09-10T20:41:14.068543+0000","last_clean":"2024-09-10T20:41:14.068543+0000","last_became_active":"2024-09-10T20:41:11.781243+0000","last_became_peered":"2024-09-10T20:41:11.781243+0000","last_unstale":"2024-09-10T20:41:14.068543+0000","last_undegraded":"2024-09-10T20:41:14.068543+0000","last_fullsized":"2024-09-10T20:41:14.068543+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:16.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069735+0000","last_change":"2024-09-10T20:41:11.777007+0000","last_active":"2024-09-10T20:41:14.069735+0000","last_peered":"2024-09-10T20:41:14.069735+0000","last_clean":"2024-09-10T20:41:14.069735+0000","last_became_active":"2024-09-10T20:41:11.776733+0000","last_became_peered":"2024-09-10T20:41:11.776733+0000","last_unstale":"2024-09-10T20:41:14.069735+0000","last_undegraded":"2024-09-10T20:41:14.069735+0000","last_fullsized":"2024-09-10T20:41:14.069735+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:33.889519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068870+0000","last_change":"2024-09-10T20:41:11.782814+0000","last_active":"2024-09-10T20:41:14.068870+0000","last_peered":"2024-09-10T20:41:14.068870+0000","last_clean":"2024-09-10T20:41:14.068870+0000","last_became_active":"2024-09-10T20:41:11.780960+0000","last_became_peered":"2024-09-10T20:41:11.780960+0000","last_unstale":"2024-09-10T20:41:14.068870+0000","last_undegraded":"2024-09-10T20:41:14.068870+0000","last_fullsized":"2024-09-10T20:41:14.068870+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:33.889519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069539+0000","last_change":"2024-09-10T20:41:11.784427+0000","last_active":"2024-09-10T20:41:14.069539+0000","last_peered":"2024-09-10T20:41:14.069539+0000","last_clean":"2024-09-10T20:41:14.069539+0000","last_became_active":"2024-09-10T20:41:11.784149+0000","last_became_peered":"2024-09-10T20:41:11.784149+0000","last_unstale":"2024-09-10T20:41:14.069539+0000","last_undegraded":"2024-09-10T20:41:14.069539+0000","last_fullsized":"2024-09-10T20:41:14.069539+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:48.609850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.783054+0000","last_change":"2024-09-10T20:41:11.785044+0000","last_active":"2024-09-10T20:41:12.783054+0000","last_peered":"2024-09-10T20:41:12.783054+0000","last_clean":"2024-09-10T20:41:12.783054+0000","last_became_active":"2024-09-10T20:41:11.784769+0000","last_became_peered":"2024-09-10T20:41:11.784769+0000","last_unstale":"2024-09-10T20:41:12.783054+0000","last_undegraded":"2024-09-10T20:41:12.783054+0000","last_fullsized":"2024-09-10T20:41:12.783054+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.775245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068229+0000","last_change":"2024-09-10T20:41:11.778315+0000","last_active":"2024-09-10T20:41:14.068229+0000","last_peered":"2024-09-10T20:41:14.068229+0000","last_clean":"2024-09-10T20:41:14.068229+0000","last_became_active":"2024-09-10T20:41:11.778030+0000","last_became_peered":"2024-09-10T20:41:11.778030+0000","last_unstale":"2024-09-10T20:41:14.068229+0000","last_undegraded":"2024-09-10T20:41:14.068229+0000","last_fullsized":"2024-09-10T20:41:14.068229+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:28.158543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782271+0000","last_change":"2024-09-10T20:41:11.777891+0000","last_active":"2024-09-10T20:41:12.782271+0000","last_peered":"2024-09-10T20:41:12.782271+0000","last_clean":"2024-09-10T20:41:12.782271+0000","last_became_active":"2024-09-10T20:41:11.777456+0000","last_became_peered":"2024-09-10T20:41:11.777456+0000","last_unstale":"2024-09-10T20:41:12.782271+0000","last_undegraded":"2024-09-10T20:41:12.782271+0000","last_fullsized":"2024-09-10T20:41:12.782271+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:18:27.727332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777446+0000","last_change":"2024-09-10T20:41:11.792592+0000","last_active":"2024-09-10T20:41:12.777446+0000","last_peered":"2024-09-10T20:41:12.777446+0000","last_clean":"2024-09-10T20:41:12.777446+0000","last_became_active":"2024-09-10T20:41:11.792335+0000","last_became_peered":"2024-09-10T20:41:11.792335+0000","last_unstale":"2024-09-10T20:41:12.777446+0000","last_undegraded":"2024-09-10T20:41:12.777446+0000","last_fullsized":"2024-09-10T20:41:12.777446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:25.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587073+0000","last_change":"2024-09-10T20:41:11.772020+0000","last_active":"2024-09-10T20:41:13.587073+0000","last_peered":"2024-09-10T20:41:13.587073+0000","last_clean":"2024-09-10T20:41:13.587073+0000","last_became_active":"2024-09-10T20:41:11.771705+0000","last_became_peered":"2024-09-10T20:41:11.771705+0000","last_unstale":"2024-09-10T20:41:13.587073+0000","last_undegraded":"2024-09-10T20:41:13.587073+0000","last_fullsized":"2024-09-10T20:41:13.587073+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:35.709377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069255+0000","last_change":"2024-09-10T20:41:11.776664+0000","last_active":"2024-09-10T20:41:14.069255+0000","last_peered":"2024-09-10T20:41:14.069255+0000","last_clean":"2024-09-10T20:41:14.069255+0000","last_became_active":"2024-09-10T20:41:11.776393+0000","last_became_peered":"2024-09-10T20:41:11.776393+0000","last_unstale":"2024-09-10T20:41:14.069255+0000","last_undegraded":"2024-09-10T20:41:14.069255+0000","last_fullsized":"2024-09-10T20:41:14.069255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:37.139676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068495+0000","last_change":"2024-09-10T20:41:11.782264+0000","last_active":"2024-09-10T20:41:14.068495+0000","last_peered":"2024-09-10T20:41:14.068495+0000","last_clean":"2024-09-10T20:41:14.068495+0000","last_became_active":"2024-09-10T20:41:11.781963+0000","last_became_peered":"2024-09-10T20:41:11.781963+0000","last_unstale":"2024-09-10T20:41:14.068495+0000","last_undegraded":"2024-09-10T20:41:14.068495+0000","last_fullsized":"2024-09-10T20:41:14.068495+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:06.499336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069595+0000","last_change":"2024-09-10T20:41:11.776213+0000","last_active":"2024-09-10T20:41:14.069595+0000","last_peered":"2024-09-10T20:41:14.069595+0000","last_clean":"2024-09-10T20:41:14.069595+0000","last_became_active":"2024-09-10T20:41:11.775935+0000","last_became_peered":"2024-09-10T20:41:11.775935+0000","last_unstale":"2024-09-10T20:41:14.069595+0000","last_undegraded":"2024-09-10T20:41:14.069595+0000","last_fullsized":"2024-09-10T20:41:14.069595+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:23.983328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782265+0000","last_change":"2024-09-10T20:41:11.779582+0000","last_active":"2024-09-10T20:41:12.782265+0000","last_peered":"2024-09-10T20:41:12.782265+0000","last_clean":"2024-09-10T20:41:12.782265+0000","last_became_active":"2024-09-10T20:41:11.779302+0000","last_became_peered":"2024-09-10T20:41:11.779302+0000","last_unstale":"2024-09-10T20:41:12.782265+0000","last_undegraded":"2024-09-10T20:41:12.782265+0000","last_fullsized":"2024-09-10T20:41:12.782265+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:15.933215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782175+0000","last_change":"2024-09-10T20:41:11.780788+0000","last_active":"2024-09-10T20:41:12.782175+0000","last_peered":"2024-09-10T20:41:12.782175+0000","last_clean":"2024-09-10T20:41:12.782175+0000","last_became_active":"2024-09-10T20:41:11.780499+0000","last_became_peered":"2024-09-10T20:41:11.780499+0000","last_unstale":"2024-09-10T20:41:12.782175+0000","last_undegraded":"2024-09-10T20:41:12.782175+0000","last_fullsized":"2024-09-10T20:41:12.782175+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:56.908307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777221+0000","last_change":"2024-09-10T20:41:11.785831+0000","last_active":"2024-09-10T20:41:12.777221+0000","last_peered":"2024-09-10T20:41:12.777221+0000","last_clean":"2024-09-10T20:41:12.777221+0000","last_became_active":"2024-09-10T20:41:11.785533+0000","last_became_peered":"2024-09-10T20:41:11.785533+0000","last_unstale":"2024-09-10T20:41:12.777221+0000","last_undegraded":"2024-09-10T20:41:12.777221+0000","last_fullsized":"2024-09-10T20:41:12.777221+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:09.332392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069121+0000","last_change":"2024-09-10T20:41:11.772959+0000","last_active":"2024-09-10T20:41:14.069121+0000","last_peered":"2024-09-10T20:41:14.069121+0000","last_clean":"2024-09-10T20:41:14.069121+0000","last_became_active":"2024-09-10T20:41:11.772679+0000","last_became_peered":"2024-09-10T20:41:11.772679+0000","last_unstale":"2024-09-10T20:41:14.069121+0000","last_undegraded":"2024-09-10T20:41:14.069121+0000","last_fullsized":"2024-09-10T20:41:14.069121+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:34.628073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782009+0000","last_change":"2024-09-10T20:41:11.776571+0000","last_active":"2024-09-10T20:41:12.782009+0000","last_peered":"2024-09-10T20:41:12.782009+0000","last_clean":"2024-09-10T20:41:12.782009+0000","last_became_active":"2024-09-10T20:41:11.775839+0000","last_became_peered":"2024-09-10T20:41:11.775839+0000","last_unstale":"2024-09-10T20:41:12.782009+0000","last_undegraded":"2024-09-10T20:41:12.782009+0000","last_fullsized":"2024-09-10T20:41:12.782009+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:57:05.163085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069258+0000","last_change":"2024-09-10T20:41:11.781270+0000","last_active":"2024-09-10T20:41:14.069258+0000","last_peered":"2024-09-10T20:41:14.069258+0000","last_clean":"2024-09-10T20:41:14.069258+0000","last_became_active":"2024-09-10T20:41:11.780765+0000","last_became_peered":"2024-09-10T20:41:11.780765+0000","last_unstale":"2024-09-10T20:41:14.069258+0000","last_undegraded":"2024-09-10T20:41:14.069258+0000","last_fullsized":"2024-09-10T20:41:14.069258+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:03.826979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782768+0000","last_change":"2024-09-10T20:41:10.767534+0000","last_active":"2024-09-10T20:41:12.782768+0000","last_peered":"2024-09-10T20:41:12.782768+0000","last_clean":"2024-09-10T20:41:12.782768+0000","last_became_active":"2024-09-10T20:41:10.767245+0000","last_became_peered":"2024-09-10T20:41:10.767245+0000","last_unstale":"2024-09-10T20:41:12.782768+0000","last_undegraded":"2024-09-10T20:41:12.782768+0000","last_fullsized":"2024-09-10T20:41:12.782768+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:51:12.527841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069825+0000","last_change":"2024-09-10T20:41:11.785522+0000","last_active":"2024-09-10T20:41:14.069825+0000","last_peered":"2024-09-10T20:41:14.069825+0000","last_clean":"2024-09-10T20:41:14.069825+0000","last_became_active":"2024-09-10T20:41:11.784987+0000","last_became_peered":"2024-09-10T20:41:11.784987+0000","last_unstale":"2024-09-10T20:41:14.069825+0000","last_undegraded":"2024-09-10T20:41:14.069825+0000","last_fullsized":"2024-09-10T20:41:14.069825+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:36:37.265641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069421+0000","last_change":"2024-09-10T20:41:10.780754+0000","last_active":"2024-09-10T20:41:14.069421+0000","last_peered":"2024-09-10T20:41:14.069421+0000","last_clean":"2024-09-10T20:41:14.069421+0000","last_became_active":"2024-09-10T20:41:10.780511+0000","last_became_peered":"2024-09-10T20:41:10.780511+0000","last_unstale":"2024-09-10T20:41:14.069421+0000","last_undegraded":"2024-09-10T20:41:14.069421+0000","last_fullsized":"2024-09-10T20:41:14.069421+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:07:23.418008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782507+0000","last_change":"2024-09-10T20:41:11.782451+0000","last_active":"2024-09-10T20:41:12.782507+0000","last_peered":"2024-09-10T20:41:12.782507+0000","last_clean":"2024-09-10T20:41:12.782507+0000","last_became_active":"2024-09-10T20:41:11.781998+0000","last_became_peered":"2024-09-10T20:41:11.781998+0000","last_unstale":"2024-09-10T20:41:12.782507+0000","last_undegraded":"2024-09-10T20:41:12.782507+0000","last_fullsized":"2024-09-10T20:41:12.782507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:27:47.283361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068920+0000","last_change":"2024-09-10T20:41:10.779916+0000","last_active":"2024-09-10T20:41:14.068920+0000","last_peered":"2024-09-10T20:41:14.068920+0000","last_clean":"2024-09-10T20:41:14.068920+0000","last_became_active":"2024-09-10T20:41:10.779621+0000","last_became_peered":"2024-09-10T20:41:10.779621+0000","last_unstale":"2024-09-10T20:41:14.068920+0000","last_undegraded":"2024-09-10T20:41:14.068920+0000","last_fullsized":"2024-09-10T20:41:14.068920+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:28.998307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069593+0000","last_change":"2024-09-10T20:41:11.778536+0000","last_active":"2024-09-10T20:41:14.069593+0000","last_peered":"2024-09-10T20:41:14.069593+0000","last_clean":"2024-09-10T20:41:14.069593+0000","last_became_active":"2024-09-10T20:41:11.778261+0000","last_became_peered":"2024-09-10T20:41:11.778261+0000","last_unstale":"2024-09-10T20:41:14.069593+0000","last_undegraded":"2024-09-10T20:41:14.069593+0000","last_fullsized":"2024-09-10T20:41:14.069593+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:30.212316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777630+0000","last_change":"2024-09-10T20:41:10.779236+0000","last_active":"2024-09-10T20:41:12.777630+0000","last_peered":"2024-09-10T20:41:12.777630+0000","last_clean":"2024-09-10T20:41:12.777630+0000","last_became_active":"2024-09-10T20:41:10.778960+0000","last_became_peered":"2024-09-10T20:41:10.778960+0000","last_unstale":"2024-09-10T20:41:12.777630+0000","last_undegraded":"2024-09-10T20:41:12.777630+0000","last_fullsized":"2024-09-10T20:41:12.777630+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:36.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782479+0000","last_change":"2024-09-10T20:41:11.779454+0000","last_active":"2024-09-10T20:41:12.782479+0000","last_peered":"2024-09-10T20:41:12.782479+0000","last_clean":"2024-09-10T20:41:12.782479+0000","last_became_active":"2024-09-10T20:41:11.779112+0000","last_became_peered":"2024-09-10T20:41:11.779112+0000","last_unstale":"2024-09-10T20:41:12.782479+0000","last_undegraded":"2024-09-10T20:41:12.782479+0000","last_fullsized":"2024-09-10T20:41:12.782479+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:39.869888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068326+0000","last_change":"2024-09-10T20:41:10.777277+0000","last_active":"2024-09-10T20:41:14.068326+0000","last_peered":"2024-09-10T20:41:14.068326+0000","last_clean":"2024-09-10T20:41:14.068326+0000","last_became_active":"2024-09-10T20:41:10.776709+0000","last_became_peered":"2024-09-10T20:41:10.776709+0000","last_unstale":"2024-09-10T20:41:14.068326+0000","last_undegraded":"2024-09-10T20:41:14.068326+0000","last_fullsized":"2024-09-10T20:41:14.068326+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:03:11.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.783337+0000","last_change":"2024-09-10T20:41:11.779336+0000","last_active":"2024-09-10T20:41:12.783337+0000","last_peered":"2024-09-10T20:41:12.783337+0000","last_clean":"2024-09-10T20:41:12.783337+0000","last_became_active":"2024-09-10T20:41:11.779073+0000","last_became_peered":"2024-09-10T20:41:11.779073+0000","last_unstale":"2024-09-10T20:41:12.783337+0000","last_undegraded":"2024-09-10T20:41:12.783337+0000","last_fullsized":"2024-09-10T20:41:12.783337+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:41.567182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068840+0000","last_change":"2024-09-10T20:41:10.783140+0000","last_active":"2024-09-10T20:41:14.068840+0000","last_peered":"2024-09-10T20:41:14.068840+0000","last_clean":"2024-09-10T20:41:14.068840+0000","last_became_active":"2024-09-10T20:41:10.782851+0000","last_became_peered":"2024-09-10T20:41:10.782851+0000","last_unstale":"2024-09-10T20:41:14.068840+0000","last_undegraded":"2024-09-10T20:41:14.068840+0000","last_fullsized":"2024-09-10T20:41:14.068840+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:34.707170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068065+0000","last_change":"2024-09-10T20:41:11.780459+0000","last_active":"2024-09-10T20:41:14.068065+0000","last_peered":"2024-09-10T20:41:14.068065+0000","last_clean":"2024-09-10T20:41:14.068065+0000","last_became_active":"2024-09-10T20:41:11.780174+0000","last_became_peered":"2024-09-10T20:41:11.780174+0000","last_unstale":"2024-09-10T20:41:14.068065+0000","last_undegraded":"2024-09-10T20:41:14.068065+0000","last_fullsized":"2024-09-10T20:41:14.068065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:03.826979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587071+0000","last_change":"2024-09-10T20:41:10.780820+0000","last_active":"2024-09-10T20:41:13.587071+0000","last_peered":"2024-09-10T20:41:13.587071+0000","last_clean":"2024-09-10T20:41:13.587071+0000","last_became_active":"2024-09-10T20:41:10.780510+0000","last_became_peered":"2024-09-10T20:41:10.780510+0000","last_unstale":"2024-09-10T20:41:13.587071+0000","last_undegraded":"2024-09-10T20:41:13.587071+0000","last_fullsized":"2024-09-10T20:41:13.587071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:40.149041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068626+0000","last_change":"2024-09-10T20:41:11.783906+0000","last_active":"2024-09-10T20:41:14.068626+0000","last_peered":"2024-09-10T20:41:14.068626+0000","last_clean":"2024-09-10T20:41:14.068626+0000","last_became_active":"2024-09-10T20:41:11.781989+0000","last_became_peered":"2024-09-10T20:41:11.781989+0000","last_unstale":"2024-09-10T20:41:14.068626+0000","last_undegraded":"2024-09-10T20:41:14.068626+0000","last_fullsized":"2024-09-10T20:41:14.068626+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:55.811802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.067747+0000","last_change":"2024-09-10T20:41:10.775128+0000","last_active":"2024-09-10T20:41:14.067747+0000","last_peered":"2024-09-10T20:41:14.067747+0000","last_clean":"2024-09-10T20:41:14.067747+0000","last_became_active":"2024-09-10T20:41:10.774832+0000","last_became_peered":"2024-09-10T20:41:10.774832+0000","last_unstale":"2024-09-10T20:41:14.067747+0000","last_undegraded":"2024-09-10T20:41:14.067747+0000","last_fullsized":"2024-09-10T20:41:14.067747+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:15:13.077576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068555+0000","last_change":"2024-09-10T20:41:11.784003+0000","last_active":"2024-09-10T20:41:14.068555+0000","last_peered":"2024-09-10T20:41:14.068555+0000","last_clean":"2024-09-10T20:41:14.068555+0000","last_became_active":"2024-09-10T20:41:11.783732+0000","last_became_peered":"2024-09-10T20:41:11.783732+0000","last_unstale":"2024-09-10T20:41:14.068555+0000","last_undegraded":"2024-09-10T20:41:14.068555+0000","last_fullsized":"2024-09-10T20:41:14.068555+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:30.212316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587436+0000","last_change":"2024-09-10T20:41:10.778435+0000","last_active":"2024-09-10T20:41:13.587436+0000","last_peered":"2024-09-10T20:41:13.587436+0000","last_clean":"2024-09-10T20:41:13.587436+0000","last_became_active":"2024-09-10T20:41:10.778171+0000","last_became_peered":"2024-09-10T20:41:10.778171+0000","last_unstale":"2024-09-10T20:41:13.587436+0000","last_undegraded":"2024-09-10T20:41:13.587436+0000","last_fullsized":"2024-09-10T20:41:13.587436+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:43:07.175378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068937+0000","last_change":"2024-09-10T20:41:11.787396+0000","last_active":"2024-09-10T20:41:14.068937+0000","last_peered":"2024-09-10T20:41:14.068937+0000","last_clean":"2024-09-10T20:41:14.068937+0000","last_became_active":"2024-09-10T20:41:11.786811+0000","last_became_peered":"2024-09-10T20:41:11.786811+0000","last_unstale":"2024-09-10T20:41:14.068937+0000","last_undegraded":"2024-09-10T20:41:14.068937+0000","last_fullsized":"2024-09-10T20:41:14.068937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:30.610959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782813+0000","last_change":"2024-09-10T20:41:10.776180+0000","last_active":"2024-09-10T20:41:12.782813+0000","last_peered":"2024-09-10T20:41:12.782813+0000","last_clean":"2024-09-10T20:41:12.782813+0000","last_became_active":"2024-09-10T20:41:10.775933+0000","last_became_peered":"2024-09-10T20:41:10.775933+0000","last_unstale":"2024-09-10T20:41:12.782813+0000","last_undegraded":"2024-09-10T20:41:12.782813+0000","last_fullsized":"2024-09-10T20:41:12.782813+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:31.931550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782708+0000","last_change":"2024-09-10T20:41:11.782283+0000","last_active":"2024-09-10T20:41:12.782708+0000","last_peered":"2024-09-10T20:41:12.782708+0000","last_clean":"2024-09-10T20:41:12.782708+0000","last_became_active":"2024-09-10T20:41:11.780342+0000","last_became_peered":"2024-09-10T20:41:11.780342+0000","last_unstale":"2024-09-10T20:41:12.782708+0000","last_undegraded":"2024-09-10T20:41:12.782708+0000","last_fullsized":"2024-09-10T20:41:12.782708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:16:59.917402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777491+0000","last_change":"2024-09-10T20:41:10.780029+0000","last_active":"2024-09-10T20:41:12.777491+0000","last_peered":"2024-09-10T20:41:12.777491+0000","last_clean":"2024-09-10T20:41:12.777491+0000","last_became_active":"2024-09-10T20:41:10.779741+0000","last_became_peered":"2024-09-10T20:41:10.779741+0000","last_unstale":"2024-09-10T20:41:12.777491+0000","last_undegraded":"2024-09-10T20:41:12.777491+0000","last_fullsized":"2024-09-10T20:41:12.777491+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:16:58.724254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777329+0000","last_change":"2024-09-10T20:41:11.784872+0000","last_active":"2024-09-10T20:41:12.777329+0000","last_peered":"2024-09-10T20:41:12.777329+0000","last_clean":"2024-09-10T20:41:12.777329+0000","last_became_active":"2024-09-10T20:41:11.784533+0000","last_became_peered":"2024-09-10T20:41:11.784533+0000","last_unstale":"2024-09-10T20:41:12.777329+0000","last_undegraded":"2024-09-10T20:41:12.777329+0000","last_fullsized":"2024-09-10T20:41:12.777329+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:16.712376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068155+0000","last_change":"2024-09-10T20:41:10.783219+0000","last_active":"2024-09-10T20:41:14.068155+0000","last_peered":"2024-09-10T20:41:14.068155+0000","last_clean":"2024-09-10T20:41:14.068155+0000","last_became_active":"2024-09-10T20:41:10.782877+0000","last_became_peered":"2024-09-10T20:41:10.782877+0000","last_unstale":"2024-09-10T20:41:14.068155+0000","last_undegraded":"2024-09-10T20:41:14.068155+0000","last_fullsized":"2024-09-10T20:41:14.068155+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:16:12.399635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587505+0000","last_change":"2024-09-10T20:41:11.769033+0000","last_active":"2024-09-10T20:41:13.587505+0000","last_peered":"2024-09-10T20:41:13.587505+0000","last_clean":"2024-09-10T20:41:13.587505+0000","last_became_active":"2024-09-10T20:41:11.768709+0000","last_became_peered":"2024-09-10T20:41:11.768709+0000","last_unstale":"2024-09-10T20:41:13.587505+0000","last_undegraded":"2024-09-10T20:41:13.587505+0000","last_fullsized":"2024-09-10T20:41:13.587505+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:30.338175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782282+0000","last_change":"2024-09-10T20:41:10.776160+0000","last_active":"2024-09-10T20:41:12.782282+0000","last_peered":"2024-09-10T20:41:12.782282+0000","last_clean":"2024-09-10T20:41:12.782282+0000","last_became_active":"2024-09-10T20:41:10.775883+0000","last_became_peered":"2024-09-10T20:41:10.775883+0000","last_unstale":"2024-09-10T20:41:12.782282+0000","last_undegraded":"2024-09-10T20:41:12.782282+0000","last_fullsized":"2024-09-10T20:41:12.782282+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:55:07.341606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068356+0000","last_change":"2024-09-10T20:41:11.778771+0000","last_active":"2024-09-10T20:41:14.068356+0000","last_peered":"2024-09-10T20:41:14.068356+0000","last_clean":"2024-09-10T20:41:14.068356+0000","last_became_active":"2024-09-10T20:41:11.778471+0000","last_became_peered":"2024-09-10T20:41:11.778471+0000","last_unstale":"2024-09-10T20:41:14.068356+0000","last_undegraded":"2024-09-10T20:41:14.068356+0000","last_fullsized":"2024-09-10T20:41:14.068356+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:30.212316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777377+0000","last_change":"2024-09-10T20:41:10.779762+0000","last_active":"2024-09-10T20:41:12.777377+0000","last_peered":"2024-09-10T20:41:12.777377+0000","last_clean":"2024-09-10T20:41:12.777377+0000","last_became_active":"2024-09-10T20:41:10.779444+0000","last_became_peered":"2024-09-10T20:41:10.779444+0000","last_unstale":"2024-09-10T20:41:12.777377+0000","last_undegraded":"2024-09-10T20:41:12.777377+0000","last_fullsized":"2024-09-10T20:41:12.777377+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:15.732507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587258+0000","last_change":"2024-09-10T20:41:11.776417+0000","last_active":"2024-09-10T20:41:13.587258+0000","last_peered":"2024-09-10T20:41:13.587258+0000","last_clean":"2024-09-10T20:41:13.587258+0000","last_became_active":"2024-09-10T20:41:11.776150+0000","last_became_peered":"2024-09-10T20:41:11.776150+0000","last_unstale":"2024-09-10T20:41:13.587258+0000","last_undegraded":"2024-09-10T20:41:13.587258+0000","last_fullsized":"2024-09-10T20:41:13.587258+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:30.000514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777762+0000","last_change":"2024-09-10T20:41:10.776282+0000","last_active":"2024-09-10T20:41:12.777762+0000","last_peered":"2024-09-10T20:41:12.777762+0000","last_clean":"2024-09-10T20:41:12.777762+0000","last_became_active":"2024-09-10T20:41:10.775927+0000","last_became_peered":"2024-09-10T20:41:10.775927+0000","last_unstale":"2024-09-10T20:41:12.777762+0000","last_undegraded":"2024-09-10T20:41:12.777762+0000","last_fullsized":"2024-09-10T20:41:12.777762+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:19.769899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777647+0000","last_change":"2024-09-10T20:41:11.791837+0000","last_active":"2024-09-10T20:41:12.777647+0000","last_peered":"2024-09-10T20:41:12.777647+0000","last_clean":"2024-09-10T20:41:12.777647+0000","last_became_active":"2024-09-10T20:41:11.791537+0000","last_became_peered":"2024-09-10T20:41:11.791537+0000","last_unstale":"2024-09-10T20:41:12.777647+0000","last_undegraded":"2024-09-10T20:41:12.777647+0000","last_fullsized":"2024-09-10T20:41:12.777647+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:00.199147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777224+0000","last_change":"2024-09-10T20:41:10.780541+0000","last_active":"2024-09-10T20:41:12.777224+0000","last_peered":"2024-09-10T20:41:12.777224+0000","last_clean":"2024-09-10T20:41:12.777224+0000","last_became_active":"2024-09-10T20:41:10.780251+0000","last_became_peered":"2024-09-10T20:41:10.780251+0000","last_unstale":"2024-09-10T20:41:12.777224+0000","last_undegraded":"2024-09-10T20:41:12.777224+0000","last_fullsized":"2024-09-10T20:41:12.777224+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:08.817507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068935+0000","last_change":"2024-09-10T20:41:11.783450+0000","last_active":"2024-09-10T20:41:14.068935+0000","last_peered":"2024-09-10T20:41:14.068935+0000","last_clean":"2024-09-10T20:41:14.068935+0000","last_became_active":"2024-09-10T20:41:11.783182+0000","last_became_peered":"2024-09-10T20:41:11.783182+0000","last_unstale":"2024-09-10T20:41:14.068935+0000","last_undegraded":"2024-09-10T20:41:14.068935+0000","last_fullsized":"2024-09-10T20:41:14.068935+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:36:37.265641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069120+0000","last_change":"2024-09-10T20:41:10.779110+0000","last_active":"2024-09-10T20:41:14.069120+0000","last_peered":"2024-09-10T20:41:14.069120+0000","last_clean":"2024-09-10T20:41:14.069120+0000","last_became_active":"2024-09-10T20:41:10.777768+0000","last_became_peered":"2024-09-10T20:41:10.777768+0000","last_unstale":"2024-09-10T20:41:14.069120+0000","last_undegraded":"2024-09-10T20:41:14.069120+0000","last_fullsized":"2024-09-10T20:41:14.069120+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:38.901865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068963+0000","last_change":"2024-09-10T20:41:11.787100+0000","last_active":"2024-09-10T20:41:14.068963+0000","last_peered":"2024-09-10T20:41:14.068963+0000","last_clean":"2024-09-10T20:41:14.068963+0000","last_became_active":"2024-09-10T20:41:11.786468+0000","last_became_peered":"2024-09-10T20:41:11.786468+0000","last_unstale":"2024-09-10T20:41:14.068963+0000","last_undegraded":"2024-09-10T20:41:14.068963+0000","last_fullsized":"2024-09-10T20:41:14.068963+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:16:46.080835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782929+0000","last_change":"2024-09-10T20:41:10.769266+0000","last_active":"2024-09-10T20:41:12.782929+0000","last_peered":"2024-09-10T20:41:12.782929+0000","last_clean":"2024-09-10T20:41:12.782929+0000","last_became_active":"2024-09-10T20:41:10.768873+0000","last_became_peered":"2024-09-10T20:41:10.768873+0000","last_unstale":"2024-09-10T20:41:12.782929+0000","last_undegraded":"2024-09-10T20:41:12.782929+0000","last_fullsized":"2024-09-10T20:41:12.782929+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:29:44.163345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069268+0000","last_change":"2024-09-10T20:41:11.776742+0000","last_active":"2024-09-10T20:41:14.069268+0000","last_peered":"2024-09-10T20:41:14.069268+0000","last_clean":"2024-09-10T20:41:14.069268+0000","last_became_active":"2024-09-10T20:41:11.776514+0000","last_became_peered":"2024-09-10T20:41:11.776514+0000","last_unstale":"2024-09-10T20:41:14.069268+0000","last_undegraded":"2024-09-10T20:41:14.069268+0000","last_fullsized":"2024-09-10T20:41:14.069268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:16:28.951190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068847+0000","last_change":"2024-09-10T20:41:10.781773+0000","last_active":"2024-09-10T20:41:14.068847+0000","last_peered":"2024-09-10T20:41:14.068847+0000","last_clean":"2024-09-10T20:41:14.068847+0000","last_became_active":"2024-09-10T20:41:10.781293+0000","last_became_peered":"2024-09-10T20:41:10.781293+0000","last_unstale":"2024-09-10T20:41:14.068847+0000","last_undegraded":"2024-09-10T20:41:14.068847+0000","last_fullsized":"2024-09-10T20:41:14.068847+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:06:25.067185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.777472+0000","last_change":"2024-09-10T20:41:11.785480+0000","last_active":"2024-09-10T20:41:12.777472+0000","last_peered":"2024-09-10T20:41:12.777472+0000","last_clean":"2024-09-10T20:41:12.777472+0000","last_became_active":"2024-09-10T20:41:11.785194+0000","last_became_peered":"2024-09-10T20:41:11.785194+0000","last_unstale":"2024-09-10T20:41:12.777472+0000","last_undegraded":"2024-09-10T20:41:12.777472+0000","last_fullsized":"2024-09-10T20:41:12.777472+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:25:31.083645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782772+0000","last_change":"2024-09-10T20:41:10.778309+0000","last_active":"2024-09-10T20:41:12.782772+0000","last_peered":"2024-09-10T20:41:12.782772+0000","last_clean":"2024-09-10T20:41:12.782772+0000","last_became_active":"2024-09-10T20:41:10.778008+0000","last_became_peered":"2024-09-10T20:41:10.778008+0000","last_unstale":"2024-09-10T20:41:12.782772+0000","last_undegraded":"2024-09-10T20:41:12.782772+0000","last_fullsized":"2024-09-10T20:41:12.782772+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:12.540221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587132+0000","last_change":"2024-09-10T20:41:11.778680+0000","last_active":"2024-09-10T20:41:13.587132+0000","last_peered":"2024-09-10T20:41:13.587132+0000","last_clean":"2024-09-10T20:41:13.587132+0000","last_became_active":"2024-09-10T20:41:11.778408+0000","last_became_peered":"2024-09-10T20:41:11.778408+0000","last_unstale":"2024-09-10T20:41:13.587132+0000","last_undegraded":"2024-09-10T20:41:13.587132+0000","last_fullsized":"2024-09-10T20:41:13.587132+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:45:12.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.783267+0000","last_change":"2024-09-10T20:41:10.777646+0000","last_active":"2024-09-10T20:41:12.783267+0000","last_peered":"2024-09-10T20:41:12.783267+0000","last_clean":"2024-09-10T20:41:12.783267+0000","last_became_active":"2024-09-10T20:41:10.777313+0000","last_became_peered":"2024-09-10T20:41:10.777313+0000","last_unstale":"2024-09-10T20:41:12.783267+0000","last_undegraded":"2024-09-10T20:41:12.783267+0000","last_fullsized":"2024-09-10T20:41:12.783267+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:19.472921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782619+0000","last_change":"2024-09-10T20:41:11.781471+0000","last_active":"2024-09-10T20:41:12.782619+0000","last_peered":"2024-09-10T20:41:12.782619+0000","last_clean":"2024-09-10T20:41:12.782619+0000","last_became_active":"2024-09-10T20:41:11.781166+0000","last_became_peered":"2024-09-10T20:41:11.781166+0000","last_unstale":"2024-09-10T20:41:12.782619+0000","last_undegraded":"2024-09-10T20:41:12.782619+0000","last_fullsized":"2024-09-10T20:41:12.782619+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:53:11.538705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782550+0000","last_change":"2024-09-10T20:41:10.769946+0000","last_active":"2024-09-10T20:41:12.782550+0000","last_peered":"2024-09-10T20:41:12.782550+0000","last_clean":"2024-09-10T20:41:12.782550+0000","last_became_active":"2024-09-10T20:41:10.769665+0000","last_became_peered":"2024-09-10T20:41:10.769665+0000","last_unstale":"2024-09-10T20:41:12.782550+0000","last_undegraded":"2024-09-10T20:41:12.782550+0000","last_fullsized":"2024-09-10T20:41:12.782550+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:01:33.615958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782125+0000","last_change":"2024-09-10T20:41:11.781698+0000","last_active":"2024-09-10T20:41:12.782125+0000","last_peered":"2024-09-10T20:41:12.782125+0000","last_clean":"2024-09-10T20:41:12.782125+0000","last_became_active":"2024-09-10T20:41:11.780032+0000","last_became_peered":"2024-09-10T20:41:11.780032+0000","last_unstale":"2024-09-10T20:41:12.782125+0000","last_undegraded":"2024-09-10T20:41:12.782125+0000","last_fullsized":"2024-09-10T20:41:12.782125+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:19.145266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587398+0000","last_change":"2024-09-10T20:41:10.775679+0000","last_active":"2024-09-10T20:41:13.587398+0000","last_peered":"2024-09-10T20:41:13.587398+0000","last_clean":"2024-09-10T20:41:13.587398+0000","last_became_active":"2024-09-10T20:41:10.775233+0000","last_became_peered":"2024-09-10T20:41:10.775233+0000","last_unstale":"2024-09-10T20:41:13.587398+0000","last_undegraded":"2024-09-10T20:41:13.587398+0000","last_fullsized":"2024-09-10T20:41:13.587398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:55.298930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069023+0000","last_change":"2024-09-10T20:41:11.782480+0000","last_active":"2024-09-10T20:41:14.069023+0000","last_peered":"2024-09-10T20:41:14.069023+0000","last_clean":"2024-09-10T20:41:14.069023+0000","last_became_active":"2024-09-10T20:41:11.780518+0000","last_became_peered":"2024-09-10T20:41:11.780518+0000","last_unstale":"2024-09-10T20:41:14.069023+0000","last_undegraded":"2024-09-10T20:41:14.069023+0000","last_fullsized":"2024-09-10T20:41:14.069023+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:22:26.577335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069226+0000","last_change":"2024-09-10T20:41:10.779846+0000","last_active":"2024-09-10T20:41:14.069226+0000","last_peered":"2024-09-10T20:41:14.069226+0000","last_clean":"2024-09-10T20:41:14.069226+0000","last_became_active":"2024-09-10T20:41:10.778661+0000","last_became_peered":"2024-09-10T20:41:10.778661+0000","last_unstale":"2024-09-10T20:41:14.069226+0000","last_undegraded":"2024-09-10T20:41:14.069226+0000","last_fullsized":"2024-09-10T20:41:14.069226+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:26:47.411890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069162+0000","last_change":"2024-09-10T20:41:11.781270+0000","last_active":"2024-09-10T20:41:14.069162+0000","last_peered":"2024-09-10T20:41:14.069162+0000","last_clean":"2024-09-10T20:41:14.069162+0000","last_became_active":"2024-09-10T20:41:11.780978+0000","last_became_peered":"2024-09-10T20:41:11.780978+0000","last_unstale":"2024-09-10T20:41:14.069162+0000","last_undegraded":"2024-09-10T20:41:14.069162+0000","last_fullsized":"2024-09-10T20:41:14.069162+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:47:22.409164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069313+0000","last_change":"2024-09-10T20:41:10.784277+0000","last_active":"2024-09-10T20:41:14.069313+0000","last_peered":"2024-09-10T20:41:14.069313+0000","last_clean":"2024-09-10T20:41:14.069313+0000","last_became_active":"2024-09-10T20:41:10.783976+0000","last_became_peered":"2024-09-10T20:41:10.783976+0000","last_unstale":"2024-09-10T20:41:14.069313+0000","last_undegraded":"2024-09-10T20:41:14.069313+0000","last_fullsized":"2024-09-10T20:41:14.069313+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:33:19.399203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069205+0000","last_change":"2024-09-10T20:41:11.785878+0000","last_active":"2024-09-10T20:41:14.069205+0000","last_peered":"2024-09-10T20:41:14.069205+0000","last_clean":"2024-09-10T20:41:14.069205+0000","last_became_active":"2024-09-10T20:41:11.785609+0000","last_became_peered":"2024-09-10T20:41:11.785609+0000","last_unstale":"2024-09-10T20:41:14.069205+0000","last_undegraded":"2024-09-10T20:41:14.069205+0000","last_fullsized":"2024-09-10T20:41:14.069205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:04.287003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.587302+0000","last_change":"2024-09-10T20:41:10.774717+0000","last_active":"2024-09-10T20:41:13.587302+0000","last_peered":"2024-09-10T20:41:13.587302+0000","last_clean":"2024-09-10T20:41:13.587302+0000","last_became_active":"2024-09-10T20:41:10.774273+0000","last_became_peered":"2024-09-10T20:41:10.774273+0000","last_unstale":"2024-09-10T20:41:13.587302+0000","last_undegraded":"2024-09-10T20:41:13.587302+0000","last_fullsized":"2024-09-10T20:41:13.587302+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:34:38.031077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.783053+0000","last_change":"2024-09-10T20:41:11.779506+0000","last_active":"2024-09-10T20:41:12.783053+0000","last_peered":"2024-09-10T20:41:12.783053+0000","last_clean":"2024-09-10T20:41:12.783053+0000","last_became_active":"2024-09-10T20:41:11.779220+0000","last_became_peered":"2024-09-10T20:41:11.779220+0000","last_unstale":"2024-09-10T20:41:12.783053+0000","last_undegraded":"2024-09-10T20:41:12.783053+0000","last_fullsized":"2024-09-10T20:41:12.783053+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:19:54.183422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068622+0000","last_change":"2024-09-10T20:41:10.777536+0000","last_active":"2024-09-10T20:41:14.068622+0000","last_peered":"2024-09-10T20:41:14.068622+0000","last_clean":"2024-09-10T20:41:14.068622+0000","last_became_active":"2024-09-10T20:41:10.777247+0000","last_became_peered":"2024-09-10T20:41:10.777247+0000","last_unstale":"2024-09-10T20:41:14.068622+0000","last_undegraded":"2024-09-10T20:41:14.068622+0000","last_fullsized":"2024-09-10T20:41:14.068622+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:14.158027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068842+0000","last_change":"2024-09-10T20:41:11.787205+0000","last_active":"2024-09-10T20:41:14.068842+0000","last_peered":"2024-09-10T20:41:14.068842+0000","last_clean":"2024-09-10T20:41:14.068842+0000","last_became_active":"2024-09-10T20:41:11.786600+0000","last_became_peered":"2024-09-10T20:41:11.786600+0000","last_unstale":"2024-09-10T20:41:14.068842+0000","last_undegraded":"2024-09-10T20:41:14.068842+0000","last_fullsized":"2024-09-10T20:41:14.068842+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:35:59.079151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068857+0000","last_change":"2024-09-10T20:41:10.779604+0000","last_active":"2024-09-10T20:41:14.068857+0000","last_peered":"2024-09-10T20:41:14.068857+0000","last_clean":"2024-09-10T20:41:14.068857+0000","last_became_active":"2024-09-10T20:41:10.778647+0000","last_became_peered":"2024-09-10T20:41:10.778647+0000","last_unstale":"2024-09-10T20:41:14.068857+0000","last_undegraded":"2024-09-10T20:41:14.068857+0000","last_fullsized":"2024-09-10T20:41:14.068857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:37:39.411088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068314+0000","last_change":"2024-09-10T20:41:11.783742+0000","last_active":"2024-09-10T20:41:14.068314+0000","last_peered":"2024-09-10T20:41:14.068314+0000","last_clean":"2024-09-10T20:41:14.068314+0000","last_became_active":"2024-09-10T20:41:11.783353+0000","last_became_peered":"2024-09-10T20:41:11.783353+0000","last_unstale":"2024-09-10T20:41:14.068314+0000","last_undegraded":"2024-09-10T20:41:14.068314+0000","last_fullsized":"2024-09-10T20:41:14.068314+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:03.826979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068234+0000","last_change":"2024-09-10T20:41:10.779772+0000","last_active":"2024-09-10T20:41:14.068234+0000","last_peered":"2024-09-10T20:41:14.068234+0000","last_clean":"2024-09-10T20:41:14.068234+0000","last_became_active":"2024-09-10T20:41:10.779487+0000","last_became_peered":"2024-09-10T20:41:10.779487+0000","last_unstale":"2024-09-10T20:41:14.068234+0000","last_undegraded":"2024-09-10T20:41:14.068234+0000","last_fullsized":"2024-09-10T20:41:14.068234+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:19.467627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782634+0000","last_change":"2024-09-10T20:41:11.786157+0000","last_active":"2024-09-10T20:41:12.782634+0000","last_peered":"2024-09-10T20:41:12.782634+0000","last_clean":"2024-09-10T20:41:12.782634+0000","last_became_active":"2024-09-10T20:41:11.785797+0000","last_became_peered":"2024-09-10T20:41:11.785797+0000","last_unstale":"2024-09-10T20:41:12.782634+0000","last_undegraded":"2024-09-10T20:41:12.782634+0000","last_fullsized":"2024-09-10T20:41:12.782634+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:52:35.052931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.068694+0000","last_change":"2024-09-10T20:41:10.778389+0000","last_active":"2024-09-10T20:41:14.068694+0000","last_peered":"2024-09-10T20:41:14.068694+0000","last_clean":"2024-09-10T20:41:14.068694+0000","last_became_active":"2024-09-10T20:41:10.778143+0000","last_became_peered":"2024-09-10T20:41:10.778143+0000","last_unstale":"2024-09-10T20:41:14.068694+0000","last_undegraded":"2024-09-10T20:41:14.068694+0000","last_fullsized":"2024-09-10T20:41:14.068694+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:13.209862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.069131+0000","last_change":"2024-09-10T20:41:11.775883+0000","last_active":"2024-09-10T20:41:14.069131+0000","last_peered":"2024-09-10T20:41:14.069131+0000","last_clean":"2024-09-10T20:41:14.069131+0000","last_became_active":"2024-09-10T20:41:11.775610+0000","last_became_peered":"2024-09-10T20:41:11.775610+0000","last_unstale":"2024-09-10T20:41:14.069131+0000","last_undegraded":"2024-09-10T20:41:14.069131+0000","last_fullsized":"2024-09-10T20:41:14.069131+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:55:40.801915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.782697+0000","last_change":"2024-09-10T20:41:10.777477+0000","last_active":"2024-09-10T20:41:12.782697+0000","last_peered":"2024-09-10T20:41:12.782697+0000","last_clean":"2024-09-10T20:41:12.782697+0000","last_became_active":"2024-09-10T20:41:10.776658+0000","last_became_peered":"2024-09-10T20:41:10.776658+0000","last_unstale":"2024-09-10T20:41:12.782697+0000","last_undegraded":"2024-09-10T20:41:12.782697+0000","last_fullsized":"2024-09-10T20:41:12.782697+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:31.180645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:12.783233+0000","last_change":"2024-09-10T20:41:11.783957+0000","last_active":"2024-09-10T20:41:12.783233+0000","last_peered":"2024-09-10T20:41:12.783233+0000","last_clean":"2024-09-10T20:41:12.783233+0000","last_became_active":"2024-09-10T20:41:11.783639+0000","last_became_peered":"2024-09-10T20:41:11.783639+0000","last_unstale":"2024-09-10T20:41:12.783233+0000","last_undegraded":"2024-09-10T20:41:12.783233+0000","last_fullsized":"2024-09-10T20:41:12.783233+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:10.754365+0000","last_clean_scrub_stamp":"2024-09-10T20:41:10.754365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:31.826862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:14.067827+0000","last_change":"2024-09-10T20:41:10.779842+0000","last_active":"2024-09-10T20:41:14.067827+0000","last_peered":"2024-09-10T20:41:14.067827+0000","last_clean":"2024-09-10T20:41:14.067827+0000","last_became_active":"2024-09-10T20:41:10.779559+0000","last_became_peered":"2024-09-10T20:41:10.779559+0000","last_unstale":"2024-09-10T20:41:14.067827+0000","last_undegraded":"2024-09-10T20:41:14.067827+0000","last_fullsized":"2024-09-10T20:41:14.067827+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:09.752159+0000","last_clean_scrub_stamp":"2024-09-10T20:41:09.752159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:21:35.297865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":175,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T20:41:13.586852+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:41:13.586852+0000","last_peered":"2024-09-10T20:41:13.586852+0000","last_clean":"2024-09-10T20:41:13.586852+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:41:13.586852+0000","last_undegraded":"2024-09-10T20:41:13.586852+0000","last_fullsized":"2024-09-10T20:41:13.586852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509483,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":667648,"data_stored":274869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509482,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":276551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509482,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1261568,"data_stored":865729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":675840,"data_stored":276391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542186,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":671744,"data_stored":274871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542187,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1261568,"data_stored":865181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542185,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":671744,"data_stored":275395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:41:16.161 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:41:16.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:41:16.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 -- 172.21.15.28:0/223168222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c15e130 msgr2=0x7f742c0d4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 --2- 172.21.15.28:0/223168222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c15e130 0x7f742c0d4b60 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f742c08e230 tx=0x7f741c02a4d0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 -- 172.21.15.28:0/223168222 shutdown_connections 2024-09-10T20:41:16.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 --2- 172.21.15.28:0/223168222 >> v2:172.21.15.159:3300/0 conn(0x7f742c0b0c90 0x7f742c0cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 --2- 172.21.15.28:0/223168222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c15e130 0x7f742c0d4b60 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 -- 172.21.15.28:0/223168222 >> 172.21.15.28:0/223168222 conn(0x7f742c0d4040 msgr2=0x7f742c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:16.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 -- 172.21.15.28:0/223168222 shutdown_connections 2024-09-10T20:41:16.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.306+0000 7f7430e60640 1 -- 172.21.15.28:0/223168222 wait complete. 2024-09-10T20:41:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.308+0000 7f7430e60640 1 Processor -- start 2024-09-10T20:41:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.308+0000 7f7430e60640 1 -- start start 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7430e60640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f742c0b0c90 0x7f742c0c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7430e60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c0b1100 0x7f742c15af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7430e60640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f742c15e130 0x7f742c15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7430e60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f742c0cfec0 con 0x7f742c0b1100 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7430e60640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f742c0cfd40 con 0x7f742c0b0c90 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7430e60640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f742c0d1160 con 0x7f742c15e130 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c0b1100 0x7f742c15af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c0b1100 0x7f742c15af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44396/0 (socket says 172.21.15.28:44396) 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 -- 172.21.15.28:0/1047015975 learned_addr learned my addr 172.21.15.28:0/1047015975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f742a59c640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f742c0b0c90 0x7f742c0c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f742ad9d640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f742c15e130 0x7f742c15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 -- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f742c15e130 msgr2=0x7f742c15b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f742c15e130 0x7f742c15b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 -- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f742c0b0c90 msgr2=0x7f742c0c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f742c0b0c90 0x7f742c0c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 -- 172.21.15.28:0/1047015975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c1f4900 con 0x7f742c0b1100 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.309+0000 7f7429d9b640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c0b1100 0x7f742c15af80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f741400d980 tx=0x7f741400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.310+0000 7f7422ffd640 1 -- 172.21.15.28:0/1047015975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414017070 con 0x7f742c0b1100 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.310+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c1f4be0 con 0x7f742c0b1100 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.310+0000 7f7422ffd640 1 -- 172.21.15.28:0/1047015975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7414009d20 con 0x7f742c0b1100 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.310+0000 7f7422ffd640 1 -- 172.21.15.28:0/1047015975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140106c0 con 0x7f742c0b1100 2024-09-10T20:41:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.310+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c1f51a0 con 0x7f742c0b1100 2024-09-10T20:41:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.311+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742c0d16f0 con 0x7f742c0b1100 2024-09-10T20:41:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.311+0000 7f7422ffd640 1 -- 172.21.15.28:0/1047015975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f7414010860 con 0x7f742c0b1100 2024-09-10T20:41:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.312+0000 7f7422ffd640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f740c078150 0x7f740c07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.312+0000 7f7422ffd640 1 -- 172.21.15.28:0/1047015975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f741409a4d0 con 0x7f742c0b1100 2024-09-10T20:41:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.312+0000 7f742a59c640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f740c078150 0x7f740c07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.313+0000 7f742a59c640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f740c078150 0x7f740c07a610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f741c033000 tx=0x7f741c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:16.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.316+0000 7f7422ffd640 1 -- 172.21.15.28:0/1047015975 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7414062600 con 0x7f742c0b1100 2024-09-10T20:41:16.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.572+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f742c1f5580 con 0x7f742c0b1100 2024-09-10T20:41:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.575+0000 7f7422ffd640 1 -- 172.21.15.28:0/1047015975 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v27) ==== 190+0+147 (secure 0 0 0) 0x7f7414098e20 con 0x7f742c0b1100 2024-09-10T20:41:16.579 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:41:16.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:41:16.579 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:41:16.579 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:41:16.579 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:41:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f740c078150 msgr2=0x7f740c07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f740c078150 0x7f740c07a610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f741c033000 tx=0x7f741c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c0b1100 msgr2=0x7f742c15af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c0b1100 0x7f742c15af80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f741400d980 tx=0x7f741400de40 comp rx=0 tx=0).stop 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 shutdown_connections 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f740c078150 0x7f740c07a610 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f742c15e130 0x7f742c15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c0b1100 0x7f742c15af80 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 --2- 172.21.15.28:0/1047015975 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f742c0b0c90 0x7f742c0c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 >> 172.21.15.28:0/1047015975 conn(0x7f742c0d4040 msgr2=0x7f742c0b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 shutdown_connections 2024-09-10T20:41:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.577+0000 7f7430e60640 1 -- 172.21.15.28:0/1047015975 wait complete. 2024-09-10T20:41:16.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.694+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/1747776867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8004d40 msgr2=0x7fb2a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.694+0000 7fb2c5b9b640 1 --2- 172.21.15.28:0/1747776867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8004d40 0x7fb2a80051c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8005fd0 tx=0x7fb2a8005830 comp rx=0 tx=0).stop 2024-09-10T20:41:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.694+0000 7fb2bffff640 1 -- 172.21.15.28:0/1747776867 reap_dead start 2024-09-10T20:41:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.694+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/1747776867 shutdown_connections 2024-09-10T20:41:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.694+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/1747776867 >> 172.21.15.28:0/1747776867 conn(0x7fb2c006dbf0 msgr2=0x7fb2c006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:16.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.694+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/1747776867 shutdown_connections 2024-09-10T20:41:16.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.694+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/1747776867 wait complete. 2024-09-10T20:41:16.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.695+0000 7fb2c5b9b640 1 Processor -- start 2024-09-10T20:41:16.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.695+0000 7fb2c5b9b640 1 -- start start 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2c5b9b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2a8004d40 0x7fb2c0151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2c5b9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015a260 0x7fb2c01592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2c5b9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2c01f7cb0 0x7fb2c01f8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2c5b9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c015e6e0 con 0x7fb2c015a260 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2c5b9b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c00b54a0 con 0x7fb2c01f7cb0 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2c5b9b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c015e860 con 0x7fb2a8004d40 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015a260 0x7fb2c01592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015a260 0x7fb2c01592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44422/0 (socket says 172.21.15.28:44422) 2024-09-10T20:41:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 -- 172.21.15.28:0/4062376677 learned_addr learned my addr 172.21.15.28:0/4062376677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2bffff640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2c01f7cb0 0x7fb2c01f8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2bf7fe640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2a8004d40 0x7fb2c0151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 -- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2a8004d40 msgr2=0x7fb2c0151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2a8004d40 0x7fb2c0151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 -- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2c01f7cb0 msgr2=0x7fb2c01f8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2c01f7cb0 0x7fb2c01f8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2beffd640 1 -- 172.21.15.28:0/4062376677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c01f5990 con 0x7fb2c015a260 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.696+0000 7fb2bffff640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2c01f7cb0 0x7fb2c01f8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.697+0000 7fb2beffd640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015a260 0x7fb2c01592e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4004650 tx=0x7fb2b400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.697+0000 7fb2a7fff640 1 -- 172.21.15.28:0/4062376677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40107f0 con 0x7fb2c015a260 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.697+0000 7fb2a7fff640 1 -- 172.21.15.28:0/4062376677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2b4010990 con 0x7fb2c015a260 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.697+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c01f8b30 con 0x7fb2c015a260 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.697+0000 7fb2a7fff640 1 -- 172.21.15.28:0/4062376677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4010dc0 con 0x7fb2c015a260 2024-09-10T20:41:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.697+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c01f5c30 con 0x7fb2c015a260 2024-09-10T20:41:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.698+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c0153510 con 0x7fb2c015a260 2024-09-10T20:41:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.699+0000 7fb2a7fff640 1 -- 172.21.15.28:0/4062376677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fb2b4023440 con 0x7fb2c015a260 2024-09-10T20:41:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.699+0000 7fb2a7fff640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb2ac078150 0x7fb2ac07a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.699+0000 7fb2a7fff640 1 -- 172.21.15.28:0/4062376677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fb2b409af80 con 0x7fb2c015a260 2024-09-10T20:41:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.699+0000 7fb2bf7fe640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb2ac078150 0x7fb2ac07a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.700+0000 7fb2bf7fe640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb2ac078150 0x7fb2ac07a610 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0000fb0 tx=0x7fb2b0001550 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:16.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.703+0000 7fb2a7fff640 1 -- 172.21.15.28:0/4062376677 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb2b4063670 con 0x7fb2c015a260 2024-09-10T20:41:16.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.926+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2c00d0a60 con 0x7fb2c015a260 2024-09-10T20:41:16.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.926+0000 7fb2a7fff640 1 -- 172.21.15.28:0/4062376677 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5223 (secure 0 0 0) 0x7fb2b4062f70 con 0x7fb2c015a260 2024-09-10T20:41:16.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:16.930 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"btime":"2024-09-10T20:41:14:782123+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/485554639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":485554639},{"type":"v1","addr":"172.21.15.159:6836","nonce":485554639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1807026586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1807026586},{"type":"v1","addr":"172.21.15.159:6837","nonce":1807026586}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/975621681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":975621681},{"type":"v1","addr":"172.21.15.28:6834","nonce":975621681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:11.137947+0000","modified":"2024-09-10T20:41:14.144276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3887400201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3887400201},{"type":"v1","addr":"172.21.15.28:6835","nonce":3887400201}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-10T20:41:16.930 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-10T20:41:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb2ac078150 msgr2=0x7fb2ac07a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb2ac078150 0x7fb2ac07a610 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0000fb0 tx=0x7fb2b0001550 comp rx=0 tx=0).stop 2024-09-10T20:41:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015a260 msgr2=0x7fb2c01592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015a260 0x7fb2c01592e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4004650 tx=0x7fb2b400ba40 comp rx=0 tx=0).stop 2024-09-10T20:41:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 shutdown_connections 2024-09-10T20:41:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb2ac078150 0x7fb2ac07a610 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2c01f7cb0 0x7fb2c01f8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015a260 0x7fb2c01592e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 --2- 172.21.15.28:0/4062376677 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2a8004d40 0x7fb2c0151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 >> 172.21.15.28:0/4062376677 conn(0x7fb2c006dbf0 msgr2=0x7fb2c006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 shutdown_connections 2024-09-10T20:41:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:16.928+0000 7fb2c5b9b640 1 -- 172.21.15.28:0/4062376677 wait complete. 2024-09-10T20:41:17.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:41:11.137947+0000', 'modified': '2024-09-10T20:41:14.144276+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5748}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5748': {'gid': 5748, 'name': 'c', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3887400201', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3887400201}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3887400201}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5748, 'qdb_cluster': [5748]} 2024-09-10T20:41:17.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:41:17.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-10T20:41:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 -- 172.21.15.28:0/1501561212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 msgr2=0x7fe9640d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 --2- 172.21.15.28:0/1501561212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe9640d0210 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe954005fd0 tx=0x7fe95402a940 comp rx=0 tx=0).stop 2024-09-10T20:41:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 -- 172.21.15.28:0/1501561212 shutdown_connections 2024-09-10T20:41:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 --2- 172.21.15.28:0/1501561212 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9640d1aa0 0x7fe9640d0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 --2- 172.21.15.28:0/1501561212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe9640d0210 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 -- 172.21.15.28:0/1501561212 >> 172.21.15.28:0/1501561212 conn(0x7fe9640d1ed0 msgr2=0x7fe9640d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 -- 172.21.15.28:0/1501561212 shutdown_connections 2024-09-10T20:41:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.452+0000 7fe969545640 1 -- 172.21.15.28:0/1501561212 wait complete. 2024-09-10T20:41:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 Processor -- start 2024-09-10T20:41:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 -- start start 2024-09-10T20:41:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe96406de00 0x7fe96415b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9640d1aa0 0x7fe964158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe96415c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9640cf240 con 0x7fe964161880 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe9640cf0c0 con 0x7fe96406de00 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe969545640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe9640ccc30 con 0x7fe9640d1aa0 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe9637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe96415c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe9637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe96415c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44444/0 (socket says 172.21.15.28:44444) 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.454+0000 7fe9637fe640 1 -- 172.21.15.28:0/3983995911 learned_addr learned my addr 172.21.15.28:0/3983995911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe9627fc640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9640d1aa0 0x7fe964158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe962ffd640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe96406de00 0x7fe96415b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe9637fe640 1 -- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9640d1aa0 msgr2=0x7fe964158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe9637fe640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9640d1aa0 0x7fe964158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe9637fe640 1 -- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe96406de00 msgr2=0x7fe96415b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe9637fe640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe96406de00 0x7fe96415b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe9637fe640 1 -- 172.21.15.28:0/3983995911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9641f8e70 con 0x7fe964161880 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe9637fe640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe96415c400 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c410 tx=0x7fe95c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe95b7fe640 1 -- 172.21.15.28:0/3983995911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c017070 con 0x7fe964161880 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe95b7fe640 1 -- 172.21.15.28:0/3983995911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe95c0035f0 con 0x7fe964161880 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9641f9150 con 0x7fe964161880 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe95b7fe640 1 -- 172.21.15.28:0/3983995911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c003ac0 con 0x7fe964161880 2024-09-10T20:41:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.455+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe964158760 con 0x7fe964161880 2024-09-10T20:41:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.457+0000 7fe95b7fe640 1 -- 172.21.15.28:0/3983995911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fe95c003c60 con 0x7fe964161880 2024-09-10T20:41:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.457+0000 7fe95b7fe640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe944078120 0x7fe94407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.458+0000 7fe95b7fe640 1 -- 172.21.15.28:0/3983995911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe95c099640 con 0x7fe964161880 2024-09-10T20:41:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.458+0000 7fe962ffd640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe944078120 0x7fe94407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.458+0000 7fe96c3ed740 1 Processor -- start 2024-09-10T20:41:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.458+0000 7fe96c3ed740 1 -- start start 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.458+0000 7fe962ffd640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe944078120 0x7fe94407a5e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe954034000 tx=0x7fe954004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a22b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a6b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a205f0 0x55b2e7a20a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2e7997820 con 0x55b2e7a6adf0 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b2e7986c50 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b2e798aa60 con 0x55b2e7a205f0 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe9627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a6b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe9627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a6b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44454/0 (socket says 172.21.15.28:44454) 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe9627fc640 1 -- 172.21.15.28:0/1023356168 learned_addr learned my addr 172.21.15.28:0/1023356168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe962ffd640 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a22b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.459+0000 7fe9637fe640 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a205f0 0x55b2e7a20a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe9627fc640 1 -- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a205f0 msgr2=0x55b2e7a20a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe9627fc640 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a205f0 0x55b2e7a20a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe9627fc640 1 -- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a4b1b0 msgr2=0x55b2e7a22b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe9627fc640 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a22b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe9627fc640 1 -- 172.21.15.28:0/1023356168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e76e8850 con 0x55b2e7a6adf0 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe9627fc640 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a6b1f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe96409fe00 tx=0x7fe94c00bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320ea766f7e6cf6b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe958ff9640 1 -- 172.21.15.28:0/1023356168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c003da0 con 0x55b2e7a6adf0 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe958ff9640 1 -- 172.21.15.28:0/1023356168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe94c015070 con 0x55b2e7a6adf0 2024-09-10T20:41:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.460+0000 7fe958ff9640 1 -- 172.21.15.28:0/1023356168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c01a6d0 con 0x55b2e7a6adf0 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1023356168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 msgr2=0x55b2e7a6b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a6b1f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe96409fe00 tx=0x7fe94c00bb60 comp rx=0 tx=0).stop 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1023356168 shutdown_connections 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a205f0 0x55b2e7a20a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a6b1f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1023356168 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a22b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1023356168 >> 172.21.15.28:0/1023356168 conn(0x55b2e7a216a0 msgr2=0x55b2e7a250c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1023356168 shutdown_connections 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1023356168 wait complete. 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 -- start start 2024-09-10T20:41:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.461+0000 7fe96c3ed740 1 Processor -- start 2024-09-10T20:41:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a34a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a68ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a4a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2e76f7620 con 0x55b2e7a6adf0 2024-09-10T20:41:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b2e792c5f0 con 0x55b2e7a693e0 2024-09-10T20:41:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b2e7a1fb30 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe96c3ed740 10 client.0 _ll_register_callbacks cb 0x55b2e79300a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:41:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe9627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a68ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe9627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a68ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44456/0 (socket says 172.21.15.28:44456) 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe9627fc640 1 -- 172.21.15.28:0/1043379271 learned_addr learned my addr 172.21.15.28:0/1043379271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe962ffd640 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a34a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.462+0000 7fe9637fe640 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a4a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe9627fc640 1 -- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 msgr2=0x55b2e7a34a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe9627fc640 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a34a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe9627fc640 1 -- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 msgr2=0x55b2e7a4a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe9627fc640 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a4a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe9627fc640 1 -- 172.21.15.28:0/1043379271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e775f6b0 con 0x55b2e7a6adf0 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe9627fc640 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a68ea0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe9640d1ea0 tx=0x7fe94c005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed673c16739aa7a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe941ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:41:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe941ffb640 1 -- 172.21.15.28:0/1043379271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c003db0 con 0x55b2e7a6adf0 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe941ffb640 1 -- 172.21.15.28:0/1043379271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe94c005020 con 0x55b2e7a6adf0 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe94c015070 con 0x55b2e7a6adf0 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe941ffb640 1 -- 172.21.15.28:0/1043379271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe94c02f050 con 0x55b2e7a6adf0 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe96c3ed740 10 client.5883 fetch_fsmap learned FSMap version 35 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55b2e7a34f40 con 0x55b2e7a6adf0 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe941ffb640 1 -- 172.21.15.28:0/1043379271 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe94c015070 con 0x55b2e7a6adf0 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe941ffb640 1 -- 172.21.15.28:0/1043379271 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7fe94c032020 con 0x55b2e7a6adf0 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.463+0000 7fe96c3ed740 10 client.5883 fetch_fsmap finished waiting for FSMap version 35 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe96c3ed740 10 client.5883 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe96c3ed740 20 client.5883 populate_metadata read hostname 'smithi028' 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a644b0 0x55b2e7a79870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe96c3ed740 4 client.5883 mds_command: new command op to 5748 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 --> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2e7999770 con 0x55b2e7a644b0 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe962ffd640 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a644b0 0x55b2e7a79870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe941ffb640 10 client.5883 ms_handle_connect on v2:172.21.15.28:6833/3887400201 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.464+0000 7fe962ffd640 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a644b0 0x55b2e7a79870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a8d090018609b01 server_cookie=c8cd18c97efba535 in_seq=0 out_seq=0 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe941ffb640 1 -- 172.21.15.28:0/1043379271 <== mds.0 v2:172.21.15.28:6833/3887400201 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b2e7999770 con 0x55b2e7a644b0 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe941ffb640 10 client.5883 handle_command_reply: tid=0 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 client.5883 shutdown 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 msgr2=0x55b2e7a68ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a68ea0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe9640d1ea0 tx=0x7fe94c005cd0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 shutdown_connections 2024-09-10T20:41:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a644b0 0x55b2e7a79870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a4a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a68ea0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe941ffb640 0 client.5883 ms_handle_reset on v2:172.21.15.28:6833/3887400201 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1043379271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a34a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 >> 172.21.15.28:0/1043379271 conn(0x55b2e7a205f0 msgr2=0x55b2e7a6a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 shutdown_connections 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1043379271 wait complete. 2024-09-10T20:41:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.465+0000 7fe96c3ed740 20 client.5883 trim_cache size 0 max 16384 2024-09-10T20:41:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.483+0000 7fe96c3ed740 1 Processor -- start 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.483+0000 7fe96c3ed740 1 -- start start 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a688e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a42430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a6adf0 0x55b2e7a42970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9640c6660 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe9640c59c0 con 0x55b2e7a693e0 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe9640c6010 con 0x55b2e7a6adf0 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a688e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a688e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44470/0 (socket says 172.21.15.28:44470) 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 -- 172.21.15.28:0/1932518718 learned_addr learned my addr 172.21.15.28:0/1932518718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe9627fc640 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a42430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe9637fe640 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a6adf0 0x55b2e7a42970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 -- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a6adf0 msgr2=0x55b2e7a42970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a6adf0 0x55b2e7a42970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 -- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 msgr2=0x55b2e7a42430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a42430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 -- 172.21.15.28:0/1932518718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e7a6a360 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.484+0000 7fe962ffd640 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a688e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe95402af30 tx=0x7fe95403d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845703ea482a32ee server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe943fff640 1 -- 172.21.15.28:0/1932518718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954030bc0 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe943fff640 1 -- 172.21.15.28:0/1932518718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe954030d60 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe943fff640 1 -- 172.21.15.28:0/1932518718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954051650 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1932518718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a4b1b0 msgr2=0x55b2e7a688e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a688e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe95402af30 tx=0x7fe95403d040 comp rx=0 tx=0).stop 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1932518718 shutdown_connections 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a6adf0 0x55b2e7a42970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a693e0 0x55b2e7a42430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/1932518718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a4b1b0 0x55b2e7a688e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1932518718 >> 172.21.15.28:0/1932518718 conn(0x55b2e7a205f0 msgr2=0x55b2e7a1e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1932518718 shutdown_connections 2024-09-10T20:41:17.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.485+0000 7fe96c3ed740 1 -- 172.21.15.28:0/1932518718 wait complete. 2024-09-10T20:41:17.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.486+0000 7fe96c3ed740 1 -- start start 2024-09-10T20:41:17.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.486+0000 7fe96c3ed740 1 Processor -- start 2024-09-10T20:41:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a8cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a693e0 0x55b2e7ad3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a42290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2e7a65620 con 0x55b2e7a693e0 2024-09-10T20:41:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b2e7a43320 con 0x55b2e7a6adf0 2024-09-10T20:41:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b2e7a648c0 con 0x55b2e7a4b1b0 2024-09-10T20:41:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 10 client.0 _ll_register_callbacks cb 0x7fe954005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a693e0 0x55b2e7ad3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a693e0 0x55b2e7ad3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44486/0 (socket says 172.21.15.28:44486) 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 -- 172.21.15.28:0/61612653 learned_addr learned my addr 172.21.15.28:0/61612653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9637fe640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a42290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe962ffd640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a8cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 -- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 msgr2=0x55b2e7a8cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a8cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 -- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a6adf0 msgr2=0x55b2e7a42290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a42290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 -- 172.21.15.28:0/61612653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e7ad4290 con 0x55b2e7a693e0 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe962ffd640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a8cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9637fe640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a42290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe9627fc640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a693e0 0x55b2e7ad3d50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe94c004d30 tx=0x7fe94c004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a19a3aa6a6443 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe941ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe941ffb640 1 -- 172.21.15.28:0/61612653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c039060 con 0x55b2e7a693e0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe941ffb640 1 -- 172.21.15.28:0/61612653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe94c022070 con 0x55b2e7a693e0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe954030d60 con 0x55b2e7a693e0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.487+0000 7fe941ffb640 1 -- 172.21.15.28:0/61612653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe94c003db0 con 0x55b2e7a693e0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 10 client.5889 fetch_fsmap learned FSMap version 35 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55b2e7a635a0 con 0x55b2e7a693e0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe941ffb640 1 -- 172.21.15.28:0/61612653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe954030d60 con 0x55b2e7a693e0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe941ffb640 1 -- 172.21.15.28:0/61612653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7fe94c032040 con 0x55b2e7a693e0 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 10 client.5889 fetch_fsmap finished waiting for FSMap version 35 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 10 client.5889 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 20 client.5889 populate_metadata read hostname 'smithi028' 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a62fd0 0x55b2e7a682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 4 client.5889 mds_command: new command op to 5748 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:41:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 --> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] -- command(tid 0: {"prefix": "status"}) -- 0x55b2e7995220 con 0x55b2e7a62fd0 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe962ffd640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a62fd0 0x55b2e7a682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe941ffb640 10 client.5889 ms_handle_connect on v2:172.21.15.28:6833/3887400201 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe962ffd640 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a62fd0 0x55b2e7a682f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84f5e637c3e54678 server_cookie=8d4a4e7a5da5bb62 in_seq=0 out_seq=0 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe941ffb640 1 -- 172.21.15.28:0/61612653 <== mds.0 v2:172.21.15.28:6833/3887400201 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55b2e7995220 con 0x55b2e7a62fd0 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe941ffb640 10 client.5889 handle_command_reply: tid=0 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.488+0000 7fe96c3ed740 1 client.5889 shutdown 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a693e0 msgr2=0x55b2e7ad3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a693e0 0x55b2e7ad3d50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe94c004d30 tx=0x7fe94c004d60 comp rx=0 tx=0).stop 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 shutdown_connections 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:6833/3887400201,v1:172.21.15.28:6835/3887400201] conn(0x55b2e7a62fd0 0x55b2e7a682f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b2e7a6adf0 0x55b2e7a42290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2e7a693e0 0x55b2e7ad3d50 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe941ffb640 0 client.5889 ms_handle_reset on v2:172.21.15.28:6833/3887400201 2024-09-10T20:41:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 --2- 172.21.15.28:0/61612653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b2e7a4b1b0 0x55b2e7a8cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 >> 172.21.15.28:0/61612653 conn(0x55b2e7a205f0 msgr2=0x55b2e7a6b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 shutdown_connections 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 1 -- 172.21.15.28:0/61612653 wait complete. 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe96c3ed740 20 client.5889 trim_cache size 0 max 16384 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5748, 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:41:17.500 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:41:17.501 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7278366829999996, 2024-09-10T20:41:17.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 35, 2024-09-10T20:41:17.501 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 56, 2024-09-10T20:41:17.501 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 56, 2024-09-10T20:41:17.501 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.428103312999999 2024-09-10T20:41:17.501 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe944078120 msgr2=0x7fe94407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe944078120 0x7fe94407a5e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe954034000 tx=0x7fe954004ab0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 msgr2=0x7fe96415c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe96415c400 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c410 tx=0x7fe95c00c8d0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 shutdown_connections 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe944078120 0x7fe94407a5e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964161880 0x7fe96415c400 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9640d1aa0 0x7fe964158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 --2- 172.21.15.28:0/3983995911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe96406de00 0x7fe96415b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 >> 172.21.15.28:0/3983995911 conn(0x7fe9640d1ed0 msgr2=0x7fe96415f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 shutdown_connections 2024-09-10T20:41:17.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.489+0000 7fe969545640 1 -- 172.21.15.28:0/3983995911 wait complete. 2024-09-10T20:41:17.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:41:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 -- 172.21.15.28:0/1094676163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c15de70 msgr2=0x7ff72c0d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1094676163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c15de70 0x7ff72c0d2b90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff72c0d01d0 tx=0x7ff71c02a5f0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 -- 172.21.15.28:0/1094676163 shutdown_connections 2024-09-10T20:41:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1094676163 >> v2:172.21.15.159:3300/0 conn(0x7ff72c162260 0x7ff72c0cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1094676163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c15de70 0x7ff72c0d2b90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 -- 172.21.15.28:0/1094676163 >> 172.21.15.28:0/1094676163 conn(0x7ff72c0d3fb0 msgr2=0x7ff72c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 -- 172.21.15.28:0/1094676163 shutdown_connections 2024-09-10T20:41:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.606+0000 7ff7329ad640 1 -- 172.21.15.28:0/1094676163 wait complete. 2024-09-10T20:41:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.607+0000 7ff7329ad640 1 Processor -- start 2024-09-10T20:41:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.607+0000 7ff7329ad640 1 -- start start 2024-09-10T20:41:17.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff7329ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c0cf780 0x7ff72c1e80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff7329ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff72c15de70 0x7ff72c15b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff7329ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72c162260 0x7ff72c1eaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff7329ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff72c0b6f80 con 0x7ff72c0cf780 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff7329ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff72c0b6e00 con 0x7ff72c162260 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff7329ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff72c0b7100 con 0x7ff72c15de70 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c0cf780 0x7ff72c1e80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c0cf780 0x7ff72c1e80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44502/0 (socket says 172.21.15.28:44502) 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 -- 172.21.15.28:0/1058908713 learned_addr learned my addr 172.21.15.28:0/1058908713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff730f4a640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72c162260 0x7ff72c1eaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 -- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff72c15de70 msgr2=0x7ff72c15b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72b7fe640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff72c15de70 0x7ff72c15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff72c15de70 0x7ff72c15b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 -- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72c162260 msgr2=0x7ff72c1eaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72c162260 0x7ff72c1eaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff72bfff640 1 -- 172.21.15.28:0/1058908713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72c1e88d0 con 0x7ff72c0cf780 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.608+0000 7ff730f4a640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72c162260 0x7ff72c1eaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.609+0000 7ff72bfff640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c0cf780 0x7ff72c1e80a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff71c02abe0 tx=0x7ff71c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.609+0000 7ff728ff9640 1 -- 172.21.15.28:0/1058908713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c03f070 con 0x7ff72c0cf780 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.609+0000 7ff728ff9640 1 -- 172.21.15.28:0/1058908713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff71c0047a0 con 0x7ff72c0cf780 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.609+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff72c1e8bb0 con 0x7ff72c0cf780 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.609+0000 7ff728ff9640 1 -- 172.21.15.28:0/1058908713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c035030 con 0x7ff72c0cf780 2024-09-10T20:41:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.609+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff72c1e9170 con 0x7ff72c0cf780 2024-09-10T20:41:17.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.610+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff72c0d2b90 con 0x7ff72c0cf780 2024-09-10T20:41:17.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.610+0000 7ff728ff9640 1 -- 172.21.15.28:0/1058908713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7ff71c004bf0 con 0x7ff72c0cf780 2024-09-10T20:41:17.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.611+0000 7ff728ff9640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff718078120 0x7ff71807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:17.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.611+0000 7ff728ff9640 1 -- 172.21.15.28:0/1058908713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ff71c0bb380 con 0x7ff72c0cf780 2024-09-10T20:41:17.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.611+0000 7ff72b7fe640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff718078120 0x7ff71807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:17.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.612+0000 7ff72b7fe640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff718078120 0x7ff71807a5e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff720001220 tx=0x7ff720009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:17.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.615+0000 7ff728ff9640 1 -- 172.21.15.28:0/1058908713 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff71c0839c0 con 0x7ff72c0cf780 2024-09-10T20:41:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.841+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff72c0b9f30 con 0x7ff72c0cf780 2024-09-10T20:41:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.841+0000 7ff728ff9640 1 -- 172.21.15.28:0/1058908713 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7ff71c0832c0 con 0x7ff72c0cf780 2024-09-10T20:41:17.845 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:41:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff718078120 msgr2=0x7ff71807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff718078120 0x7ff71807a5e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff720001220 tx=0x7ff720009040 comp rx=0 tx=0).stop 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c0cf780 msgr2=0x7ff72c1e80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c0cf780 0x7ff72c1e80a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff71c02abe0 tx=0x7ff71c0038a0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 shutdown_connections 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ff718078120 0x7ff71807a5e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff72c162260 0x7ff72c1eaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff72c15de70 0x7ff72c15b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 --2- 172.21.15.28:0/1058908713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72c0cf780 0x7ff72c1e80a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.843+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 >> 172.21.15.28:0/1058908713 conn(0x7ff72c0d3fb0 msgr2=0x7ff72c0b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.844+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 shutdown_connections 2024-09-10T20:41:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:17.844+0000 7ff7329ad640 1 -- 172.21.15.28:0/1058908713 wait complete. 2024-09-10T20:41:17.858 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:41:17.859 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:41:17.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:17.860 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:41:17.876 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:41:17.876 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42553sec preferred_lft 42553sec 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:17.878 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:17.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:17.879 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:41:17.941 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:41:17.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:17.942 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:41:18.000 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:41:18.000 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:41:18.000 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:41:18.000 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:41:18.000 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:41:18.000 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:41:18.000 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:41:18.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:41:18.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:41:18.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:41:18.001 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:41:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:41:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:41:18.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:18.129 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:41:18.144 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:41:18.145 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:41:18.145 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:41:18.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:18.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:41:18.211 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:41:18.211 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:41:18.267 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:41:18.267 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:41:18.267 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:41:18.322 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:41:18.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:18.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:41:18.408 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:41:18.409 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:41:18.409 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:41:18.417 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:41:18.418 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:41:18.418 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:41:18.418 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:41:18.418 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:41:18.424 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:41:18.424 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:41:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:41:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:41:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:41:18.427 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:41:18.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:18.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:41:18.460 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:41:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:41:18.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:41:18.563 INFO:teuthology.orchestra.run.smithi028.stdout:client5273 2024-09-10T20:41:18.564 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:18.564 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:18.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:41:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.723+0000 7effcd93d640 1 -- 172.21.15.28:0/351372482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb4004d40 msgr2=0x7effb40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.723+0000 7effcd93d640 1 --2- 172.21.15.28:0/351372482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb4004d40 0x7effb40051c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7effc0005fd0 tx=0x7effb4005830 comp rx=0 tx=0).stop 2024-09-10T20:41:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.723+0000 7effc77fe640 1 -- 172.21.15.28:0/351372482 reap_dead start 2024-09-10T20:41:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.723+0000 7effcd93d640 1 -- 172.21.15.28:0/351372482 shutdown_connections 2024-09-10T20:41:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.723+0000 7effcd93d640 1 -- 172.21.15.28:0/351372482 >> 172.21.15.28:0/351372482 conn(0x7effc80d1e40 msgr2=0x7effc80d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.723+0000 7effcd93d640 1 -- 172.21.15.28:0/351372482 shutdown_connections 2024-09-10T20:41:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.723+0000 7effcd93d640 1 -- 172.21.15.28:0/351372482 wait complete. 2024-09-10T20:41:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.724+0000 7effcd93d640 1 Processor -- start 2024-09-10T20:41:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.724+0000 7effcd93d640 1 -- start start 2024-09-10T20:41:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effcd93d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effb4004d40 0x7effc81e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effcd93d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effc815a580 0x7effc8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effcd93d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc809f280 0x7effc809f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effcd93d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7effc80b13e0 con 0x7effc809f280 2024-09-10T20:41:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effcd93d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7effc80cdcb0 con 0x7effb4004d40 2024-09-10T20:41:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effcd93d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7effc80b1560 con 0x7effc815a580 2024-09-10T20:41:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc809f280 0x7effc809f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc809f280 0x7effc809f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44522/0 (socket says 172.21.15.28:44522) 2024-09-10T20:41:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 -- 172.21.15.28:0/3541650066 learned_addr learned my addr 172.21.15.28:0/3541650066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc6ffd640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effb4004d40 0x7effc81e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc67fc640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effc815a580 0x7effc8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 -- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effc815a580 msgr2=0x7effc8159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effc815a580 0x7effc8159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 -- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effb4004d40 msgr2=0x7effc81e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effb4004d40 0x7effc81e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.725+0000 7effc77fe640 1 -- 172.21.15.28:0/3541650066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc81f4a20 con 0x7effc809f280 2024-09-10T20:41:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.726+0000 7effc77fe640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc809f280 0x7effc809f700 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7effc002aea0 tx=0x7effc0033b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.726+0000 7effbffff640 1 -- 172.21.15.28:0/3541650066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0038040 con 0x7effc809f280 2024-09-10T20:41:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.726+0000 7effbffff640 1 -- 172.21.15.28:0/3541650066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effc002bc10 con 0x7effc809f280 2024-09-10T20:41:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.726+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc81f4ce0 con 0x7effc809f280 2024-09-10T20:41:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.726+0000 7effbffff640 1 -- 172.21.15.28:0/3541650066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0009200 con 0x7effc809f280 2024-09-10T20:41:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.726+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc81f52a0 con 0x7effc809f280 2024-09-10T20:41:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.727+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc80b7aa0 con 0x7effc809f280 2024-09-10T20:41:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.727+0000 7effbffff640 1 -- 172.21.15.28:0/3541650066 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7effc003b070 con 0x7effc809f280 2024-09-10T20:41:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.728+0000 7effbffff640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7effa8078120 0x7effa807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.728+0000 7effbffff640 1 -- 172.21.15.28:0/3541650066 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7effc00ba770 con 0x7effc809f280 2024-09-10T20:41:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.728+0000 7effc6ffd640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7effa8078120 0x7effa807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.729+0000 7effc6ffd640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7effa8078120 0x7effa807a5e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7effb8009cd0 tx=0x7effb8005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.732+0000 7effbffff640 1 -- 172.21.15.28:0/3541650066 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7effc80b7aa0 con 0x7effc809f280 2024-09-10T20:41:18.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.954+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7effc80cfa10 con 0x7effc809f280 2024-09-10T20:41:18.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.954+0000 7effbffff640 1 -- 172.21.15.28:0/3541650066 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7effc004a020 con 0x7effc809f280 2024-09-10T20:41:18.958 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.956+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7effa8078120 msgr2=0x7effa807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.956+0000 7effcd93d640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7effa8078120 0x7effa807a5e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7effb8009cd0 tx=0x7effb8005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.956+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc809f280 msgr2=0x7effc809f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.956+0000 7effcd93d640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc809f280 0x7effc809f700 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7effc002aea0 tx=0x7effc0033b80 comp rx=0 tx=0).stop 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 shutdown_connections 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7effa8078120 0x7effa807a5e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc809f280 0x7effc809f700 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7effc815a580 0x7effc8159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 --2- 172.21.15.28:0/3541650066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7effb4004d40 0x7effc81e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 >> 172.21.15.28:0/3541650066 conn(0x7effc80d1e40 msgr2=0x7effc80b4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 shutdown_connections 2024-09-10T20:41:18.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:18.957+0000 7effcd93d640 1 -- 172.21.15.28:0/3541650066 wait complete. 2024-09-10T20:41:18.971 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:41:18.971 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:41:18.971 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:41:18.971 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:41:18.972 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:41:18.972 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:41:18.972 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:41:18.972 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:41:18.972 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:41:18.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:18.972 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:41:18.992 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:41:18.992 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:41:18.992 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:41:18.993 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42570sec preferred_lft 42570sec 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:41:18.994 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:18.995 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:41:18.995 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:41:18.995 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:41:18.995 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:41:18.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:18.995 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:41:19.061 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:41:19.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:19.063 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:41:19.121 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:41:19.122 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:41:19.122 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:41:19.122 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:41:19.122 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:41:19.122 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:41:19.122 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:41:19.122 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:41:19.122 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:41:19.122 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:41:19.122 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:41:19.213 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:41:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:41:19.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:41:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:41:19.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:19.254 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:41:19.269 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:41:19.269 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:41:19.269 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:41:19.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:19.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:41:19.334 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:41:19.334 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:41:19.390 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:41:19.391 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:41:19.391 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:41:19.445 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:41:19.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:19.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:41:19.529 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:41:19.529 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:41:19.529 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:41:19.538 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:41:19.538 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:41:19.538 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:41:19.538 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:41:19.538 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:41:19.543 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:41:19.544 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:41:19.545 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:41:19.545 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:41:19.545 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:41:19.545 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:41:19.547 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:41:19.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:19.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:41:19.612 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:41:19.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:41:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:41:19.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:41:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:41:19.708 INFO:teuthology.orchestra.run.smithi159.stdout:client5282 2024-09-10T20:41:19.709 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:19.709 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:19.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:41:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 -- 172.21.15.28:0/1291430490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524070440 msgr2=0x7f15240ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 --2- 172.21.15.28:0/1291430490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524070440 0x7f15240ca620 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f152406db30 tx=0x7f151c02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:41:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 -- 172.21.15.28:0/1291430490 shutdown_connections 2024-09-10T20:41:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 --2- 172.21.15.28:0/1291430490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524070440 0x7f15240ca620 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 --2- 172.21.15.28:0/1291430490 >> v2:172.21.15.159:3300/0 conn(0x7f1524151cf0 0x7f152415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:19.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 -- 172.21.15.28:0/1291430490 >> 172.21.15.28:0/1291430490 conn(0x7f15240cfc80 msgr2=0x7f15240d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:19.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 -- 172.21.15.28:0/1291430490 shutdown_connections 2024-09-10T20:41:19.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.847+0000 7f1529c47640 1 -- 172.21.15.28:0/1291430490 wait complete. 2024-09-10T20:41:19.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.848+0000 7f1529c47640 1 Processor -- start 2024-09-10T20:41:19.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1529c47640 1 -- start start 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1529c47640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1524070440 0x7f1524156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1529c47640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1524072db0 0x7f15240b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1529c47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524151cf0 0x7f15240b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1529c47640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15240caef0 con 0x7f1524151cf0 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1529c47640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f15240cad70 con 0x7f1524072db0 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1529c47640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f15240cb070 con 0x7f1524070440 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1523fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524151cf0 0x7f15240b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1523fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524151cf0 0x7f15240b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44554/0 (socket says 172.21.15.28:44554) 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.849+0000 7f1523fff640 1 -- 172.21.15.28:0/1526674111 learned_addr learned my addr 172.21.15.28:0/1526674111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f15237fe640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1524070440 0x7f1524156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1522ffd640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1524072db0 0x7f15240b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1523fff640 1 -- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1524070440 msgr2=0x7f1524156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1523fff640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1524070440 0x7f1524156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1523fff640 1 -- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1524072db0 msgr2=0x7f15240b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1523fff640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1524072db0 0x7f15240b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1523fff640 1 -- 172.21.15.28:0/1526674111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1524155890 con 0x7f1524151cf0 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1522ffd640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1524072db0 0x7f15240b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1523fff640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524151cf0 0x7f15240b2b70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f151c02aea0 tx=0x7f151c0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f150bfff640 1 -- 172.21.15.28:0/1526674111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c040070 con 0x7f1524151cf0 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f150bfff640 1 -- 172.21.15.28:0/1526674111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f151c004040 con 0x7f1524151cf0 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15240b1530 con 0x7f1524151cf0 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f150bfff640 1 -- 172.21.15.28:0/1526674111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c004470 con 0x7f1524151cf0 2024-09-10T20:41:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.850+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15240b3830 con 0x7f1524151cf0 2024-09-10T20:41:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.851+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15240ca620 con 0x7f1524151cf0 2024-09-10T20:41:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.852+0000 7f150bfff640 1 -- 172.21.15.28:0/1526674111 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f151c04d030 con 0x7f1524151cf0 2024-09-10T20:41:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.853+0000 7f150bfff640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1510078120 0x7f151007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.853+0000 7f150bfff640 1 -- 172.21.15.28:0/1526674111 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f151c0bb5e0 con 0x7f1524151cf0 2024-09-10T20:41:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.853+0000 7f15237fe640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1510078120 0x7f151007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.853+0000 7f15237fe640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1510078120 0x7f151007a5e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1514009b30 tx=0x7f1514005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:19.857+0000 7f150bfff640 1 -- 172.21.15.28:0/1526674111 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f151c083d50 con 0x7f1524151cf0 2024-09-10T20:41:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.083+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f1524069a00 con 0x7f1524151cf0 2024-09-10T20:41:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.804+0000 7f150bfff640 1 -- 172.21.15.28:0/1526674111 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v36) ==== 136+0+0 (secure 0 0 0) 0x7f151c083650 con 0x7f1524151cf0 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1510078120 msgr2=0x7f151007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1510078120 0x7f151007a5e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1514009b30 tx=0x7f1514005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524151cf0 msgr2=0x7f15240b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524151cf0 0x7f15240b2b70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f151c02aea0 tx=0x7f151c0316f0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 shutdown_connections 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f1510078120 0x7f151007a5e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524151cf0 0x7f15240b2b70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1524072db0 0x7f15240b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 --2- 172.21.15.28:0/1526674111 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1524070440 0x7f1524156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 >> 172.21.15.28:0/1526674111 conn(0x7f15240cfc80 msgr2=0x7f15240ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 shutdown_connections 2024-09-10T20:41:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.808+0000 7f1529c47640 1 -- 172.21.15.28:0/1526674111 wait complete. 2024-09-10T20:41:20.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:41:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.935+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3765577359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415df70 msgr2=0x7f6bc40b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.935+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3765577359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415df70 0x7f6bc40b2ba0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4005fd0 tx=0x7f6bb402a670 comp rx=0 tx=0).stop 2024-09-10T20:41:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.935+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3765577359 shutdown_connections 2024-09-10T20:41:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.935+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3765577359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415df70 0x7f6bc40b2ba0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.935+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3765577359 >> 172.21.15.28:0/3765577359 conn(0x7f6bc406cf00 msgr2=0x7f6bc406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.935+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3765577359 shutdown_connections 2024-09-10T20:41:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.935+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3765577359 wait complete. 2024-09-10T20:41:20.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.936+0000 7f6bcbb16640 1 Processor -- start 2024-09-10T20:41:20.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.936+0000 7f6bcbb16640 1 -- start start 2024-09-10T20:41:20.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bcbb16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc40b4d10 0x7f6bc40af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:20.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bcbb16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415a190 0x7f6bc4147770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:20.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bcbb16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc415df70 0x7f6bc406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:20.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bcbb16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc415a900 con 0x7f6bc415a190 2024-09-10T20:41:20.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bcbb16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc415a780 con 0x7f6bc40b4d10 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bcbb16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc415aa80 con 0x7f6bc415df70 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415a190 0x7f6bc4147770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415a190 0x7f6bc4147770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41516/0 (socket says 172.21.15.28:41516) 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 -- 172.21.15.28:0/3618881943 learned_addr learned my addr 172.21.15.28:0/3618881943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc98b2640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc40b4d10 0x7f6bc40af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bca0b3640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc415df70 0x7f6bc406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 -- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc415df70 msgr2=0x7f6bc406ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc415df70 0x7f6bc406ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 -- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc40b4d10 msgr2=0x7f6bc40af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc40b4d10 0x7f6bc40af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 -- 172.21.15.28:0/3618881943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc41f58f0 con 0x7f6bc415a190 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc98b2640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc40b4d10 0x7f6bc40af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bca0b3640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc415df70 0x7f6bc406ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.937+0000 7f6bc90b1640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415a190 0x7f6bc4147770 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4034040 tx=0x7f6bb40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.938+0000 7f6bba7fc640 1 -- 172.21.15.28:0/3618881943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb403e030 con 0x7f6bc415a190 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.938+0000 7f6bba7fc640 1 -- 172.21.15.28:0/3618881943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bb40047a0 con 0x7f6bc415a190 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.938+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc41f5bd0 con 0x7f6bc415a190 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.938+0000 7f6bba7fc640 1 -- 172.21.15.28:0/3618881943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4035040 con 0x7f6bc415a190 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.938+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc41f6190 con 0x7f6bc415a190 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.939+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc415af50 con 0x7f6bc415a190 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.939+0000 7f6bba7fc640 1 -- 172.21.15.28:0/3618881943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f6bb4004b90 con 0x7f6bc415a190 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.940+0000 7f6bba7fc640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6ba8078120 0x7f6ba807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.940+0000 7f6bba7fc640 1 -- 172.21.15.28:0/3618881943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f6bb40bac10 con 0x7f6bc415a190 2024-09-10T20:41:20.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.940+0000 7f6bc98b2640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6ba8078120 0x7f6ba807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:20.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.940+0000 7f6bc98b2640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6ba8078120 0x7f6ba807a5e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0007fd0 tx=0x7f6bc0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:20.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:20.944+0000 7f6bba7fc640 1 -- 172.21.15.28:0/3618881943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6bb4083370 con 0x7f6bc415a190 2024-09-10T20:41:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.154+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f6bc41f56a0 con 0x7f6ba8078120 2024-09-10T20:41:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.156+0000 7f6bba7fc640 1 -- 172.21.15.28:0/3618881943 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f6bc41f56a0 con 0x7f6ba8078120 2024-09-10T20:41:21.160 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:41:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:41:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:21.161 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:41:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6ba8078120 msgr2=0x7f6ba807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6ba8078120 0x7f6ba807a5e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0007fd0 tx=0x7f6bc0009040 comp rx=0 tx=0).stop 2024-09-10T20:41:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415a190 msgr2=0x7f6bc4147770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415a190 0x7f6bc4147770 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4034040 tx=0x7f6bb40039e0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 shutdown_connections 2024-09-10T20:41:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6ba8078120 0x7f6ba807a5e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6bc415df70 0x7f6bc406ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc415a190 0x7f6bc4147770 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 --2- 172.21.15.28:0/3618881943 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6bc40b4d10 0x7f6bc40af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.158+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 >> 172.21.15.28:0/3618881943 conn(0x7f6bc406cf00 msgr2=0x7f6bc41537c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.159+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 shutdown_connections 2024-09-10T20:41:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.159+0000 7f6bcbb16640 1 -- 172.21.15.28:0/3618881943 wait complete. 2024-09-10T20:41:21.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:41:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.278+0000 7f28c5233640 1 -- 172.21.15.28:0/888052566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac003c10 msgr2=0x7f28ac004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.278+0000 7f28c5233640 1 --2- 172.21.15.28:0/888052566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac003c10 0x7f28ac004010 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006fb0 tx=0x7f28ac005830 comp rx=0 tx=0).stop 2024-09-10T20:41:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.278+0000 7f28bf59e640 1 -- 172.21.15.28:0/888052566 reap_dead start 2024-09-10T20:41:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.278+0000 7f28c5233640 1 -- 172.21.15.28:0/888052566 shutdown_connections 2024-09-10T20:41:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.278+0000 7f28c5233640 1 -- 172.21.15.28:0/888052566 >> 172.21.15.28:0/888052566 conn(0x7f28c006d110 msgr2=0x7f28c00b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.278+0000 7f28c5233640 1 -- 172.21.15.28:0/888052566 shutdown_connections 2024-09-10T20:41:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.278+0000 7f28c5233640 1 -- 172.21.15.28:0/888052566 wait complete. 2024-09-10T20:41:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.280+0000 7f28c5233640 1 Processor -- start 2024-09-10T20:41:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.280+0000 7f28c5233640 1 -- start start 2024-09-10T20:41:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28ac003c10 0x7f28c0151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28c0153d30 0x7f28c00b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c00b4310 0x7f28c00b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28c0163c60 con 0x7f28c00b4310 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f28c0163ae0 con 0x7f28ac003c10 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f28c0163de0 con 0x7f28c0153d30 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c00b4310 0x7f28c00b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c00b4310 0x7f28c00b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41524/0 (socket says 172.21.15.28:41524) 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 -- 172.21.15.28:0/3878069327 learned_addr learned my addr 172.21.15.28:0/3878069327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bed9d640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28ac003c10 0x7f28c0151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28b7fff640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28c0153d30 0x7f28c00b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 -- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28c0153d30 msgr2=0x7f28c00b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28c0153d30 0x7f28c00b31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 -- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28ac003c10 msgr2=0x7f28c0151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28ac003c10 0x7f28c0151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 -- 172.21.15.28:0/3878069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c0152700 con 0x7f28c00b4310 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bed9d640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28ac003c10 0x7f28c0151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28bf59e640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c00b4310 0x7f28c00b3730 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f28c00b2510 tx=0x7f28b800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28b77fe640 1 -- 172.21.15.28:0/3878069327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8003970 con 0x7f28c00b4310 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28b77fe640 1 -- 172.21.15.28:0/3878069327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28b8003b10 con 0x7f28c00b4310 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c01529e0 con 0x7f28c00b4310 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28b77fe640 1 -- 172.21.15.28:0/3878069327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8004030 con 0x7f28c00b4310 2024-09-10T20:41:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.281+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c0151bb0 con 0x7f28c00b4310 2024-09-10T20:41:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.283+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c015b170 con 0x7f28c00b4310 2024-09-10T20:41:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.283+0000 7f28b77fe640 1 -- 172.21.15.28:0/3878069327 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7f28b8022060 con 0x7f28c00b4310 2024-09-10T20:41:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.284+0000 7f28b77fe640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f28a0078120 0x7f28a007a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.284+0000 7f28b77fe640 1 -- 172.21.15.28:0/3878069327 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f28b809a3c0 con 0x7f28c00b4310 2024-09-10T20:41:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.284+0000 7f28bed9d640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f28a0078120 0x7f28a007a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.285+0000 7f28bed9d640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f28a0078120 0x7f28a007a5e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f28b0033000 tx=0x7f28b002d000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.288+0000 7f28b77fe640 1 -- 172.21.15.28:0/3878069327 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f28b8062b30 con 0x7f28c00b4310 2024-09-10T20:41:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.474+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f28c00b2540 con 0x7f28c00b4310 2024-09-10T20:41:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.475+0000 7f28b77fe640 1 -- 172.21.15.28:0/3878069327 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f28b8062430 con 0x7f28c00b4310 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f28a0078120 msgr2=0x7f28a007a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f28a0078120 0x7f28a007a5e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f28b0033000 tx=0x7f28b002d000 comp rx=0 tx=0).stop 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c00b4310 msgr2=0x7f28c00b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c00b4310 0x7f28c00b3730 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f28c00b2510 tx=0x7f28b800f3c0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 shutdown_connections 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f28a0078120 0x7f28a007a5e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c00b4310 0x7f28c00b3730 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28c0153d30 0x7f28c00b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 --2- 172.21.15.28:0/3878069327 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f28ac003c10 0x7f28c0151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 >> 172.21.15.28:0/3878069327 conn(0x7f28c006d110 msgr2=0x7f28c0156aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:21.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 shutdown_connections 2024-09-10T20:41:21.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.477+0000 7f28c5233640 1 -- 172.21.15.28:0/3878069327 wait complete. 2024-09-10T20:41:21.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:21.492 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvol45350) 2024-09-10T20:41:21.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:21.524 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:41:21.583 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:41:21.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:21.584 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/_nogroup/subvol45350) 2024-09-10T20:41:21.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol66221 2024-09-10T20:41:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 -- 172.21.15.28:0/2300938360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 msgr2=0x7fc6640b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 --2- 172.21.15.28:0/2300938360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b1830 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc6640d1e80 tx=0x7fc64c02a4a0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 -- 172.21.15.28:0/2300938360 shutdown_connections 2024-09-10T20:41:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 --2- 172.21.15.28:0/2300938360 >> v2:172.21.15.159:3300/0 conn(0x7fc664161b50 0x7fc6640b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 --2- 172.21.15.28:0/2300938360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b1830 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 -- 172.21.15.28:0/2300938360 >> 172.21.15.28:0/2300938360 conn(0x7fc6640d4070 msgr2=0x7fc6640d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 -- 172.21.15.28:0/2300938360 shutdown_connections 2024-09-10T20:41:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.799+0000 7fc669448640 1 -- 172.21.15.28:0/2300938360 wait complete. 2024-09-10T20:41:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 Processor -- start 2024-09-10T20:41:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 -- start start 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc664161770 0x7fc6641f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc664161b50 0x7fc6641504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6640b2dd0 con 0x7fc6640d27f0 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc6640d4de0 con 0x7fc664161b50 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc669448640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc6640b2f50 con 0x7fc664161770 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.801+0000 7fc662ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41548/0 (socket says 172.21.15.28:41548) 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 -- 172.21.15.28:0/3370419965 learned_addr learned my addr 172.21.15.28:0/3370419965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc6627fc640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc664161770 0x7fc6641f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc664161b50 0x7fc6641504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 -- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc664161770 msgr2=0x7fc6641f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc664161770 0x7fc6641f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 -- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc664161b50 msgr2=0x7fc6641504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc664161b50 0x7fc6641504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 -- 172.21.15.28:0/3370419965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6640cac10 con 0x7fc6640d27f0 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc6627fc640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc664161770 0x7fc6641f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc664161b50 0x7fc6641504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b9ac0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c4c0 tx=0x7fc65400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc65b7fe640 1 -- 172.21.15.28:0/3370419965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654016020 con 0x7fc6640d27f0 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc65b7fe640 1 -- 172.21.15.28:0/3370419965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc654012940 con 0x7fc6640d27f0 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc65b7fe640 1 -- 172.21.15.28:0/3370419965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654012d90 con 0x7fc6640d27f0 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6641e86f0 con 0x7fc6640d27f0 2024-09-10T20:41:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.802+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6641e8cb0 con 0x7fc6640d27f0 2024-09-10T20:41:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.803+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6640d20e0 con 0x7fc6640d27f0 2024-09-10T20:41:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.804+0000 7fc65b7fe640 1 -- 172.21.15.28:0/3370419965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 100754+0+0 (secure 0 0 0) 0x7fc65401a3b0 con 0x7fc6640d27f0 2024-09-10T20:41:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.804+0000 7fc65b7fe640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc644078120 0x7fc64407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.804+0000 7fc65b7fe640 1 -- 172.21.15.28:0/3370419965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fc65409a1f0 con 0x7fc6640d27f0 2024-09-10T20:41:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.805+0000 7fc6627fc640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc644078120 0x7fc64407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.805+0000 7fc6627fc640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc644078120 0x7fc64407a5e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc64c034000 tx=0x7fc64c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:21.809+0000 7fc65b7fe640 1 -- 172.21.15.28:0/3370419965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc654020030 con 0x7fc6640d27f0 2024-09-10T20:41:22.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.023+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol66221", "target": ["mon-mgr", ""]}) -- 0x7fc66406d770 con 0x7fc644078120 2024-09-10T20:41:22.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:22.027+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:22.032 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:22.028+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:22.032 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:22.028+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:22.032 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:22.028+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:22.032 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:22.028+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.064+0000 7fc65b7fe640 1 -- 172.21.15.28:0/3370419965 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc66406d770 con 0x7fc644078120 2024-09-10T20:41:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc644078120 msgr2=0x7fc64407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc644078120 0x7fc64407a5e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc64c034000 tx=0x7fc64c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 msgr2=0x7fc6640b9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b9ac0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c4c0 tx=0x7fc65400c980 comp rx=0 tx=0).stop 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 shutdown_connections 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc644078120 0x7fc64407a5e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc664161b50 0x7fc6641504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc664161770 0x7fc6641f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 --2- 172.21.15.28:0/3370419965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6640d27f0 0x7fc6640b9ac0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 >> 172.21.15.28:0/3370419965 conn(0x7fc6640d4070 msgr2=0x7fc664160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 shutdown_connections 2024-09-10T20:41:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:22.066+0000 7fc669448640 1 -- 172.21.15.28:0/3370419965 wait complete. 2024-09-10T20:41:22.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:22.081 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cp ./volumes/_nogroup/subvol66221/.meta ./volumes/_nogroup/subvol45350) 2024-09-10T20:41:27.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol45350 2024-09-10T20:41:27.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 -- 172.21.15.28:0/1426724295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60070f90 msgr2=0x7ffa600cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 --2- 172.21.15.28:0/1426724295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60070f90 0x7ffa600cabd0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffa6006e640 tx=0x7ffa4802a560 comp rx=0 tx=0).stop 2024-09-10T20:41:27.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 -- 172.21.15.28:0/1426724295 shutdown_connections 2024-09-10T20:41:27.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 --2- 172.21.15.28:0/1426724295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60070f90 0x7ffa600cabd0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 --2- 172.21.15.28:0/1426724295 >> v2:172.21.15.159:3300/0 conn(0x7ffa60158a10 0x7ffa6015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 -- 172.21.15.28:0/1426724295 >> 172.21.15.28:0/1426724295 conn(0x7ffa600d0090 msgr2=0x7ffa600d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 -- 172.21.15.28:0/1426724295 shutdown_connections 2024-09-10T20:41:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.193+0000 7ffa659ad640 1 -- 172.21.15.28:0/1426724295 wait complete. 2024-09-10T20:41:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.194+0000 7ffa659ad640 1 Processor -- start 2024-09-10T20:41:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.194+0000 7ffa659ad640 1 -- start start 2024-09-10T20:41:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa659ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa60070f90 0x7ffa601f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa659ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa600cb110 0x7ffa601e4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa659ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60158a10 0x7ffa60073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa659ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa600cf770 con 0x7ffa60158a10 2024-09-10T20:41:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa659ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffa600ca790 con 0x7ffa60070f90 2024-09-10T20:41:27.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa659ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6015f170 con 0x7ffa600cb110 2024-09-10T20:41:27.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60158a10 0x7ffa60073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60158a10 0x7ffa60073620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41564/0 (socket says 172.21.15.28:41564) 2024-09-10T20:41:27.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa5f7fe640 1 -- 172.21.15.28:0/2829780826 learned_addr learned my addr 172.21.15.28:0/2829780826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:27.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa5effd640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa60070f90 0x7ffa601f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.195+0000 7ffa567fc640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa600cb110 0x7ffa601e4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5f7fe640 1 -- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa600cb110 msgr2=0x7ffa601e4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5f7fe640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa600cb110 0x7ffa601e4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5f7fe640 1 -- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa60070f90 msgr2=0x7ffa601f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5f7fe640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa60070f90 0x7ffa601f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5f7fe640 1 -- 172.21.15.28:0/2829780826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa601450e0 con 0x7ffa60158a10 2024-09-10T20:41:27.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5f7fe640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60158a10 0x7ffa60073620 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800c770 tx=0x7ffa5800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:27.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5cff9640 1 -- 172.21.15.28:0/2829780826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58003da0 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5cff9640 1 -- 172.21.15.28:0/2829780826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa58005070 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa601e4a80 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa5cff9640 1 -- 172.21.15.28:0/2829780826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58005500 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.196+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa60144600 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.197+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6015f620 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.198+0000 7ffa5cff9640 1 -- 172.21.15.28:0/2829780826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 10) ==== 100805+0+0 (secure 0 0 0) 0x7ffa5800a5e0 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.198+0000 7ffa5cff9640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffa400781e0 0x7ffa4007a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.198+0000 7ffa5cff9640 1 -- 172.21.15.28:0/2829780826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ffa58099ff0 con 0x7ffa60158a10 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.199+0000 7ffa5effd640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffa400781e0 0x7ffa4007a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.199+0000 7ffa5effd640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffa400781e0 0x7ffa4007a6a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009b30 tx=0x7ffa50005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:27.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.203+0000 7ffa5cff9640 1 -- 172.21.15.28:0/2829780826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffa58061fd0 con 0x7ffa60158a10 2024-09-10T20:41:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.415+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol45350", "target": ["mon-mgr", ""]}) -- 0x7ffa600c9fe0 con 0x7ffa400781e0 2024-09-10T20:41:27.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.443+0000 7ffa5cff9640 1 -- 172.21.15.28:0/2829780826 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7ffa600c9fe0 con 0x7ffa400781e0 2024-09-10T20:41:27.447 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol45350 2024-09-10T20:41:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffa400781e0 msgr2=0x7ffa4007a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffa400781e0 0x7ffa4007a6a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009b30 tx=0x7ffa50005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60158a10 msgr2=0x7ffa60073620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60158a10 0x7ffa60073620 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800c770 tx=0x7ffa5800cc30 comp rx=0 tx=0).stop 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 shutdown_connections 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffa400781e0 0x7ffa4007a6a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa60158a10 0x7ffa60073620 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa600cb110 0x7ffa601e4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 --2- 172.21.15.28:0/2829780826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa60070f90 0x7ffa601f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 >> 172.21.15.28:0/2829780826 conn(0x7ffa600d0090 msgr2=0x7ffa600afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 shutdown_connections 2024-09-10T20:41:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.445+0000 7ffa659ad640 1 -- 172.21.15.28:0/2829780826 wait complete. 2024-09-10T20:41:27.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol45350 2024-09-10T20:41:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.566+0000 7f4d662ed640 1 -- 172.21.15.28:0/2255081645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d48003fd0 msgr2=0x7f4d48004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.566+0000 7f4d662ed640 1 --2- 172.21.15.28:0/2255081645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d48003fd0 0x7f4d48004800 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009b50 tx=0x7f4d48005830 comp rx=0 tx=0).stop 2024-09-10T20:41:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.566+0000 7f4d6488a640 1 -- 172.21.15.28:0/2255081645 reap_dead start 2024-09-10T20:41:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.566+0000 7f4d662ed640 1 -- 172.21.15.28:0/2255081645 shutdown_connections 2024-09-10T20:41:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.566+0000 7f4d662ed640 1 -- 172.21.15.28:0/2255081645 >> 172.21.15.28:0/2255081645 conn(0x7f4d600d1f30 msgr2=0x7f4d600d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.566+0000 7f4d662ed640 1 -- 172.21.15.28:0/2255081645 shutdown_connections 2024-09-10T20:41:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.566+0000 7f4d662ed640 1 -- 172.21.15.28:0/2255081645 wait complete. 2024-09-10T20:41:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.567+0000 7f4d662ed640 1 Processor -- start 2024-09-10T20:41:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.567+0000 7f4d662ed640 1 -- start start 2024-09-10T20:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d662ed640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d48003fd0 0x7f4d600c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d662ed640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d600c5a60 0x7f4d601f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d662ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d601f66e0 0x7f4d601f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d662ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d600d1030 con 0x7f4d601f66e0 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d662ed640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4d600d0eb0 con 0x7f4d48003fd0 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d662ed640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4d600ce750 con 0x7f4d600c5a60 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d601f66e0 0x7f4d601f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d601f66e0 0x7f4d601f6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41594/0 (socket says 172.21.15.28:41594) 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 -- 172.21.15.28:0/1459246443 learned_addr learned my addr 172.21.15.28:0/1459246443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d5ffff640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d48003fd0 0x7f4d600c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d5f7fe640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d600c5a60 0x7f4d601f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 -- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d600c5a60 msgr2=0x7f4d601f5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d600c5a60 0x7f4d601f5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 -- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d48003fd0 msgr2=0x7f4d600c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d48003fd0 0x7f4d600c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.568+0000 7f4d6488a640 1 -- 172.21.15.28:0/1459246443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d600c8ca0 con 0x7f4d601f66e0 2024-09-10T20:41:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.569+0000 7f4d6488a640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d601f66e0 0x7f4d601f6010 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800e880 tx=0x7f4d5800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.569+0000 7f4d5cff9640 1 -- 172.21.15.28:0/1459246443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58015020 con 0x7f4d601f66e0 2024-09-10T20:41:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.569+0000 7f4d5cff9640 1 -- 172.21.15.28:0/1459246443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d5800cce0 con 0x7f4d601f66e0 2024-09-10T20:41:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.569+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d600cadc0 con 0x7f4d601f66e0 2024-09-10T20:41:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.569+0000 7f4d5cff9640 1 -- 172.21.15.28:0/1459246443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d580116c0 con 0x7f4d601f66e0 2024-09-10T20:41:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.569+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d60158c50 con 0x7f4d601f66e0 2024-09-10T20:41:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.570+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d6014abe0 con 0x7f4d601f66e0 2024-09-10T20:41:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.571+0000 7f4d5cff9640 1 -- 172.21.15.28:0/1459246443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 10) ==== 100805+0+0 (secure 0 0 0) 0x7f4d58011860 con 0x7f4d601f66e0 2024-09-10T20:41:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.571+0000 7f4d5cff9640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d4c0781b0 0x7f4d4c07a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.571+0000 7f4d5cff9640 1 -- 172.21.15.28:0/1459246443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f4d5809ac90 con 0x7f4d601f66e0 2024-09-10T20:41:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.571+0000 7f4d5ffff640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d4c0781b0 0x7f4d4c07a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.572+0000 7f4d5ffff640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d4c0781b0 0x7f4d4c07a670 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000da40 tx=0x7f4d5000d390 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.575+0000 7f4d5cff9640 1 -- 172.21.15.28:0/1459246443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4d6014abe0 con 0x7f4d601f66e0 2024-09-10T20:41:27.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.794+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol45350", "target": ["mon-mgr", ""]}) -- 0x7f4d6015ab30 con 0x7f4d4c0781b0 2024-09-10T20:41:27.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.800+0000 7f4d5cff9640 1 -- 172.21.15.28:0/1459246443 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f4d6015ab30 con 0x7f4d4c0781b0 2024-09-10T20:41:27.803 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol45350 2024-09-10T20:41:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d4c0781b0 msgr2=0x7f4d4c07a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d4c0781b0 0x7f4d4c07a670 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000da40 tx=0x7f4d5000d390 comp rx=0 tx=0).stop 2024-09-10T20:41:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d601f66e0 msgr2=0x7f4d601f6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d601f66e0 0x7f4d601f6010 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800e880 tx=0x7f4d5800ed40 comp rx=0 tx=0).stop 2024-09-10T20:41:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 shutdown_connections 2024-09-10T20:41:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d4c0781b0 0x7f4d4c07a670 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d601f66e0 0x7f4d601f6010 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d600c5a60 0x7f4d601f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 --2- 172.21.15.28:0/1459246443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d48003fd0 0x7f4d600c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 >> 172.21.15.28:0/1459246443 conn(0x7f4d600d1f30 msgr2=0x7f4d6015da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 shutdown_connections 2024-09-10T20:41:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:27.802+0000 7f4d662ed640 1 -- 172.21.15.28:0/1459246443 wait complete. 2024-09-10T20:41:27.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:27.817 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/0a5e9240dd2c8367b6b40b96abbbe6ee.meta) 2024-09-10T20:41:31.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:41:31.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:41:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:41:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:41:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:41:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol45350 2024-09-10T20:41:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:41:32.071 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:32.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume authorize cephfs subvol45350 alice 2024-09-10T20:41:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.190+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3647003810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c161850 msgr2=0x7fdb3c0d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.190+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3647003810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c161850 0x7fdb3c0d4a80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdb30005fd0 tx=0x7fdb3002a670 comp rx=0 tx=0).stop 2024-09-10T20:41:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.190+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3647003810 shutdown_connections 2024-09-10T20:41:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.190+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3647003810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c161850 0x7fdb3c0d4a80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.190+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3647003810 >> 172.21.15.28:0/3647003810 conn(0x7fdb3c0d3f90 msgr2=0x7fdb3c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.191+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3647003810 shutdown_connections 2024-09-10T20:41:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.191+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3647003810 wait complete. 2024-09-10T20:41:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.192+0000 7fdb41fa2640 1 Processor -- start 2024-09-10T20:41:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.192+0000 7fdb41fa2640 1 -- start start 2024-09-10T20:41:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb41fa2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb3c0b4c20 0x7fdb3c14cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb41fa2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb3c161850 0x7fdb3c0b8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb41fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c162000 0x7fdb3c0b8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb41fa2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c0d24f0 con 0x7fdb3c162000 2024-09-10T20:41:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb41fa2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c0d2370 con 0x7fdb3c0b4c20 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb41fa2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdb3c0d2670 con 0x7fdb3c161850 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c162000 0x7fdb3c0b8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c162000 0x7fdb3c0b8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36166/0 (socket says 172.21.15.28:36166) 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 -- 172.21.15.28:0/3510594119 learned_addr learned my addr 172.21.15.28:0/3510594119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3b7fe640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb3c0b4c20 0x7fdb3c14cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3affd640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb3c161850 0x7fdb3c0b8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 -- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb3c161850 msgr2=0x7fdb3c0b8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb3c161850 0x7fdb3c0b8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 -- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb3c0b4c20 msgr2=0x7fdb3c14cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb3c0b4c20 0x7fdb3c14cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 -- 172.21.15.28:0/3510594119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c1f7fa0 con 0x7fdb3c162000 2024-09-10T20:41:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.193+0000 7fdb3bfff640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c162000 0x7fdb3c0b8fc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400c780 tx=0x7fdb3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.194+0000 7fdb23fff640 1 -- 172.21.15.28:0/3510594119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34017070 con 0x7fdb3c162000 2024-09-10T20:41:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.194+0000 7fdb23fff640 1 -- 172.21.15.28:0/3510594119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb34005070 con 0x7fdb3c162000 2024-09-10T20:41:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.194+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3c1f8240 con 0x7fdb3c162000 2024-09-10T20:41:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.194+0000 7fdb23fff640 1 -- 172.21.15.28:0/3510594119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34005500 con 0x7fdb3c162000 2024-09-10T20:41:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.194+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb3c1f8800 con 0x7fdb3c162000 2024-09-10T20:41:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.195+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb3c0b9f10 con 0x7fdb3c162000 2024-09-10T20:41:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.195+0000 7fdb23fff640 1 -- 172.21.15.28:0/3510594119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 10) ==== 100805+0+0 (secure 0 0 0) 0x7fdb3400a5e0 con 0x7fdb3c162000 2024-09-10T20:41:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.196+0000 7fdb23fff640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdb280781b0 0x7fdb2807a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.196+0000 7fdb23fff640 1 -- 172.21.15.28:0/3510594119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fdb34099ab0 con 0x7fdb3c162000 2024-09-10T20:41:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.196+0000 7fdb3b7fe640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdb280781b0 0x7fdb2807a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.197+0000 7fdb3b7fe640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdb280781b0 0x7fdb2807a670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c001720 tx=0x7fdb2c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:32.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.200+0000 7fdb23fff640 1 -- 172.21.15.28:0/3510594119 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdb34061b80 con 0x7fdb3c162000 2024-09-10T20:41:32.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.412+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvol45350", "auth_id": "alice", "target": ["mon-mgr", ""]}) -- 0x7fdb3c0d4a00 con 0x7fdb280781b0 2024-09-10T20:41:32.425 INFO:tasks.ceph.mgr.x.smithi159.stderr:Creating meta for ID alice with tenant None 2024-09-10T20:41:32.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.450+0000 7fdb23fff640 1 -- 172.21.15.28:0/3510594119 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdb3c0d4a00 con 0x7fdb280781b0 2024-09-10T20:41:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdb280781b0 msgr2=0x7fdb2807a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdb280781b0 0x7fdb2807a670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c001720 tx=0x7fdb2c009040 comp rx=0 tx=0).stop 2024-09-10T20:41:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c162000 msgr2=0x7fdb3c0b8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c162000 0x7fdb3c0b8fc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400c780 tx=0x7fdb3400cc40 comp rx=0 tx=0).stop 2024-09-10T20:41:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 shutdown_connections 2024-09-10T20:41:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdb280781b0 0x7fdb2807a670 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb3c162000 0x7fdb3c0b8fc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdb3c161850 0x7fdb3c0b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 --2- 172.21.15.28:0/3510594119 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdb3c0b4c20 0x7fdb3c14cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 >> 172.21.15.28:0/3510594119 conn(0x7fdb3c0d3f90 msgr2=0x7fdb3c0ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:32.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 shutdown_connections 2024-09-10T20:41:32.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.452+0000 7fdb41fa2640 1 -- 172.21.15.28:0/3510594119 wait complete. 2024-09-10T20:41:32.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get client.alice --format=json-pretty 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.570+0000 7f990e9c6640 1 -- 172.21.15.28:0/215051014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990815ed20 msgr2=0x7f99080cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.570+0000 7f990e9c6640 1 --2- 172.21.15.28:0/215051014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990815ed20 0x7f99080cb210 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9908074830 tx=0x7f990002aa30 comp rx=0 tx=0).stop 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.570+0000 7f990e9c6640 1 -- 172.21.15.28:0/215051014 shutdown_connections 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.571+0000 7f990e9c6640 1 --2- 172.21.15.28:0/215051014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990815ed20 0x7f99080cb210 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.571+0000 7f990e9c6640 1 --2- 172.21.15.28:0/215051014 >> v2:172.21.15.159:3300/0 conn(0x7f99080cf0f0 0x7f990815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.571+0000 7f990e9c6640 1 -- 172.21.15.28:0/215051014 >> 172.21.15.28:0/215051014 conn(0x7f99080d0420 msgr2=0x7f99080d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.571+0000 7f990e9c6640 1 -- 172.21.15.28:0/215051014 shutdown_connections 2024-09-10T20:41:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.571+0000 7f990e9c6640 1 -- 172.21.15.28:0/215051014 wait complete. 2024-09-10T20:41:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.572+0000 7f990e9c6640 1 Processor -- start 2024-09-10T20:41:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.572+0000 7f990e9c6640 1 -- start start 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990e9c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99080cf0f0 0x7f99080b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990e9c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9908151cc0 0x7f990806f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990e9c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f990815ed20 0x7f99080ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990e9c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f990815f430 con 0x7f99080cf0f0 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990e9c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9908152120 con 0x7f990815ed20 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990e9c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f990815f5b0 con 0x7f9908151cc0 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99080cf0f0 0x7f99080b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99080cf0f0 0x7f99080b33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36182/0 (socket says 172.21.15.28:36182) 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 -- 172.21.15.28:0/370922841 learned_addr learned my addr 172.21.15.28:0/370922841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990cf63640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f990815ed20 0x7f99080ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f99077fe640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9908151cc0 0x7f990806f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 -- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9908151cc0 msgr2=0x7f990806f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9908151cc0 0x7f990806f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 -- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f990815ed20 msgr2=0x7f99080ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f990815ed20 0x7f99080ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 -- 172.21.15.28:0/370922841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99081f7cf0 con 0x7f99080cf0f0 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9907fff640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99080cf0f0 0x7f99080b33f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f98f800c410 tx=0x7f98f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9904ff9640 1 -- 172.21.15.28:0/370922841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8017070 con 0x7f99080cf0f0 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9904ff9640 1 -- 172.21.15.28:0/370922841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98f80035f0 con 0x7f99080cf0f0 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99081f7f90 con 0x7f99080cf0f0 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.573+0000 7f9904ff9640 1 -- 172.21.15.28:0/370922841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8003a80 con 0x7f99080cf0f0 2024-09-10T20:41:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.574+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99081f8550 con 0x7f99080cf0f0 2024-09-10T20:41:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.574+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990815e7e0 con 0x7f99080cf0f0 2024-09-10T20:41:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.575+0000 7f9904ff9640 1 -- 172.21.15.28:0/370922841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 10) ==== 100805+0+0 (secure 0 0 0) 0x7f98f8003c20 con 0x7f99080cf0f0 2024-09-10T20:41:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.576+0000 7f9904ff9640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98f40781b0 0x7f98f407a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.576+0000 7f9904ff9640 1 -- 172.21.15.28:0/370922841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f98f8099950 con 0x7f99080cf0f0 2024-09-10T20:41:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.576+0000 7f99077fe640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98f40781b0 0x7f98f407a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.576+0000 7f99077fe640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98f40781b0 0x7f98f407a670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f99080b65f0 tx=0x7f98fc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:32.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.580+0000 7f9904ff9640 1 -- 172.21.15.28:0/370922841 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f98f80010c0 con 0x7f99080cf0f0 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.835+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"} v 0) -- 0x7f99081f4750 con 0x7f99080cf0f0 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.835+0000 7f9904ff9640 1 -- 172.21.15.28:0/370922841 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"}]=0 v28) ==== 107+0+280 (secure 0 0 0) 0x7f98f80618f0 con 0x7f99080cf0f0 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.alice", 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD8ruBm/eTUGRAAjibYtCfoMLWr9ThEVaTxnQ==", 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol45350", 2024-09-10T20:41:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:41:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw pool=cephfs_data" 2024-09-10T20:41:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:32.840 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:41:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98f40781b0 msgr2=0x7f98f407a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98f40781b0 0x7f98f407a670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f99080b65f0 tx=0x7f98fc008040 comp rx=0 tx=0).stop 2024-09-10T20:41:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99080cf0f0 msgr2=0x7f99080b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99080cf0f0 0x7f99080b33f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f98f800c410 tx=0x7f98f800c8d0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 shutdown_connections 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f98f40781b0 0x7f98f407a670 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f990815ed20 0x7f99080ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9908151cc0 0x7f990806f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 --2- 172.21.15.28:0/370922841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99080cf0f0 0x7f99080b33f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 >> 172.21.15.28:0/370922841 conn(0x7f99080d0420 msgr2=0x7f9908158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 shutdown_connections 2024-09-10T20:41:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.837+0000 7f990e9c6640 1 -- 172.21.15.28:0/370922841 wait complete. 2024-09-10T20:41:32.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol45350 2024-09-10T20:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 -- 172.21.15.28:0/2863250778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840b3580 msgr2=0x7f558406e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 --2- 172.21.15.28:0/2863250778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840b3580 0x7f558406e400 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f558415d260 tx=0x7f556c02a560 comp rx=0 tx=0).stop 2024-09-10T20:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 -- 172.21.15.28:0/2863250778 shutdown_connections 2024-09-10T20:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 --2- 172.21.15.28:0/2863250778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840b3580 0x7f558406e400 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 --2- 172.21.15.28:0/2863250778 >> v2:172.21.15.159:3300/0 conn(0x7f558406e940 0x7f55840ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 -- 172.21.15.28:0/2863250778 >> 172.21.15.28:0/2863250778 conn(0x7f558406d110 msgr2=0x7f55840b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:32.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 -- 172.21.15.28:0/2863250778 shutdown_connections 2024-09-10T20:41:32.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.957+0000 7f5589353640 1 -- 172.21.15.28:0/2863250778 wait complete. 2024-09-10T20:41:32.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.958+0000 7f5589353640 1 Processor -- start 2024-09-10T20:41:32.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.958+0000 7f5589353640 1 -- start start 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5589353640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558406e940 0x7f558413b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5589353640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584152e00 0x7f558413dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5589353640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f558413e2a0 0x7f558413e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5589353640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f558415f820 con 0x7f558406e940 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5589353640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f55840cf5b0 con 0x7f558413e2a0 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5589353640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f558415f9a0 con 0x7f5584152e00 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558406e940 0x7f558413b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558406e940 0x7f558413b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36198/0 (socket says 172.21.15.28:36198) 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 -- 172.21.15.28:0/123254926 learned_addr learned my addr 172.21.15.28:0/123254926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f557a7fc640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584152e00 0x7f558413dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f55837fe640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f558413e2a0 0x7f558413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 -- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584152e00 msgr2=0x7f558413dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584152e00 0x7f558413dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 -- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f558413e2a0 msgr2=0x7f558413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f558413e2a0 0x7f558413e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.959+0000 7f5582ffd640 1 -- 172.21.15.28:0/123254926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558413ec70 con 0x7f558406e940 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.960+0000 7f5582ffd640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558406e940 0x7f558413b3f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f557400c3d0 tx=0x7f557400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.960+0000 7f557bfff640 1 -- 172.21.15.28:0/123254926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574017070 con 0x7f558406e940 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.960+0000 7f557bfff640 1 -- 172.21.15.28:0/123254926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55740053c0 con 0x7f558406e940 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.960+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558415c350 con 0x7f558406e940 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.960+0000 7f557bfff640 1 -- 172.21.15.28:0/123254926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574003850 con 0x7f558406e940 2024-09-10T20:41:32.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.960+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558413bee0 con 0x7f558406e940 2024-09-10T20:41:32.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.961+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558413bcd0 con 0x7f558406e940 2024-09-10T20:41:32.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.961+0000 7f557bfff640 1 -- 172.21.15.28:0/123254926 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 10) ==== 100805+0+0 (secure 0 0 0) 0x7f55740039f0 con 0x7f558406e940 2024-09-10T20:41:32.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.962+0000 7f557bfff640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f55640781b0 0x7f556407a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:32.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.962+0000 7f557bfff640 1 -- 172.21.15.28:0/123254926 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5574099790 con 0x7f558406e940 2024-09-10T20:41:32.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.962+0000 7f557a7fc640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f55640781b0 0x7f556407a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:32.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.963+0000 7f557a7fc640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f55640781b0 0x7f556407a670 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f556c033000 tx=0x7f556c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:32.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:32.966+0000 7f557bfff640 1 -- 172.21.15.28:0/123254926 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5574061810 con 0x7f558406e940 2024-09-10T20:41:33.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.178+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol45350", "target": ["mon-mgr", ""]}) -- 0x7f5584160270 con 0x7f55640781b0 2024-09-10T20:41:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.192+0000 7f557bfff640 1 -- 172.21.15.28:0/123254926 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5584160270 con 0x7f55640781b0 2024-09-10T20:41:33.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f55640781b0 msgr2=0x7f556407a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f55640781b0 0x7f556407a670 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f556c033000 tx=0x7f556c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:33.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558406e940 msgr2=0x7f558413b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558406e940 0x7f558413b3f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f557400c3d0 tx=0x7f557400c890 comp rx=0 tx=0).stop 2024-09-10T20:41:33.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 shutdown_connections 2024-09-10T20:41:33.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f55640781b0 0x7f556407a670 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f558413e2a0 0x7f558413e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5584152e00 0x7f558413dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 --2- 172.21.15.28:0/123254926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558406e940 0x7f558413b3f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 >> 172.21.15.28:0/123254926 conn(0x7f558406d110 msgr2=0x7f5584150720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 shutdown_connections 2024-09-10T20:41:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.194+0000 7f5589353640 1 -- 172.21.15.28:0/123254926 wait complete. 2024-09-10T20:41:33.199 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.195+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.199 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.195+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.199 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.195+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.200 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.195+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.200 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.195+0000 7f97d4bcf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol66221 2024-09-10T20:41:33.214 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.210+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.214 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.210+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.214 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.210+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.214 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.210+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.214 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:41:33.210+0000 7f97d5bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:41:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.316+0000 7fc525f24640 1 -- 172.21.15.28:0/40348739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508004d40 msgr2=0x7fc5080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.316+0000 7fc525f24640 1 --2- 172.21.15.28:0/40348739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508004d40 0x7fc5080051c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc51800c640 tx=0x7fc508005830 comp rx=0 tx=0).stop 2024-09-10T20:41:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.316+0000 7fc51ffff640 1 -- 172.21.15.28:0/40348739 reap_dead start 2024-09-10T20:41:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.316+0000 7fc525f24640 1 -- 172.21.15.28:0/40348739 shutdown_connections 2024-09-10T20:41:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.316+0000 7fc525f24640 1 -- 172.21.15.28:0/40348739 >> 172.21.15.28:0/40348739 conn(0x7fc5200d1e60 msgr2=0x7fc5200d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.316+0000 7fc525f24640 1 -- 172.21.15.28:0/40348739 shutdown_connections 2024-09-10T20:41:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.317+0000 7fc525f24640 1 -- 172.21.15.28:0/40348739 wait complete. 2024-09-10T20:41:33.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.318+0000 7fc525f24640 1 Processor -- start 2024-09-10T20:41:33.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.318+0000 7fc525f24640 1 -- start start 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc525f24640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc508004d40 0x7fc5200c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc525f24640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5201f2060 0x7fc5200b6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc525f24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5200b6d70 0x7fc5200afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc525f24640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5200cf770 con 0x7fc5200b6d70 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc525f24640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc5200cf5f0 con 0x7fc5201f2060 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc525f24640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc5200b2990 con 0x7fc508004d40 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5200b6d70 0x7fc5200afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5200b6d70 0x7fc5200afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36212/0 (socket says 172.21.15.28:36212) 2024-09-10T20:41:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 -- 172.21.15.28:0/904192966 learned_addr learned my addr 172.21.15.28:0/904192966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51effd640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5201f2060 0x7fc5200b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51f7fe640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc508004d40 0x7fc5200c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 -- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc508004d40 msgr2=0x7fc5200c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc508004d40 0x7fc5200c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 -- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5201f2060 msgr2=0x7fc5200b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5201f2060 0x7fc5200b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 -- 172.21.15.28:0/904192966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5200c5990 con 0x7fc5200b6d70 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51f7fe640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc508004d40 0x7fc5200c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51effd640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5201f2060 0x7fc5200b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc51ffff640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5200b6d70 0x7fc5200afb10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc5180012e0 tx=0x7fc518004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.319+0000 7fc507fff640 1 -- 172.21.15.28:0/904192966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51802b7e0 con 0x7fc5200b6d70 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.320+0000 7fc507fff640 1 -- 172.21.15.28:0/904192966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc51802b980 con 0x7fc5200b6d70 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.320+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5200c5c70 con 0x7fc5200b6d70 2024-09-10T20:41:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.320+0000 7fc507fff640 1 -- 172.21.15.28:0/904192966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51802bdb0 con 0x7fc5200b6d70 2024-09-10T20:41:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.320+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5200c6230 con 0x7fc5200b6d70 2024-09-10T20:41:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.321+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5200c68f0 con 0x7fc5200b6d70 2024-09-10T20:41:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.321+0000 7fc507fff640 1 -- 172.21.15.28:0/904192966 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 10) ==== 100805+0+0 (secure 0 0 0) 0x7fc5180060a0 con 0x7fc5200b6d70 2024-09-10T20:41:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.322+0000 7fc507fff640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc50c0781b0 0x7fc50c07a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.322+0000 7fc507fff640 1 -- 172.21.15.28:0/904192966 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fc5180b9cb0 con 0x7fc5200b6d70 2024-09-10T20:41:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.322+0000 7fc51f7fe640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc50c0781b0 0x7fc50c07a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.322+0000 7fc51f7fe640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc50c0781b0 0x7fc50c07a670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc510009b40 tx=0x7fc510005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:33.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.326+0000 7fc507fff640 1 -- 172.21.15.28:0/904192966 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc5200c68f0 con 0x7fc5200b6d70 2024-09-10T20:41:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.538+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol66221", "target": ["mon-mgr", ""]}) -- 0x7fc5200aed60 con 0x7fc50c0781b0 2024-09-10T20:41:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.555+0000 7fc507fff640 1 -- 172.21.15.28:0/904192966 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc5200aed60 con 0x7fc50c0781b0 2024-09-10T20:41:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc50c0781b0 msgr2=0x7fc50c07a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc50c0781b0 0x7fc50c07a670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc510009b40 tx=0x7fc510005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5200b6d70 msgr2=0x7fc5200afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5200b6d70 0x7fc5200afb10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc5180012e0 tx=0x7fc518004bb0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 shutdown_connections 2024-09-10T20:41:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc50c0781b0 0x7fc50c07a670 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5200b6d70 0x7fc5200afb10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5201f2060 0x7fc5200b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 --2- 172.21.15.28:0/904192966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc508004d40 0x7fc5200c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 >> 172.21.15.28:0/904192966 conn(0x7fc5200d1e60 msgr2=0x7fc5200b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 shutdown_connections 2024-09-10T20:41:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.558+0000 7fc525f24640 1 -- 172.21.15.28:0/904192966 wait complete. 2024-09-10T20:41:33.573 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-10T20:41:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:41:33.627 INFO:teuthology.orchestra.run.smithi028.stdout:2 2024-09-10T20:41:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:41:33.634 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:41:33.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:41:33.689 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:41:33.740 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:41:33.740 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:41:33.740 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:33.740 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:33.757 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:33.757 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:33.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:41:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.956+0000 7f58f37f8640 1 -- 172.21.15.28:0/1029911568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d4003a00 msgr2=0x7f58d4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.956+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1029911568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d4003a00 0x7f58d4003de0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009ed0 tx=0x7f58d4005830 comp rx=0 tx=0).stop 2024-09-10T20:41:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.956+0000 7f58f37f8640 1 -- 172.21.15.28:0/1029911568 shutdown_connections 2024-09-10T20:41:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.956+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1029911568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d4003a00 0x7f58d4003de0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.956+0000 7f58f37f8640 1 -- 172.21.15.28:0/1029911568 >> 172.21.15.28:0/1029911568 conn(0x7f58ec0d1e40 msgr2=0x7f58ec0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.956+0000 7f58f37f8640 1 -- 172.21.15.28:0/1029911568 shutdown_connections 2024-09-10T20:41:33.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.956+0000 7f58f37f8640 1 -- 172.21.15.28:0/1029911568 wait complete. 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.957+0000 7f58f37f8640 1 Processor -- start 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.957+0000 7f58f37f8640 1 -- start start 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f37f8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4003a00 0x7f58ec15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f37f8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58ec0cacb0 0x7f58ec1532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f37f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ec160bc0 0x7f58ec161070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f37f8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec0d1350 con 0x7f58ec160bc0 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f37f8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec0d11d0 con 0x7f58ec0cacb0 2024-09-10T20:41:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f37f8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f58ec15e490 con 0x7f58d4003a00 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ec160bc0 0x7f58ec161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ec160bc0 0x7f58ec161070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36222/0 (socket says 172.21.15.28:36222) 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 -- 172.21.15.28:0/1941788874 learned_addr learned my addr 172.21.15.28:0/1941788874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f0d93640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58ec0cacb0 0x7f58ec1532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1594640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4003a00 0x7f58ec15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 -- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4003a00 msgr2=0x7f58ec15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4003a00 0x7f58ec15b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 -- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58ec0cacb0 msgr2=0x7f58ec1532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58ec0cacb0 0x7f58ec1532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 -- 172.21.15.28:0/1941788874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ec09ee40 con 0x7f58ec160bc0 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.958+0000 7f58f1d95640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ec160bc0 0x7f58ec161070 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f58ec161a90 tx=0x7f58e400ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.959+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4015020 con 0x7f58ec160bc0 2024-09-10T20:41:33.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.959+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58e40109a0 con 0x7f58ec160bc0 2024-09-10T20:41:33.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.959+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ec0cb140 con 0x7f58ec160bc0 2024-09-10T20:41:33.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.959+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4010df0 con 0x7f58ec160bc0 2024-09-10T20:41:33.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.959+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ec09f3a0 con 0x7f58ec160bc0 2024-09-10T20:41:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.960+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58ec0cea80 con 0x7f58ec160bc0 2024-09-10T20:41:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.960+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 10) ==== 100805+0+0 (secure 0 0 0) 0x7f58e40193b0 con 0x7f58ec160bc0 2024-09-10T20:41:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.961+0000 7f58e9ffb640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f58d80781b0 0x7f58d807a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.961+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f58e409a620 con 0x7f58ec160bc0 2024-09-10T20:41:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.961+0000 7f58f1594640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f58d80781b0 0x7f58d807a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.962+0000 7f58f1594640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f58d80781b0 0x7f58d807a670 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58dc035000 tx=0x7f58dc004a40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:33.965+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f58e401f030 con 0x7f58ec160bc0 2024-09-10T20:41:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.110+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f58e405e960 con 0x7f58ec160bc0 2024-09-10T20:41:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.162+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f58ec0cb420 con 0x7f58ec160bc0 2024-09-10T20:41:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.162+0000 7f58e9ffb640 1 -- 172.21.15.28:0/1941788874 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f58ec09ee40 con 0x7f58ec160bc0 2024-09-10T20:41:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:41:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.164+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f58d80781b0 msgr2=0x7f58d807a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.164+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f58d80781b0 0x7f58d807a670 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58dc035000 tx=0x7f58dc004a40 comp rx=0 tx=0).stop 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ec160bc0 msgr2=0x7f58ec161070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ec160bc0 0x7f58ec161070 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f58ec161a90 tx=0x7f58e400ea20 comp rx=0 tx=0).stop 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 shutdown_connections 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f58d80781b0 0x7f58d807a670 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ec160bc0 0x7f58ec161070 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58ec0cacb0 0x7f58ec1532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 --2- 172.21.15.28:0/1941788874 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58d4003a00 0x7f58ec15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 >> 172.21.15.28:0/1941788874 conn(0x7f58ec0d1e40 msgr2=0x7f58ec0cd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 shutdown_connections 2024-09-10T20:41:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.165+0000 7f58f37f8640 1 -- 172.21.15.28:0/1941788874 wait complete. 2024-09-10T20:41:34.179 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:41:34.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:34.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:41:34.233 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:41:34.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:34.233 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:41:34.256 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:41:34.256 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:34.256 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:34.273 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:34.273 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:34.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.431+0000 7f7a38594640 1 -- 172.21.15.28:0/1849486356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300cd3d0 msgr2=0x7f7a300cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.431+0000 7f7a38594640 1 --2- 172.21.15.28:0/1849486356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300cd3d0 0x7f7a300cc570 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7a300ce0d0 tx=0x7f7a2c02a530 comp rx=0 tx=0).stop 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.432+0000 7f7a38594640 1 -- 172.21.15.28:0/1849486356 shutdown_connections 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.432+0000 7f7a38594640 1 --2- 172.21.15.28:0/1849486356 >> v2:172.21.15.159:3300/0 conn(0x7f7a300d07a0 0x7f7a300b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.432+0000 7f7a38594640 1 --2- 172.21.15.28:0/1849486356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300cd3d0 0x7f7a300cc570 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.432+0000 7f7a38594640 1 -- 172.21.15.28:0/1849486356 >> 172.21.15.28:0/1849486356 conn(0x7f7a3006cf00 msgr2=0x7f7a3006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.432+0000 7f7a38594640 1 -- 172.21.15.28:0/1849486356 shutdown_connections 2024-09-10T20:41:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.432+0000 7f7a38594640 1 -- 172.21.15.28:0/1849486356 wait complete. 2024-09-10T20:41:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.433+0000 7f7a38594640 1 Processor -- start 2024-09-10T20:41:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.433+0000 7f7a38594640 1 -- start start 2024-09-10T20:41:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a38594640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a300b4340 0x7f7a300726c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a38594640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a300cd3d0 0x7f7a3014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a38594640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300d07a0 0x7f7a30073e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a38594640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300b33c0 con 0x7f7a300d07a0 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a38594640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300b4aa0 con 0x7f7a300cd3d0 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a38594640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7a300b3540 con 0x7f7a300b4340 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300d07a0 0x7f7a30073e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300d07a0 0x7f7a30073e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36236/0 (socket says 172.21.15.28:36236) 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 -- 172.21.15.28:0/632343069 learned_addr learned my addr 172.21.15.28:0/632343069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36330640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a300b4340 0x7f7a300726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a35b2f640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a300cd3d0 0x7f7a3014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 -- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a300b4340 msgr2=0x7f7a300726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a300b4340 0x7f7a300726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 -- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a300cd3d0 msgr2=0x7f7a3014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a300cd3d0 0x7f7a3014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 -- 172.21.15.28:0/632343069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a30072dc0 con 0x7f7a300d07a0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a36b31640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300d07a0 0x7f7a30073e60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800cab0 tx=0x7f7a2800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a22ffd640 1 -- 172.21.15.28:0/632343069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28007d90 con 0x7f7a300d07a0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a22ffd640 1 -- 172.21.15.28:0/632343069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a28003990 con 0x7f7a300d07a0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a22ffd640 1 -- 172.21.15.28:0/632343069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28003dc0 con 0x7f7a300d07a0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.434+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a300730a0 con 0x7f7a300d07a0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.435+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a30073660 con 0x7f7a300d07a0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.435+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a300b39f0 con 0x7f7a300d07a0 2024-09-10T20:41:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.436+0000 7f7a22ffd640 1 -- 172.21.15.28:0/632343069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f7a28003f60 con 0x7f7a300d07a0 2024-09-10T20:41:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.437+0000 7f7a22ffd640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7a14078300 0x7f7a1407a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.437+0000 7f7a22ffd640 1 -- 172.21.15.28:0/632343069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f7a2809aaa0 con 0x7f7a300d07a0 2024-09-10T20:41:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.437+0000 7f7a36330640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7a14078300 0x7f7a1407a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.437+0000 7f7a36330640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7a14078300 0x7f7a1407a7c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c034000 tx=0x7f7a2c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:34.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.441+0000 7f7a22ffd640 1 -- 172.21.15.28:0/632343069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7a28014030 con 0x7f7a300d07a0 2024-09-10T20:41:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.640+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7a3006d720 con 0x7f7a300d07a0 2024-09-10T20:41:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.640+0000 7f7a22ffd640 1 -- 172.21.15.28:0/632343069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f7a280629c0 con 0x7f7a300d07a0 2024-09-10T20:41:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7a14078300 msgr2=0x7f7a1407a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7a14078300 0x7f7a1407a7c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c034000 tx=0x7f7a2c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300d07a0 msgr2=0x7f7a30073e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300d07a0 0x7f7a30073e60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800cab0 tx=0x7f7a2800cf70 comp rx=0 tx=0).stop 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 shutdown_connections 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7a14078300 0x7f7a1407a7c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a300d07a0 0x7f7a30073e60 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a300cd3d0 0x7f7a3014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 --2- 172.21.15.28:0/632343069 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a300b4340 0x7f7a300726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 >> 172.21.15.28:0/632343069 conn(0x7f7a3006cf00 msgr2=0x7f7a3009be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 shutdown_connections 2024-09-10T20:41:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.643+0000 7f7a38594640 1 -- 172.21.15.28:0/632343069 wait complete. 2024-09-10T20:41:34.657 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:41:34.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:34.658 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:41:34.687 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:41:34.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:41:34.687 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:41:34.743 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:41:34.743 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:34.743 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:34.760 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:41:34.760 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:41:34.760 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:34.760 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:34.801 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:41:34.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.902+0000 7eff56072640 1 -- 172.21.15.28:0/1135068374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500cd3d0 msgr2=0x7eff500cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.902+0000 7eff56072640 1 --2- 172.21.15.28:0/1135068374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500cd3d0 0x7eff500cc570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7eff40009830 tx=0x7eff4002a4a0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.903+0000 7eff56072640 1 -- 172.21.15.28:0/1135068374 shutdown_connections 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.903+0000 7eff56072640 1 --2- 172.21.15.28:0/1135068374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff500b4340 0x7eff500b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.903+0000 7eff56072640 1 --2- 172.21.15.28:0/1135068374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500cd3d0 0x7eff500cc570 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.903+0000 7eff56072640 1 -- 172.21.15.28:0/1135068374 >> 172.21.15.28:0/1135068374 conn(0x7eff5006cf00 msgr2=0x7eff5006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.903+0000 7eff56072640 1 -- 172.21.15.28:0/1135068374 shutdown_connections 2024-09-10T20:41:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.903+0000 7eff56072640 1 -- 172.21.15.28:0/1135068374 wait complete. 2024-09-10T20:41:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.904+0000 7eff56072640 1 Processor -- start 2024-09-10T20:41:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.904+0000 7eff56072640 1 -- start start 2024-09-10T20:41:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff56072640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500b4340 0x7eff500b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff56072640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff500cd3d0 0x7eff501589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff56072640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff500d0090 0x7eff501f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff56072640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff500d0980 con 0x7eff500cd3d0 2024-09-10T20:41:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff56072640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7eff500d0800 con 0x7eff500b4340 2024-09-10T20:41:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff56072640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7eff500b3590 con 0x7eff500d0090 2024-09-10T20:41:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff500cd3d0 0x7eff501589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff500cd3d0 0x7eff501589f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36248/0 (socket says 172.21.15.28:36248) 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 -- 172.21.15.28:0/1442327043 learned_addr learned my addr 172.21.15.28:0/1442327043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4f7fe640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500b4340 0x7eff500b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4ffff640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff500d0090 0x7eff501f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 -- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff500d0090 msgr2=0x7eff501f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff500d0090 0x7eff501f5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 -- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500b4340 msgr2=0x7eff500b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500b4340 0x7eff500b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 -- 172.21.15.28:0/1442327043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff500b0820 con 0x7eff500cd3d0 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4f7fe640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500b4340 0x7eff500b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff4effd640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff500cd3d0 0x7eff501589f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7eff4400e9f0 tx=0x7eff4400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.905+0000 7eff37fff640 1 -- 172.21.15.28:0/1442327043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4400cd80 con 0x7eff500cd3d0 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.906+0000 7eff37fff640 1 -- 172.21.15.28:0/1442327043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff44010430 con 0x7eff500cd3d0 2024-09-10T20:41:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.906+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff500af780 con 0x7eff500cd3d0 2024-09-10T20:41:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.906+0000 7eff37fff640 1 -- 172.21.15.28:0/1442327043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44010840 con 0x7eff500cd3d0 2024-09-10T20:41:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.906+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff500afd40 con 0x7eff500cd3d0 2024-09-10T20:41:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.906+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff500b3ad0 con 0x7eff500cd3d0 2024-09-10T20:41:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.907+0000 7eff37fff640 1 -- 172.21.15.28:0/1442327043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7eff440109e0 con 0x7eff500cd3d0 2024-09-10T20:41:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.908+0000 7eff37fff640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7eff3c078300 0x7eff3c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.908+0000 7eff37fff640 1 -- 172.21.15.28:0/1442327043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7eff4409af60 con 0x7eff500cd3d0 2024-09-10T20:41:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.908+0000 7eff4f7fe640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7eff3c078300 0x7eff3c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.908+0000 7eff4f7fe640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7eff3c078300 0x7eff3c07a7c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7eff40034000 tx=0x7eff40004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:34.912+0000 7eff37fff640 1 -- 172.21.15.28:0/1442327043 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7eff44063580 con 0x7eff500cd3d0 2024-09-10T20:41:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.132+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff5006d620 con 0x7eff500cd3d0 2024-09-10T20:41:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.133+0000 7eff37fff640 1 -- 172.21.15.28:0/1442327043 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5223 (secure 0 0 0) 0x7eff44062e80 con 0x7eff500cd3d0 2024-09-10T20:41:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:35.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"btime":"2024-09-10T20:41:20:801101+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/485554639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":485554639},{"type":"v1","addr":"172.21.15.159:6836","nonce":485554639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1807026586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1807026586},{"type":"v1","addr":"172.21.15.159:6837","nonce":1807026586}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/975621681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":975621681},{"type":"v1","addr":"172.21.15.28:6834","nonce":975621681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:11.137947+0000","modified":"2024-09-10T20:41:14.144276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3887400201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3887400201},{"type":"v1","addr":"172.21.15.28:6835","nonce":3887400201}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-10T20:41:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-09-10T20:41:35.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7eff3c078300 msgr2=0x7eff3c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7eff3c078300 0x7eff3c07a7c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7eff40034000 tx=0x7eff40004730 comp rx=0 tx=0).stop 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff500cd3d0 msgr2=0x7eff501589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff500cd3d0 0x7eff501589f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7eff4400e9f0 tx=0x7eff4400eeb0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 shutdown_connections 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7eff3c078300 0x7eff3c07a7c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff500d0090 0x7eff501f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff500cd3d0 0x7eff501589f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 --2- 172.21.15.28:0/1442327043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff500b4340 0x7eff500b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.135+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 >> 172.21.15.28:0/1442327043 conn(0x7eff5006cf00 msgr2=0x7eff5009bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.136+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 shutdown_connections 2024-09-10T20:41:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.136+0000 7eff56072640 1 -- 172.21.15.28:0/1442327043 wait complete. 2024-09-10T20:41:35.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.252+0000 7f969e0a9640 1 -- 172.21.15.28:0/2280300997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9680003a00 msgr2=0x7f9680003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.252+0000 7f969e0a9640 1 --2- 172.21.15.28:0/2280300997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9680003a00 0x7f9680003e20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9688009b50 tx=0x7f9680005830 comp rx=0 tx=0).stop 2024-09-10T20:41:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.252+0000 7f9697fff640 1 -- 172.21.15.28:0/2280300997 reap_dead start 2024-09-10T20:41:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.252+0000 7f969e0a9640 1 -- 172.21.15.28:0/2280300997 shutdown_connections 2024-09-10T20:41:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.252+0000 7f969e0a9640 1 -- 172.21.15.28:0/2280300997 >> 172.21.15.28:0/2280300997 conn(0x7f969806cf00 msgr2=0x7f969806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.252+0000 7f969e0a9640 1 -- 172.21.15.28:0/2280300997 shutdown_connections 2024-09-10T20:41:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.252+0000 7f969e0a9640 1 -- 172.21.15.28:0/2280300997 wait complete. 2024-09-10T20:41:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.253+0000 7f969e0a9640 1 Processor -- start 2024-09-10T20:41:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f969e0a9640 1 -- start start 2024-09-10T20:41:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f969e0a9640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9680003a00 0x7f969814f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f969e0a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f969815d0c0 0x7f969814c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f969e0a9640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f969815d500 0x7f969814c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f969e0a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96980d07a0 con 0x7f969815d0c0 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f969e0a9640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f96980d0620 con 0x7f9680003a00 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f969e0a9640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f96980ce030 con 0x7f969815d500 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f9696ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f969815d0c0 0x7f969814c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f9696ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f969815d0c0 0x7f969814c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36268/0 (socket says 172.21.15.28:36268) 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f9696ffd640 1 -- 172.21.15.28:0/1955357976 learned_addr learned my addr 172.21.15.28:0/1955357976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f96977fe640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9680003a00 0x7f969814f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.254+0000 7f9697fff640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f969815d500 0x7f969814c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f9696ffd640 1 -- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f969815d500 msgr2=0x7f969814c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f9696ffd640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f969815d500 0x7f969814c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f9696ffd640 1 -- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9680003a00 msgr2=0x7f969814f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f9696ffd640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9680003a00 0x7f969814f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f9696ffd640 1 -- 172.21.15.28:0/1955357976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96980b14f0 con 0x7f969815d0c0 2024-09-10T20:41:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f9696ffd640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f969815d0c0 0x7f969814c0e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f968c001400 tx=0x7f968c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f967ffff640 1 -- 172.21.15.28:0/1955357976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c015070 con 0x7f969815d0c0 2024-09-10T20:41:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f967ffff640 1 -- 172.21.15.28:0/1955357976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f968c011900 con 0x7f969815d0c0 2024-09-10T20:41:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f967ffff640 1 -- 172.21.15.28:0/1955357976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c011d50 con 0x7f969815d0c0 2024-09-10T20:41:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96980b17d0 con 0x7f969815d0c0 2024-09-10T20:41:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.255+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96980b1d90 con 0x7f969815d0c0 2024-09-10T20:41:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.256+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96980b2280 con 0x7f969815d0c0 2024-09-10T20:41:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.257+0000 7f967ffff640 1 -- 172.21.15.28:0/1955357976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f968c010420 con 0x7f969815d0c0 2024-09-10T20:41:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.257+0000 7f967ffff640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9684078300 0x7f968407a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.257+0000 7f967ffff640 1 -- 172.21.15.28:0/1955357976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f968c09b5b0 con 0x7f969815d0c0 2024-09-10T20:41:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.257+0000 7f96977fe640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9684078300 0x7f968407a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.258+0000 7f96977fe640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9684078300 0x7f968407a7c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9688036000 tx=0x7f96880046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.261+0000 7f967ffff640 1 -- 172.21.15.28:0/1955357976 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f968c063450 con 0x7f969815d0c0 2024-09-10T20:41:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.487+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96980b2ba0 con 0x7f969815d0c0 2024-09-10T20:41:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.488+0000 7f967ffff640 1 -- 172.21.15.28:0/1955357976 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5223 (secure 0 0 0) 0x7f968c067e80 con 0x7f969815d0c0 2024-09-10T20:41:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:35.492 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"btime":"2024-09-10T20:41:20:801101+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/485554639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":485554639},{"type":"v1","addr":"172.21.15.159:6836","nonce":485554639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1807026586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1807026586},{"type":"v1","addr":"172.21.15.159:6837","nonce":1807026586}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/975621681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":975621681},{"type":"v1","addr":"172.21.15.28:6834","nonce":975621681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:11.137947+0000","modified":"2024-09-10T20:41:14.144276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3887400201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3887400201},{"type":"v1","addr":"172.21.15.28:6835","nonce":3887400201}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-10T20:41:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-09-10T20:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9684078300 msgr2=0x7f968407a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9684078300 0x7f968407a7c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9688036000 tx=0x7f96880046c0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f969815d0c0 msgr2=0x7f969814c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f969815d0c0 0x7f969814c0e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f968c001400 tx=0x7f968c0039c0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 shutdown_connections 2024-09-10T20:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9684078300 0x7f968407a7c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f969815d500 0x7f969814c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f969815d0c0 0x7f969814c0e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 --2- 172.21.15.28:0/1955357976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9680003a00 0x7f969814f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 >> 172.21.15.28:0/1955357976 conn(0x7f969806cf00 msgr2=0x7f9698072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 shutdown_connections 2024-09-10T20:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.490+0000 7f969e0a9640 1 -- 172.21.15.28:0/1955357976 wait complete. 2024-09-10T20:41:35.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.606+0000 7fe5f1416640 1 -- 172.21.15.28:0/68454336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec162060 msgr2=0x7fe5ec0d16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.606+0000 7fe5f1416640 1 --2- 172.21.15.28:0/68454336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec162060 0x7fe5ec0d16e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009830 tx=0x7fe5e002a5c0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.606+0000 7fe5f1416640 1 -- 172.21.15.28:0/68454336 shutdown_connections 2024-09-10T20:41:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.606+0000 7fe5f1416640 1 --2- 172.21.15.28:0/68454336 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec162060 0x7fe5ec0d16e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.606+0000 7fe5f1416640 1 -- 172.21.15.28:0/68454336 >> 172.21.15.28:0/68454336 conn(0x7fe5ec06dbf0 msgr2=0x7fe5ec06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.606+0000 7fe5f1416640 1 -- 172.21.15.28:0/68454336 shutdown_connections 2024-09-10T20:41:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.606+0000 7fe5f1416640 1 -- 172.21.15.28:0/68454336 wait complete. 2024-09-10T20:41:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.607+0000 7fe5f1416640 1 Processor -- start 2024-09-10T20:41:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5f1416640 1 -- start start 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5f1416640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5ec146a80 0x7fe5ec0add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5f1416640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec146e60 0x7fe5ec0b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5f1416640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ec162060 0x7fe5ec0b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5f1416640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec06f8e0 con 0x7fe5ec162060 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5f1416640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec06f760 con 0x7fe5ec146e60 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5f1416640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe5ec06fa60 con 0x7fe5ec146a80 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5eb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ec162060 0x7fe5ec0b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5eb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ec162060 0x7fe5ec0b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36274/0 (socket says 172.21.15.28:36274) 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5eb7fe640 1 -- 172.21.15.28:0/3617864138 learned_addr learned my addr 172.21.15.28:0/3617864138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.608+0000 7fe5ea7fc640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec146e60 0x7fe5ec0b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5eaffd640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5ec146a80 0x7fe5ec0add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5eb7fe640 1 -- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5ec146a80 msgr2=0x7fe5ec0add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5eb7fe640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5ec146a80 0x7fe5ec0add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5eb7fe640 1 -- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec146e60 msgr2=0x7fe5ec0b22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5eb7fe640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec146e60 0x7fe5ec0b22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5eb7fe640 1 -- 172.21.15.28:0/3617864138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec1413c0 con 0x7fe5ec162060 2024-09-10T20:41:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5eb7fe640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ec162060 0x7fe5ec0b2800 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400c7d0 tx=0x7fe5e400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5d37fe640 1 -- 172.21.15.28:0/3617864138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4003b70 con 0x7fe5ec162060 2024-09-10T20:41:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5d37fe640 1 -- 172.21.15.28:0/3617864138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5e4003d30 con 0x7fe5ec162060 2024-09-10T20:41:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5d37fe640 1 -- 172.21.15.28:0/3617864138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e400aab0 con 0x7fe5ec162060 2024-09-10T20:41:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec0ae670 con 0x7fe5ec162060 2024-09-10T20:41:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.609+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec0aec30 con 0x7fe5ec162060 2024-09-10T20:41:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.610+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ec070700 con 0x7fe5ec162060 2024-09-10T20:41:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.611+0000 7fe5d37fe640 1 -- 172.21.15.28:0/3617864138 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fe5e400ac50 con 0x7fe5ec162060 2024-09-10T20:41:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.611+0000 7fe5d37fe640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe5d80782d0 0x7fe5d807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.611+0000 7fe5d37fe640 1 -- 172.21.15.28:0/3617864138 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe5e409a0a0 con 0x7fe5ec162060 2024-09-10T20:41:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.612+0000 7fe5eaffd640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe5d80782d0 0x7fe5d807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.612+0000 7fe5eaffd640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe5d80782d0 0x7fe5d807a790 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc007fd0 tx=0x7fe5dc007eb0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.616+0000 7fe5d37fe640 1 -- 172.21.15.28:0/3617864138 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe5e4061fc0 con 0x7fe5ec162060 2024-09-10T20:41:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.806+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5ec06cfd0 con 0x7fe5ec162060 2024-09-10T20:41:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.807+0000 7fe5d37fe640 1 -- 172.21.15.28:0/3617864138 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16725 (secure 0 0 0) 0x7fe5e40669f0 con 0x7fe5ec162060 2024-09-10T20:41:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:35.811 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":56,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:12.768390+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:08.769779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:10.096656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.809+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe5d80782d0 msgr2=0x7fe5d807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.809+0000 7fe5f1416640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe5d80782d0 0x7fe5d807a790 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc007fd0 tx=0x7fe5dc007eb0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.809+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ec162060 msgr2=0x7fe5ec0b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.809+0000 7fe5f1416640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ec162060 0x7fe5ec0b2800 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400c7d0 tx=0x7fe5e400cc90 comp rx=0 tx=0).stop 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 shutdown_connections 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe5d80782d0 0x7fe5d807a790 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ec162060 0x7fe5ec0b2800 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5ec146e60 0x7fe5ec0b22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 --2- 172.21.15.28:0/3617864138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe5ec146a80 0x7fe5ec0add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 >> 172.21.15.28:0/3617864138 conn(0x7fe5ec06dbf0 msgr2=0x7fe5ec15fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 shutdown_connections 2024-09-10T20:41:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.810+0000 7fe5f1416640 1 -- 172.21.15.28:0/3617864138 wait complete. 2024-09-10T20:41:35.824 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:41:35.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.926+0000 7fd2b982b640 1 -- 172.21.15.28:0/3705229227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c003a00 msgr2=0x7fd29c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.926+0000 7fd2b982b640 1 --2- 172.21.15.28:0/3705229227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c003a00 0x7fd29c003de0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400ac40 tx=0x7fd29c005830 comp rx=0 tx=0).stop 2024-09-10T20:41:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.926+0000 7fd2b982b640 1 -- 172.21.15.28:0/3705229227 shutdown_connections 2024-09-10T20:41:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.926+0000 7fd2b982b640 1 --2- 172.21.15.28:0/3705229227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c003a00 0x7fd29c003de0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.926+0000 7fd2b982b640 1 -- 172.21.15.28:0/3705229227 >> 172.21.15.28:0/3705229227 conn(0x7fd2b406cf00 msgr2=0x7fd2b406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.926+0000 7fd2b982b640 1 -- 172.21.15.28:0/3705229227 shutdown_connections 2024-09-10T20:41:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.926+0000 7fd2b982b640 1 -- 172.21.15.28:0/3705229227 wait complete. 2024-09-10T20:41:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.927+0000 7fd2b982b640 1 Processor -- start 2024-09-10T20:41:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.927+0000 7fd2b982b640 1 -- start start 2024-09-10T20:41:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b982b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c003a00 0x7fd2b41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b982b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b40cae30 0x7fd2b40ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b982b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2b41f8190 0x7fd2b41f8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b982b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b40b4780 con 0x7fd2b40cae30 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b982b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b40b4600 con 0x7fd2b41f8190 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b982b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b40b4900 con 0x7fd29c003a00 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b40cae30 0x7fd2b40ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b40cae30 0x7fd2b40ad8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36294/0 (socket says 172.21.15.28:36294) 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b27fc640 1 -- 172.21.15.28:0/1159016413 learned_addr learned my addr 172.21.15.28:0/1159016413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b37fe640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2b41f8190 0x7fd2b41f8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.928+0000 7fd2b2ffd640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c003a00 0x7fd2b41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b27fc640 1 -- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c003a00 msgr2=0x7fd2b41576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b27fc640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c003a00 0x7fd2b41576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b27fc640 1 -- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2b41f8190 msgr2=0x7fd2b41f8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b27fc640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2b41f8190 0x7fd2b41f8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b27fc640 1 -- 172.21.15.28:0/1159016413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b40cb380 con 0x7fd2b40cae30 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b2ffd640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c003a00 0x7fd2b41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b27fc640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b40cae30 0x7fd2b40ad8a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8000c00 tx=0x7fd2a8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd29bfff640 1 -- 172.21.15.28:0/1159016413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8015020 con 0x7fd2b40cae30 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd29bfff640 1 -- 172.21.15.28:0/1159016413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2a8011960 con 0x7fd2b40cae30 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b40cb660 con 0x7fd2b40cae30 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd29bfff640 1 -- 172.21.15.28:0/1159016413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8011d50 con 0x7fd2b40cae30 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.929+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b41f7c70 con 0x7fd2b40cae30 2024-09-10T20:41:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.930+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b40d0260 con 0x7fd2b40cae30 2024-09-10T20:41:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.931+0000 7fd29bfff640 1 -- 172.21.15.28:0/1159016413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fd2a801a400 con 0x7fd2b40cae30 2024-09-10T20:41:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.931+0000 7fd29bfff640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd2a00782d0 0x7fd2a007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.931+0000 7fd29bfff640 1 -- 172.21.15.28:0/1159016413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fd2a809b100 con 0x7fd2b40cae30 2024-09-10T20:41:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.931+0000 7fd2b2ffd640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd2a00782d0 0x7fd2a007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.932+0000 7fd2b2ffd640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd2a00782d0 0x7fd2a007a790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4035000 tx=0x7fd2a40046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:35.935+0000 7fd29bfff640 1 -- 172.21.15.28:0/1159016413 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd2a801f020 con 0x7fd2b40cae30 2024-09-10T20:41:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.158+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2b41f7e40 con 0x7fd2b40cae30 2024-09-10T20:41:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.158+0000 7fd29bfff640 1 -- 172.21.15.28:0/1159016413 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5223 (secure 0 0 0) 0x7fd2a80636e0 con 0x7fd2b40cae30 2024-09-10T20:41:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:36.162 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"btime":"2024-09-10T20:41:20:801101+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/485554639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":485554639},{"type":"v1","addr":"172.21.15.159:6836","nonce":485554639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1807026586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1807026586},{"type":"v1","addr":"172.21.15.159:6837","nonce":1807026586}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/975621681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":975621681},{"type":"v1","addr":"172.21.15.28:6834","nonce":975621681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:11.137947+0000","modified":"2024-09-10T20:41:14.144276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3887400201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3887400201},{"type":"v1","addr":"172.21.15.28:6835","nonce":3887400201}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-10T20:41:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-09-10T20:41:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd2a00782d0 msgr2=0x7fd2a007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd2a00782d0 0x7fd2a007a790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4035000 tx=0x7fd2a40046c0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b40cae30 msgr2=0x7fd2b40ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b40cae30 0x7fd2b40ad8a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8000c00 tx=0x7fd2a8001710 comp rx=0 tx=0).stop 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 shutdown_connections 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd2a00782d0 0x7fd2a007a790 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2b41f8190 0x7fd2b41f8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b40cae30 0x7fd2b40ad8a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 --2- 172.21.15.28:0/1159016413 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c003a00 0x7fd2b41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 >> 172.21.15.28:0/1159016413 conn(0x7fd2b406cf00 msgr2=0x7fd2b41546c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 shutdown_connections 2024-09-10T20:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.160+0000 7fd2b982b640 1 -- 172.21.15.28:0/1159016413 wait complete. 2024-09-10T20:41:36.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3454666605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40cfc20 msgr2=0x7fa9e40d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3454666605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40cfc20 0x7fa9e40d0000 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009830 tx=0x7fa9d402a480 comp rx=0 tx=0).stop 2024-09-10T20:41:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3454666605 shutdown_connections 2024-09-10T20:41:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3454666605 >> v2:172.21.15.159:3300/0 conn(0x7fa9e40b2a20 0x7fa9e415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3454666605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40cfc20 0x7fa9e40d0000 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3454666605 >> 172.21.15.28:0/3454666605 conn(0x7fa9e40d1bd0 msgr2=0x7fa9e40d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3454666605 shutdown_connections 2024-09-10T20:41:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.277+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3454666605 wait complete. 2024-09-10T20:41:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.278+0000 7fa9e3fff640 1 Processor -- start 2024-09-10T20:41:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.278+0000 7fa9e3fff640 1 -- start start 2024-09-10T20:41:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40b2a20 0x7fa9e40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e40cfc20 0x7fa9e406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9e4160350 0x7fa9e4154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e3fff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e41607a0 con 0x7fa9e40cfc20 2024-09-10T20:41:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e3fff640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e40b2750 con 0x7fa9e4160350 2024-09-10T20:41:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e3fff640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e4160920 con 0x7fa9e40b2a20 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e40cfc20 0x7fa9e406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e2ffd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40b2a20 0x7fa9e40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e40cfc20 0x7fa9e406dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36308/0 (socket says 172.21.15.28:36308) 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 -- 172.21.15.28:0/3990474155 learned_addr learned my addr 172.21.15.28:0/3990474155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9e37fe640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9e4160350 0x7fa9e4154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 -- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40b2a20 msgr2=0x7fa9e40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40b2a20 0x7fa9e40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 -- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9e4160350 msgr2=0x7fa9e4154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9e4160350 0x7fa9e4154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.279+0000 7fa9dbfff640 1 -- 172.21.15.28:0/3990474155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e40b5c70 con 0x7fa9e40cfc20 2024-09-10T20:41:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.280+0000 7fa9dbfff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e40cfc20 0x7fa9e406dbf0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00ea60 tx=0x7fa9cc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.280+0000 7fa9db7fe640 1 -- 172.21.15.28:0/3990474155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00cde0 con 0x7fa9e40cfc20 2024-09-10T20:41:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.280+0000 7fa9db7fe640 1 -- 172.21.15.28:0/3990474155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9cc004020 con 0x7fa9e40cfc20 2024-09-10T20:41:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.280+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e40b5f50 con 0x7fa9e40cfc20 2024-09-10T20:41:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.280+0000 7fa9db7fe640 1 -- 172.21.15.28:0/3990474155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc004450 con 0x7fa9e40cfc20 2024-09-10T20:41:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.280+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e406e130 con 0x7fa9e40cfc20 2024-09-10T20:41:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.281+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e415dac0 con 0x7fa9e40cfc20 2024-09-10T20:41:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.282+0000 7fa9db7fe640 1 -- 172.21.15.28:0/3990474155 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fa9cc0035f0 con 0x7fa9e40cfc20 2024-09-10T20:41:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.282+0000 7fa9db7fe640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9c4078300 0x7fa9c407a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.282+0000 7fa9db7fe640 1 -- 172.21.15.28:0/3990474155 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fa9cc09ad60 con 0x7fa9e40cfc20 2024-09-10T20:41:36.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.283+0000 7fa9e2ffd640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9c4078300 0x7fa9c407a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.283+0000 7fa9e2ffd640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9c4078300 0x7fa9c407a7c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4034000 tx=0x7fa9d4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:36.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.287+0000 7fa9db7fe640 1 -- 172.21.15.28:0/3990474155 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa9cc062c00 con 0x7fa9e40cfc20 2024-09-10T20:41:36.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.513+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9e415fe10 con 0x7fa9e40cfc20 2024-09-10T20:41:36.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.513+0000 7fa9db7fe640 1 -- 172.21.15.28:0/3990474155 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5223 (secure 0 0 0) 0x7fa9cc067630 con 0x7fa9e40cfc20 2024-09-10T20:41:36.518 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:36.518 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"btime":"2024-09-10T20:41:20:801101+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/485554639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":485554639},{"type":"v1","addr":"172.21.15.159:6836","nonce":485554639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1807026586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1807026586},{"type":"v1","addr":"172.21.15.159:6837","nonce":1807026586}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/975621681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":975621681},{"type":"v1","addr":"172.21.15.28:6834","nonce":975621681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:11.137947+0000","modified":"2024-09-10T20:41:14.144276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5748},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5748":{"gid":5748,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3887400201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3887400201},{"type":"v1","addr":"172.21.15.28:6835","nonce":3887400201}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5748,"qdb_cluster":[5748]},"id":4}]} 2024-09-10T20:41:36.518 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-09-10T20:41:36.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9c4078300 msgr2=0x7fa9c407a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9c4078300 0x7fa9c407a7c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4034000 tx=0x7fa9d4004730 comp rx=0 tx=0).stop 2024-09-10T20:41:36.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e40cfc20 msgr2=0x7fa9e406dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e40cfc20 0x7fa9e406dbf0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00ea60 tx=0x7fa9cc00ef20 comp rx=0 tx=0).stop 2024-09-10T20:41:36.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 shutdown_connections 2024-09-10T20:41:36.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa9c4078300 0x7fa9c407a7c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa9e4160350 0x7fa9e4154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e40cfc20 0x7fa9e406dbf0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 --2- 172.21.15.28:0/3990474155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9e40b2a20 0x7fa9e40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 >> 172.21.15.28:0/3990474155 conn(0x7fa9e40d1bd0 msgr2=0x7fa9e40cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:36.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 shutdown_connections 2024-09-10T20:41:36.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.516+0000 7fa9e3fff640 1 -- 172.21.15.28:0/3990474155 wait complete. 2024-09-10T20:41:36.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.633+0000 7f4d2c310640 1 -- 172.21.15.28:0/1505423966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240cd3d0 msgr2=0x7f4d240cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.633+0000 7f4d2c310640 1 --2- 172.21.15.28:0/1505423966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240cd3d0 0x7f4d240cc570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4d14005fd0 tx=0x7f4d1402aa90 comp rx=0 tx=0).stop 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.634+0000 7f4d2c310640 1 -- 172.21.15.28:0/1505423966 shutdown_connections 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.634+0000 7f4d2c310640 1 --2- 172.21.15.28:0/1505423966 >> v2:172.21.15.159:3300/0 conn(0x7f4d240b4340 0x7f4d240b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.634+0000 7f4d2c310640 1 --2- 172.21.15.28:0/1505423966 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240cd3d0 0x7f4d240cc570 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.634+0000 7f4d2c310640 1 -- 172.21.15.28:0/1505423966 >> 172.21.15.28:0/1505423966 conn(0x7f4d2406cf00 msgr2=0x7f4d2406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.634+0000 7f4d2c310640 1 -- 172.21.15.28:0/1505423966 shutdown_connections 2024-09-10T20:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.634+0000 7f4d2c310640 1 -- 172.21.15.28:0/1505423966 wait complete. 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.635+0000 7f4d2c310640 1 Processor -- start 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.635+0000 7f4d2c310640 1 -- start start 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2c310640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240b4340 0x7f4d24073c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2c310640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d240cd3d0 0x7f4d241f8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2c310640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d240d0090 0x7f4d2414ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2c310640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d240d09d0 con 0x7f4d240cd3d0 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2c310640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4d240d0850 con 0x7f4d240d0090 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2c310640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4d240b3590 con 0x7f4d240b4340 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d240cd3d0 0x7f4d241f8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d240cd3d0 0x7f4d241f8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36320/0 (socket says 172.21.15.28:36320) 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 -- 172.21.15.28:0/2820723054 learned_addr learned my addr 172.21.15.28:0/2820723054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2a0ac640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240b4340 0x7f4d24073c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2a8ad640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d240d0090 0x7f4d2414ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 -- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240b4340 msgr2=0x7f4d24073c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240b4340 0x7f4d24073c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 -- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d240d0090 msgr2=0x7f4d2414ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d240d0090 0x7f4d2414ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 -- 172.21.15.28:0/2820723054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d241f9030 con 0x7f4d240cd3d0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d298ab640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d240cd3d0 0x7f4d241f8af0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800e9f0 tx=0x7f4d1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d22ffd640 1 -- 172.21.15.28:0/2820723054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800cd80 con 0x7f4d240cd3d0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d24072bf0 con 0x7f4d240cd3d0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d22ffd640 1 -- 172.21.15.28:0/2820723054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d18004020 con 0x7f4d240cd3d0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.636+0000 7f4d22ffd640 1 -- 172.21.15.28:0/2820723054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18004430 con 0x7f4d240cd3d0 2024-09-10T20:41:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.637+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d240731b0 con 0x7f4d240cd3d0 2024-09-10T20:41:36.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.637+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d240b3a20 con 0x7f4d240cd3d0 2024-09-10T20:41:36.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.638+0000 7f4d22ffd640 1 -- 172.21.15.28:0/2820723054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f4d180035f0 con 0x7f4d240cd3d0 2024-09-10T20:41:36.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.639+0000 7f4d22ffd640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d100782d0 0x7f4d1007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.639+0000 7f4d22ffd640 1 -- 172.21.15.28:0/2820723054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f4d1809aff0 con 0x7f4d240cd3d0 2024-09-10T20:41:36.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.639+0000 7f4d2a0ac640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d100782d0 0x7f4d1007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.639+0000 7f4d2a0ac640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d100782d0 0x7f4d1007a790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4d14035000 tx=0x7f4d14039040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:36.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.643+0000 7f4d22ffd640 1 -- 172.21.15.28:0/2820723054 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4d241f9030 con 0x7f4d240cd3d0 2024-09-10T20:41:36.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.830+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d2406d660 con 0x7f4d240cd3d0 2024-09-10T20:41:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.831+0000 7f4d22ffd640 1 -- 172.21.15.28:0/2820723054 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16725 (secure 0 0 0) 0x7f4d18062f10 con 0x7f4d240cd3d0 2024-09-10T20:41:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:36.835 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":56,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:12.768390+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:08.769779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:10.096656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d100782d0 msgr2=0x7f4d1007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d100782d0 0x7f4d1007a790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4d14035000 tx=0x7f4d14039040 comp rx=0 tx=0).stop 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d240cd3d0 msgr2=0x7f4d241f8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d240cd3d0 0x7f4d241f8af0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800e9f0 tx=0x7f4d1800eeb0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 shutdown_connections 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d100782d0 0x7f4d1007a790 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d240d0090 0x7f4d2414ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d240cd3d0 0x7f4d241f8af0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 --2- 172.21.15.28:0/2820723054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d240b4340 0x7f4d24073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 >> 172.21.15.28:0/2820723054 conn(0x7f4d2406cf00 msgr2=0x7f4d2409ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 shutdown_connections 2024-09-10T20:41:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.833+0000 7f4d2c310640 1 -- 172.21.15.28:0/2820723054 wait complete. 2024-09-10T20:41:36.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.949+0000 7f11b8391640 1 -- 172.21.15.28:0/490381761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b015c050 msgr2=0x7f11b00cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.949+0000 7f11b8391640 1 --2- 172.21.15.28:0/490381761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b015c050 0x7f11b00cfa10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f11a8005fd0 tx=0x7f11a802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.950+0000 7f11b8391640 1 -- 172.21.15.28:0/490381761 shutdown_connections 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.950+0000 7f11b8391640 1 --2- 172.21.15.28:0/490381761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11b00d2960 0x7f11b00ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.950+0000 7f11b8391640 1 --2- 172.21.15.28:0/490381761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b015c050 0x7f11b00cfa10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.950+0000 7f11b8391640 1 -- 172.21.15.28:0/490381761 >> 172.21.15.28:0/490381761 conn(0x7f11b00d1e40 msgr2=0x7f11b00d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.950+0000 7f11b8391640 1 -- 172.21.15.28:0/490381761 shutdown_connections 2024-09-10T20:41:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.950+0000 7f11b8391640 1 -- 172.21.15.28:0/490381761 wait complete. 2024-09-10T20:41:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.951+0000 7f11b8391640 1 Processor -- start 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.951+0000 7f11b8391640 1 -- start start 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b8391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11b00cd9b0 0x7f11b00ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b8391640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b00d2960 0x7f11b00c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b8391640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11b015c050 0x7f11b00c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b8391640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11b00b1af0 con 0x7f11b00cd9b0 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b8391640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f11b00cf110 con 0x7f11b00d2960 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b8391640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f11b00b1c70 con 0x7f11b015c050 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11b00cd9b0 0x7f11b00ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11b00cd9b0 0x7f11b00ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36322/0 (socket says 172.21.15.28:36322) 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 -- 172.21.15.28:0/2688986282 learned_addr learned my addr 172.21.15.28:0/2688986282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b692e640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11b015c050 0x7f11b00c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b592c640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b00d2960 0x7f11b00c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 -- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11b015c050 msgr2=0x7f11b00c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11b015c050 0x7f11b00c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 -- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b00d2960 msgr2=0x7f11b00c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b00d2960 0x7f11b00c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 -- 172.21.15.28:0/2688986282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b00c6d00 con 0x7f11b00cd9b0 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b612d640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11b00cd9b0 0x7f11b00ca600 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f11a8034040 tx=0x7f11a80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11aeffd640 1 -- 172.21.15.28:0/2688986282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a803e030 con 0x7f11b00cd9b0 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11aeffd640 1 -- 172.21.15.28:0/2688986282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11a8003ae0 con 0x7f11b00cd9b0 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.952+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b00c9560 con 0x7f11b00cd9b0 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.953+0000 7f11aeffd640 1 -- 172.21.15.28:0/2688986282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a8035040 con 0x7f11b00cd9b0 2024-09-10T20:41:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.953+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b00c9b20 con 0x7f11b00cd9b0 2024-09-10T20:41:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.953+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b00724d0 con 0x7f11b00cd9b0 2024-09-10T20:41:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.954+0000 7f11aeffd640 1 -- 172.21.15.28:0/2688986282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f11a8031400 con 0x7f11b00cd9b0 2024-09-10T20:41:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.955+0000 7f11aeffd640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f119c078300 0x7f119c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.955+0000 7f11aeffd640 1 -- 172.21.15.28:0/2688986282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f11a80ba9a0 con 0x7f11b00cd9b0 2024-09-10T20:41:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.955+0000 7f11b592c640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f119c078300 0x7f119c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.956+0000 7f11b592c640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f119c078300 0x7f119c07a7c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f11b00af8c0 tx=0x7f11a4005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:36.959+0000 7f11aeffd640 1 -- 172.21.15.28:0/2688986282 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f11b00724d0 con 0x7f11b00cd9b0 2024-09-10T20:41:37.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.181+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f11b0076360 con 0x7f11b00cd9b0 2024-09-10T20:41:37.186 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:41:37.181+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:41:37.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.182+0000 7f11aeffd640 1 -- 172.21.15.28:0/2688986282 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7f11b00c6d00 con 0x7f11b00cd9b0 2024-09-10T20:41:37.187 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:41:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f119c078300 msgr2=0x7f119c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f119c078300 0x7f119c07a7c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f11b00af8c0 tx=0x7f11a4005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11b00cd9b0 msgr2=0x7f11b00ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11b00cd9b0 0x7f11b00ca600 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f11a8034040 tx=0x7f11a80035f0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 shutdown_connections 2024-09-10T20:41:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f119c078300 0x7f119c07a7c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11b015c050 0x7f11b00c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11b00d2960 0x7f11b00c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 --2- 172.21.15.28:0/2688986282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11b00cd9b0 0x7f11b00ca600 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 >> 172.21.15.28:0/2688986282 conn(0x7f11b00d1e40 msgr2=0x7f11b0071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 shutdown_connections 2024-09-10T20:41:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.186+0000 7f11b8391640 1 -- 172.21.15.28:0/2688986282 wait complete. 2024-09-10T20:41:37.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:41:37.229 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:41:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 -- 172.21.15.28:0/1019852174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd104072d70 msgr2=0x7fd10415c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 --2- 172.21.15.28:0/1019852174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd104072d70 0x7fd10415c060 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd1040733c0 tx=0x7fd0f402a530 comp rx=0 tx=0).stop 2024-09-10T20:41:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 -- 172.21.15.28:0/1019852174 shutdown_connections 2024-09-10T20:41:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 --2- 172.21.15.28:0/1019852174 >> v2:172.21.15.159:3300/0 conn(0x7fd10406fd80 0x7fd10415c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 --2- 172.21.15.28:0/1019852174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd104072d70 0x7fd10415c060 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 -- 172.21.15.28:0/1019852174 >> 172.21.15.28:0/1019852174 conn(0x7fd1040b1800 msgr2=0x7fd1040b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 -- 172.21.15.28:0/1019852174 shutdown_connections 2024-09-10T20:41:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.307+0000 7fd10c70e640 1 -- 172.21.15.28:0/1019852174 wait complete. 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.308+0000 7fd10c70e640 1 Processor -- start 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.308+0000 7fd10c70e640 1 -- start start 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10c70e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10406fd80 0x7fd10414dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10c70e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd104072d70 0x7fd1041f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10c70e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1041551d0 0x7fd10414fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10c70e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd10406e5f0 con 0x7fd1041551d0 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10c70e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd10415cae0 con 0x7fd104072d70 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10c70e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd10414e510 con 0x7fd10406fd80 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1041551d0 0x7fd10414fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1041551d0 0x7fd10414fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36356/0 (socket says 172.21.15.28:36356) 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 -- 172.21.15.28:0/3396039777 learned_addr learned my addr 172.21.15.28:0/3396039777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd109ca9640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd104072d70 0x7fd1041f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10a4aa640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10406fd80 0x7fd10414dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 -- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10406fd80 msgr2=0x7fd10414dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10406fd80 0x7fd10414dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 -- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd104072d70 msgr2=0x7fd1041f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd104072d70 0x7fd1041f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 -- 172.21.15.28:0/3396039777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1041f8100 con 0x7fd1041551d0 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10a4aa640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10406fd80 0x7fd10414dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd10acab640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1041551d0 0x7fd10414fdf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00cce0 tx=0x7fd0fc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.309+0000 7fd0faffd640 1 -- 172.21.15.28:0/3396039777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc00f450 con 0x7fd1041551d0 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.310+0000 7fd0faffd640 1 -- 172.21.15.28:0/3396039777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0fc00f700 con 0x7fd1041551d0 2024-09-10T20:41:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.310+0000 7fd0faffd640 1 -- 172.21.15.28:0/3396039777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc00fb50 con 0x7fd1041551d0 2024-09-10T20:41:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.310+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10414c830 con 0x7fd1041551d0 2024-09-10T20:41:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.310+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1041f83a0 con 0x7fd1041551d0 2024-09-10T20:41:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.311+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10415d030 con 0x7fd1041551d0 2024-09-10T20:41:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.311+0000 7fd0faffd640 1 -- 172.21.15.28:0/3396039777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fd0fc0213f0 con 0x7fd1041551d0 2024-09-10T20:41:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.312+0000 7fd0faffd640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd0e80782d0 0x7fd0e807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.312+0000 7fd0faffd640 1 -- 172.21.15.28:0/3396039777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7fd0fc09b110 con 0x7fd1041551d0 2024-09-10T20:41:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.312+0000 7fd10a4aa640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd0e80782d0 0x7fd0e807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.313+0000 7fd10a4aa640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd0e80782d0 0x7fd0e807a790 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd1000016a0 tx=0x7fd100009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.316+0000 7fd0faffd640 1 -- 172.21.15.28:0/3396039777 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd0fc014030 con 0x7fd1041551d0 2024-09-10T20:41:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:37.539+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd1041591a0 con 0x7fd1041551d0 2024-09-10T20:41:38.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.186+0000 7fd0faffd640 1 -- 172.21.15.28:0/3396039777 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7fd0fc063590 con 0x7fd1041551d0 2024-09-10T20:41:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd0e80782d0 msgr2=0x7fd0e807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd0e80782d0 0x7fd0e807a790 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd1000016a0 tx=0x7fd100009040 comp rx=0 tx=0).stop 2024-09-10T20:41:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1041551d0 msgr2=0x7fd10414fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1041551d0 0x7fd10414fdf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00cce0 tx=0x7fd0fc00a620 comp rx=0 tx=0).stop 2024-09-10T20:41:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 shutdown_connections 2024-09-10T20:41:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd0e80782d0 0x7fd0e807a790 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1041551d0 0x7fd10414fdf0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd104072d70 0x7fd1041f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 --2- 172.21.15.28:0/3396039777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10406fd80 0x7fd10414dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 >> 172.21.15.28:0/3396039777 conn(0x7fd1040b1800 msgr2=0x7fd1040b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 shutdown_connections 2024-09-10T20:41:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.189+0000 7fd10c70e640 1 -- 172.21.15.28:0/3396039777 wait complete. 2024-09-10T20:41:38.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:41:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.357+0000 7fd91dacd640 1 -- 172.21.15.28:0/4207952757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 msgr2=0x7fd9180d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.357+0000 7fd91dacd640 1 --2- 172.21.15.28:0/4207952757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 0x7fd9180d4a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009860 tx=0x7fd90c02a620 comp rx=0 tx=0).stop 2024-09-10T20:41:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.357+0000 7fd91dacd640 1 -- 172.21.15.28:0/4207952757 shutdown_connections 2024-09-10T20:41:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.357+0000 7fd91dacd640 1 --2- 172.21.15.28:0/4207952757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 0x7fd9180d4a80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.357+0000 7fd91dacd640 1 -- 172.21.15.28:0/4207952757 >> 172.21.15.28:0/4207952757 conn(0x7fd9180d3f90 msgr2=0x7fd9180d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.357+0000 7fd91dacd640 1 -- 172.21.15.28:0/4207952757 shutdown_connections 2024-09-10T20:41:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.357+0000 7fd91dacd640 1 -- 172.21.15.28:0/4207952757 wait complete. 2024-09-10T20:41:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.358+0000 7fd91dacd640 1 Processor -- start 2024-09-10T20:41:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.358+0000 7fd91dacd640 1 -- start start 2024-09-10T20:41:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd91dacd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9180b4c20 0x7fd9181f5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd91dacd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 0x7fd9180cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd91dacd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918162000 0x7fd9180cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd91dacd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9180d24f0 con 0x7fd918162000 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd91dacd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd9180d2370 con 0x7fd9180b4c20 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd91dacd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd9180d2670 con 0x7fd918161850 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd917fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918162000 0x7fd9180cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd917fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918162000 0x7fd9180cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36364/0 (socket says 172.21.15.28:36364) 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd917fff640 1 -- 172.21.15.28:0/2618826468 learned_addr learned my addr 172.21.15.28:0/2618826468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd916ffd640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 0x7fd9180cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.359+0000 7fd9177fe640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9180b4c20 0x7fd9181f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd917fff640 1 -- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 msgr2=0x7fd9180cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd917fff640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 0x7fd9180cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd917fff640 1 -- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9180b4c20 msgr2=0x7fd9181f5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd917fff640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9180b4c20 0x7fd9181f5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd917fff640 1 -- 172.21.15.28:0/2618826468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9180b7eb0 con 0x7fd918162000 2024-09-10T20:41:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd917fff640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918162000 0x7fd9180cb5a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd91000cba0 tx=0x7fd91000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd8fffff640 1 -- 172.21.15.28:0/2618826468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910016020 con 0x7fd918162000 2024-09-10T20:41:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd8fffff640 1 -- 172.21.15.28:0/2618826468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd91000ae50 con 0x7fd918162000 2024-09-10T20:41:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd8fffff640 1 -- 172.21.15.28:0/2618826468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910012680 con 0x7fd918162000 2024-09-10T20:41:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9180cdd60 con 0x7fd918162000 2024-09-10T20:41:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.360+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9180ce040 con 0x7fd918162000 2024-09-10T20:41:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.361+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9180b9f10 con 0x7fd918162000 2024-09-10T20:41:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.362+0000 7fd8fffff640 1 -- 172.21.15.28:0/2618826468 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fd910012820 con 0x7fd918162000 2024-09-10T20:41:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.362+0000 7fd8fffff640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd9040782d0 0x7fd90407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.362+0000 7fd8fffff640 1 -- 172.21.15.28:0/2618826468 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7fd91009ac90 con 0x7fd918162000 2024-09-10T20:41:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.363+0000 7fd9177fe640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd9040782d0 0x7fd90407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.363+0000 7fd9177fe640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd9040782d0 0x7fd90407a790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd908001120 tx=0x7fd908009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.366+0000 7fd8fffff640 1 -- 172.21.15.28:0/2618826468 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd910020020 con 0x7fd918162000 2024-09-10T20:41:38.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:38.555+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd9180d2c70 con 0x7fd918162000 2024-09-10T20:41:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.187+0000 7fd8fffff640 1 -- 172.21.15.28:0/2618826468 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) ==== 181+0+0 (secure 0 0 0) 0x7fd910062bc0 con 0x7fd918162000 2024-09-10T20:41:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:41:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd9040782d0 msgr2=0x7fd90407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd9040782d0 0x7fd90407a790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd908001120 tx=0x7fd908009040 comp rx=0 tx=0).stop 2024-09-10T20:41:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918162000 msgr2=0x7fd9180cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918162000 0x7fd9180cb5a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd91000cba0 tx=0x7fd91000a430 comp rx=0 tx=0).stop 2024-09-10T20:41:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 shutdown_connections 2024-09-10T20:41:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fd9040782d0 0x7fd90407a790 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918162000 0x7fd9180cb5a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd918161850 0x7fd9180cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 --2- 172.21.15.28:0/2618826468 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9180b4c20 0x7fd9181f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.191+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 >> 172.21.15.28:0/2618826468 conn(0x7fd9180d3f90 msgr2=0x7fd91815d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.192+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 shutdown_connections 2024-09-10T20:41:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.192+0000 7fd91dacd640 1 -- 172.21.15.28:0/2618826468 wait complete. 2024-09-10T20:41:39.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:41:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.321+0000 7fdd2876b640 1 -- 172.21.15.28:0/30278913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 msgr2=0x7fdd2015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.321+0000 7fdd2876b640 1 --2- 172.21.15.28:0/30278913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd2015bd70 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005fd0 tx=0x7fdd1002aa90 comp rx=0 tx=0).stop 2024-09-10T20:41:39.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.321+0000 7fdd2876b640 1 -- 172.21.15.28:0/30278913 shutdown_connections 2024-09-10T20:41:39.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.321+0000 7fdd2876b640 1 --2- 172.21.15.28:0/30278913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd2015bd70 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:39.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.321+0000 7fdd2876b640 1 -- 172.21.15.28:0/30278913 >> 172.21.15.28:0/30278913 conn(0x7fdd200cfc80 msgr2=0x7fdd200d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:39.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.321+0000 7fdd2876b640 1 -- 172.21.15.28:0/30278913 shutdown_connections 2024-09-10T20:41:39.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.321+0000 7fdd2876b640 1 -- 172.21.15.28:0/30278913 wait complete. 2024-09-10T20:41:39.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.323+0000 7fdd2876b640 1 Processor -- start 2024-09-10T20:41:39.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.323+0000 7fdd2876b640 1 -- start start 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.323+0000 7fdd2876b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd20070440 0x7fdd20159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.323+0000 7fdd2876b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20072db0 0x7fdd201f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.323+0000 7fdd2876b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd201f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.323+0000 7fdd2876b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd200cb0c0 con 0x7fdd20151cf0 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd2876b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdd200caf40 con 0x7fdd20070440 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd2876b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdd200cb240 con 0x7fdd20072db0 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd201f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd201f4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36390/0 (socket says 172.21.15.28:36390) 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 -- 172.21.15.28:0/2747594620 learned_addr learned my addr 172.21.15.28:0/2747594620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26507640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd20070440 0x7fdd20159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 -- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20072db0 msgr2=0x7fdd201f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20072db0 0x7fdd201f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 -- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd20070440 msgr2=0x7fdd20159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd20070440 0x7fdd20159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 -- 172.21.15.28:0/2747594620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd200b3500 con 0x7fdd20151cf0 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26507640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd20070440 0x7fdd20159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd26d08640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd201f4900 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800c390 tx=0x7fdd1800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd1effd640 1 -- 172.21.15.28:0/2747594620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18017070 con 0x7fdd20151cf0 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd1effd640 1 -- 172.21.15.28:0/2747594620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd180035f0 con 0x7fdd20151cf0 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd1effd640 1 -- 172.21.15.28:0/2747594620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18003a40 con 0x7fdd20151cf0 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd200b37e0 con 0x7fdd20151cf0 2024-09-10T20:41:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.324+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd200b4080 con 0x7fdd20151cf0 2024-09-10T20:41:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.325+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd200cb750 con 0x7fdd20151cf0 2024-09-10T20:41:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.326+0000 7fdd1effd640 1 -- 172.21.15.28:0/2747594620 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fdd18003be0 con 0x7fdd20151cf0 2024-09-10T20:41:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.327+0000 7fdd1effd640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd0c078300 0x7fdd0c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.327+0000 7fdd1effd640 1 -- 172.21.15.28:0/2747594620 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5357+0+0 (secure 0 0 0) 0x7fdd18099840 con 0x7fdd20151cf0 2024-09-10T20:41:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.327+0000 7fdd26507640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd0c078300 0x7fdd0c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.328+0000 7fdd26507640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd0c078300 0x7fdd0c07a7c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdd10035000 tx=0x7fdd10004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:39.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.331+0000 7fdd1effd640 1 -- 172.21.15.28:0/2747594620 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdd18061f80 con 0x7fdd20151cf0 2024-09-10T20:41:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:39.515+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdd200b39b0 con 0x7fdd20151cf0 2024-09-10T20:41:40.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.193+0000 7fdd1effd640 1 -- 172.21.15.28:0/2747594620 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) ==== 169+0+0 (secure 0 0 0) 0x7fdd18061880 con 0x7fdd20151cf0 2024-09-10T20:41:40.199 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:41:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.197+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd0c078300 msgr2=0x7fdd0c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.197+0000 7fdd2876b640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd0c078300 0x7fdd0c07a7c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdd10035000 tx=0x7fdd10004ab0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.197+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 msgr2=0x7fdd201f4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.197+0000 7fdd2876b640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd201f4900 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800c390 tx=0x7fdd1800c850 comp rx=0 tx=0).stop 2024-09-10T20:41:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 shutdown_connections 2024-09-10T20:41:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fdd0c078300 0x7fdd0c07a7c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd20151cf0 0x7fdd201f4900 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd20072db0 0x7fdd201f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 --2- 172.21.15.28:0/2747594620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd20070440 0x7fdd20159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 >> 172.21.15.28:0/2747594620 conn(0x7fdd200cfc80 msgr2=0x7fdd200cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 shutdown_connections 2024-09-10T20:41:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.198+0000 7fdd2876b640 1 -- 172.21.15.28:0/2747594620 wait complete. 2024-09-10T20:41:40.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:41:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 -- 172.21.15.28:0/2230653921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 msgr2=0x7f4314161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 --2- 172.21.15.28:0/2230653921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314161b50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f43140d01d0 tx=0x7f430802a530 comp rx=0 tx=0).stop 2024-09-10T20:41:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 -- 172.21.15.28:0/2230653921 shutdown_connections 2024-09-10T20:41:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 --2- 172.21.15.28:0/2230653921 >> v2:172.21.15.159:3300/0 conn(0x7f4314162090 0x7f43140b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 --2- 172.21.15.28:0/2230653921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314161b50 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 -- 172.21.15.28:0/2230653921 >> 172.21.15.28:0/2230653921 conn(0x7f43140d3fb0 msgr2=0x7f43140d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 -- 172.21.15.28:0/2230653921 shutdown_connections 2024-09-10T20:41:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.326+0000 7f431ac08640 1 -- 172.21.15.28:0/2230653921 wait complete. 2024-09-10T20:41:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 Processor -- start 2024-09-10T20:41:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 -- start start 2024-09-10T20:41:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314160880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f431415de70 0x7f43140b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4314162090 0x7f43140b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f431415e3d0 con 0x7f43140d2b90 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f431415e250 con 0x7f4314162090 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.328+0000 7f431ac08640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f43140b7340 con 0x7f431415de70 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36402/0 (socket says 172.21.15.28:36402) 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 -- 172.21.15.28:0/368526566 learned_addr learned my addr 172.21.15.28:0/368526566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f4313fff640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f431415de70 0x7f43140b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43191a5640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4314162090 0x7f43140b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 -- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f431415de70 msgr2=0x7f43140b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f431415de70 0x7f43140b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 -- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4314162090 msgr2=0x7f43140b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4314162090 0x7f43140b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 -- 172.21.15.28:0/368526566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431415f7e0 con 0x7f43140d2b90 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43191a5640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4314162090 0x7f43140b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43189a4640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314160880 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f430400c500 tx=0x7f430400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43117fa640 1 -- 172.21.15.28:0/368526566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304017070 con 0x7f43140d2b90 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43117fa640 1 -- 172.21.15.28:0/368526566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43040035f0 con 0x7f43140d2b90 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431415fac0 con 0x7f43140d2b90 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f43117fa640 1 -- 172.21.15.28:0/368526566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304003a40 con 0x7f43140d2b90 2024-09-10T20:41:40.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.329+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4314160080 con 0x7f43140d2b90 2024-09-10T20:41:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.331+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43140b7890 con 0x7f43140d2b90 2024-09-10T20:41:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.331+0000 7f43117fa640 1 -- 172.21.15.28:0/368526566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f4304003be0 con 0x7f43140d2b90 2024-09-10T20:41:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.332+0000 7f43117fa640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43000782d0 0x7f430007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.332+0000 7f43117fa640 1 -- 172.21.15.28:0/368526566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f43040997e0 con 0x7f43140d2b90 2024-09-10T20:41:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.332+0000 7f4313fff640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43000782d0 0x7f430007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.332+0000 7f4313fff640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43000782d0 0x7f430007a790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4308034000 tx=0x7f4308004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.336+0000 7f43117fa640 1 -- 172.21.15.28:0/368526566 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4304062150 con 0x7f43140d2b90 2024-09-10T20:41:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.519+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f43140b9f30 con 0x7f43140d2b90 2024-09-10T20:41:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.522+0000 7f43117fa640 1 -- 172.21.15.28:0/368526566 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v33) ==== 71+0+0 (secure 0 0 0) 0x7f4304061a50 con 0x7f43140d2b90 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43000782d0 msgr2=0x7f430007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43000782d0 0x7f430007a790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4308034000 tx=0x7f4308004730 comp rx=0 tx=0).stop 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 msgr2=0x7f4314160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314160880 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f430400c500 tx=0x7f430400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 shutdown_connections 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f43000782d0 0x7f430007a790 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4314162090 0x7f43140b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f431415de70 0x7f43140b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 --2- 172.21.15.28:0/368526566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43140d2b90 0x7f4314160880 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 >> 172.21.15.28:0/368526566 conn(0x7f43140d3fb0 msgr2=0x7f43140b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 shutdown_connections 2024-09-10T20:41:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.525+0000 7f431ac08640 1 -- 172.21.15.28:0/368526566 wait complete. 2024-09-10T20:41:40.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/758005864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f0415de70 msgr2=0x7f0f040d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/758005864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f0415de70 0x7f0f040d2b90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4005fd0 tx=0x7f0ef402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/758005864 shutdown_connections 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/758005864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f040cf780 0x7f0f040cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/758005864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f0415de70 0x7f0f040d2b90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/758005864 >> 172.21.15.28:0/758005864 conn(0x7f0f040d3fb0 msgr2=0x7f0f040d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/758005864 shutdown_connections 2024-09-10T20:41:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.642+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/758005864 wait complete. 2024-09-10T20:41:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.643+0000 7f0f0a5b5640 1 Processor -- start 2024-09-10T20:41:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.643+0000 7f0f0a5b5640 1 -- start start 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f0a5b5640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f040cf780 0x7f0f040cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f0a5b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0415de70 0x7f0f04151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f0a5b5640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f04161b50 0x7f0f04152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f0a5b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f04162350 con 0x7f0f0415de70 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f0a5b5640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0f041621d0 con 0x7f0f04161b50 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f0a5b5640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0f041624d0 con 0x7f0f040cf780 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0415de70 0x7f0f04151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0415de70 0x7f0f04151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35980/0 (socket says 172.21.15.28:35980) 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 -- 172.21.15.28:0/581056144 learned_addr learned my addr 172.21.15.28:0/581056144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f03fff640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f040cf780 0x7f0f040cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f08b52640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f04161b50 0x7f0f04152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 -- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f040cf780 msgr2=0x7f0f040cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f040cf780 0x7f0f040cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 -- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f04161b50 msgr2=0x7f0f04152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f04161b50 0x7f0f04152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 -- 172.21.15.28:0/581056144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0409e0f0 con 0x7f0f0415de70 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.644+0000 7f0f037fe640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0415de70 0x7f0f04151fe0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800e9c0 tx=0x7f0ef800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.645+0000 7f0f00ff9640 1 -- 172.21.15.28:0/581056144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef800cde0 con 0x7f0f0415de70 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.645+0000 7f0f00ff9640 1 -- 172.21.15.28:0/581056144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ef8010520 con 0x7f0f0415de70 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.645+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0409e3d0 con 0x7f0f0415de70 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.645+0000 7f0f00ff9640 1 -- 172.21.15.28:0/581056144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8010970 con 0x7f0f0415de70 2024-09-10T20:41:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.645+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0409e990 con 0x7f0f0415de70 2024-09-10T20:41:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.646+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f040cfbc0 con 0x7f0f0415de70 2024-09-10T20:41:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.646+0000 7f0f00ff9640 1 -- 172.21.15.28:0/581056144 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f0ef8010b10 con 0x7f0f0415de70 2024-09-10T20:41:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.647+0000 7f0f00ff9640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ef0078300 0x7f0ef007a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.647+0000 7f0f00ff9640 1 -- 172.21.15.28:0/581056144 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f0ef809aaa0 con 0x7f0f0415de70 2024-09-10T20:41:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.647+0000 7f0f03fff640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ef0078300 0x7f0ef007a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.648+0000 7f0f03fff640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ef0078300 0x7f0ef007a7c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4033000 tx=0x7f0ef4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.651+0000 7f0f00ff9640 1 -- 172.21.15.28:0/581056144 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0ef8062dc0 con 0x7f0f0415de70 2024-09-10T20:41:40.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:40.915+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f0f040b4f70 con 0x7f0f0415de70 2024-09-10T20:41:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.195+0000 7f0f00ff9640 1 -- 172.21.15.28:0/581056144 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v192) ==== 161+0+0 (secure 0 0 0) 0x7f0ef80677f0 con 0x7f0f0415de70 2024-09-10T20:41:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ef0078300 msgr2=0x7f0ef007a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ef0078300 0x7f0ef007a7c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4033000 tx=0x7f0ef4004730 comp rx=0 tx=0).stop 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0415de70 msgr2=0x7f0f04151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0415de70 0x7f0f04151fe0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800e9c0 tx=0x7f0ef800ee80 comp rx=0 tx=0).stop 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 shutdown_connections 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f0ef0078300 0x7f0ef007a7c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0f04161b50 0x7f0f04152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0415de70 0x7f0f04151fe0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 --2- 172.21.15.28:0/581056144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0f040cf780 0x7f0f040cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.197+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 >> 172.21.15.28:0/581056144 conn(0x7f0f040d3fb0 msgr2=0x7f0f040b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.198+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 shutdown_connections 2024-09-10T20:41:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.198+0000 7f0f0a5b5640 1 -- 172.21.15.28:0/581056144 wait complete. 2024-09-10T20:41:41.214 INFO:tasks.cephfs_test_runner:Validate handcrafted .meta file on legacy subvol root doesn't break the system ... ok 2024-09-10T20:41:41.214 INFO:tasks.cephfs_test_runner:Starting test: test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:41:41.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:41.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 -- 172.21.15.28:0/3655388684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c44151cc0 msgr2=0x7f2c440cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 --2- 172.21.15.28:0/3655388684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c44151cc0 0x7f2c440cf0f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2c44068cb0 tx=0x7f2c3c02a5f0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 -- 172.21.15.28:0/3655388684 shutdown_connections 2024-09-10T20:41:41.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 --2- 172.21.15.28:0/3655388684 >> v2:172.21.15.159:3300/0 conn(0x7f2c4415eef0 0x7f2c440cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 --2- 172.21.15.28:0/3655388684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c44151cc0 0x7f2c440cf0f0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 -- 172.21.15.28:0/3655388684 >> 172.21.15.28:0/3655388684 conn(0x7f2c440d0420 msgr2=0x7f2c440d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 -- 172.21.15.28:0/3655388684 shutdown_connections 2024-09-10T20:41:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.316+0000 7f2c4c552640 1 -- 172.21.15.28:0/3655388684 wait complete. 2024-09-10T20:41:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.317+0000 7f2c4c552640 1 Processor -- start 2024-09-10T20:41:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.317+0000 7f2c4c552640 1 -- start start 2024-09-10T20:41:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4c552640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c440738b0 0x7f2c440b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4c552640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c44151cc0 0x7f2c44147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4c552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c4415eef0 0x7f2c44155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4c552640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44073cf0 con 0x7f2c4415eef0 2024-09-10T20:41:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4c552640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4415f640 con 0x7f2c440738b0 2024-09-10T20:41:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4c552640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2c44073e70 con 0x7f2c44151cc0 2024-09-10T20:41:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4aaef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c4415eef0 0x7f2c44155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4aaef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c4415eef0 0x7f2c44155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36000/0 (socket says 172.21.15.28:36000) 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.318+0000 7f2c4aaef640 1 -- 172.21.15.28:0/2217623150 learned_addr learned my addr 172.21.15.28:0/2217623150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c49aed640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c44151cc0 0x7f2c44147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4aaef640 1 -- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c44151cc0 msgr2=0x7f2c44147b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4aaef640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c44151cc0 0x7f2c44147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4a2ee640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c440738b0 0x7f2c440b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4aaef640 1 -- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c440738b0 msgr2=0x7f2c440b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4aaef640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c440738b0 0x7f2c440b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4aaef640 1 -- 172.21.15.28:0/2217623150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c440b1c60 con 0x7f2c4415eef0 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4aaef640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c4415eef0 0x7f2c44155e70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400b970 tx=0x7f2c3400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c42ffd640 1 -- 172.21.15.28:0/2217623150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c34019070 con 0x7f2c4415eef0 2024-09-10T20:41:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c42ffd640 1 -- 172.21.15.28:0/2217623150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c3400cc50 con 0x7f2c4415eef0 2024-09-10T20:41:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c44152ea0 con 0x7f2c4415eef0 2024-09-10T20:41:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c42ffd640 1 -- 172.21.15.28:0/2217623150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c340146a0 con 0x7f2c4415eef0 2024-09-10T20:41:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.319+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c440b13d0 con 0x7f2c4415eef0 2024-09-10T20:41:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.320+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c440cf0f0 con 0x7f2c4415eef0 2024-09-10T20:41:41.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.321+0000 7f2c42ffd640 1 -- 172.21.15.28:0/2217623150 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f2c3401d3b0 con 0x7f2c4415eef0 2024-09-10T20:41:41.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.321+0000 7f2c42ffd640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2c30078300 0x7f2c3007a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.321+0000 7f2c42ffd640 1 -- 172.21.15.28:0/2217623150 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f2c3409a0b0 con 0x7f2c4415eef0 2024-09-10T20:41:41.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.321+0000 7f2c4a2ee640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2c30078300 0x7f2c3007a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.322+0000 7f2c4a2ee640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2c30078300 0x7f2c3007a7c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c033000 tx=0x7f2c3c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:41.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.325+0000 7f2c42ffd640 1 -- 172.21.15.28:0/2217623150 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2c34062990 con 0x7f2c4415eef0 2024-09-10T20:41:41.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.511+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c44068d10 con 0x7f2c4415eef0 2024-09-10T20:41:41.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.512+0000 7f2c42ffd640 1 -- 172.21.15.28:0/2217623150 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+13032 (secure 0 0 0) 0x7f2c34062290 con 0x7f2c4415eef0 2024-09-10T20:41:41.516 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":59,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:40.190166+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3887400201":"2024-09-11T20:41:37.183124+0000","172.21.15.28:6833/3887400201":"2024-09-11T20:41:37.183124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2c30078300 msgr2=0x7f2c3007a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2c30078300 0x7f2c3007a7c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c033000 tx=0x7f2c3c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c4415eef0 msgr2=0x7f2c44155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c4415eef0 0x7f2c44155e70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400b970 tx=0x7f2c3400be30 comp rx=0 tx=0).stop 2024-09-10T20:41:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 shutdown_connections 2024-09-10T20:41:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2c30078300 0x7f2c3007a7c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c4415eef0 0x7f2c44155e70 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2c44151cc0 0x7f2c44147b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 --2- 172.21.15.28:0/2217623150 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2c440738b0 0x7f2c440b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 >> 172.21.15.28:0/2217623150 conn(0x7f2c440d0420 msgr2=0x7f2c440cd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 shutdown_connections 2024-09-10T20:41:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.514+0000 7f2c4c552640 1 -- 172.21.15.28:0/2217623150 wait complete. 2024-09-10T20:41:41.530 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:41:41.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:41:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.634+0000 7f70509e8640 1 -- 172.21.15.28:0/2284448656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7038004000 msgr2=0x7f7038004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.634+0000 7f70509e8640 1 --2- 172.21.15.28:0/2284448656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7038004000 0x7f7038004800 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7034009b50 tx=0x7f7038005830 comp rx=0 tx=0).stop 2024-09-10T20:41:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.634+0000 7f704ad9d640 1 -- 172.21.15.28:0/2284448656 reap_dead start 2024-09-10T20:41:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.634+0000 7f70509e8640 1 -- 172.21.15.28:0/2284448656 shutdown_connections 2024-09-10T20:41:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.634+0000 7f70509e8640 1 -- 172.21.15.28:0/2284448656 >> 172.21.15.28:0/2284448656 conn(0x7f704c06d110 msgr2=0x7f704c0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.634+0000 7f70509e8640 1 -- 172.21.15.28:0/2284448656 shutdown_connections 2024-09-10T20:41:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.634+0000 7f70509e8640 1 -- 172.21.15.28:0/2284448656 wait complete. 2024-09-10T20:41:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.635+0000 7f70509e8640 1 Processor -- start 2024-09-10T20:41:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.635+0000 7f70509e8640 1 -- start start 2024-09-10T20:41:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f70509e8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038004000 0x7f704c140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f70509e8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704c146420 0x7f704c141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f70509e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704c142390 0x7f704c143ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f70509e8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f704c0b32e0 con 0x7f704c142390 2024-09-10T20:41:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f70509e8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f704c0b3160 con 0x7f7038004000 2024-09-10T20:41:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f70509e8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f704c073290 con 0x7f704c146420 2024-09-10T20:41:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704c142390 0x7f704c143ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704c142390 0x7f704c143ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36022/0 (socket says 172.21.15.28:36022) 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 -- 172.21.15.28:0/2670782425 learned_addr learned my addr 172.21.15.28:0/2670782425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704a59c640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038004000 0x7f704c140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f7049d9b640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704c146420 0x7f704c141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 -- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704c146420 msgr2=0x7f704c141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704c146420 0x7f704c141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 -- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038004000 msgr2=0x7f704c140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038004000 0x7f704c140780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.636+0000 7f704ad9d640 1 -- 172.21.15.28:0/2670782425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c140fd0 con 0x7f704c142390 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.637+0000 7f704ad9d640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704c142390 0x7f704c143ef0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f70440012e0 tx=0x7f704400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.637+0000 7f7042ffd640 1 -- 172.21.15.28:0/2670782425 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704400cc20 con 0x7f704c142390 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.637+0000 7f7042ffd640 1 -- 172.21.15.28:0/2670782425 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7044017070 con 0x7f704c142390 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.637+0000 7f7042ffd640 1 -- 172.21.15.28:0/2670782425 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70440116c0 con 0x7f704c142390 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.637+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f704c1412b0 con 0x7f704c142390 2024-09-10T20:41:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.637+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f704c141870 con 0x7f704c142390 2024-09-10T20:41:41.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.638+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f704c141ab0 con 0x7f704c142390 2024-09-10T20:41:41.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.638+0000 7f7042ffd640 1 -- 172.21.15.28:0/2670782425 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f7044011860 con 0x7f704c142390 2024-09-10T20:41:41.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.639+0000 7f7042ffd640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f702c078300 0x7f702c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.639+0000 7f7042ffd640 1 -- 172.21.15.28:0/2670782425 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f704409a4e0 con 0x7f704c142390 2024-09-10T20:41:41.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.639+0000 7f704a59c640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f702c078300 0x7f702c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.639+0000 7f704a59c640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f702c078300 0x7f702c07a7c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f703c000c00 tx=0x7f703c006b70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:41.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.643+0000 7f7042ffd640 1 -- 172.21.15.28:0/2670782425 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f704c141ab0 con 0x7f704c142390 2024-09-10T20:41:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.827+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f704c0b2300 con 0x7f704c142390 2024-09-10T20:41:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.827+0000 7f7042ffd640 1 -- 172.21.15.28:0/2670782425 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) ==== 93+0+10 (secure 0 0 0) 0x7f704400ee60 con 0x7f704c142390 2024-09-10T20:41:41.833 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:41:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f702c078300 msgr2=0x7f702c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f702c078300 0x7f702c07a7c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f703c000c00 tx=0x7f703c006b70 comp rx=0 tx=0).stop 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704c142390 msgr2=0x7f704c143ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704c142390 0x7f704c143ef0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f70440012e0 tx=0x7f704400ed30 comp rx=0 tx=0).stop 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 shutdown_connections 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f702c078300 0x7f702c07a7c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704c142390 0x7f704c143ef0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704c146420 0x7f704c141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 --2- 172.21.15.28:0/2670782425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7038004000 0x7f704c140780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 >> 172.21.15.28:0/2670782425 conn(0x7f704c06d110 msgr2=0x7f704c157390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 shutdown_connections 2024-09-10T20:41:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.829+0000 7f70509e8640 1 -- 172.21.15.28:0/2670782425 wait complete. 2024-09-10T20:41:41.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-09-10T20:41:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.945+0000 7f281022a640 1 -- 172.21.15.28:0/3115063985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 msgr2=0x7f28080cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.945+0000 7f281022a640 1 --2- 172.21.15.28:0/3115063985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f28080cea80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f280806eb50 tx=0x7f280402a4a0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.945+0000 7f281022a640 1 -- 172.21.15.28:0/3115063985 shutdown_connections 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.945+0000 7f281022a640 1 --2- 172.21.15.28:0/3115063985 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28080d1170 0x7f280806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.945+0000 7f281022a640 1 --2- 172.21.15.28:0/3115063985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f28080cea80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.945+0000 7f281022a640 1 -- 172.21.15.28:0/3115063985 >> 172.21.15.28:0/3115063985 conn(0x7f28080d4050 msgr2=0x7f28080d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.945+0000 7f281022a640 1 -- 172.21.15.28:0/3115063985 shutdown_connections 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.946+0000 7f281022a640 1 -- 172.21.15.28:0/3115063985 wait complete. 2024-09-10T20:41:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.947+0000 7f281022a640 1 Processor -- start 2024-09-10T20:41:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.947+0000 7f281022a640 1 -- start start 2024-09-10T20:41:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.947+0000 7f281022a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f280806eba0 0x7f2808148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f281022a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28080d1170 0x7f280815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f281022a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f280815fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f281022a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28080cf140 con 0x7f28081617e0 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f281022a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f28080cefc0 con 0x7f280806eba0 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f281022a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f280806de40 con 0x7f28080d1170 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f280815fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f280815fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36034/0 (socket says 172.21.15.28:36034) 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 -- 172.21.15.28:0/3243639234 learned_addr learned my addr 172.21.15.28:0/3243639234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280d7c5640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28080d1170 0x7f280815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280dfc6640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f280806eba0 0x7f2808148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 -- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28080d1170 msgr2=0x7f280815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28080d1170 0x7f280815f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 -- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f280806eba0 msgr2=0x7f2808148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f280806eba0 0x7f2808148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 -- 172.21.15.28:0/3243639234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28081f5d20 con 0x7f28081617e0 2024-09-10T20:41:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f280e7c7640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f280815fee0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f280000e9d0 tx=0x7f280000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f27fa7fc640 1 -- 172.21.15.28:0/3243639234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280000cd80 con 0x7f28081617e0 2024-09-10T20:41:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f27fa7fc640 1 -- 172.21.15.28:0/3243639234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2800004c30 con 0x7f28081617e0 2024-09-10T20:41:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28080b6f90 con 0x7f28081617e0 2024-09-10T20:41:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.948+0000 7f27fa7fc640 1 -- 172.21.15.28:0/3243639234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800005280 con 0x7f28081617e0 2024-09-10T20:41:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.949+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28080b7550 con 0x7f28081617e0 2024-09-10T20:41:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.949+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28080cea80 con 0x7f28081617e0 2024-09-10T20:41:41.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.950+0000 7f27fa7fc640 1 -- 172.21.15.28:0/3243639234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f280000f040 con 0x7f28081617e0 2024-09-10T20:41:41.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.951+0000 7f27fa7fc640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27ec078300 0x7f27ec07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:41.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.951+0000 7f27fa7fc640 1 -- 172.21.15.28:0/3243639234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f280009a6f0 con 0x7f28081617e0 2024-09-10T20:41:41.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.951+0000 7f280dfc6640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27ec078300 0x7f27ec07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.952+0000 7f280dfc6640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27ec078300 0x7f27ec07a7c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2804033000 tx=0x7f2804004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:41.955+0000 7f27fa7fc640 1 -- 172.21.15.28:0/3243639234 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2800016030 con 0x7f28081617e0 2024-09-10T20:41:42.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:42.211+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7f28080b6930 con 0x7f28081617e0 2024-09-10T20:41:43.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.200+0000 7f27fa7fc640 1 -- 172.21.15.28:0/3243639234 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v194) ==== 134+0+0 (secure 0 0 0) 0x7f2800062910 con 0x7f28081617e0 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.203+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27ec078300 msgr2=0x7f27ec07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27ec078300 0x7f27ec07a7c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2804033000 tx=0x7f2804004730 comp rx=0 tx=0).stop 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 msgr2=0x7f280815fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f280815fee0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f280000e9d0 tx=0x7f280000ee90 comp rx=0 tx=0).stop 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 shutdown_connections 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f27ec078300 0x7f27ec07a7c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28081617e0 0x7f280815fee0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f28080d1170 0x7f280815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 --2- 172.21.15.28:0/3243639234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f280806eba0 0x7f2808148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 >> 172.21.15.28:0/3243639234 conn(0x7f28080d4050 msgr2=0x7f28080b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 shutdown_connections 2024-09-10T20:41:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.204+0000 7f281022a640 1 -- 172.21.15.28:0/3243639234 wait complete. 2024-09-10T20:41:43.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:41:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 -- 172.21.15.28:0/4060668240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000b21b0 msgr2=0x7f580006dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 --2- 172.21.15.28:0/4060668240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000b21b0 0x7f580006dbf0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009830 tx=0x7f57f002a510 comp rx=0 tx=0).stop 2024-09-10T20:41:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 -- 172.21.15.28:0/4060668240 shutdown_connections 2024-09-10T20:41:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 --2- 172.21.15.28:0/4060668240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000b21b0 0x7f580006dbf0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 --2- 172.21.15.28:0/4060668240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58000d03f0 0x7f58000d0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 -- 172.21.15.28:0/4060668240 >> 172.21.15.28:0/4060668240 conn(0x7f58000d1e40 msgr2=0x7f58000d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 -- 172.21.15.28:0/4060668240 shutdown_connections 2024-09-10T20:41:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.329+0000 7f580819c640 1 -- 172.21.15.28:0/4060668240 wait complete. 2024-09-10T20:41:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.330+0000 7f580819c640 1 Processor -- start 2024-09-10T20:41:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.330+0000 7f580819c640 1 -- start start 2024-09-10T20:41:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f580819c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58000b21b0 0x7f580015eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f580819c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58000d03f0 0x7f580015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f580819c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000d07d0 0x7f58000c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f580819c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58000aedf0 con 0x7f58000d07d0 2024-09-10T20:41:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f580819c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f58000aec70 con 0x7f58000d03f0 2024-09-10T20:41:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f580819c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f58000aef70 con 0x7f58000b21b0 2024-09-10T20:41:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f5806739640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000d07d0 0x7f58000c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f5806739640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000d07d0 0x7f58000c7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36048/0 (socket says 172.21.15.28:36048) 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.331+0000 7f5806739640 1 -- 172.21.15.28:0/1027052701 learned_addr learned my addr 172.21.15.28:0/1027052701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5805f38640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58000b21b0 0x7f580015eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5805737640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58000d03f0 0x7f580015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5806739640 1 -- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58000b21b0 msgr2=0x7f580015eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5806739640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58000b21b0 0x7f580015eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5806739640 1 -- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58000d03f0 msgr2=0x7f580015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5806739640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58000d03f0 0x7f580015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5806739640 1 -- 172.21.15.28:0/1027052701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580009c820 con 0x7f58000d07d0 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f5806739640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000d07d0 0x7f58000c7b00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f57f800c6c0 tx=0x7f57f800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f57f67fc640 1 -- 172.21.15.28:0/1027052701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f8017070 con 0x7f58000d07d0 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f57f67fc640 1 -- 172.21.15.28:0/1027052701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57f8003c60 con 0x7f58000d07d0 2024-09-10T20:41:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580009cb00 con 0x7f58000d07d0 2024-09-10T20:41:43.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f57f67fc640 1 -- 172.21.15.28:0/1027052701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f80052d0 con 0x7f58000d07d0 2024-09-10T20:41:43.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.332+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580015c240 con 0x7f58000d07d0 2024-09-10T20:41:43.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.333+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f580009e9c0 con 0x7f58000d07d0 2024-09-10T20:41:43.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.334+0000 7f57f67fc640 1 -- 172.21.15.28:0/1027052701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f57f8005470 con 0x7f58000d07d0 2024-09-10T20:41:43.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.334+0000 7f57f67fc640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f57e40782d0 0x7f57e407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.335+0000 7f57f67fc640 1 -- 172.21.15.28:0/1027052701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f57f80998f0 con 0x7f58000d07d0 2024-09-10T20:41:43.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.335+0000 7f5805f38640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f57e40782d0 0x7f57e407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.335+0000 7f5805f38640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f57e40782d0 0x7f57e407a790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00ba40 tx=0x7f57fc005cc0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:43.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.339+0000 7f57f67fc640 1 -- 172.21.15.28:0/1027052701 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f58000b3750 con 0x7f58000d07d0 2024-09-10T20:41:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.527+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f58000d0d90 con 0x7f58000d07d0 2024-09-10T20:41:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.528+0000 7f57f67fc640 1 -- 172.21.15.28:0/1027052701 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v33) ==== 59+0+1998 (secure 0 0 0) 0x7f57f8062260 con 0x7f58000d07d0 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:41:43.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:41:43.534 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:41:43.535 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:41:43.535 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:41:43.535 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:41:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f57e40782d0 msgr2=0x7f57e407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f57e40782d0 0x7f57e407a790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00ba40 tx=0x7f57fc005cc0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000d07d0 msgr2=0x7f58000c7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000d07d0 0x7f58000c7b00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f57f800c6c0 tx=0x7f57f800cb80 comp rx=0 tx=0).stop 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 shutdown_connections 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f57e40782d0 0x7f57e407a790 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58000d07d0 0x7f58000c7b00 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58000d03f0 0x7f580015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 --2- 172.21.15.28:0/1027052701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58000b21b0 0x7f580015eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 >> 172.21.15.28:0/1027052701 conn(0x7f58000d1e40 msgr2=0x7f580006dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 shutdown_connections 2024-09-10T20:41:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.530+0000 7f580819c640 1 -- 172.21.15.28:0/1027052701 wait complete. 2024-09-10T20:41:43.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:41:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 -- 172.21.15.28:0/1572148138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151aa0 msgr2=0x7f401815b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 --2- 172.21.15.28:0/1572148138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151aa0 0x7f401815b2f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4008009830 tx=0x7f400802a480 comp rx=0 tx=0).stop 2024-09-10T20:41:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 -- 172.21.15.28:0/1572148138 shutdown_connections 2024-09-10T20:41:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 --2- 172.21.15.28:0/1572148138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4018151e80 0x7f401815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 --2- 172.21.15.28:0/1572148138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151aa0 0x7f401815b2f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 -- 172.21.15.28:0/1572148138 >> 172.21.15.28:0/1572148138 conn(0x7f40180d0c70 msgr2=0x7f40180d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 -- 172.21.15.28:0/1572148138 shutdown_connections 2024-09-10T20:41:43.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.649+0000 7f401d35d640 1 -- 172.21.15.28:0/1572148138 wait complete. 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.650+0000 7f401d35d640 1 Processor -- start 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.650+0000 7f401d35d640 1 -- start start 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f401d35d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40180d05e0 0x7f40181eee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f401d35d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4018151aa0 0x7f40181f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f401d35d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151e80 0x7f40180ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f401d35d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4018070640 con 0x7f4018151e80 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f401d35d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4018073fe0 con 0x7f40180d05e0 2024-09-10T20:41:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f401d35d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f40180707c0 con 0x7f4018151aa0 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f40177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151e80 0x7f40180ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f40177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151e80 0x7f40180ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36068/0 (socket says 172.21.15.28:36068) 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f40177fe640 1 -- 172.21.15.28:0/3856361670 learned_addr learned my addr 172.21.15.28:0/3856361670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f40167fc640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4018151aa0 0x7f40181f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f4016ffd640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40180d05e0 0x7f40181eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f40177fe640 1 -- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4018151aa0 msgr2=0x7f40181f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.651+0000 7f40177fe640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4018151aa0 0x7f40181f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f40177fe640 1 -- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40180d05e0 msgr2=0x7f40181eee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f40177fe640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40180d05e0 0x7f40181eee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f40177fe640 1 -- 172.21.15.28:0/3856361670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40180c7d40 con 0x7f4018151e80 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f40177fe640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151e80 0x7f40180ca3d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f401000cbc0 tx=0x7f401000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f3fff7fe640 1 -- 172.21.15.28:0/3856361670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010016020 con 0x7f4018151e80 2024-09-10T20:41:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f3fff7fe640 1 -- 172.21.15.28:0/3856361670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f401000ae30 con 0x7f4018151e80 2024-09-10T20:41:43.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f3fff7fe640 1 -- 172.21.15.28:0/3856361670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010012660 con 0x7f4018151e80 2024-09-10T20:41:43.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40180c8020 con 0x7f4018151e80 2024-09-10T20:41:43.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.652+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40180c85e0 con 0x7f4018151e80 2024-09-10T20:41:43.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.653+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f401806a0f0 con 0x7f4018151e80 2024-09-10T20:41:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.654+0000 7f3fff7fe640 1 -- 172.21.15.28:0/3856361670 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f401001b400 con 0x7f4018151e80 2024-09-10T20:41:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.654+0000 7f3fff7fe640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078300 0x7f400407a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.654+0000 7f3fff7fe640 1 -- 172.21.15.28:0/3856361670 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f401009a8a0 con 0x7f4018151e80 2024-09-10T20:41:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.654+0000 7f4016ffd640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078300 0x7f400407a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.655+0000 7f4016ffd640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078300 0x7f400407a7c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4008034000 tx=0x7f4008004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.658+0000 7f3fff7fe640 1 -- 172.21.15.28:0/3856361670 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f401806a0f0 con 0x7f4018151e80 2024-09-10T20:41:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.846+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f4018158e10 con 0x7f4018151e80 2024-09-10T20:41:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.850+0000 7f3fff7fe640 1 -- 172.21.15.28:0/3856361670 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f4010020020 con 0x7f4018151e80 2024-09-10T20:41:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078300 msgr2=0x7f400407a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078300 0x7f400407a7c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4008034000 tx=0x7f4008004730 comp rx=0 tx=0).stop 2024-09-10T20:41:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151e80 msgr2=0x7f40180ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151e80 0x7f40180ca3d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f401000cbc0 tx=0x7f401000a430 comp rx=0 tx=0).stop 2024-09-10T20:41:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 shutdown_connections 2024-09-10T20:41:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4004078300 0x7f400407a7c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018151e80 0x7f40180ca3d0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4018151aa0 0x7f40181f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 --2- 172.21.15.28:0/3856361670 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40180d05e0 0x7f40181eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 >> 172.21.15.28:0/3856361670 conn(0x7f40180d0c70 msgr2=0x7f4018159820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 shutdown_connections 2024-09-10T20:41:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:43.852+0000 7f401d35d640 1 -- 172.21.15.28:0/3856361670 wait complete. 2024-09-10T20:41:43.868 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:41:43.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:41:43.913 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:41:43.959 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:41:43.959 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:43.959 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:43.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:41:43.976 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:43.976 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:43.994 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:41:43.994 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:41:43.994 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:44.032 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:41:44.032 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:41:44.032 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:41:44.052 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:41:44.052 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:41:44.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 -- 172.21.15.28:0/2706349718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc06d020 msgr2=0x7fc1cc15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/2706349718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc06d020 0x7fc1cc15cbe0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4005fd0 tx=0x7fc1c402aa00 comp rx=0 tx=0).stop 2024-09-10T20:41:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 -- 172.21.15.28:0/2706349718 shutdown_connections 2024-09-10T20:41:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/2706349718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc06d020 0x7fc1cc15cbe0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/2706349718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1cc15bf20 0x7fc1cc06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 -- 172.21.15.28:0/2706349718 >> 172.21.15.28:0/2706349718 conn(0x7fc1cc06dbf0 msgr2=0x7fc1cc06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 -- 172.21.15.28:0/2706349718 shutdown_connections 2024-09-10T20:41:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.173+0000 7fc1d39bb640 1 -- 172.21.15.28:0/2706349718 wait complete. 2024-09-10T20:41:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.174+0000 7fc1d39bb640 1 Processor -- start 2024-09-10T20:41:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.174+0000 7fc1d39bb640 1 -- start start 2024-09-10T20:41:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d39bb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1cc06d020 0x7fc1cc09bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d39bb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1cc15bb40 0x7fc1cc1f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d39bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc15bf20 0x7fc1cc0b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d39bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc15d4b0 con 0x7fc1cc15bf20 2024-09-10T20:41:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d39bb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc15d330 con 0x7fc1cc15bb40 2024-09-10T20:41:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d39bb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc1cc15d630 con 0x7fc1cc06d020 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc15bf20 0x7fc1cc0b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc15bf20 0x7fc1cc0b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36098/0 (socket says 172.21.15.28:36098) 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 -- 172.21.15.28:0/709032051 learned_addr learned my addr 172.21.15.28:0/709032051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d0f56640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1cc15bb40 0x7fc1cc1f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1757640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1cc06d020 0x7fc1cc09bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 -- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1cc06d020 msgr2=0x7fc1cc09bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1cc06d020 0x7fc1cc09bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 -- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1cc15bb40 msgr2=0x7fc1cc1f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1cc15bb40 0x7fc1cc1f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.175+0000 7fc1d1f58640 1 -- 172.21.15.28:0/709032051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc1f5190 con 0x7fc1cc15bf20 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.176+0000 7fc1d1f58640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc15bf20 0x7fc1cc0b4b40 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009d90 tx=0x7fc1c4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.176+0000 7fc1c1ffb640 1 -- 172.21.15.28:0/709032051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4047070 con 0x7fc1cc15bf20 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.176+0000 7fc1c1ffb640 1 -- 172.21.15.28:0/709032051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1c4004250 con 0x7fc1cc15bf20 2024-09-10T20:41:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.176+0000 7fc1c1ffb640 1 -- 172.21.15.28:0/709032051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4033050 con 0x7fc1cc15bf20 2024-09-10T20:41:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.176+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1cc1f5470 con 0x7fc1cc15bf20 2024-09-10T20:41:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.176+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1cc09b830 con 0x7fc1cc15bf20 2024-09-10T20:41:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.177+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1cc1f56b0 con 0x7fc1cc15bf20 2024-09-10T20:41:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.178+0000 7fc1c1ffb640 1 -- 172.21.15.28:0/709032051 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fc1c4004680 con 0x7fc1cc15bf20 2024-09-10T20:41:44.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.178+0000 7fc1c1ffb640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc1b00782d0 0x7fc1b007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.178+0000 7fc1c1ffb640 1 -- 172.21.15.28:0/709032051 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fc1c40bb5f0 con 0x7fc1cc15bf20 2024-09-10T20:41:44.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.178+0000 7fc1d1757640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc1b00782d0 0x7fc1b007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.179+0000 7fc1d1757640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc1b00782d0 0x7fc1b007a790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0b34a0 tx=0x7fc1c8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:44.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.182+0000 7fc1c1ffb640 1 -- 172.21.15.28:0/709032051 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc1cc1f56b0 con 0x7fc1cc15bf20 2024-09-10T20:41:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.404+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1cc09ba30 con 0x7fc1cc15bf20 2024-09-10T20:41:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.404+0000 7fc1c1ffb640 1 -- 172.21.15.28:0/709032051 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+3911 (secure 0 0 0) 0x7fc1cc09ba30 con 0x7fc1cc15bf20 2024-09-10T20:41:44.410 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:44.410 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":38,"btime":"2024-09-10T20:41:38:184881+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/485554639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":485554639},{"type":"v1","addr":"172.21.15.159:6836","nonce":485554639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5195,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/1807026586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":1807026586},{"type":"v1","addr":"172.21.15.159:6837","nonce":1807026586}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/975621681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":975621681},{"type":"v1","addr":"172.21.15.28:6834","nonce":975621681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1580238478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1580238478},{"type":"v1","addr":"172.21.15.28:6835","nonce":1580238478}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[]} 2024-09-10T20:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 38 2024-09-10T20:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc1b00782d0 msgr2=0x7fc1b007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc1b00782d0 0x7fc1b007a790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0b34a0 tx=0x7fc1c8009040 comp rx=0 tx=0).stop 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc15bf20 msgr2=0x7fc1cc0b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc15bf20 0x7fc1cc0b4b40 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009d90 tx=0x7fc1c4003f10 comp rx=0 tx=0).stop 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 shutdown_connections 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc1b00782d0 0x7fc1b007a790 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc15bf20 0x7fc1cc0b4b40 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1cc15bb40 0x7fc1cc1f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 --2- 172.21.15.28:0/709032051 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1cc06d020 0x7fc1cc09bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 >> 172.21.15.28:0/709032051 conn(0x7fc1cc06dbf0 msgr2=0x7fc1cc075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 shutdown_connections 2024-09-10T20:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.407+0000 7fc1d39bb640 1 -- 172.21.15.28:0/709032051 wait complete. 2024-09-10T20:41:44.424 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:41:44.424 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:41:44.424 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:41:44.424 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:44.425 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:41:44.425 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:41:44.425 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:41:44.425 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:44.425 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:41:44.425 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:41:44.426 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:41:44.426 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:44.426 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:41:44.426 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:41:44.426 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:41:44.426 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:41:44.436 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:41:44.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:41:44.438 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:41:44.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:41:44.439 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:41:44.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:41:44.440 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:41:44.440 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:41:44.479 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:41:44.479 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:44.480 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:41:44.481 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:44.481 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:41:44.482 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:44.482 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:41:44.482 DEBUG:teuthology.parallel:result is None 2024-09-10T20:41:44.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:41:44.557 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:41:44.557 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:41:44.563 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:41:44.563 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:41:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.587+0000 7f0800952640 1 -- 172.21.15.28:0/1015220402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc0d2b90 msgr2=0x7f07fc161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.587+0000 7f0800952640 1 --2- 172.21.15.28:0/1015220402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc0d2b90 0x7f07fc161b50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009830 tx=0x7f07f002a510 comp rx=0 tx=0).stop 2024-09-10T20:41:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.588+0000 7f0800952640 1 -- 172.21.15.28:0/1015220402 shutdown_connections 2024-09-10T20:41:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.588+0000 7f0800952640 1 --2- 172.21.15.28:0/1015220402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc0d2b90 0x7f07fc161b50 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.588+0000 7f0800952640 1 -- 172.21.15.28:0/1015220402 >> 172.21.15.28:0/1015220402 conn(0x7f07fc0d3fb0 msgr2=0x7f07fc0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.588+0000 7f0800952640 1 -- 172.21.15.28:0/1015220402 shutdown_connections 2024-09-10T20:41:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.588+0000 7f0800952640 1 -- 172.21.15.28:0/1015220402 wait complete. 2024-09-10T20:41:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.589+0000 7f0800952640 1 Processor -- start 2024-09-10T20:41:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.589+0000 7f0800952640 1 -- start start 2024-09-10T20:41:44.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f0800952640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc0d2b90 0x7f07fc09e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f0800952640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc15de70 0x7f07fc0cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f0800952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc162090 0x7f07fc160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f0800952640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc15e3d0 con 0x7f07fc162090 2024-09-10T20:41:44.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f0800952640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc15e250 con 0x7f07fc0d2b90 2024-09-10T20:41:44.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f0800952640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f07fc0b7510 con 0x7f07fc15de70 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc162090 0x7f07fc160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fa59c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc0d2b90 0x7f07fc09e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc162090 0x7f07fc160ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36142/0 (socket says 172.21.15.28:36142) 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fad9d640 1 -- 172.21.15.28:0/277695347 learned_addr learned my addr 172.21.15.28:0/277695347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07f9d9b640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc15de70 0x7f07fc0cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fa59c640 1 -- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc15de70 msgr2=0x7f07fc0cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fa59c640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc15de70 0x7f07fc0cc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fa59c640 1 -- 172.21.15.28:0/277695347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc162090 msgr2=0x7f07fc160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fa59c640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc162090 0x7f07fc160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.590+0000 7f07fa59c640 1 -- 172.21.15.28:0/277695347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07fc15e760 con 0x7f07fc0d2b90 2024-09-10T20:41:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.591+0000 7f07fa59c640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc0d2b90 0x7f07fc09e600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00c3c0 tx=0x7f07ec00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.591+0000 7f07eaffd640 1 -- 172.21.15.28:0/277695347 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec011790 con 0x7f07fc0d2b90 2024-09-10T20:41:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.591+0000 7f07eaffd640 1 -- 172.21.15.28:0/277695347 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07ec017070 con 0x7f07fc0d2b90 2024-09-10T20:41:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.591+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07fc15ea40 con 0x7f07fc0d2b90 2024-09-10T20:41:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.591+0000 7f07eaffd640 1 -- 172.21.15.28:0/277695347 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec011ce0 con 0x7f07fc0d2b90 2024-09-10T20:41:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.591+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07fc15f2e0 con 0x7f07fc0d2b90 2024-09-10T20:41:44.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.592+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07fc15ee30 con 0x7f07fc0d2b90 2024-09-10T20:41:44.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.593+0000 7f07eaffd640 1 -- 172.21.15.28:0/277695347 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f07ec01c460 con 0x7f07fc0d2b90 2024-09-10T20:41:44.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.593+0000 7f07eaffd640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f07dc0782d0 0x7f07dc07a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:44.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.593+0000 7f07eaffd640 1 -- 172.21.15.28:0/277695347 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f07ec099d10 con 0x7f07fc0d2b90 2024-09-10T20:41:44.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.593+0000 7f07f9d9b640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f07dc0782d0 0x7f07dc07a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:44.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.594+0000 7f07f9d9b640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f07dc0782d0 0x7f07dc07a790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f07f0034000 tx=0x7f07f0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:44.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.598+0000 7f07eaffd640 1 -- 172.21.15.28:0/277695347 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f07ec061fa0 con 0x7f07fc0d2b90 2024-09-10T20:41:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:44.796+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f07fc0b7a00 con 0x7f07fc0d2b90 2024-09-10T20:41:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.879+0000 7f07eaffd640 1 -- 172.21.15.28:0/277695347 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) ==== 97+0+0 (secure 0 0 0) 0x7f07ec0669d0 con 0x7f07fc0d2b90 2024-09-10T20:41:45.884 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:41:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.881+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f07dc0782d0 msgr2=0x7f07dc07a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.881+0000 7f0800952640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f07dc0782d0 0x7f07dc07a790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f07f0034000 tx=0x7f07f0004730 comp rx=0 tx=0).stop 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.881+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc0d2b90 msgr2=0x7f07fc09e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.881+0000 7f0800952640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc0d2b90 0x7f07fc09e600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00c3c0 tx=0x7f07ec00c880 comp rx=0 tx=0).stop 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 shutdown_connections 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f07dc0782d0 0x7f07dc07a790 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07fc162090 0x7f07fc160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07fc15de70 0x7f07fc0cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 --2- 172.21.15.28:0/277695347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07fc0d2b90 0x7f07fc09e600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 >> 172.21.15.28:0/277695347 conn(0x7f07fc0d3fb0 msgr2=0x7f07fc06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 shutdown_connections 2024-09-10T20:41:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:45.882+0000 7f0800952640 1 -- 172.21.15.28:0/277695347 wait complete. 2024-09-10T20:41:45.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:41:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.007+0000 7f165862a640 1 -- 172.21.15.28:0/1215529798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156510 msgr2=0x7f16500caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.007+0000 7f165862a640 1 --2- 172.21.15.28:0/1215529798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156510 0x7f16500caa70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1648009830 tx=0x7f164802a5c0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.007+0000 7f165862a640 1 -- 172.21.15.28:0/1215529798 shutdown_connections 2024-09-10T20:41:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.007+0000 7f165862a640 1 --2- 172.21.15.28:0/1215529798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156510 0x7f16500caa70 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.007+0000 7f165862a640 1 -- 172.21.15.28:0/1215529798 >> 172.21.15.28:0/1215529798 conn(0x7f16500d0420 msgr2=0x7f16500d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.007+0000 7f165862a640 1 -- 172.21.15.28:0/1215529798 shutdown_connections 2024-09-10T20:41:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.007+0000 7f165862a640 1 -- 172.21.15.28:0/1215529798 wait complete. 2024-09-10T20:41:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.008+0000 7f165862a640 1 Processor -- start 2024-09-10T20:41:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.008+0000 7f165862a640 1 -- start start 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f165862a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f165006d1f0 0x7f16500b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f165862a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156130 0x7f16500b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f165862a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1650156510 0x7f165014e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f165862a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f165006f2c0 con 0x7f1650156130 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f165862a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f165006f140 con 0x7f1650156510 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f165862a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1650073900 con 0x7f165006d1f0 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156130 0x7f16500b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156130 0x7f16500b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36154/0 (socket says 172.21.15.28:36154) 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 -- 172.21.15.28:0/213460815 learned_addr learned my addr 172.21.15.28:0/213460815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f16563c6640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f165006d1f0 0x7f16500b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1656bc7640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1650156510 0x7f165014e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 -- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f165006d1f0 msgr2=0x7f16500b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f165006d1f0 0x7f16500b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 -- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1650156510 msgr2=0x7f165014e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1650156510 0x7f165014e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.009+0000 7f1655bc5640 1 -- 172.21.15.28:0/213460815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16500b4440 con 0x7f1650156130 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.010+0000 7f1655bc5640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156130 0x7f16500b5a70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f164400c500 tx=0x7f164400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.010+0000 7f164effd640 1 -- 172.21.15.28:0/213460815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16440052d0 con 0x7f1650156130 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.010+0000 7f164effd640 1 -- 172.21.15.28:0/213460815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1644005470 con 0x7f1650156130 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.010+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16500c5360 con 0x7f1650156130 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.010+0000 7f164effd640 1 -- 172.21.15.28:0/213460815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1644012870 con 0x7f1650156130 2024-09-10T20:41:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.010+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16500b49c0 con 0x7f1650156130 2024-09-10T20:41:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.011+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1650073db0 con 0x7f1650156130 2024-09-10T20:41:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.012+0000 7f164effd640 1 -- 172.21.15.28:0/213460815 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f16440035f0 con 0x7f1650156130 2024-09-10T20:41:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.012+0000 7f164effd640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f163c078300 0x7f163c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.012+0000 7f164effd640 1 -- 172.21.15.28:0/213460815 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f1644099d20 con 0x7f1650156130 2024-09-10T20:41:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.013+0000 7f16563c6640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f163c078300 0x7f163c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.013+0000 7f16563c6640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f163c078300 0x7f163c07a7c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16400012e0 tx=0x7f1640009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:46.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.016+0000 7f164effd640 1 -- 172.21.15.28:0/213460815 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1644062560 con 0x7f1650156130 2024-09-10T20:41:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.271+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f16500d11c0 con 0x7f1650156130 2024-09-10T20:41:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.271+0000 7f164effd640 1 -- 172.21.15.28:0/213460815 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) ==== 80+0+5617 (secure 0 0 0) 0x7f1644061e60 con 0x7f1650156130 2024-09-10T20:41:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:46.276 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:41:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:41:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.278 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.279 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.280 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.282 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:41:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:41:46.284 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:41:46.285 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:41:46.286 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.alice", 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD8ruBm/eTUGRAAjibYtCfoMLWr9ThEVaTxnQ==", 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.288 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol45350", 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw pool=cephfs_data" 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:41:46.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:41:46.290 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:41:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f163c078300 msgr2=0x7f163c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f163c078300 0x7f163c07a7c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16400012e0 tx=0x7f1640009040 comp rx=0 tx=0).stop 2024-09-10T20:41:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156130 msgr2=0x7f16500b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156130 0x7f16500b5a70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f164400c500 tx=0x7f164400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 shutdown_connections 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f163c078300 0x7f163c07a7c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1650156510 0x7f165014e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1650156130 0x7f16500b5a70 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 --2- 172.21.15.28:0/213460815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f165006d1f0 0x7f16500b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 >> 172.21.15.28:0/213460815 conn(0x7f16500d0420 msgr2=0x7f165015e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 shutdown_connections 2024-09-10T20:41:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.273+0000 7f165862a640 1 -- 172.21.15.28:0/213460815 wait complete. 2024-09-10T20:41:46.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.alice 2024-09-10T20:41:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 -- 172.21.15.28:0/350005712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 msgr2=0x7f45e40b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/350005712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e40b4600 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f45cc005fd0 tx=0x7f45cc02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 -- 172.21.15.28:0/350005712 shutdown_connections 2024-09-10T20:41:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/350005712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e40b4600 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/350005712 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45e415a190 0x7f45e40b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 -- 172.21.15.28:0/350005712 >> 172.21.15.28:0/350005712 conn(0x7f45e406cf00 msgr2=0x7f45e406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:46.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 -- 172.21.15.28:0/350005712 shutdown_connections 2024-09-10T20:41:46.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.394+0000 7f45e8bd3640 1 -- 172.21.15.28:0/350005712 wait complete. 2024-09-10T20:41:46.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.395+0000 7f45e8bd3640 1 Processor -- start 2024-09-10T20:41:46.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.395+0000 7f45e8bd3640 1 -- start start 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45e40b4b40 0x7f45e4152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45e415a190 0x7f45e41f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e4072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45e40d0270 con 0x7f45e415df70 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f45e40d00f0 con 0x7f45e40b4b40 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f45e40d03f0 con 0x7f45e415a190 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e4072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e4072ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36162/0 (socket says 172.21.15.28:36162) 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 -- 172.21.15.28:0/2737292965 learned_addr learned my addr 172.21.15.28:0/2737292965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e1d9b640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45e415a190 0x7f45e41f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e259c640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45e40b4b40 0x7f45e4152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 -- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45e415a190 msgr2=0x7f45e41f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45e415a190 0x7f45e41f5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 -- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45e40b4b40 msgr2=0x7f45e4152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45e40b4b40 0x7f45e4152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 -- 172.21.15.28:0/2737292965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e409c9d0 con 0x7f45e415df70 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e2d9d640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e4072ea0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f45d400c3f0 tx=0x7f45d400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45daffd640 1 -- 172.21.15.28:0/2737292965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4017070 con 0x7f45e415df70 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45daffd640 1 -- 172.21.15.28:0/2737292965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45d40053c0 con 0x7f45e415df70 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e409cc70 con 0x7f45e415df70 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45daffd640 1 -- 172.21.15.28:0/2737292965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4003850 con 0x7f45e415df70 2024-09-10T20:41:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.397+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e409d230 con 0x7f45e415df70 2024-09-10T20:41:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.398+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e40b2ba0 con 0x7f45e415df70 2024-09-10T20:41:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.399+0000 7f45daffd640 1 -- 172.21.15.28:0/2737292965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f45d40039f0 con 0x7f45e415df70 2024-09-10T20:41:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.399+0000 7f45daffd640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f45c40782d0 0x7f45c407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.399+0000 7f45daffd640 1 -- 172.21.15.28:0/2737292965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f45d40996f0 con 0x7f45e415df70 2024-09-10T20:41:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.400+0000 7f45e259c640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f45c40782d0 0x7f45c407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.400+0000 7f45e259c640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f45c40782d0 0x7f45c407a790 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f45dc005c40 tx=0x7f45dc005bd0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.404+0000 7f45daffd640 1 -- 172.21.15.28:0/2737292965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f45d4062020 con 0x7f45e415df70 2024-09-10T20:41:46.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.663+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.alice"} v 0) -- 0x7f45e41e8db0 con 0x7f45e415df70 2024-09-10T20:41:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.665+0000 7f45daffd640 1 -- 172.21.15.28:0/2737292965 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.alice"}]=0 v29) ==== 82+0+0 (secure 0 0 0) 0x7f45d4061920 con 0x7f45e415df70 2024-09-10T20:41:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f45c40782d0 msgr2=0x7f45c407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f45c40782d0 0x7f45c407a790 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f45dc005c40 tx=0x7f45dc005bd0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 msgr2=0x7f45e4072ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e4072ea0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f45d400c3f0 tx=0x7f45d400c8b0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 shutdown_connections 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f45c40782d0 0x7f45c407a790 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e415df70 0x7f45e4072ea0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45e415a190 0x7f45e41f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 --2- 172.21.15.28:0/2737292965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45e40b4b40 0x7f45e4152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 >> 172.21.15.28:0/2737292965 conn(0x7f45e406cf00 msgr2=0x7f45e40726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 shutdown_connections 2024-09-10T20:41:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.667+0000 7f45e8bd3640 1 -- 172.21.15.28:0/2737292965 wait complete. 2024-09-10T20:41:46.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.784+0000 7fa3b052b640 1 -- 172.21.15.28:0/1406285857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa394003a30 msgr2=0x7fa394003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.784+0000 7fa3b052b640 1 --2- 172.21.15.28:0/1406285857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa394003a30 0x7fa394003e50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009b50 tx=0x7fa394005830 comp rx=0 tx=0).stop 2024-09-10T20:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.784+0000 7fa3aeac8640 1 -- 172.21.15.28:0/1406285857 reap_dead start 2024-09-10T20:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.784+0000 7fa3b052b640 1 -- 172.21.15.28:0/1406285857 shutdown_connections 2024-09-10T20:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.784+0000 7fa3b052b640 1 -- 172.21.15.28:0/1406285857 >> 172.21.15.28:0/1406285857 conn(0x7fa3a80d0420 msgr2=0x7fa3a80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.784+0000 7fa3b052b640 1 -- 172.21.15.28:0/1406285857 shutdown_connections 2024-09-10T20:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.785+0000 7fa3b052b640 1 -- 172.21.15.28:0/1406285857 wait complete. 2024-09-10T20:41:46.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 Processor -- start 2024-09-10T20:41:46.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 -- start start 2024-09-10T20:41:46.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa394003a30 0x7fa3a8152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3a814a190 0x7fa3a8153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3a80b58b0 0x7fa3a80b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a806ef30 con 0x7fa3a80b58b0 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a80cacc0 con 0x7fa3a814a190 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.786+0000 7fa3b052b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a806f0b0 con 0x7fa394003a30 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3a80b58b0 0x7fa3a80b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3a80b58b0 0x7fa3a80b5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36180/0 (socket says 172.21.15.28:36180) 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 -- 172.21.15.28:0/2789101976 learned_addr learned my addr 172.21.15.28:0/2789101976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3ae2c7640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa394003a30 0x7fa3a8152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 -- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa394003a30 msgr2=0x7fa3a8152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa394003a30 0x7fa3a8152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3adac6640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3a814a190 0x7fa3a8153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 -- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3a814a190 msgr2=0x7fa3a8153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3a814a190 0x7fa3a8153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 -- 172.21.15.28:0/2789101976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a814a610 con 0x7fa3a80b58b0 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3adac6640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3a814a190 0x7fa3a8153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3aeac8640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3a80b58b0 0x7fa3a80b5d30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa394004780 tx=0x7fa3a00019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa39effd640 1 -- 172.21.15.28:0/2789101976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a00107d0 con 0x7fa3a80b58b0 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa39effd640 1 -- 172.21.15.28:0/2789101976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3a0010970 con 0x7fa3a80b58b0 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a81529a0 con 0x7fa3a80b58b0 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa39effd640 1 -- 172.21.15.28:0/2789101976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0010dc0 con 0x7fa3a80b58b0 2024-09-10T20:41:46.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.787+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a80b3520 con 0x7fa3a80b58b0 2024-09-10T20:41:46.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.788+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a806cae0 con 0x7fa3a80b58b0 2024-09-10T20:41:46.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.789+0000 7fa39effd640 1 -- 172.21.15.28:0/2789101976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fa3a001a400 con 0x7fa3a80b58b0 2024-09-10T20:41:46.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.789+0000 7fa39effd640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa38c078300 0x7fa38c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:46.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.789+0000 7fa39effd640 1 -- 172.21.15.28:0/2789101976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7fa3a009a5e0 con 0x7fa3a80b58b0 2024-09-10T20:41:46.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.790+0000 7fa3ae2c7640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa38c078300 0x7fa38c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:46.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.790+0000 7fa3ae2c7640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa38c078300 0x7fa38c07a7c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4035000 tx=0x7fa3a402d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.794+0000 7fa39effd640 1 -- 172.21.15.28:0/2789101976 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa3a0062e60 con 0x7fa3a80b58b0 2024-09-10T20:41:46.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.984+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa3a80d11c0 con 0x7fa3a80b58b0 2024-09-10T20:41:46.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.984+0000 7fa39effd640 1 -- 172.21.15.28:0/2789101976 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) ==== 84+0+65 (secure 0 0 0) 0x7fa3a0062760 con 0x7fa3a80b58b0 2024-09-10T20:41:46.988 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:46.988 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:41:46.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:41:46.988 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:41:46.988 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:41:46.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:41:46.989 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:41:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa38c078300 msgr2=0x7fa38c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa38c078300 0x7fa38c07a7c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4035000 tx=0x7fa3a402d040 comp rx=0 tx=0).stop 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3a80b58b0 msgr2=0x7fa3a80b5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3a80b58b0 0x7fa3a80b5d30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa394004780 tx=0x7fa3a00019b0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 shutdown_connections 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa38c078300 0x7fa38c07a7c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3a80b58b0 0x7fa3a80b5d30 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3a814a190 0x7fa3a8153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.986+0000 7fa3b052b640 1 --2- 172.21.15.28:0/2789101976 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa394003a30 0x7fa3a8152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.987+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 >> 172.21.15.28:0/2789101976 conn(0x7fa3a80d0420 msgr2=0x7fa3a8070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.987+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 shutdown_connections 2024-09-10T20:41:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:46.987+0000 7fa3b052b640 1 -- 172.21.15.28:0/2789101976 wait complete. 2024-09-10T20:41:47.001 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:41:47.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:41:47.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 -- 172.21.15.28:0/782919323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 msgr2=0x7f9edc0ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:47.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/782919323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc0ca620 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0af790 tx=0x7f9ed402aa30 comp rx=0 tx=0).stop 2024-09-10T20:41:47.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 -- 172.21.15.28:0/782919323 shutdown_connections 2024-09-10T20:41:47.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/782919323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc0ca620 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:47.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/782919323 >> v2:172.21.15.159:3300/0 conn(0x7f9edc15bd70 0x7f9edc06fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:47.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 -- 172.21.15.28:0/782919323 >> 172.21.15.28:0/782919323 conn(0x7f9edc0cfc80 msgr2=0x7f9edc0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:47.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 -- 172.21.15.28:0/782919323 shutdown_connections 2024-09-10T20:41:47.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.107+0000 7f9ee1e60640 1 -- 172.21.15.28:0/782919323 wait complete. 2024-09-10T20:41:47.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.108+0000 7f9ee1e60640 1 Processor -- start 2024-09-10T20:41:47.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.108+0000 7f9ee1e60640 1 -- start start 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9ee1e60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc1f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9ee1e60640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc151cf0 0x7f9edc156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9ee1e60640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc15bd70 0x7f9edc0b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9ee1e60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc0caef0 con 0x7f9edc070270 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9ee1e60640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc0cad70 con 0x7f9edc151cf0 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9ee1e60640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9edc0cb070 con 0x7f9edc15bd70 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc1f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc1f51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36206/0 (socket says 172.21.15.28:36206) 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 -- 172.21.15.28:0/3237727583 learned_addr learned my addr 172.21.15.28:0/3237727583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edbfff640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc15bd70 0x7f9edc0b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edaffd640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc151cf0 0x7f9edc156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 -- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc15bd70 msgr2=0x7f9edc0b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc15bd70 0x7f9edc0b4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 -- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc151cf0 msgr2=0x7f9edc156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc151cf0 0x7f9edc156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edb7fe640 1 -- 172.21.15.28:0/3237727583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc1f5720 con 0x7f9edc070270 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.109+0000 7f9edaffd640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc151cf0 0x7f9edc156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.110+0000 7f9edb7fe640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc1f51e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c460 tx=0x7f9ecc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.110+0000 7f9ec3fff640 1 -- 172.21.15.28:0/3237727583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc016020 con 0x7f9edc070270 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.110+0000 7f9ec3fff640 1 -- 172.21.15.28:0/3237727583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ecc012960 con 0x7f9edc070270 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.110+0000 7f9ec3fff640 1 -- 172.21.15.28:0/3237727583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc012db0 con 0x7f9edc070270 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.110+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9edc1570c0 con 0x7f9edc070270 2024-09-10T20:41:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.110+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9edc157680 con 0x7f9edc070270 2024-09-10T20:41:47.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.111+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9edc0ca620 con 0x7f9edc070270 2024-09-10T20:41:47.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.111+0000 7f9ec3fff640 1 -- 172.21.15.28:0/3237727583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f9ecc01a3b0 con 0x7f9edc070270 2024-09-10T20:41:47.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.112+0000 7f9ec3fff640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ec80782d0 0x7f9ec807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:47.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.112+0000 7f9ec3fff640 1 -- 172.21.15.28:0/3237727583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f9ecc099d40 con 0x7f9edc070270 2024-09-10T20:41:47.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.112+0000 7f9edaffd640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ec80782d0 0x7f9ec807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:47.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.113+0000 7f9edaffd640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ec80782d0 0x7f9ec807a790 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00058c0 tx=0x7f9ed0005c90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.116+0000 7f9ec3fff640 1 -- 172.21.15.28:0/3237727583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ecc020070 con 0x7f9edc070270 2024-09-10T20:41:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:47.303+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9edc15dfe0 con 0x7f9edc070270 2024-09-10T20:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.221+0000 7f9ec3fff640 1 -- 172.21.15.28:0/3237727583 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) ==== 138+0+0 (secure 0 0 0) 0x7f9ecc062700 con 0x7f9edc070270 2024-09-10T20:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:41:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.225+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ec80782d0 msgr2=0x7f9ec807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.225+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ec80782d0 0x7f9ec807a790 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00058c0 tx=0x7f9ed0005c90 comp rx=0 tx=0).stop 2024-09-10T20:41:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.225+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 msgr2=0x7f9edc1f51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.225+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc1f51e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c460 tx=0x7f9ecc00c920 comp rx=0 tx=0).stop 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 shutdown_connections 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f9ec80782d0 0x7f9ec807a790 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc15bd70 0x7f9edc0b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc151cf0 0x7f9edc156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 --2- 172.21.15.28:0/3237727583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc070270 0x7f9edc1f51e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 >> 172.21.15.28:0/3237727583 conn(0x7f9edc0cfc80 msgr2=0x7f9edc0d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 shutdown_connections 2024-09-10T20:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.226+0000 7f9ee1e60640 1 -- 172.21.15.28:0/3237727583 wait complete. 2024-09-10T20:41:48.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:41:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 -- 172.21.15.28:0/580988721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 msgr2=0x7fbfd8070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/580988721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 0x7fbfd8070880 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009830 tx=0x7fbfc802a510 comp rx=0 tx=0).stop 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 -- 172.21.15.28:0/580988721 shutdown_connections 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/580988721 >> v2:172.21.15.159:3300/0 conn(0x7fbfd80b1140 0x7fbfd8070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/580988721 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 0x7fbfd8070880 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 -- 172.21.15.28:0/580988721 >> 172.21.15.28:0/580988721 conn(0x7fbfd80d1da0 msgr2=0x7fbfd80d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 -- 172.21.15.28:0/580988721 shutdown_connections 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.363+0000 7fbfdcec6640 1 -- 172.21.15.28:0/580988721 wait complete. 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.365+0000 7fbfdcec6640 1 Processor -- start 2024-09-10T20:41:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.365+0000 7fbfdcec6640 1 -- start start 2024-09-10T20:41:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.365+0000 7fbfdcec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfd8071300 0x7fbfd806e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.365+0000 7fbfdcec6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfd80b1140 0x7fbfd806f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.365+0000 7fbfdcec6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 0x7fbfd815b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.365+0000 7fbfdcec6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80cc900 con 0x7fbfd8071300 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.365+0000 7fbfdcec6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80cc780 con 0x7fbfd80b1140 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfdcec6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd80d0420 con 0x7fbfd815f4a0 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfd8071300 0x7fbfd806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfd8071300 0x7fbfd806e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36214/0 (socket says 172.21.15.28:36214) 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 -- 172.21.15.28:0/1980919164 learned_addr learned my addr 172.21.15.28:0/1980919164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd5d9b640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfd80b1140 0x7fbfd806f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd6d9d640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 0x7fbfd815b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 -- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 msgr2=0x7fbfd815b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 0x7fbfd815b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 -- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfd80b1140 msgr2=0x7fbfd806f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfd80b1140 0x7fbfd806f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 -- 172.21.15.28:0/1980919164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd815bbc0 con 0x7fbfd8071300 2024-09-10T20:41:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfd659c640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfd8071300 0x7fbfd806e600 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009950 tx=0x7fbfc8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfc6ffd640 1 -- 172.21.15.28:0/1980919164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8031d60 con 0x7fbfd8071300 2024-09-10T20:41:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfc6ffd640 1 -- 172.21.15.28:0/1980919164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfc8033e10 con 0x7fbfd8071300 2024-09-10T20:41:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd806eca0 con 0x7fbfd8071300 2024-09-10T20:41:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.366+0000 7fbfc6ffd640 1 -- 172.21.15.28:0/1980919164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8005260 con 0x7fbfd8071300 2024-09-10T20:41:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.367+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd80c90c0 con 0x7fbfd8071300 2024-09-10T20:41:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.368+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd80d1680 con 0x7fbfd8071300 2024-09-10T20:41:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.368+0000 7fbfc6ffd640 1 -- 172.21.15.28:0/1980919164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fbfc8005400 con 0x7fbfd8071300 2024-09-10T20:41:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.369+0000 7fbfc6ffd640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfb80782d0 0x7fbfb807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.369+0000 7fbfc6ffd640 1 -- 172.21.15.28:0/1980919164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5227+0+0 (secure 0 0 0) 0x7fbfc80bb340 con 0x7fbfd8071300 2024-09-10T20:41:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.369+0000 7fbfd5d9b640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfb80782d0 0x7fbfb807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.370+0000 7fbfd5d9b640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfb80782d0 0x7fbfb807a790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0095f0 tx=0x7fbfcc005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.374+0000 7fbfc6ffd640 1 -- 172.21.15.28:0/1980919164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbfd80d1680 con 0x7fbfd8071300 2024-09-10T20:41:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:48.561+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbfd80d0920 con 0x7fbfd8071300 2024-09-10T20:41:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.223+0000 7fbfc6ffd640 1 -- 172.21.15.28:0/1980919164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) ==== 170+0+0 (secure 0 0 0) 0x7fbfc803f030 con 0x7fbfd8071300 2024-09-10T20:41:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfb80782d0 msgr2=0x7fbfb807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfb80782d0 0x7fbfb807a790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0095f0 tx=0x7fbfcc005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfd8071300 msgr2=0x7fbfd806e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfd8071300 0x7fbfd806e600 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009950 tx=0x7fbfc8033c20 comp rx=0 tx=0).stop 2024-09-10T20:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 shutdown_connections 2024-09-10T20:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fbfb80782d0 0x7fbfb807a790 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbfd815f4a0 0x7fbfd815b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbfd80b1140 0x7fbfd806f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 --2- 172.21.15.28:0/1980919164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfd8071300 0x7fbfd806e600 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.226+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 >> 172.21.15.28:0/1980919164 conn(0x7fbfd80d1da0 msgr2=0x7fbfd80b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.227+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 shutdown_connections 2024-09-10T20:41:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.227+0000 7fbfdcec6640 1 -- 172.21.15.28:0/1980919164 wait complete. 2024-09-10T20:41:49.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:41:49.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.366+0000 7f4d82d52640 1 -- 172.21.15.28:0/2160545878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d68004800 msgr2=0x7f4d68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:49.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.366+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2160545878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d68004800 0x7f4d68004c80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4d64001690 tx=0x7f4d68005830 comp rx=0 tx=0).stop 2024-09-10T20:41:49.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.366+0000 7f4d82d52640 1 -- 172.21.15.28:0/2160545878 shutdown_connections 2024-09-10T20:41:49.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.366+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2160545878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d68004800 0x7f4d68004c80 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:49.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.366+0000 7f4d82d52640 1 -- 172.21.15.28:0/2160545878 >> 172.21.15.28:0/2160545878 conn(0x7f4d7c0d4070 msgr2=0x7f4d7c0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:49.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.366+0000 7f4d82d52640 1 -- 172.21.15.28:0/2160545878 shutdown_connections 2024-09-10T20:41:49.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.366+0000 7f4d82d52640 1 -- 172.21.15.28:0/2160545878 wait complete. 2024-09-10T20:41:49.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.367+0000 7f4d82d52640 1 Processor -- start 2024-09-10T20:41:49.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.368+0000 7f4d82d52640 1 -- start start 2024-09-10T20:41:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.368+0000 7f4d82d52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d68004800 0x7f4d7c0d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.368+0000 7f4d82d52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d7c0b9bd0 0x7f4d7c0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.368+0000 7f4d82d52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c0a1090 0x7f4d7c0a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.368+0000 7f4d82d52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c0d4e00 con 0x7f4d7c0a1090 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.368+0000 7f4d82d52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c0b2fe0 con 0x7f4d7c0b9bd0 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.368+0000 7f4d82d52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7c0d20e0 con 0x7f4d68004800 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c0a1090 0x7f4d7c0a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c0a1090 0x7f4d7c0a14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36238/0 (socket says 172.21.15.28:36238) 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 -- 172.21.15.28:0/2418083529 learned_addr learned my addr 172.21.15.28:0/2418083529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d80aee640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d68004800 0x7f4d7c0d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d7bfff640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d7c0b9bd0 0x7f4d7c0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 -- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d68004800 msgr2=0x7f4d7c0d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d68004800 0x7f4d7c0d0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 -- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d7c0b9bd0 msgr2=0x7f4d7c0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d7c0b9bd0 0x7f4d7c0cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 -- 172.21.15.28:0/2418083529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c1f8630 con 0x7f4d7c0a1090 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d7bfff640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d7c0b9bd0 0x7f4d7c0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d812ef640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c0a1090 0x7f4d7c0a14d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0b32b0 tx=0x7f4d74011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d797fa640 1 -- 172.21.15.28:0/2418083529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74004230 con 0x7f4d7c0a1090 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d797fa640 1 -- 172.21.15.28:0/2418083529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d740043f0 con 0x7f4d7c0a1090 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c1f8910 con 0x7f4d7c0a1090 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d797fa640 1 -- 172.21.15.28:0/2418083529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7400fb50 con 0x7f4d7c0a1090 2024-09-10T20:41:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.369+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c1f8ed0 con 0x7f4d7c0a1090 2024-09-10T20:41:49.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.370+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d7c0b3870 con 0x7f4d7c0a1090 2024-09-10T20:41:49.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.371+0000 7f4d797fa640 1 -- 172.21.15.28:0/2418083529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f4d74018030 con 0x7f4d7c0a1090 2024-09-10T20:41:49.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.372+0000 7f4d797fa640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d5c0782d0 0x7f4d5c07a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:49.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.372+0000 7f4d797fa640 1 -- 172.21.15.28:0/2418083529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5609+0+0 (secure 0 0 0) 0x7f4d7409a9e0 con 0x7f4d7c0a1090 2024-09-10T20:41:49.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.372+0000 7f4d80aee640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d5c0782d0 0x7f4d5c07a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:49.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.373+0000 7f4d80aee640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d5c0782d0 0x7f4d5c07a790 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0d1960 tx=0x7f4d6c006b20 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:49.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.376+0000 7f4d797fa640 1 -- 172.21.15.28:0/2418083529 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4d74062f70 con 0x7f4d7c0a1090 2024-09-10T20:41:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:49.601+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4d7c06d910 con 0x7f4d7c0a1090 2024-09-10T20:41:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.259+0000 7f4d797fa640 1 -- 172.21.15.28:0/2418083529 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v40) ==== 174+0+0 (secure 0 0 0) 0x7f4d74062870 con 0x7f4d7c0a1090 2024-09-10T20:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 10 and data pool 11 2024-09-10T20:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.270+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d5c0782d0 msgr2=0x7f4d5c07a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.270+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d5c0782d0 0x7f4d5c07a790 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0d1960 tx=0x7f4d6c006b20 comp rx=0 tx=0).stop 2024-09-10T20:41:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.270+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c0a1090 msgr2=0x7f4d7c0a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.270+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c0a1090 0x7f4d7c0a14d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0b32b0 tx=0x7f4d74011ba0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.272+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 shutdown_connections 2024-09-10T20:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.273+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d5c0782d0 0x7f4d5c07a790 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.273+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c0a1090 0x7f4d7c0a14d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.273+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d7c0b9bd0 0x7f4d7c0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.273+0000 7f4d82d52640 1 --2- 172.21.15.28:0/2418083529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d68004800 0x7f4d7c0d0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.273+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 >> 172.21.15.28:0/2418083529 conn(0x7f4d7c0d4070 msgr2=0x7f4d7c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.273+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 shutdown_connections 2024-09-10T20:41:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.273+0000 7f4d82d52640 1 -- 172.21.15.28:0/2418083529 wait complete. 2024-09-10T20:41:50.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 -- 172.21.15.28:0/2426047804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80151cc0 msgr2=0x7f2f800cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/2426047804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80151cc0 0x7f2f800cf0f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2f80074830 tx=0x7f2f7c02a4d0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 -- 172.21.15.28:0/2426047804 shutdown_connections 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/2426047804 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f80073cf0 0x7f2f8015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/2426047804 >> v2:172.21.15.159:3300/0 conn(0x7f2f800738b0 0x7f2f8015e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/2426047804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80151cc0 0x7f2f800cf0f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 -- 172.21.15.28:0/2426047804 >> 172.21.15.28:0/2426047804 conn(0x7f2f800d0420 msgr2=0x7f2f800d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 -- 172.21.15.28:0/2426047804 shutdown_connections 2024-09-10T20:41:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.402+0000 7f2f87e3c640 1 -- 172.21.15.28:0/2426047804 wait complete. 2024-09-10T20:41:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.403+0000 7f2f87e3c640 1 Processor -- start 2024-09-10T20:41:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.403+0000 7f2f87e3c640 1 -- start start 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f87e3c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f800738b0 0x7f2f801f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f87e3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80073cf0 0x7f2f80147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f87e3c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f80151cc0 0x7f2f800b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f87e3c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f800cb210 con 0x7f2f80073cf0 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f87e3c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8015f640 con 0x7f2f80151cc0 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f87e3c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2f800cb390 con 0x7f2f800738b0 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80073cf0 0x7f2f80147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80073cf0 0x7f2f80147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58220/0 (socket says 172.21.15.28:58220) 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 -- 172.21.15.28:0/671594521 learned_addr learned my addr 172.21.15.28:0/671594521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f85bd8640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f800738b0 0x7f2f801f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f863d9640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f80151cc0 0x7f2f800b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 -- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f800738b0 msgr2=0x7f2f801f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f800738b0 0x7f2f801f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 -- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f80151cc0 msgr2=0x7f2f800b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f80151cc0 0x7f2f800b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 -- 172.21.15.28:0/671594521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f800b0810 con 0x7f2f80073cf0 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.404+0000 7f2f853d7640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80073cf0 0x7f2f80147d20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2f70005940 tx=0x7f2f7000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.405+0000 7f2f767fc640 1 -- 172.21.15.28:0/671594521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70007bb0 con 0x7f2f80073cf0 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.405+0000 7f2f767fc640 1 -- 172.21.15.28:0/671594521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f70007e40 con 0x7f2f80073cf0 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.405+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f801f2f70 con 0x7f2f80073cf0 2024-09-10T20:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.405+0000 7f2f767fc640 1 -- 172.21.15.28:0/671594521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7000f6b0 con 0x7f2f80073cf0 2024-09-10T20:41:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.405+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f801f3530 con 0x7f2f80073cf0 2024-09-10T20:41:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.406+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f800cf0f0 con 0x7f2f80073cf0 2024-09-10T20:41:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.406+0000 7f2f767fc640 1 -- 172.21.15.28:0/671594521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f2f7000f850 con 0x7f2f80073cf0 2024-09-10T20:41:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.407+0000 7f2f767fc640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f640782d0 0x7f2f6407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.407+0000 7f2f767fc640 1 -- 172.21.15.28:0/671594521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f2f7009a9f0 con 0x7f2f80073cf0 2024-09-10T20:41:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.407+0000 7f2f85bd8640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f640782d0 0x7f2f6407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.408+0000 7f2f85bd8640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f640782d0 0x7f2f6407a790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c033000 tx=0x7f2f7c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.411+0000 7f2f767fc640 1 -- 172.21.15.28:0/671594521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2f70014030 con 0x7f2f80073cf0 2024-09-10T20:41:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.597+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f8015b770 con 0x7f2f80073cf0 2024-09-10T20:41:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.598+0000 7f2f767fc640 1 -- 172.21.15.28:0/671594521 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16681 (secure 0 0 0) 0x7f2f70062990 con 0x7f2f80073cf0 2024-09-10T20:41:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:50.603 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:49.602899+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:47.307952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:48.566196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f640782d0 msgr2=0x7f2f6407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f640782d0 0x7f2f6407a790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c033000 tx=0x7f2f7c004730 comp rx=0 tx=0).stop 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80073cf0 msgr2=0x7f2f80147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80073cf0 0x7f2f80147d20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2f70005940 tx=0x7f2f7000cd60 comp rx=0 tx=0).stop 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 shutdown_connections 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f2f640782d0 0x7f2f6407a790 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f80151cc0 0x7f2f800b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f80073cf0 0x7f2f80147d20 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 --2- 172.21.15.28:0/671594521 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f800738b0 0x7f2f801f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 >> 172.21.15.28:0/671594521 conn(0x7f2f800d0420 msgr2=0x7f2f8015e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 shutdown_connections 2024-09-10T20:41:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.601+0000 7f2f87e3c640 1 -- 172.21.15.28:0/671594521 wait complete. 2024-09-10T20:41:50.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.717+0000 7f04986b6640 1 -- 172.21.15.28:0/2692732611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490162260 msgr2=0x7f04900cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.717+0000 7f04986b6640 1 --2- 172.21.15.28:0/2692732611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490162260 0x7f04900cfbc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0480005fd0 tx=0x7f048002aa90 comp rx=0 tx=0).stop 2024-09-10T20:41:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.717+0000 7f04986b6640 1 -- 172.21.15.28:0/2692732611 shutdown_connections 2024-09-10T20:41:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.717+0000 7f04986b6640 1 --2- 172.21.15.28:0/2692732611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490162260 0x7f04900cfbc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.717+0000 7f04986b6640 1 --2- 172.21.15.28:0/2692732611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f049015de70 0x7f04900d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.717+0000 7f04986b6640 1 -- 172.21.15.28:0/2692732611 >> 172.21.15.28:0/2692732611 conn(0x7f04900d3fb0 msgr2=0x7f04900d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.717+0000 7f04986b6640 1 -- 172.21.15.28:0/2692732611 shutdown_connections 2024-09-10T20:41:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.718+0000 7f04986b6640 1 -- 172.21.15.28:0/2692732611 wait complete. 2024-09-10T20:41:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 Processor -- start 2024-09-10T20:41:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 -- start start 2024-09-10T20:41:50.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04900cf780 0x7f04901f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f049015de70 0x7f049015c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0490162260 0x7f04900d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04900b7010 con 0x7f04900cf780 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f049015e490 con 0x7f0490162260 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.719+0000 7f04986b6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f04900b7190 con 0x7f049015de70 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04900cf780 0x7f04901f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04900cf780 0x7f04901f5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58238/0 (socket says 172.21.15.28:58238) 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 -- 172.21.15.28:0/1306781229 learned_addr learned my addr 172.21.15.28:0/1306781229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0495c51640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f049015de70 0x7f049015c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496c53640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0490162260 0x7f04900d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 -- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f049015de70 msgr2=0x7f049015c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f049015de70 0x7f049015c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 -- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0490162260 msgr2=0x7f04900d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0490162260 0x7f04900d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 -- 172.21.15.28:0/1306781229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04900b4b80 con 0x7f04900cf780 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496c53640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0490162260 0x7f04900d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f0496452640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04900cf780 0x7f04901f5340 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f048800d7a0 tx=0x7f048800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f048effd640 1 -- 172.21.15.28:0/1306781229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0488017070 con 0x7f04900cf780 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f048effd640 1 -- 172.21.15.28:0/1306781229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0488009d20 con 0x7f04900cf780 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0490161010 con 0x7f04900cf780 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f048effd640 1 -- 172.21.15.28:0/1306781229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0488005280 con 0x7f04900cf780 2024-09-10T20:41:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.720+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04901f5cd0 con 0x7f04900cf780 2024-09-10T20:41:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.721+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04900d2b90 con 0x7f04900cf780 2024-09-10T20:41:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.722+0000 7f048effd640 1 -- 172.21.15.28:0/1306781229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f0488005420 con 0x7f04900cf780 2024-09-10T20:41:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.722+0000 7f048effd640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f047c078300 0x7f047c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.722+0000 7f048effd640 1 -- 172.21.15.28:0/1306781229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f048809a670 con 0x7f04900cf780 2024-09-10T20:41:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.723+0000 7f0495c51640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f047c078300 0x7f047c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.723+0000 7f0495c51640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f047c078300 0x7f047c07a7c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04900d17e0 tx=0x7f0484009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.727+0000 7f048effd640 1 -- 172.21.15.28:0/1306781229 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0488062be0 con 0x7f04900cf780 2024-09-10T20:41:50.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.911+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04900cff10 con 0x7f04900cf780 2024-09-10T20:41:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.912+0000 7f048effd640 1 -- 172.21.15.28:0/1306781229 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16681 (secure 0 0 0) 0x7f048806b300 con 0x7f04900cf780 2024-09-10T20:41:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:50.992 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:49.602899+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:47.307952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:48.566196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f047c078300 msgr2=0x7f047c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f047c078300 0x7f047c07a7c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04900d17e0 tx=0x7f0484009040 comp rx=0 tx=0).stop 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04900cf780 msgr2=0x7f04901f5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04900cf780 0x7f04901f5340 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f048800d7a0 tx=0x7f048800dc60 comp rx=0 tx=0).stop 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 shutdown_connections 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f047c078300 0x7f047c07a7c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0490162260 0x7f04900d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f049015de70 0x7f049015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 --2- 172.21.15.28:0/1306781229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04900cf780 0x7f04901f5340 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:50.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 >> 172.21.15.28:0/1306781229 conn(0x7f04900d3fb0 msgr2=0x7f04900b9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:50.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 shutdown_connections 2024-09-10T20:41:50.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:50.914+0000 7f04986b6640 1 -- 172.21.15.28:0/1306781229 wait complete. 2024-09-10T20:41:50.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:41:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.099+0000 7fe88f3bb640 1 -- 172.21.15.28:0/1157444238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880d27f0 msgr2=0x7fe8880b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.099+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/1157444238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880d27f0 0x7fe8880b1830 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009830 tx=0x7fe87c02a480 comp rx=0 tx=0).stop 2024-09-10T20:41:51.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.099+0000 7fe88f3bb640 1 -- 172.21.15.28:0/1157444238 shutdown_connections 2024-09-10T20:41:51.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.099+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/1157444238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880d27f0 0x7fe8880b1830 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:51.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.099+0000 7fe88f3bb640 1 -- 172.21.15.28:0/1157444238 >> 172.21.15.28:0/1157444238 conn(0x7fe8880d4070 msgr2=0x7fe8880d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:51.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.099+0000 7fe88f3bb640 1 -- 172.21.15.28:0/1157444238 shutdown_connections 2024-09-10T20:41:51.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.099+0000 7fe88f3bb640 1 -- 172.21.15.28:0/1157444238 wait complete. 2024-09-10T20:41:51.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.100+0000 7fe88f3bb640 1 Processor -- start 2024-09-10T20:41:51.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.100+0000 7fe88f3bb640 1 -- start start 2024-09-10T20:41:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88f3bb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8880d27f0 0x7fe8880ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88f3bb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe888161770 0x7fe8880b9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88f3bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888161b50 0x7fe8881f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88f3bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8880b2f50 con 0x7fe888161b50 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88f3bb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe8880b2dd0 con 0x7fe888161770 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88f3bb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe8880b30d0 con 0x7fe8880d27f0 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88d958640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888161b50 0x7fe8881f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88d958640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888161b50 0x7fe8881f5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58254/0 (socket says 172.21.15.28:58254) 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88d958640 1 -- 172.21.15.28:0/91686583 learned_addr learned my addr 172.21.15.28:0/91686583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88c956640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe888161770 0x7fe8880b9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.101+0000 7fe88d157640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8880d27f0 0x7fe8880ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88d958640 1 -- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8880d27f0 msgr2=0x7fe8880ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88d958640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8880d27f0 0x7fe8880ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88d958640 1 -- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe888161770 msgr2=0x7fe8880b9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88d958640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe888161770 0x7fe8880b9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88d958640 1 -- 172.21.15.28:0/91686583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8881f58b0 con 0x7fe888161b50 2024-09-10T20:41:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88c956640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe888161770 0x7fe8880b9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88d958640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888161b50 0x7fe8881f5be0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b410 tx=0x7fe88000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe885ffb640 1 -- 172.21.15.28:0/91686583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880015030 con 0x7fe888161b50 2024-09-10T20:41:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe885ffb640 1 -- 172.21.15.28:0/91686583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe880004020 con 0x7fe888161b50 2024-09-10T20:41:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8880b6ea0 con 0x7fe888161b50 2024-09-10T20:41:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe885ffb640 1 -- 172.21.15.28:0/91686583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880004470 con 0x7fe888161b50 2024-09-10T20:41:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.102+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8880b7460 con 0x7fe888161b50 2024-09-10T20:41:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.103+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8880b2690 con 0x7fe888161b50 2024-09-10T20:41:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.104+0000 7fe885ffb640 1 -- 172.21.15.28:0/91686583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fe880022060 con 0x7fe888161b50 2024-09-10T20:41:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.104+0000 7fe885ffb640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe8740782d0 0x7fe87407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.104+0000 7fe885ffb640 1 -- 172.21.15.28:0/91686583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fe880099da0 con 0x7fe888161b50 2024-09-10T20:41:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.105+0000 7fe88d157640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe8740782d0 0x7fe87407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.105+0000 7fe88d157640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe8740782d0 0x7fe87407a790 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe8780012e0 tx=0x7fe878009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.109+0000 7fe885ffb640 1 -- 172.21.15.28:0/91686583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe880062410 con 0x7fe888161b50 2024-09-10T20:41:51.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:51.335+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe8881f6120 con 0x7fe888161b50 2024-09-10T20:41:52.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.275+0000 7fe885ffb640 1 -- 172.21.15.28:0/91686583 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) ==== 118+0+0 (secure 0 0 0) 0x7fe880061d10 con 0x7fe888161b50 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe8740782d0 msgr2=0x7fe87407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe8740782d0 0x7fe87407a790 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe8780012e0 tx=0x7fe878009040 comp rx=0 tx=0).stop 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888161b50 msgr2=0x7fe8881f5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888161b50 0x7fe8881f5be0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b410 tx=0x7fe88000b8d0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 shutdown_connections 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fe8740782d0 0x7fe87407a790 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe888161b50 0x7fe8881f5be0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe888161770 0x7fe8880b9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 --2- 172.21.15.28:0/91686583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8880d27f0 0x7fe8880ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 >> 172.21.15.28:0/91686583 conn(0x7fe8880d4070 msgr2=0x7fe88806ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 shutdown_connections 2024-09-10T20:41:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.279+0000 7fe88f3bb640 1 -- 172.21.15.28:0/91686583 wait complete. 2024-09-10T20:41:52.296 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:41:52.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:41:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 -- 172.21.15.28:0/3495667236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161770 msgr2=0x7f70340d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 --2- 172.21.15.28:0/3495667236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161770 0x7f70340d20e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7024005fd0 tx=0x7f702402a970 comp rx=0 tx=0).stop 2024-09-10T20:41:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 -- 172.21.15.28:0/3495667236 shutdown_connections 2024-09-10T20:41:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 --2- 172.21.15.28:0/3495667236 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7034161b50 0x7f70340d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 --2- 172.21.15.28:0/3495667236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161770 0x7f70340d20e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 -- 172.21.15.28:0/3495667236 >> 172.21.15.28:0/3495667236 conn(0x7f70340d4070 msgr2=0x7f70340d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 -- 172.21.15.28:0/3495667236 shutdown_connections 2024-09-10T20:41:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.407+0000 7f703bc93640 1 -- 172.21.15.28:0/3495667236 wait complete. 2024-09-10T20:41:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.409+0000 7f703bc93640 1 Processor -- start 2024-09-10T20:41:52.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.409+0000 7f703bc93640 1 -- start start 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.409+0000 7f703bc93640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70340d4b90 0x7f70341f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.409+0000 7f703bc93640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7034161770 0x7f70340cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.409+0000 7f703bc93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161b50 0x7f70341573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703bc93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70340b2f50 con 0x7f7034161b50 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703bc93640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f70340b2dd0 con 0x7f70340d4b90 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703bc93640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f70340b30d0 con 0x7f7034161770 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161b50 0x7f70341573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161b50 0x7f70341573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58268/0 (socket says 172.21.15.28:58268) 2024-09-10T20:41:52.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 -- 172.21.15.28:0/3507250129 learned_addr learned my addr 172.21.15.28:0/3507250129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f7039a2f640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70340d4b90 0x7f70341f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703922e640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7034161770 0x7f70340cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 -- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7034161770 msgr2=0x7f70340cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7034161770 0x7f70340cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 -- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70340d4b90 msgr2=0x7f70341f90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70340d4b90 0x7f70341f90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 -- 172.21.15.28:0/3507250129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70341f95e0 con 0x7f7034161b50 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703a230640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161b50 0x7f70341573b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c410 tx=0x7f702c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f70327fc640 1 -- 172.21.15.28:0/3507250129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c011790 con 0x7f7034161b50 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f70327fc640 1 -- 172.21.15.28:0/3507250129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f702c017070 con 0x7f7034161b50 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70340c9d50 con 0x7f7034161b50 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.410+0000 7f70327fc640 1 -- 172.21.15.28:0/3507250129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c011d00 con 0x7f7034161b50 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.411+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70340b7220 con 0x7f7034161b50 2024-09-10T20:41:52.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.412+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70340b3870 con 0x7f7034161b50 2024-09-10T20:41:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.412+0000 7f70327fc640 1 -- 172.21.15.28:0/3507250129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f702c01c460 con 0x7f7034161b50 2024-09-10T20:41:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.413+0000 7f70327fc640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f70200782d0 0x7f702007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.413+0000 7f70327fc640 1 -- 172.21.15.28:0/3507250129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f702c099f20 con 0x7f7034161b50 2024-09-10T20:41:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.413+0000 7f7039a2f640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f70200782d0 0x7f702007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.414+0000 7f7039a2f640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f70200782d0 0x7f702007a790 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7024035000 tx=0x7f702403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.417+0000 7f70327fc640 1 -- 172.21.15.28:0/3507250129 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f702c021020 con 0x7f7034161b50 2024-09-10T20:41:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:52.640+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f7034160540 con 0x7f7034161b50 2024-09-10T20:41:53.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.276+0000 7f70327fc640 1 -- 172.21.15.28:0/3507250129 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f70340c9d50 con 0x7f7034161b50 2024-09-10T20:41:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f70200782d0 msgr2=0x7f702007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f70200782d0 0x7f702007a790 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7024035000 tx=0x7f702403c040 comp rx=0 tx=0).stop 2024-09-10T20:41:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161b50 msgr2=0x7f70341573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161b50 0x7f70341573b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c410 tx=0x7f702c00c8d0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 shutdown_connections 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f70200782d0 0x7f702007a790 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7034161b50 0x7f70341573b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7034161770 0x7f70340cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 --2- 172.21.15.28:0/3507250129 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70340d4b90 0x7f70341f90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 >> 172.21.15.28:0/3507250129 conn(0x7f70340d4070 msgr2=0x7f70340d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 shutdown_connections 2024-09-10T20:41:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.279+0000 7f703bc93640 1 -- 172.21.15.28:0/3507250129 wait complete. 2024-09-10T20:41:53.296 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:41:53.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.405+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3406452748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec0cf0f0 msgr2=0x7fa4ec15e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.405+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3406452748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec0cf0f0 0x7fa4ec15e7e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005fd0 tx=0x7fa4e002a530 comp rx=0 tx=0).stop 2024-09-10T20:41:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.406+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3406452748 shutdown_connections 2024-09-10T20:41:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.406+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3406452748 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4ec15ed20 0x7fa4ec0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.406+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3406452748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec0cf0f0 0x7fa4ec15e7e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.406+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3406452748 >> 172.21.15.28:0/3406452748 conn(0x7fa4ec0d0420 msgr2=0x7fa4ec0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.406+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3406452748 shutdown_connections 2024-09-10T20:41:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.406+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3406452748 wait complete. 2024-09-10T20:41:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.407+0000 7fa4f3c7f640 1 Processor -- start 2024-09-10T20:41:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.407+0000 7fa4f3c7f640 1 -- start start 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f3c7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4ec0cf0f0 0x7fa4ec1f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f3c7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec151cc0 0x7fa4ec0b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f3c7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4ec15ed20 0x7fa4ec1e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f3c7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec15f5b0 con 0x7fa4ec151cc0 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f3c7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec15f430 con 0x7fa4ec15ed20 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f3c7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ec0cb750 con 0x7fa4ec0cf0f0 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec151cc0 0x7fa4ec0b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec151cc0 0x7fa4ec0b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58290/0 (socket says 172.21.15.28:58290) 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 -- 172.21.15.28:0/3997578997 learned_addr learned my addr 172.21.15.28:0/3997578997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f221c640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4ec15ed20 0x7fa4ec1e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f1a1b640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4ec0cf0f0 0x7fa4ec1f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 -- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4ec0cf0f0 msgr2=0x7fa4ec1f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4ec0cf0f0 0x7fa4ec1f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 -- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4ec15ed20 msgr2=0x7fa4ec1e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4ec15ed20 0x7fa4ec1e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 -- 172.21.15.28:0/3997578997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ec1f8e50 con 0x7fa4ec151cc0 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f1a1b640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4ec0cf0f0 0x7fa4ec1f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4f121a640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec151cc0 0x7fa4ec0b1950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009850 tx=0x7fa4e00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4ea7fc640 1 -- 172.21.15.28:0/3997578997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e003f070 con 0x7fa4ec151cc0 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.408+0000 7fa4ea7fc640 1 -- 172.21.15.28:0/3997578997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4e0004890 con 0x7fa4ec151cc0 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.409+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ec0b3a50 con 0x7fa4ec151cc0 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.409+0000 7fa4ea7fc640 1 -- 172.21.15.28:0/3997578997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0035030 con 0x7fa4ec151cc0 2024-09-10T20:41:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.409+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ec0b4010 con 0x7fa4ec151cc0 2024-09-10T20:41:53.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.410+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ec0cfb90 con 0x7fa4ec151cc0 2024-09-10T20:41:53.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.410+0000 7fa4ea7fc640 1 -- 172.21.15.28:0/3997578997 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fa4e0004c60 con 0x7fa4ec151cc0 2024-09-10T20:41:53.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.411+0000 7fa4ea7fc640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa4d8078300 0x7fa4d807a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.411+0000 7fa4ea7fc640 1 -- 172.21.15.28:0/3997578997 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fa4e00bb500 con 0x7fa4ec151cc0 2024-09-10T20:41:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.411+0000 7fa4f1a1b640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa4d8078300 0x7fa4d807a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.412+0000 7fa4f1a1b640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa4d8078300 0x7fa4d807a7c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0045b0 tx=0x7fa4dc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.415+0000 7fa4ea7fc640 1 -- 172.21.15.28:0/3997578997 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa4ec0cfb90 con 0x7fa4ec151cc0 2024-09-10T20:41:53.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.639+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4ec068d10 con 0x7fa4ec151cc0 2024-09-10T20:41:53.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.639+0000 7fa4ea7fc640 1 -- 172.21.15.28:0/3997578997 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5228 (secure 0 0 0) 0x7fa4e008c030 con 0x7fa4ec151cc0 2024-09-10T20:41:53.643 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:53.643 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":44,"btime":"2024-09-10T20:41:53:275679+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:41:53.644 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 44 2024-09-10T20:41:53.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.641+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa4d8078300 msgr2=0x7fa4d807a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.641+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa4d8078300 0x7fa4d807a7c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0045b0 tx=0x7fa4dc009040 comp rx=0 tx=0).stop 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.641+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec151cc0 msgr2=0x7fa4ec0b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.641+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec151cc0 0x7fa4ec0b1950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009850 tx=0x7fa4e00039e0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 shutdown_connections 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fa4d8078300 0x7fa4d807a7c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4ec15ed20 0x7fa4ec1e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ec151cc0 0x7fa4ec0b1950 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 --2- 172.21.15.28:0/3997578997 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4ec0cf0f0 0x7fa4ec1f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 >> 172.21.15.28:0/3997578997 conn(0x7fa4ec0d0420 msgr2=0x7fa4ec0cbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 shutdown_connections 2024-09-10T20:41:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.642+0000 7fa4f3c7f640 1 -- 172.21.15.28:0/3997578997 wait complete. 2024-09-10T20:41:53.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:41:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 -- 172.21.15.28:0/3880889690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f624006d1f0 msgr2=0x7f624006ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 --2- 172.21.15.28:0/3880889690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f624006d1f0 0x7f624006ef30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6238005fd0 tx=0x7f623802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 -- 172.21.15.28:0/3880889690 shutdown_connections 2024-09-10T20:41:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 --2- 172.21.15.28:0/3880889690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f624006d1f0 0x7f624006ef30 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 --2- 172.21.15.28:0/3880889690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6240156130 0x7f62400caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 -- 172.21.15.28:0/3880889690 >> 172.21.15.28:0/3880889690 conn(0x7f62400d0420 msgr2=0x7f62400d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 -- 172.21.15.28:0/3880889690 shutdown_connections 2024-09-10T20:41:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.758+0000 7f6247cef640 1 -- 172.21.15.28:0/3880889690 wait complete. 2024-09-10T20:41:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.759+0000 7f6247cef640 1 Processor -- start 2024-09-10T20:41:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.759+0000 7f6247cef640 1 -- start start 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f6247cef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f624006d1f0 0x7f62401512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f6247cef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6240156130 0x7f62400c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f6247cef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6240156510 0x7f62400b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f6247cef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62400733b0 con 0x7f6240156130 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f6247cef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6240073230 con 0x7f624006d1f0 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f6247cef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6240073530 con 0x7f6240156510 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6240156130 0x7f62400c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6240156130 0x7f62400c9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58322/0 (socket says 172.21.15.28:58322) 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 -- 172.21.15.28:0/176524633 learned_addr learned my addr 172.21.15.28:0/176524633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f6245a8b640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f624006d1f0 0x7f62401512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624628c640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6240156510 0x7f62400b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 -- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6240156510 msgr2=0x7f62400b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6240156510 0x7f62400b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 -- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f624006d1f0 msgr2=0x7f62401512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f624006d1f0 0x7f62401512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.760+0000 7f624528a640 1 -- 172.21.15.28:0/176524633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6240151800 con 0x7f6240156130 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.761+0000 7f624528a640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6240156130 0x7f62400c9f40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f623000c3f0 tx=0x7f623000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.761+0000 7f62367fc640 1 -- 172.21.15.28:0/176524633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230017070 con 0x7f6240156130 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.761+0000 7f62367fc640 1 -- 172.21.15.28:0/176524633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62300035f0 con 0x7f6240156130 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.761+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62401f6810 con 0x7f6240156130 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.761+0000 7f62367fc640 1 -- 172.21.15.28:0/176524633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230003a40 con 0x7f6240156130 2024-09-10T20:41:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.761+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62400c9a20 con 0x7f6240156130 2024-09-10T20:41:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.762+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f624006ef30 con 0x7f6240156130 2024-09-10T20:41:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.763+0000 7f62367fc640 1 -- 172.21.15.28:0/176524633 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f6230003be0 con 0x7f6240156130 2024-09-10T20:41:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.763+0000 7f62367fc640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f62240782d0 0x7f622407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.764+0000 7f62367fc640 1 -- 172.21.15.28:0/176524633 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f62300998e0 con 0x7f6240156130 2024-09-10T20:41:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.764+0000 7f6245a8b640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f62240782d0 0x7f622407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.764+0000 7f6245a8b640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f62240782d0 0x7f622407a790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f623c009bd0 tx=0x7f623c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.768+0000 7f62367fc640 1 -- 172.21.15.28:0/176524633 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6230061870 con 0x7f6240156130 2024-09-10T20:41:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.954+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62400b1b90 con 0x7f6240156130 2024-09-10T20:41:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.955+0000 7f62367fc640 1 -- 172.21.15.28:0/176524633 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16668 (secure 0 0 0) 0x7f62300662a0 con 0x7f6240156130 2024-09-10T20:41:53.959 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":68,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:41:51.252136+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:47.307952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:48.566196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:41:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.957+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f62240782d0 msgr2=0x7f622407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.957+0000 7f6247cef640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f62240782d0 0x7f622407a790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f623c009bd0 tx=0x7f623c005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.957+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6240156130 msgr2=0x7f62400c9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.957+0000 7f6247cef640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6240156130 0x7f62400c9f40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f623000c3f0 tx=0x7f623000c8b0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 shutdown_connections 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f62240782d0 0x7f622407a790 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6240156510 0x7f62400b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6240156130 0x7f62400c9f40 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 --2- 172.21.15.28:0/176524633 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f624006d1f0 0x7f62401512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 >> 172.21.15.28:0/176524633 conn(0x7f62400d0420 msgr2=0x7f62400d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 shutdown_connections 2024-09-10T20:41:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:53.958+0000 7f6247cef640 1 -- 172.21.15.28:0/176524633 wait complete. 2024-09-10T20:41:53.972 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:41:53.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 -- 172.21.15.28:0/2408460323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 msgr2=0x7f4d140afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 --2- 172.21.15.28:0/2408460323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d140afba0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4d08005fd0 tx=0x7f4d0802a580 comp rx=0 tx=0).stop 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 -- 172.21.15.28:0/2408460323 shutdown_connections 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 --2- 172.21.15.28:0/2408460323 >> v2:172.21.15.159:3300/0 conn(0x7f4d1406cae0 0x7f4d1406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 --2- 172.21.15.28:0/2408460323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d140afba0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 --2- 172.21.15.28:0/2408460323 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d1415bb40 0x7f4d140af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 -- 172.21.15.28:0/2408460323 >> 172.21.15.28:0/2408460323 conn(0x7f4d1406dbf0 msgr2=0x7f4d1406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 -- 172.21.15.28:0/2408460323 shutdown_connections 2024-09-10T20:41:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.074+0000 7f4d1a156640 1 -- 172.21.15.28:0/2408460323 wait complete. 2024-09-10T20:41:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.075+0000 7f4d1a156640 1 Processor -- start 2024-09-10T20:41:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.075+0000 7f4d1a156640 1 -- start start 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d1a156640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d1406cae0 0x7f4d141f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d1a156640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d1415bb40 0x7f4d1409d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d1a156640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d14156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d1a156640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d140b4320 con 0x7f4d1415bf20 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d1a156640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4d140b41a0 con 0x7f4d1415bb40 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d1a156640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4d140b44a0 con 0x7f4d1406cae0 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d14156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d14156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58340/0 (socket says 172.21.15.28:58340) 2024-09-10T20:41:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 -- 172.21.15.28:0/1714369645 learned_addr learned my addr 172.21.15.28:0/1714369645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d137fe640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d1406cae0 0x7f4d141f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d12ffd640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d1415bb40 0x7f4d1409d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 -- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d1406cae0 msgr2=0x7f4d141f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d1406cae0 0x7f4d141f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 -- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d1415bb40 msgr2=0x7f4d1409d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d1415bb40 0x7f4d1409d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d13fff640 1 -- 172.21.15.28:0/1714369645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1409dc80 con 0x7f4d1415bf20 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d12ffd640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d1415bb40 0x7f4d1409d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.076+0000 7f4d137fe640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d1406cae0 0x7f4d141f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.077+0000 7f4d13fff640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d14156c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00cce0 tx=0x7f4d0c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.077+0000 7f4cfbfff640 1 -- 172.21.15.28:0/1714369645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c00f450 con 0x7f4d1415bf20 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.077+0000 7f4cfbfff640 1 -- 172.21.15.28:0/1714369645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d0c00f700 con 0x7f4d1415bf20 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.077+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1409df60 con 0x7f4d1415bf20 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.077+0000 7f4cfbfff640 1 -- 172.21.15.28:0/1714369645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c00fb50 con 0x7f4d1415bf20 2024-09-10T20:41:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.077+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d1409e520 con 0x7f4d1415bf20 2024-09-10T20:41:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.078+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d140b4950 con 0x7f4d1415bf20 2024-09-10T20:41:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.079+0000 7f4cfbfff640 1 -- 172.21.15.28:0/1714369645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f4d0c0213f0 con 0x7f4d1415bf20 2024-09-10T20:41:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.079+0000 7f4cfbfff640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d000782d0 0x7f4d0007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.079+0000 7f4cfbfff640 1 -- 172.21.15.28:0/1714369645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f4d0c09b0d0 con 0x7f4d1415bf20 2024-09-10T20:41:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.079+0000 7f4d137fe640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d000782d0 0x7f4d0007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.080+0000 7f4d137fe640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d000782d0 0x7f4d0007a790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d04005d70 tx=0x7f4d04005d00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.083+0000 7f4cfbfff640 1 -- 172.21.15.28:0/1714369645 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4d0c014030 con 0x7f4d1415bf20 2024-09-10T20:41:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.266+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d140d1570 con 0x7f4d000782d0 2024-09-10T20:41:54.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.271+0000 7f4cfbfff640 1 -- 172.21.15.28:0/1714369645 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249747 (secure 0 0 0) 0x7f4d140d1570 con 0x7f4d000782d0 2024-09-10T20:41:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:54.279 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d000782d0 msgr2=0x7f4d0007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d000782d0 0x7f4d0007a790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d04005d70 tx=0x7f4d04005d00 comp rx=0 tx=0).stop 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 msgr2=0x7f4d14156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d14156c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00cce0 tx=0x7f4d0c00a620 comp rx=0 tx=0).stop 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 shutdown_connections 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4d000782d0 0x7f4d0007a790 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d1415bf20 0x7f4d14156c80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d1415bb40 0x7f4d1409d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 --2- 172.21.15.28:0/1714369645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d1406cae0 0x7f4d141f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 >> 172.21.15.28:0/1714369645 conn(0x7f4d1406dbf0 msgr2=0x7f4d14073dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:54.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 shutdown_connections 2024-09-10T20:41:54.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.274+0000 7f4d1a156640 1 -- 172.21.15.28:0/1714369645 wait complete. 2024-09-10T20:41:54.289 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-10T20:41:54.029282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591042,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":132,"num_write_kb":1631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244464,"kb_used_data":7408,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730256,"statfs":{"total":773094113280,"available":772843782144,"internally_reserved":0,"allocated":7585792,"data_stored":3836678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.805315"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264924+0000","last_change":"2024-09-10T20:41:50.262634+0000","last_active":"2024-09-10T20:41:50.264924+0000","last_peered":"2024-09-10T20:41:50.264924+0000","last_clean":"2024-09-10T20:41:50.264924+0000","last_became_active":"2024-09-10T20:41:50.262341+0000","last_became_peered":"2024-09-10T20:41:50.262341+0000","last_unstale":"2024-09-10T20:41:50.264924+0000","last_undegraded":"2024-09-10T20:41:50.264924+0000","last_fullsized":"2024-09-10T20:41:50.264924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:09:16.358398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270227+0000","last_change":"2024-09-10T20:41:50.268290+0000","last_active":"2024-09-10T20:41:50.270227+0000","last_peered":"2024-09-10T20:41:50.270227+0000","last_clean":"2024-09-10T20:41:50.270227+0000","last_became_active":"2024-09-10T20:41:50.268020+0000","last_became_peered":"2024-09-10T20:41:50.268020+0000","last_unstale":"2024-09-10T20:41:50.270227+0000","last_undegraded":"2024-09-10T20:41:50.270227+0000","last_fullsized":"2024-09-10T20:41:50.270227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:44.595286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232984+0000","last_change":"2024-09-10T20:41:49.232984+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232984+0000","last_undegraded":"2024-09-10T20:41:49.232984+0000","last_fullsized":"2024-09-10T20:41:49.232984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233187+0000","last_change":"2024-09-10T20:41:49.233187+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233187+0000","last_undegraded":"2024-09-10T20:41:49.233187+0000","last_fullsized":"2024-09-10T20:41:49.233187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230680+0000","last_change":"2024-09-10T20:41:49.230680+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230680+0000","last_undegraded":"2024-09-10T20:41:49.230680+0000","last_fullsized":"2024-09-10T20:41:49.230680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267661+0000","last_change":"2024-09-10T20:41:50.262868+0000","last_active":"2024-09-10T20:41:50.267661+0000","last_peered":"2024-09-10T20:41:50.267661+0000","last_clean":"2024-09-10T20:41:50.267661+0000","last_became_active":"2024-09-10T20:41:50.262182+0000","last_became_peered":"2024-09-10T20:41:50.262182+0000","last_unstale":"2024-09-10T20:41:50.267661+0000","last_undegraded":"2024-09-10T20:41:50.267661+0000","last_fullsized":"2024-09-10T20:41:50.267661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:10.294469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278278+0000","last_change":"2024-09-10T20:41:50.269379+0000","last_active":"2024-09-10T20:41:50.278278+0000","last_peered":"2024-09-10T20:41:50.278278+0000","last_clean":"2024-09-10T20:41:50.278278+0000","last_became_active":"2024-09-10T20:41:50.268955+0000","last_became_peered":"2024-09-10T20:41:50.268955+0000","last_unstale":"2024-09-10T20:41:50.278278+0000","last_undegraded":"2024-09-10T20:41:50.278278+0000","last_fullsized":"2024-09-10T20:41:50.278278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:55.335452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264003+0000","last_change":"2024-09-10T20:41:50.261207+0000","last_active":"2024-09-10T20:41:50.264003+0000","last_peered":"2024-09-10T20:41:50.264003+0000","last_clean":"2024-09-10T20:41:50.264003+0000","last_became_active":"2024-09-10T20:41:50.260866+0000","last_became_peered":"2024-09-10T20:41:50.260866+0000","last_unstale":"2024-09-10T20:41:50.264003+0000","last_undegraded":"2024-09-10T20:41:50.264003+0000","last_fullsized":"2024-09-10T20:41:50.264003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:48.716401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232777+0000","last_change":"2024-09-10T20:41:49.232777+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232777+0000","last_undegraded":"2024-09-10T20:41:49.232777+0000","last_fullsized":"2024-09-10T20:41:49.232777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278111+0000","last_change":"2024-09-10T20:41:50.270275+0000","last_active":"2024-09-10T20:41:50.278111+0000","last_peered":"2024-09-10T20:41:50.278111+0000","last_clean":"2024-09-10T20:41:50.278111+0000","last_became_active":"2024-09-10T20:41:50.269980+0000","last_became_peered":"2024-09-10T20:41:50.269980+0000","last_unstale":"2024-09-10T20:41:50.278111+0000","last_undegraded":"2024-09-10T20:41:50.278111+0000","last_fullsized":"2024-09-10T20:41:50.278111+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:12.357126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263972+0000","last_change":"2024-09-10T20:41:50.260076+0000","last_active":"2024-09-10T20:41:50.263972+0000","last_peered":"2024-09-10T20:41:50.263972+0000","last_clean":"2024-09-10T20:41:50.263972+0000","last_became_active":"2024-09-10T20:41:50.259752+0000","last_became_peered":"2024-09-10T20:41:50.259752+0000","last_unstale":"2024-09-10T20:41:50.263972+0000","last_undegraded":"2024-09-10T20:41:50.263972+0000","last_fullsized":"2024-09-10T20:41:50.263972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:08.756456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228409+0000","last_change":"2024-09-10T20:41:49.228409+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228409+0000","last_undegraded":"2024-09-10T20:41:49.228409+0000","last_fullsized":"2024-09-10T20:41:49.228409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232209+0000","last_change":"2024-09-10T20:41:49.232209+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232209+0000","last_undegraded":"2024-09-10T20:41:49.232209+0000","last_fullsized":"2024-09-10T20:41:49.232209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229613+0000","last_change":"2024-09-10T20:41:49.229613+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229613+0000","last_undegraded":"2024-09-10T20:41:49.229613+0000","last_fullsized":"2024-09-10T20:41:49.229613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230334+0000","last_change":"2024-09-10T20:41:49.230334+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230334+0000","last_undegraded":"2024-09-10T20:41:49.230334+0000","last_fullsized":"2024-09-10T20:41:49.230334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.262988+0000","last_change":"2024-09-10T20:41:50.259865+0000","last_active":"2024-09-10T20:41:50.262988+0000","last_peered":"2024-09-10T20:41:50.262988+0000","last_clean":"2024-09-10T20:41:50.262988+0000","last_became_active":"2024-09-10T20:41:50.259492+0000","last_became_peered":"2024-09-10T20:41:50.259492+0000","last_unstale":"2024-09-10T20:41:50.262988+0000","last_undegraded":"2024-09-10T20:41:50.262988+0000","last_fullsized":"2024-09-10T20:41:50.262988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:32:07.918850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228536+0000","last_change":"2024-09-10T20:41:49.228536+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228536+0000","last_undegraded":"2024-09-10T20:41:49.228536+0000","last_fullsized":"2024-09-10T20:41:49.228536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230656+0000","last_change":"2024-09-10T20:41:49.230656+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230656+0000","last_undegraded":"2024-09-10T20:41:49.230656+0000","last_fullsized":"2024-09-10T20:41:49.230656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269422+0000","last_change":"2024-09-10T20:41:50.264629+0000","last_active":"2024-09-10T20:41:50.269422+0000","last_peered":"2024-09-10T20:41:50.269422+0000","last_clean":"2024-09-10T20:41:50.269422+0000","last_became_active":"2024-09-10T20:41:50.264345+0000","last_became_peered":"2024-09-10T20:41:50.264345+0000","last_unstale":"2024-09-10T20:41:50.269422+0000","last_undegraded":"2024-09-10T20:41:50.269422+0000","last_fullsized":"2024-09-10T20:41:50.269422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:33:47.084724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228954+0000","last_change":"2024-09-10T20:41:49.228954+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228954+0000","last_undegraded":"2024-09-10T20:41:49.228954+0000","last_fullsized":"2024-09-10T20:41:49.228954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257652+0000","last_change":"2024-09-10T20:41:50.255889+0000","last_active":"2024-09-10T20:41:50.257652+0000","last_peered":"2024-09-10T20:41:50.257652+0000","last_clean":"2024-09-10T20:41:50.257652+0000","last_became_active":"2024-09-10T20:41:50.255572+0000","last_became_peered":"2024-09-10T20:41:50.255572+0000","last_unstale":"2024-09-10T20:41:50.257652+0000","last_undegraded":"2024-09-10T20:41:50.257652+0000","last_fullsized":"2024-09-10T20:41:50.257652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:38.194068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274208+0000","last_change":"2024-09-10T20:41:50.265660+0000","last_active":"2024-09-10T20:41:50.274208+0000","last_peered":"2024-09-10T20:41:50.274208+0000","last_clean":"2024-09-10T20:41:50.274208+0000","last_became_active":"2024-09-10T20:41:50.265364+0000","last_became_peered":"2024-09-10T20:41:50.265364+0000","last_unstale":"2024-09-10T20:41:50.274208+0000","last_undegraded":"2024-09-10T20:41:50.274208+0000","last_fullsized":"2024-09-10T20:41:50.274208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:52.067493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269174+0000","last_change":"2024-09-10T20:41:50.263991+0000","last_active":"2024-09-10T20:41:50.269174+0000","last_peered":"2024-09-10T20:41:50.269174+0000","last_clean":"2024-09-10T20:41:50.269174+0000","last_became_active":"2024-09-10T20:41:50.263690+0000","last_became_peered":"2024-09-10T20:41:50.263690+0000","last_unstale":"2024-09-10T20:41:50.269174+0000","last_undegraded":"2024-09-10T20:41:50.269174+0000","last_fullsized":"2024-09-10T20:41:50.269174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:08.679923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264022+0000","last_change":"2024-09-10T20:41:50.261825+0000","last_active":"2024-09-10T20:41:50.264022+0000","last_peered":"2024-09-10T20:41:50.264022+0000","last_clean":"2024-09-10T20:41:50.264022+0000","last_became_active":"2024-09-10T20:41:50.261516+0000","last_became_peered":"2024-09-10T20:41:50.261516+0000","last_unstale":"2024-09-10T20:41:50.264022+0000","last_undegraded":"2024-09-10T20:41:50.264022+0000","last_fullsized":"2024-09-10T20:41:50.264022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:26.740959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278057+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.278057+0000","last_peered":"2024-09-10T20:41:50.278057+0000","last_clean":"2024-09-10T20:41:50.278057+0000","last_became_active":"2024-09-10T20:41:50.272144+0000","last_became_peered":"2024-09-10T20:41:50.272144+0000","last_unstale":"2024-09-10T20:41:50.278057+0000","last_undegraded":"2024-09-10T20:41:50.278057+0000","last_fullsized":"2024-09-10T20:41:50.278057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:05.044942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.275708+0000","last_change":"2024-09-10T20:41:50.273908+0000","last_active":"2024-09-10T20:41:50.275708+0000","last_peered":"2024-09-10T20:41:50.275708+0000","last_clean":"2024-09-10T20:41:50.275708+0000","last_became_active":"2024-09-10T20:41:50.273619+0000","last_became_peered":"2024-09-10T20:41:50.273619+0000","last_unstale":"2024-09-10T20:41:50.275708+0000","last_undegraded":"2024-09-10T20:41:50.275708+0000","last_fullsized":"2024-09-10T20:41:50.275708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:57.474072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229786+0000","last_change":"2024-09-10T20:41:49.229786+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229786+0000","last_undegraded":"2024-09-10T20:41:49.229786+0000","last_fullsized":"2024-09-10T20:41:49.229786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269719+0000","last_change":"2024-09-10T20:41:50.267808+0000","last_active":"2024-09-10T20:41:50.269719+0000","last_peered":"2024-09-10T20:41:50.269719+0000","last_clean":"2024-09-10T20:41:50.269719+0000","last_became_active":"2024-09-10T20:41:50.267481+0000","last_became_peered":"2024-09-10T20:41:50.267481+0000","last_unstale":"2024-09-10T20:41:50.269719+0000","last_undegraded":"2024-09-10T20:41:50.269719+0000","last_fullsized":"2024-09-10T20:41:50.269719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:18:49.925720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231774+0000","last_change":"2024-09-10T20:41:49.231774+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231774+0000","last_undegraded":"2024-09-10T20:41:49.231774+0000","last_fullsized":"2024-09-10T20:41:49.231774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.265287+0000","last_change":"2024-09-10T20:41:50.263690+0000","last_active":"2024-09-10T20:41:50.265287+0000","last_peered":"2024-09-10T20:41:50.265287+0000","last_clean":"2024-09-10T20:41:50.265287+0000","last_became_active":"2024-09-10T20:41:50.263407+0000","last_became_peered":"2024-09-10T20:41:50.263407+0000","last_unstale":"2024-09-10T20:41:50.265287+0000","last_undegraded":"2024-09-10T20:41:50.265287+0000","last_fullsized":"2024-09-10T20:41:50.265287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:16.841293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269657+0000","last_change":"2024-09-10T20:41:50.266413+0000","last_active":"2024-09-10T20:41:50.269657+0000","last_peered":"2024-09-10T20:41:50.269657+0000","last_clean":"2024-09-10T20:41:50.269657+0000","last_became_active":"2024-09-10T20:41:50.266150+0000","last_became_peered":"2024-09-10T20:41:50.266150+0000","last_unstale":"2024-09-10T20:41:50.269657+0000","last_undegraded":"2024-09-10T20:41:50.269657+0000","last_fullsized":"2024-09-10T20:41:50.269657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:20.996250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270272+0000","last_change":"2024-09-10T20:41:50.267692+0000","last_active":"2024-09-10T20:41:50.270272+0000","last_peered":"2024-09-10T20:41:50.270272+0000","last_clean":"2024-09-10T20:41:50.270272+0000","last_became_active":"2024-09-10T20:41:50.267181+0000","last_became_peered":"2024-09-10T20:41:50.267181+0000","last_unstale":"2024-09-10T20:41:50.270272+0000","last_undegraded":"2024-09-10T20:41:50.270272+0000","last_fullsized":"2024-09-10T20:41:50.270272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:14.639786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263938+0000","last_change":"2024-09-10T20:41:50.260450+0000","last_active":"2024-09-10T20:41:50.263938+0000","last_peered":"2024-09-10T20:41:50.263938+0000","last_clean":"2024-09-10T20:41:50.263938+0000","last_became_active":"2024-09-10T20:41:50.260093+0000","last_became_peered":"2024-09-10T20:41:50.260093+0000","last_unstale":"2024-09-10T20:41:50.263938+0000","last_undegraded":"2024-09-10T20:41:50.263938+0000","last_fullsized":"2024-09-10T20:41:50.263938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:16.794344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281382+0000","last_change":"2024-09-10T20:41:49.244556+0000","last_active":"2024-09-10T20:41:50.281382+0000","last_peered":"2024-09-10T20:41:50.281382+0000","last_clean":"2024-09-10T20:41:50.281382+0000","last_became_active":"2024-09-10T20:41:49.244277+0000","last_became_peered":"2024-09-10T20:41:49.244277+0000","last_unstale":"2024-09-10T20:41:50.281382+0000","last_undegraded":"2024-09-10T20:41:50.281382+0000","last_fullsized":"2024-09-10T20:41:50.281382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:24.745124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232330+0000","last_change":"2024-09-10T20:41:49.232330+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232330+0000","last_undegraded":"2024-09-10T20:41:49.232330+0000","last_fullsized":"2024-09-10T20:41:49.232330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242361+0000","last_change":"2024-09-10T20:41:49.236963+0000","last_active":"2024-09-10T20:41:49.242361+0000","last_peered":"2024-09-10T20:41:49.242361+0000","last_clean":"2024-09-10T20:41:49.242361+0000","last_became_active":"2024-09-10T20:41:49.236654+0000","last_became_peered":"2024-09-10T20:41:49.236654+0000","last_unstale":"2024-09-10T20:41:49.242361+0000","last_undegraded":"2024-09-10T20:41:49.242361+0000","last_fullsized":"2024-09-10T20:41:49.242361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:48.203284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278810+0000","last_change":"2024-09-10T20:41:50.277395+0000","last_active":"2024-09-10T20:41:50.278810+0000","last_peered":"2024-09-10T20:41:50.278810+0000","last_clean":"2024-09-10T20:41:50.278810+0000","last_became_active":"2024-09-10T20:41:50.277101+0000","last_became_peered":"2024-09-10T20:41:50.277101+0000","last_unstale":"2024-09-10T20:41:50.278810+0000","last_undegraded":"2024-09-10T20:41:50.278810+0000","last_fullsized":"2024-09-10T20:41:50.278810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:47.799998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282196+0000","last_change":"2024-09-10T20:41:49.238213+0000","last_active":"2024-09-10T20:41:50.282196+0000","last_peered":"2024-09-10T20:41:50.282196+0000","last_clean":"2024-09-10T20:41:50.282196+0000","last_became_active":"2024-09-10T20:41:49.237949+0000","last_became_peered":"2024-09-10T20:41:49.237949+0000","last_unstale":"2024-09-10T20:41:50.282196+0000","last_undegraded":"2024-09-10T20:41:50.282196+0000","last_fullsized":"2024-09-10T20:41:50.282196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:45:22.532346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.266258+0000","last_change":"2024-09-10T20:41:50.262357+0000","last_active":"2024-09-10T20:41:50.266258+0000","last_peered":"2024-09-10T20:41:50.266258+0000","last_clean":"2024-09-10T20:41:50.266258+0000","last_became_active":"2024-09-10T20:41:50.262081+0000","last_became_peered":"2024-09-10T20:41:50.262081+0000","last_unstale":"2024-09-10T20:41:50.266258+0000","last_undegraded":"2024-09-10T20:41:50.266258+0000","last_fullsized":"2024-09-10T20:41:50.266258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:44.966942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.279900+0000","last_change":"2024-09-10T20:41:49.247040+0000","last_active":"2024-09-10T20:41:50.279900+0000","last_peered":"2024-09-10T20:41:50.279900+0000","last_clean":"2024-09-10T20:41:50.279900+0000","last_became_active":"2024-09-10T20:41:49.246731+0000","last_became_peered":"2024-09-10T20:41:49.246731+0000","last_unstale":"2024-09-10T20:41:50.279900+0000","last_undegraded":"2024-09-10T20:41:50.279900+0000","last_fullsized":"2024-09-10T20:41:50.279900+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:37.379165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263903+0000","last_change":"2024-09-10T20:41:50.261513+0000","last_active":"2024-09-10T20:41:50.263903+0000","last_peered":"2024-09-10T20:41:50.263903+0000","last_clean":"2024-09-10T20:41:50.263903+0000","last_became_active":"2024-09-10T20:41:50.261203+0000","last_became_peered":"2024-09-10T20:41:50.261203+0000","last_unstale":"2024-09-10T20:41:50.263903+0000","last_undegraded":"2024-09-10T20:41:50.263903+0000","last_fullsized":"2024-09-10T20:41:50.263903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:39.097791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242512+0000","last_change":"2024-09-10T20:41:49.237350+0000","last_active":"2024-09-10T20:41:49.242512+0000","last_peered":"2024-09-10T20:41:49.242512+0000","last_clean":"2024-09-10T20:41:49.242512+0000","last_became_active":"2024-09-10T20:41:49.237086+0000","last_became_peered":"2024-09-10T20:41:49.237086+0000","last_unstale":"2024-09-10T20:41:49.242512+0000","last_undegraded":"2024-09-10T20:41:49.242512+0000","last_fullsized":"2024-09-10T20:41:49.242512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:13.283338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276471+0000","last_change":"2024-09-10T20:41:50.275485+0000","last_active":"2024-09-10T20:41:50.276471+0000","last_peered":"2024-09-10T20:41:50.276471+0000","last_clean":"2024-09-10T20:41:50.276471+0000","last_became_active":"2024-09-10T20:41:50.275179+0000","last_became_peered":"2024-09-10T20:41:50.275179+0000","last_unstale":"2024-09-10T20:41:50.276471+0000","last_undegraded":"2024-09-10T20:41:50.276471+0000","last_fullsized":"2024-09-10T20:41:50.276471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:34.056711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278185+0000","last_change":"2024-09-10T20:41:49.245982+0000","last_active":"2024-09-10T20:41:50.278185+0000","last_peered":"2024-09-10T20:41:50.278185+0000","last_clean":"2024-09-10T20:41:50.278185+0000","last_became_active":"2024-09-10T20:41:49.245669+0000","last_became_peered":"2024-09-10T20:41:49.245669+0000","last_unstale":"2024-09-10T20:41:50.278185+0000","last_undegraded":"2024-09-10T20:41:50.278185+0000","last_fullsized":"2024-09-10T20:41:50.278185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:21:05.620306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.234051+0000","last_change":"2024-09-10T20:41:49.234051+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.234051+0000","last_undegraded":"2024-09-10T20:41:49.234051+0000","last_fullsized":"2024-09-10T20:41:49.234051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.250481+0000","last_change":"2024-09-10T20:41:49.246041+0000","last_active":"2024-09-10T20:41:49.250481+0000","last_peered":"2024-09-10T20:41:49.250481+0000","last_clean":"2024-09-10T20:41:49.250481+0000","last_became_active":"2024-09-10T20:41:49.245783+0000","last_became_peered":"2024-09-10T20:41:49.245783+0000","last_unstale":"2024-09-10T20:41:49.250481+0000","last_undegraded":"2024-09-10T20:41:49.250481+0000","last_fullsized":"2024-09-10T20:41:49.250481+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:23.096775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229857+0000","last_change":"2024-09-10T20:41:49.229857+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229857+0000","last_undegraded":"2024-09-10T20:41:49.229857+0000","last_fullsized":"2024-09-10T20:41:49.229857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238563+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238256+0000","last_became_peered":"2024-09-10T20:41:49.238256+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:16.403282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278144+0000","last_change":"2024-09-10T20:41:50.273036+0000","last_active":"2024-09-10T20:41:50.278144+0000","last_peered":"2024-09-10T20:41:50.278144+0000","last_clean":"2024-09-10T20:41:50.278144+0000","last_became_active":"2024-09-10T20:41:50.272751+0000","last_became_peered":"2024-09-10T20:41:50.272751+0000","last_unstale":"2024-09-10T20:41:50.278144+0000","last_undegraded":"2024-09-10T20:41:50.278144+0000","last_fullsized":"2024-09-10T20:41:50.278144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:48:00.876771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252771+0000","last_change":"2024-09-10T20:41:49.239816+0000","last_active":"2024-09-10T20:41:50.252771+0000","last_peered":"2024-09-10T20:41:50.252771+0000","last_clean":"2024-09-10T20:41:50.252771+0000","last_became_active":"2024-09-10T20:41:49.239509+0000","last_became_peered":"2024-09-10T20:41:49.239509+0000","last_unstale":"2024-09-10T20:41:50.252771+0000","last_undegraded":"2024-09-10T20:41:50.252771+0000","last_fullsized":"2024-09-10T20:41:50.252771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:30.409719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270329+0000","last_change":"2024-09-10T20:41:50.268159+0000","last_active":"2024-09-10T20:41:50.270329+0000","last_peered":"2024-09-10T20:41:50.270329+0000","last_clean":"2024-09-10T20:41:50.270329+0000","last_became_active":"2024-09-10T20:41:50.267522+0000","last_became_peered":"2024-09-10T20:41:50.267522+0000","last_unstale":"2024-09-10T20:41:50.270329+0000","last_undegraded":"2024-09-10T20:41:50.270329+0000","last_fullsized":"2024-09-10T20:41:50.270329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.176084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247689+0000","last_change":"2024-09-10T20:41:49.243018+0000","last_active":"2024-09-10T20:41:50.247689+0000","last_peered":"2024-09-10T20:41:50.247689+0000","last_clean":"2024-09-10T20:41:50.247689+0000","last_became_active":"2024-09-10T20:41:49.242751+0000","last_became_peered":"2024-09-10T20:41:49.242751+0000","last_unstale":"2024-09-10T20:41:50.247689+0000","last_undegraded":"2024-09-10T20:41:50.247689+0000","last_fullsized":"2024-09-10T20:41:50.247689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:47.281477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274705+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.274705+0000","last_peered":"2024-09-10T20:41:50.274705+0000","last_clean":"2024-09-10T20:41:50.274705+0000","last_became_active":"2024-09-10T20:41:50.272083+0000","last_became_peered":"2024-09-10T20:41:50.272083+0000","last_unstale":"2024-09-10T20:41:50.274705+0000","last_undegraded":"2024-09-10T20:41:50.274705+0000","last_fullsized":"2024-09-10T20:41:50.274705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242314+0000","last_change":"2024-09-10T20:41:49.235825+0000","last_active":"2024-09-10T20:41:49.242314+0000","last_peered":"2024-09-10T20:41:49.242314+0000","last_clean":"2024-09-10T20:41:49.242314+0000","last_became_active":"2024-09-10T20:41:49.235466+0000","last_became_peered":"2024-09-10T20:41:49.235466+0000","last_unstale":"2024-09-10T20:41:49.242314+0000","last_undegraded":"2024-09-10T20:41:49.242314+0000","last_fullsized":"2024-09-10T20:41:49.242314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:08.240386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229403+0000","last_change":"2024-09-10T20:41:49.229403+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229403+0000","last_undegraded":"2024-09-10T20:41:49.229403+0000","last_fullsized":"2024-09-10T20:41:49.229403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245167+0000","last_change":"2024-09-10T20:41:49.241236+0000","last_active":"2024-09-10T20:41:49.245167+0000","last_peered":"2024-09-10T20:41:49.245167+0000","last_clean":"2024-09-10T20:41:49.245167+0000","last_became_active":"2024-09-10T20:41:49.240686+0000","last_became_peered":"2024-09-10T20:41:49.240686+0000","last_unstale":"2024-09-10T20:41:49.245167+0000","last_undegraded":"2024-09-10T20:41:49.245167+0000","last_fullsized":"2024-09-10T20:41:49.245167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:40:43.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267631+0000","last_change":"2024-09-10T20:41:50.262537+0000","last_active":"2024-09-10T20:41:50.267631+0000","last_peered":"2024-09-10T20:41:50.267631+0000","last_clean":"2024-09-10T20:41:50.267631+0000","last_became_active":"2024-09-10T20:41:50.261672+0000","last_became_peered":"2024-09-10T20:41:50.261672+0000","last_unstale":"2024-09-10T20:41:50.267631+0000","last_undegraded":"2024-09-10T20:41:50.267631+0000","last_fullsized":"2024-09-10T20:41:50.267631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:57:22.115806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245187+0000","last_change":"2024-09-10T20:41:49.241419+0000","last_active":"2024-09-10T20:41:49.245187+0000","last_peered":"2024-09-10T20:41:49.245187+0000","last_clean":"2024-09-10T20:41:49.245187+0000","last_became_active":"2024-09-10T20:41:49.241125+0000","last_became_peered":"2024-09-10T20:41:49.241125+0000","last_unstale":"2024-09-10T20:41:49.245187+0000","last_undegraded":"2024-09-10T20:41:49.245187+0000","last_fullsized":"2024-09-10T20:41:49.245187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:45:50.357901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278395+0000","last_change":"2024-09-10T20:41:50.272126+0000","last_active":"2024-09-10T20:41:50.278395+0000","last_peered":"2024-09-10T20:41:50.278395+0000","last_clean":"2024-09-10T20:41:50.278395+0000","last_became_active":"2024-09-10T20:41:50.271842+0000","last_became_peered":"2024-09-10T20:41:50.271842+0000","last_unstale":"2024-09-10T20:41:50.278395+0000","last_undegraded":"2024-09-10T20:41:50.278395+0000","last_fullsized":"2024-09-10T20:41:50.278395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:17:24.548442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254957+0000","last_change":"2024-09-10T20:41:49.248639+0000","last_active":"2024-09-10T20:41:50.254957+0000","last_peered":"2024-09-10T20:41:50.254957+0000","last_clean":"2024-09-10T20:41:50.254957+0000","last_became_active":"2024-09-10T20:41:49.248352+0000","last_became_peered":"2024-09-10T20:41:49.248352+0000","last_unstale":"2024-09-10T20:41:50.254957+0000","last_undegraded":"2024-09-10T20:41:50.254957+0000","last_fullsized":"2024-09-10T20:41:50.254957+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:41.288742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228516+0000","last_change":"2024-09-10T20:41:49.228516+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228516+0000","last_undegraded":"2024-09-10T20:41:49.228516+0000","last_fullsized":"2024-09-10T20:41:49.228516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.251913+0000","last_change":"2024-09-10T20:41:49.249615+0000","last_active":"2024-09-10T20:41:50.251913+0000","last_peered":"2024-09-10T20:41:50.251913+0000","last_clean":"2024-09-10T20:41:50.251913+0000","last_became_active":"2024-09-10T20:41:49.249414+0000","last_became_peered":"2024-09-10T20:41:49.249414+0000","last_unstale":"2024-09-10T20:41:50.251913+0000","last_undegraded":"2024-09-10T20:41:50.251913+0000","last_fullsized":"2024-09-10T20:41:50.251913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:36.540914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231389+0000","last_change":"2024-09-10T20:41:49.231389+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231389+0000","last_undegraded":"2024-09-10T20:41:49.231389+0000","last_fullsized":"2024-09-10T20:41:49.231389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282862+0000","last_change":"2024-09-10T20:41:49.244860+0000","last_active":"2024-09-10T20:41:50.282862+0000","last_peered":"2024-09-10T20:41:50.282862+0000","last_clean":"2024-09-10T20:41:50.282862+0000","last_became_active":"2024-09-10T20:41:49.244532+0000","last_became_peered":"2024-09-10T20:41:49.244532+0000","last_unstale":"2024-09-10T20:41:50.282862+0000","last_undegraded":"2024-09-10T20:41:50.282862+0000","last_fullsized":"2024-09-10T20:41:50.282862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:33.273565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257492+0000","last_change":"2024-09-10T20:41:50.255850+0000","last_active":"2024-09-10T20:41:50.257492+0000","last_peered":"2024-09-10T20:41:50.257492+0000","last_clean":"2024-09-10T20:41:50.257492+0000","last_became_active":"2024-09-10T20:41:50.254971+0000","last_became_peered":"2024-09-10T20:41:50.254971+0000","last_unstale":"2024-09-10T20:41:50.257492+0000","last_undegraded":"2024-09-10T20:41:50.257492+0000","last_fullsized":"2024-09-10T20:41:50.257492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:24:15.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283814+0000","last_change":"2024-09-10T20:41:49.239536+0000","last_active":"2024-09-10T20:41:50.283814+0000","last_peered":"2024-09-10T20:41:50.283814+0000","last_clean":"2024-09-10T20:41:50.283814+0000","last_became_active":"2024-09-10T20:41:49.239253+0000","last_became_peered":"2024-09-10T20:41:49.239253+0000","last_unstale":"2024-09-10T20:41:50.283814+0000","last_undegraded":"2024-09-10T20:41:50.283814+0000","last_fullsized":"2024-09-10T20:41:50.283814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:15.552579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254349+0000","last_change":"2024-09-10T20:41:50.251514+0000","last_active":"2024-09-10T20:41:50.254349+0000","last_peered":"2024-09-10T20:41:50.254349+0000","last_clean":"2024-09-10T20:41:50.254349+0000","last_became_active":"2024-09-10T20:41:50.251241+0000","last_became_peered":"2024-09-10T20:41:50.251241+0000","last_unstale":"2024-09-10T20:41:50.254349+0000","last_undegraded":"2024-09-10T20:41:50.254349+0000","last_fullsized":"2024-09-10T20:41:50.254349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.239713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283432+0000","last_change":"2024-09-10T20:41:49.248393+0000","last_active":"2024-09-10T20:41:50.283432+0000","last_peered":"2024-09-10T20:41:50.283432+0000","last_clean":"2024-09-10T20:41:50.283432+0000","last_became_active":"2024-09-10T20:41:49.248103+0000","last_became_peered":"2024-09-10T20:41:49.248103+0000","last_unstale":"2024-09-10T20:41:50.283432+0000","last_undegraded":"2024-09-10T20:41:50.283432+0000","last_fullsized":"2024-09-10T20:41:50.283432+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:54.196476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231812+0000","last_change":"2024-09-10T20:41:49.231812+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231812+0000","last_undegraded":"2024-09-10T20:41:49.231812+0000","last_fullsized":"2024-09-10T20:41:49.231812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276182+0000","last_change":"2024-09-10T20:41:49.239383+0000","last_active":"2024-09-10T20:41:50.276182+0000","last_peered":"2024-09-10T20:41:50.276182+0000","last_clean":"2024-09-10T20:41:50.276182+0000","last_became_active":"2024-09-10T20:41:49.239097+0000","last_became_peered":"2024-09-10T20:41:49.239097+0000","last_unstale":"2024-09-10T20:41:50.276182+0000","last_undegraded":"2024-09-10T20:41:50.276182+0000","last_fullsized":"2024-09-10T20:41:50.276182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:51.420841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232592+0000","last_change":"2024-09-10T20:41:49.232592+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232592+0000","last_undegraded":"2024-09-10T20:41:49.232592+0000","last_fullsized":"2024-09-10T20:41:49.232592+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257075+0000","last_change":"2024-09-10T20:41:49.243612+0000","last_active":"2024-09-10T20:41:50.257075+0000","last_peered":"2024-09-10T20:41:50.257075+0000","last_clean":"2024-09-10T20:41:50.257075+0000","last_became_active":"2024-09-10T20:41:49.243065+0000","last_became_peered":"2024-09-10T20:41:49.243065+0000","last_unstale":"2024-09-10T20:41:50.257075+0000","last_undegraded":"2024-09-10T20:41:50.257075+0000","last_fullsized":"2024-09-10T20:41:50.257075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:20:31.645185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233707+0000","last_change":"2024-09-10T20:41:49.233707+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233707+0000","last_undegraded":"2024-09-10T20:41:49.233707+0000","last_fullsized":"2024-09-10T20:41:49.233707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252699+0000","last_change":"2024-09-10T20:41:49.241358+0000","last_active":"2024-09-10T20:41:50.252699+0000","last_peered":"2024-09-10T20:41:50.252699+0000","last_clean":"2024-09-10T20:41:50.252699+0000","last_became_active":"2024-09-10T20:41:49.241098+0000","last_became_peered":"2024-09-10T20:41:49.241098+0000","last_unstale":"2024-09-10T20:41:50.252699+0000","last_undegraded":"2024-09-10T20:41:50.252699+0000","last_fullsized":"2024-09-10T20:41:50.252699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:53:49.000467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233626+0000","last_change":"2024-09-10T20:41:49.233626+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233626+0000","last_undegraded":"2024-09-10T20:41:49.233626+0000","last_fullsized":"2024-09-10T20:41:49.233626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247275+0000","last_change":"2024-09-10T20:41:49.234909+0000","last_active":"2024-09-10T20:41:50.247275+0000","last_peered":"2024-09-10T20:41:50.247275+0000","last_clean":"2024-09-10T20:41:50.247275+0000","last_became_active":"2024-09-10T20:41:49.234598+0000","last_became_peered":"2024-09-10T20:41:49.234598+0000","last_unstale":"2024-09-10T20:41:50.247275+0000","last_undegraded":"2024-09-10T20:41:50.247275+0000","last_fullsized":"2024-09-10T20:41:50.247275+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:00.862079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229401+0000","last_change":"2024-09-10T20:41:49.229401+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229401+0000","last_undegraded":"2024-09-10T20:41:49.229401+0000","last_fullsized":"2024-09-10T20:41:49.229401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248123+0000","last_change":"2024-09-10T20:41:49.245102+0000","last_active":"2024-09-10T20:41:49.248123+0000","last_peered":"2024-09-10T20:41:49.248123+0000","last_clean":"2024-09-10T20:41:49.248123+0000","last_became_active":"2024-09-10T20:41:49.244790+0000","last_became_peered":"2024-09-10T20:41:49.244790+0000","last_unstale":"2024-09-10T20:41:49.248123+0000","last_undegraded":"2024-09-10T20:41:49.248123+0000","last_fullsized":"2024-09-10T20:41:49.248123+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:26.633006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230237+0000","last_change":"2024-09-10T20:41:49.230237+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230237+0000","last_undegraded":"2024-09-10T20:41:49.230237+0000","last_fullsized":"2024-09-10T20:41:49.230237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281880+0000","last_change":"2024-09-10T20:41:49.239083+0000","last_active":"2024-09-10T20:41:50.281880+0000","last_peered":"2024-09-10T20:41:50.281880+0000","last_clean":"2024-09-10T20:41:50.281880+0000","last_became_active":"2024-09-10T20:41:49.238741+0000","last_became_peered":"2024-09-10T20:41:49.238741+0000","last_unstale":"2024-09-10T20:41:50.281880+0000","last_undegraded":"2024-09-10T20:41:50.281880+0000","last_fullsized":"2024-09-10T20:41:50.281880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:09:56.852921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.277987+0000","last_change":"2024-09-10T20:41:50.270889+0000","last_active":"2024-09-10T20:41:50.277987+0000","last_peered":"2024-09-10T20:41:50.277987+0000","last_clean":"2024-09-10T20:41:50.277987+0000","last_became_active":"2024-09-10T20:41:50.270603+0000","last_became_peered":"2024-09-10T20:41:50.270603+0000","last_unstale":"2024-09-10T20:41:50.277987+0000","last_undegraded":"2024-09-10T20:41:50.277987+0000","last_fullsized":"2024-09-10T20:41:50.277987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:23.399016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252628+0000","last_change":"2024-09-10T20:41:49.249494+0000","last_active":"2024-09-10T20:41:50.252628+0000","last_peered":"2024-09-10T20:41:50.252628+0000","last_clean":"2024-09-10T20:41:50.252628+0000","last_became_active":"2024-09-10T20:41:49.249207+0000","last_became_peered":"2024-09-10T20:41:49.249207+0000","last_unstale":"2024-09-10T20:41:50.252628+0000","last_undegraded":"2024-09-10T20:41:50.252628+0000","last_fullsized":"2024-09-10T20:41:50.252628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:47.078899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.268605+0000","last_change":"2024-09-10T20:41:50.265954+0000","last_active":"2024-09-10T20:41:50.268605+0000","last_peered":"2024-09-10T20:41:50.268605+0000","last_clean":"2024-09-10T20:41:50.268605+0000","last_became_active":"2024-09-10T20:41:50.265644+0000","last_became_peered":"2024-09-10T20:41:50.265644+0000","last_unstale":"2024-09-10T20:41:50.268605+0000","last_undegraded":"2024-09-10T20:41:50.268605+0000","last_fullsized":"2024-09-10T20:41:50.268605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:22.562059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238772+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238312+0000","last_became_peered":"2024-09-10T20:41:49.238312+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:20.684231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278050+0000","last_change":"2024-09-10T20:41:50.265096+0000","last_active":"2024-09-10T20:41:50.278050+0000","last_peered":"2024-09-10T20:41:50.278050+0000","last_clean":"2024-09-10T20:41:50.278050+0000","last_became_active":"2024-09-10T20:41:50.264830+0000","last_became_peered":"2024-09-10T20:41:50.264830+0000","last_unstale":"2024-09-10T20:41:50.278050+0000","last_undegraded":"2024-09-10T20:41:50.278050+0000","last_fullsized":"2024-09-10T20:41:50.278050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:15.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242477+0000","last_change":"2024-09-10T20:41:49.236972+0000","last_active":"2024-09-10T20:41:49.242477+0000","last_peered":"2024-09-10T20:41:49.242477+0000","last_clean":"2024-09-10T20:41:49.242477+0000","last_became_active":"2024-09-10T20:41:49.236675+0000","last_became_peered":"2024-09-10T20:41:49.236675+0000","last_unstale":"2024-09-10T20:41:49.242477+0000","last_undegraded":"2024-09-10T20:41:49.242477+0000","last_fullsized":"2024-09-10T20:41:49.242477+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:08.249515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229009+0000","last_change":"2024-09-10T20:41:49.229009+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229009+0000","last_undegraded":"2024-09-10T20:41:49.229009+0000","last_fullsized":"2024-09-10T20:41:49.229009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.239295+0000","last_change":"2024-09-10T20:41:49.235872+0000","last_active":"2024-09-10T20:41:49.239295+0000","last_peered":"2024-09-10T20:41:49.239295+0000","last_clean":"2024-09-10T20:41:49.239295+0000","last_became_active":"2024-09-10T20:41:49.235579+0000","last_became_peered":"2024-09-10T20:41:49.235579+0000","last_unstale":"2024-09-10T20:41:49.239295+0000","last_undegraded":"2024-09-10T20:41:49.239295+0000","last_fullsized":"2024-09-10T20:41:49.239295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:32.310476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263923+0000","last_change":"2024-09-10T20:41:50.260226+0000","last_active":"2024-09-10T20:41:50.263923+0000","last_peered":"2024-09-10T20:41:50.263923+0000","last_clean":"2024-09-10T20:41:50.263923+0000","last_became_active":"2024-09-10T20:41:50.259625+0000","last_became_peered":"2024-09-10T20:41:50.259625+0000","last_unstale":"2024-09-10T20:41:50.263923+0000","last_undegraded":"2024-09-10T20:41:50.263923+0000","last_fullsized":"2024-09-10T20:41:50.263923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:38.991086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248086+0000","last_change":"2024-09-10T20:41:49.242417+0000","last_active":"2024-09-10T20:41:49.248086+0000","last_peered":"2024-09-10T20:41:49.248086+0000","last_clean":"2024-09-10T20:41:49.248086+0000","last_became_active":"2024-09-10T20:41:49.242161+0000","last_became_peered":"2024-09-10T20:41:49.242161+0000","last_unstale":"2024-09-10T20:41:49.248086+0000","last_undegraded":"2024-09-10T20:41:49.248086+0000","last_fullsized":"2024-09-10T20:41:49.248086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:13.922975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233118+0000","last_change":"2024-09-10T20:41:49.233118+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233118+0000","last_undegraded":"2024-09-10T20:41:49.233118+0000","last_fullsized":"2024-09-10T20:41:49.233118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247637+0000","last_change":"2024-09-10T20:41:49.240524+0000","last_active":"2024-09-10T20:41:50.247637+0000","last_peered":"2024-09-10T20:41:50.247637+0000","last_clean":"2024-09-10T20:41:50.247637+0000","last_became_active":"2024-09-10T20:41:49.240245+0000","last_became_peered":"2024-09-10T20:41:49.240245+0000","last_unstale":"2024-09-10T20:41:50.247637+0000","last_undegraded":"2024-09-10T20:41:50.247637+0000","last_fullsized":"2024-09-10T20:41:50.247637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:14.912119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269687+0000","last_change":"2024-09-10T20:41:50.267529+0000","last_active":"2024-09-10T20:41:50.269687+0000","last_peered":"2024-09-10T20:41:50.269687+0000","last_clean":"2024-09-10T20:41:50.269687+0000","last_became_active":"2024-09-10T20:41:50.267222+0000","last_became_peered":"2024-09-10T20:41:50.267222+0000","last_unstale":"2024-09-10T20:41:50.269687+0000","last_undegraded":"2024-09-10T20:41:50.269687+0000","last_fullsized":"2024-09-10T20:41:50.269687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:21:30.112985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.253694+0000","last_change":"2024-09-10T20:41:49.236467+0000","last_active":"2024-09-10T20:41:50.253694+0000","last_peered":"2024-09-10T20:41:50.253694+0000","last_clean":"2024-09-10T20:41:50.253694+0000","last_became_active":"2024-09-10T20:41:49.236170+0000","last_became_peered":"2024-09-10T20:41:49.236170+0000","last_unstale":"2024-09-10T20:41:50.253694+0000","last_undegraded":"2024-09-10T20:41:50.253694+0000","last_fullsized":"2024-09-10T20:41:50.253694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:12.514693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229765+0000","last_change":"2024-09-10T20:41:49.229765+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229765+0000","last_undegraded":"2024-09-10T20:41:49.229765+0000","last_fullsized":"2024-09-10T20:41:49.229765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.280647+0000","last_change":"2024-09-10T20:41:49.249357+0000","last_active":"2024-09-10T20:41:50.280647+0000","last_peered":"2024-09-10T20:41:50.280647+0000","last_clean":"2024-09-10T20:41:50.280647+0000","last_became_active":"2024-09-10T20:41:49.248955+0000","last_became_peered":"2024-09-10T20:41:49.248955+0000","last_unstale":"2024-09-10T20:41:50.280647+0000","last_undegraded":"2024-09-10T20:41:50.280647+0000","last_fullsized":"2024-09-10T20:41:50.280647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:26.862222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.250227+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:41:50.250227+0000","last_peered":"2024-09-10T20:41:50.250227+0000","last_clean":"2024-09-10T20:41:50.250227+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:41:50.250227+0000","last_undegraded":"2024-09-10T20:41:50.250227+0000","last_fullsized":"2024-09-10T20:41:50.250227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509490,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":819200,"data_stored":337112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509489,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509489,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1400832,"data_stored":927354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509489,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":811008,"data_stored":337056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542194,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":782336,"data_stored":323944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1388544,"data_stored":920579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:41:54.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:41:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.431+0000 7f3383717640 1 -- 172.21.15.28:0/3949567495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3368003a50 msgr2=0x7f3368003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.431+0000 7f3383717640 1 --2- 172.21.15.28:0/3949567495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3368003a50 0x7f3368003e70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f33780016c0 tx=0x7f3368005830 comp rx=0 tx=0).stop 2024-09-10T20:41:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.431+0000 7f3383717640 1 -- 172.21.15.28:0/3949567495 shutdown_connections 2024-09-10T20:41:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.431+0000 7f3383717640 1 --2- 172.21.15.28:0/3949567495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3368003a50 0x7f3368003e70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.431+0000 7f3383717640 1 -- 172.21.15.28:0/3949567495 >> 172.21.15.28:0/3949567495 conn(0x7f337c0d3fb0 msgr2=0x7f337c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.432+0000 7f3383717640 1 -- 172.21.15.28:0/3949567495 shutdown_connections 2024-09-10T20:41:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.432+0000 7f3383717640 1 -- 172.21.15.28:0/3949567495 wait complete. 2024-09-10T20:41:54.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.433+0000 7f3383717640 1 Processor -- start 2024-09-10T20:41:54.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.433+0000 7f3383717640 1 -- start start 2024-09-10T20:41:54.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3383717640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3368003a50 0x7f337c1f8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3383717640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f337c1f7dd0 0x7f337c0a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3383717640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f337c1f46d0 0x7f337c0b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3383717640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f337c161b50 con 0x7f337c1f7dd0 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3383717640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f337c15e460 con 0x7f337c1f46d0 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3383717640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f337c161cd0 con 0x7f3368003a50 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f337c1f7dd0 0x7f337c0a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f337c1f7dd0 0x7f337c0a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58354/0 (socket says 172.21.15.28:58354) 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 -- 172.21.15.28:0/550669466 learned_addr learned my addr 172.21.15.28:0/550669466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f33814b3640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3368003a50 0x7f337c1f8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3381cb4640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f337c1f46d0 0x7f337c0b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 -- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3368003a50 msgr2=0x7f337c1f8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3368003a50 0x7f337c1f8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 -- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f337c1f46d0 msgr2=0x7f337c0b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f337c1f46d0 0x7f337c0b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 -- 172.21.15.28:0/550669466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c0a0b40 con 0x7f337c1f7dd0 2024-09-10T20:41:54.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3380cb2640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f337c1f7dd0 0x7f337c0a15c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f336c011780 tx=0x7f336c011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:54.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3371ffb640 1 -- 172.21.15.28:0/550669466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c004230 con 0x7f337c1f7dd0 2024-09-10T20:41:54.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3371ffb640 1 -- 172.21.15.28:0/550669466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f336c00f940 con 0x7f337c1f7dd0 2024-09-10T20:41:54.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c1f50e0 con 0x7f337c1f7dd0 2024-09-10T20:41:54.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.434+0000 7f3371ffb640 1 -- 172.21.15.28:0/550669466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c0044c0 con 0x7f337c1f7dd0 2024-09-10T20:41:54.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.435+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c0a0dc0 con 0x7f337c1f7dd0 2024-09-10T20:41:54.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.436+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337c0cf8f0 con 0x7f337c1f7dd0 2024-09-10T20:41:54.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.436+0000 7f3371ffb640 1 -- 172.21.15.28:0/550669466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f336c0035f0 con 0x7f337c1f7dd0 2024-09-10T20:41:54.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.437+0000 7f3371ffb640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f33600782d0 0x7f336007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.437+0000 7f3371ffb640 1 -- 172.21.15.28:0/550669466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f336c09ae10 con 0x7f337c1f7dd0 2024-09-10T20:41:54.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.437+0000 7f33814b3640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f33600782d0 0x7f336007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.438+0000 7f33814b3640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f33600782d0 0x7f336007a790 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3378032000 tx=0x7f337802c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:54.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.441+0000 7f3371ffb640 1 -- 172.21.15.28:0/550669466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f337c0a0b40 con 0x7f337c1f7dd0 2024-09-10T20:41:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.623+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f337c0d1500 con 0x7f33600782d0 2024-09-10T20:41:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.627+0000 7f3371ffb640 1 -- 172.21.15.28:0/550669466 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249747 (secure 0 0 0) 0x7f337c0d1500 con 0x7f33600782d0 2024-09-10T20:41:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f33600782d0 msgr2=0x7f336007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f33600782d0 0x7f336007a790 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3378032000 tx=0x7f337802c040 comp rx=0 tx=0).stop 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f337c1f7dd0 msgr2=0x7f337c0a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f337c1f7dd0 0x7f337c0a15c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f336c011780 tx=0x7f336c011b80 comp rx=0 tx=0).stop 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 shutdown_connections 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f33600782d0 0x7f336007a790 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f337c1f46d0 0x7f337c0b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f337c1f7dd0 0x7f337c0a15c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 --2- 172.21.15.28:0/550669466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3368003a50 0x7f337c1f8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.630+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 >> 172.21.15.28:0/550669466 conn(0x7f337c0d3fb0 msgr2=0x7f337c0b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.631+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 shutdown_connections 2024-09-10T20:41:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.631+0000 7f3383717640 1 -- 172.21.15.28:0/550669466 wait complete. 2024-09-10T20:41:54.646 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-10T20:41:54.029282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591042,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":132,"num_write_kb":1631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244464,"kb_used_data":7408,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730256,"statfs":{"total":773094113280,"available":772843782144,"internally_reserved":0,"allocated":7585792,"data_stored":3836678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.805315"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264924+0000","last_change":"2024-09-10T20:41:50.262634+0000","last_active":"2024-09-10T20:41:50.264924+0000","last_peered":"2024-09-10T20:41:50.264924+0000","last_clean":"2024-09-10T20:41:50.264924+0000","last_became_active":"2024-09-10T20:41:50.262341+0000","last_became_peered":"2024-09-10T20:41:50.262341+0000","last_unstale":"2024-09-10T20:41:50.264924+0000","last_undegraded":"2024-09-10T20:41:50.264924+0000","last_fullsized":"2024-09-10T20:41:50.264924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:09:16.358398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270227+0000","last_change":"2024-09-10T20:41:50.268290+0000","last_active":"2024-09-10T20:41:50.270227+0000","last_peered":"2024-09-10T20:41:50.270227+0000","last_clean":"2024-09-10T20:41:50.270227+0000","last_became_active":"2024-09-10T20:41:50.268020+0000","last_became_peered":"2024-09-10T20:41:50.268020+0000","last_unstale":"2024-09-10T20:41:50.270227+0000","last_undegraded":"2024-09-10T20:41:50.270227+0000","last_fullsized":"2024-09-10T20:41:50.270227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:44.595286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232984+0000","last_change":"2024-09-10T20:41:49.232984+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232984+0000","last_undegraded":"2024-09-10T20:41:49.232984+0000","last_fullsized":"2024-09-10T20:41:49.232984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233187+0000","last_change":"2024-09-10T20:41:49.233187+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233187+0000","last_undegraded":"2024-09-10T20:41:49.233187+0000","last_fullsized":"2024-09-10T20:41:49.233187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230680+0000","last_change":"2024-09-10T20:41:49.230680+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230680+0000","last_undegraded":"2024-09-10T20:41:49.230680+0000","last_fullsized":"2024-09-10T20:41:49.230680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267661+0000","last_change":"2024-09-10T20:41:50.262868+0000","last_active":"2024-09-10T20:41:50.267661+0000","last_peered":"2024-09-10T20:41:50.267661+0000","last_clean":"2024-09-10T20:41:50.267661+0000","last_became_active":"2024-09-10T20:41:50.262182+0000","last_became_peered":"2024-09-10T20:41:50.262182+0000","last_unstale":"2024-09-10T20:41:50.267661+0000","last_undegraded":"2024-09-10T20:41:50.267661+0000","last_fullsized":"2024-09-10T20:41:50.267661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:10.294469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278278+0000","last_change":"2024-09-10T20:41:50.269379+0000","last_active":"2024-09-10T20:41:50.278278+0000","last_peered":"2024-09-10T20:41:50.278278+0000","last_clean":"2024-09-10T20:41:50.278278+0000","last_became_active":"2024-09-10T20:41:50.268955+0000","last_became_peered":"2024-09-10T20:41:50.268955+0000","last_unstale":"2024-09-10T20:41:50.278278+0000","last_undegraded":"2024-09-10T20:41:50.278278+0000","last_fullsized":"2024-09-10T20:41:50.278278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:55.335452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264003+0000","last_change":"2024-09-10T20:41:50.261207+0000","last_active":"2024-09-10T20:41:50.264003+0000","last_peered":"2024-09-10T20:41:50.264003+0000","last_clean":"2024-09-10T20:41:50.264003+0000","last_became_active":"2024-09-10T20:41:50.260866+0000","last_became_peered":"2024-09-10T20:41:50.260866+0000","last_unstale":"2024-09-10T20:41:50.264003+0000","last_undegraded":"2024-09-10T20:41:50.264003+0000","last_fullsized":"2024-09-10T20:41:50.264003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:48.716401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232777+0000","last_change":"2024-09-10T20:41:49.232777+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232777+0000","last_undegraded":"2024-09-10T20:41:49.232777+0000","last_fullsized":"2024-09-10T20:41:49.232777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278111+0000","last_change":"2024-09-10T20:41:50.270275+0000","last_active":"2024-09-10T20:41:50.278111+0000","last_peered":"2024-09-10T20:41:50.278111+0000","last_clean":"2024-09-10T20:41:50.278111+0000","last_became_active":"2024-09-10T20:41:50.269980+0000","last_became_peered":"2024-09-10T20:41:50.269980+0000","last_unstale":"2024-09-10T20:41:50.278111+0000","last_undegraded":"2024-09-10T20:41:50.278111+0000","last_fullsized":"2024-09-10T20:41:50.278111+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:12.357126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263972+0000","last_change":"2024-09-10T20:41:50.260076+0000","last_active":"2024-09-10T20:41:50.263972+0000","last_peered":"2024-09-10T20:41:50.263972+0000","last_clean":"2024-09-10T20:41:50.263972+0000","last_became_active":"2024-09-10T20:41:50.259752+0000","last_became_peered":"2024-09-10T20:41:50.259752+0000","last_unstale":"2024-09-10T20:41:50.263972+0000","last_undegraded":"2024-09-10T20:41:50.263972+0000","last_fullsized":"2024-09-10T20:41:50.263972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:08.756456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228409+0000","last_change":"2024-09-10T20:41:49.228409+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228409+0000","last_undegraded":"2024-09-10T20:41:49.228409+0000","last_fullsized":"2024-09-10T20:41:49.228409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232209+0000","last_change":"2024-09-10T20:41:49.232209+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232209+0000","last_undegraded":"2024-09-10T20:41:49.232209+0000","last_fullsized":"2024-09-10T20:41:49.232209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229613+0000","last_change":"2024-09-10T20:41:49.229613+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229613+0000","last_undegraded":"2024-09-10T20:41:49.229613+0000","last_fullsized":"2024-09-10T20:41:49.229613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230334+0000","last_change":"2024-09-10T20:41:49.230334+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230334+0000","last_undegraded":"2024-09-10T20:41:49.230334+0000","last_fullsized":"2024-09-10T20:41:49.230334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.262988+0000","last_change":"2024-09-10T20:41:50.259865+0000","last_active":"2024-09-10T20:41:50.262988+0000","last_peered":"2024-09-10T20:41:50.262988+0000","last_clean":"2024-09-10T20:41:50.262988+0000","last_became_active":"2024-09-10T20:41:50.259492+0000","last_became_peered":"2024-09-10T20:41:50.259492+0000","last_unstale":"2024-09-10T20:41:50.262988+0000","last_undegraded":"2024-09-10T20:41:50.262988+0000","last_fullsized":"2024-09-10T20:41:50.262988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:32:07.918850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228536+0000","last_change":"2024-09-10T20:41:49.228536+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228536+0000","last_undegraded":"2024-09-10T20:41:49.228536+0000","last_fullsized":"2024-09-10T20:41:49.228536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230656+0000","last_change":"2024-09-10T20:41:49.230656+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230656+0000","last_undegraded":"2024-09-10T20:41:49.230656+0000","last_fullsized":"2024-09-10T20:41:49.230656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269422+0000","last_change":"2024-09-10T20:41:50.264629+0000","last_active":"2024-09-10T20:41:50.269422+0000","last_peered":"2024-09-10T20:41:50.269422+0000","last_clean":"2024-09-10T20:41:50.269422+0000","last_became_active":"2024-09-10T20:41:50.264345+0000","last_became_peered":"2024-09-10T20:41:50.264345+0000","last_unstale":"2024-09-10T20:41:50.269422+0000","last_undegraded":"2024-09-10T20:41:50.269422+0000","last_fullsized":"2024-09-10T20:41:50.269422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:33:47.084724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228954+0000","last_change":"2024-09-10T20:41:49.228954+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228954+0000","last_undegraded":"2024-09-10T20:41:49.228954+0000","last_fullsized":"2024-09-10T20:41:49.228954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257652+0000","last_change":"2024-09-10T20:41:50.255889+0000","last_active":"2024-09-10T20:41:50.257652+0000","last_peered":"2024-09-10T20:41:50.257652+0000","last_clean":"2024-09-10T20:41:50.257652+0000","last_became_active":"2024-09-10T20:41:50.255572+0000","last_became_peered":"2024-09-10T20:41:50.255572+0000","last_unstale":"2024-09-10T20:41:50.257652+0000","last_undegraded":"2024-09-10T20:41:50.257652+0000","last_fullsized":"2024-09-10T20:41:50.257652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:38.194068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274208+0000","last_change":"2024-09-10T20:41:50.265660+0000","last_active":"2024-09-10T20:41:50.274208+0000","last_peered":"2024-09-10T20:41:50.274208+0000","last_clean":"2024-09-10T20:41:50.274208+0000","last_became_active":"2024-09-10T20:41:50.265364+0000","last_became_peered":"2024-09-10T20:41:50.265364+0000","last_unstale":"2024-09-10T20:41:50.274208+0000","last_undegraded":"2024-09-10T20:41:50.274208+0000","last_fullsized":"2024-09-10T20:41:50.274208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:52.067493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269174+0000","last_change":"2024-09-10T20:41:50.263991+0000","last_active":"2024-09-10T20:41:50.269174+0000","last_peered":"2024-09-10T20:41:50.269174+0000","last_clean":"2024-09-10T20:41:50.269174+0000","last_became_active":"2024-09-10T20:41:50.263690+0000","last_became_peered":"2024-09-10T20:41:50.263690+0000","last_unstale":"2024-09-10T20:41:50.269174+0000","last_undegraded":"2024-09-10T20:41:50.269174+0000","last_fullsized":"2024-09-10T20:41:50.269174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:08.679923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264022+0000","last_change":"2024-09-10T20:41:50.261825+0000","last_active":"2024-09-10T20:41:50.264022+0000","last_peered":"2024-09-10T20:41:50.264022+0000","last_clean":"2024-09-10T20:41:50.264022+0000","last_became_active":"2024-09-10T20:41:50.261516+0000","last_became_peered":"2024-09-10T20:41:50.261516+0000","last_unstale":"2024-09-10T20:41:50.264022+0000","last_undegraded":"2024-09-10T20:41:50.264022+0000","last_fullsized":"2024-09-10T20:41:50.264022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:26.740959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278057+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.278057+0000","last_peered":"2024-09-10T20:41:50.278057+0000","last_clean":"2024-09-10T20:41:50.278057+0000","last_became_active":"2024-09-10T20:41:50.272144+0000","last_became_peered":"2024-09-10T20:41:50.272144+0000","last_unstale":"2024-09-10T20:41:50.278057+0000","last_undegraded":"2024-09-10T20:41:50.278057+0000","last_fullsized":"2024-09-10T20:41:50.278057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:05.044942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.275708+0000","last_change":"2024-09-10T20:41:50.273908+0000","last_active":"2024-09-10T20:41:50.275708+0000","last_peered":"2024-09-10T20:41:50.275708+0000","last_clean":"2024-09-10T20:41:50.275708+0000","last_became_active":"2024-09-10T20:41:50.273619+0000","last_became_peered":"2024-09-10T20:41:50.273619+0000","last_unstale":"2024-09-10T20:41:50.275708+0000","last_undegraded":"2024-09-10T20:41:50.275708+0000","last_fullsized":"2024-09-10T20:41:50.275708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:57.474072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229786+0000","last_change":"2024-09-10T20:41:49.229786+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229786+0000","last_undegraded":"2024-09-10T20:41:49.229786+0000","last_fullsized":"2024-09-10T20:41:49.229786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269719+0000","last_change":"2024-09-10T20:41:50.267808+0000","last_active":"2024-09-10T20:41:50.269719+0000","last_peered":"2024-09-10T20:41:50.269719+0000","last_clean":"2024-09-10T20:41:50.269719+0000","last_became_active":"2024-09-10T20:41:50.267481+0000","last_became_peered":"2024-09-10T20:41:50.267481+0000","last_unstale":"2024-09-10T20:41:50.269719+0000","last_undegraded":"2024-09-10T20:41:50.269719+0000","last_fullsized":"2024-09-10T20:41:50.269719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:18:49.925720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231774+0000","last_change":"2024-09-10T20:41:49.231774+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231774+0000","last_undegraded":"2024-09-10T20:41:49.231774+0000","last_fullsized":"2024-09-10T20:41:49.231774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.265287+0000","last_change":"2024-09-10T20:41:50.263690+0000","last_active":"2024-09-10T20:41:50.265287+0000","last_peered":"2024-09-10T20:41:50.265287+0000","last_clean":"2024-09-10T20:41:50.265287+0000","last_became_active":"2024-09-10T20:41:50.263407+0000","last_became_peered":"2024-09-10T20:41:50.263407+0000","last_unstale":"2024-09-10T20:41:50.265287+0000","last_undegraded":"2024-09-10T20:41:50.265287+0000","last_fullsized":"2024-09-10T20:41:50.265287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:16.841293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269657+0000","last_change":"2024-09-10T20:41:50.266413+0000","last_active":"2024-09-10T20:41:50.269657+0000","last_peered":"2024-09-10T20:41:50.269657+0000","last_clean":"2024-09-10T20:41:50.269657+0000","last_became_active":"2024-09-10T20:41:50.266150+0000","last_became_peered":"2024-09-10T20:41:50.266150+0000","last_unstale":"2024-09-10T20:41:50.269657+0000","last_undegraded":"2024-09-10T20:41:50.269657+0000","last_fullsized":"2024-09-10T20:41:50.269657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:20.996250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270272+0000","last_change":"2024-09-10T20:41:50.267692+0000","last_active":"2024-09-10T20:41:50.270272+0000","last_peered":"2024-09-10T20:41:50.270272+0000","last_clean":"2024-09-10T20:41:50.270272+0000","last_became_active":"2024-09-10T20:41:50.267181+0000","last_became_peered":"2024-09-10T20:41:50.267181+0000","last_unstale":"2024-09-10T20:41:50.270272+0000","last_undegraded":"2024-09-10T20:41:50.270272+0000","last_fullsized":"2024-09-10T20:41:50.270272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:14.639786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263938+0000","last_change":"2024-09-10T20:41:50.260450+0000","last_active":"2024-09-10T20:41:50.263938+0000","last_peered":"2024-09-10T20:41:50.263938+0000","last_clean":"2024-09-10T20:41:50.263938+0000","last_became_active":"2024-09-10T20:41:50.260093+0000","last_became_peered":"2024-09-10T20:41:50.260093+0000","last_unstale":"2024-09-10T20:41:50.263938+0000","last_undegraded":"2024-09-10T20:41:50.263938+0000","last_fullsized":"2024-09-10T20:41:50.263938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:16.794344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281382+0000","last_change":"2024-09-10T20:41:49.244556+0000","last_active":"2024-09-10T20:41:50.281382+0000","last_peered":"2024-09-10T20:41:50.281382+0000","last_clean":"2024-09-10T20:41:50.281382+0000","last_became_active":"2024-09-10T20:41:49.244277+0000","last_became_peered":"2024-09-10T20:41:49.244277+0000","last_unstale":"2024-09-10T20:41:50.281382+0000","last_undegraded":"2024-09-10T20:41:50.281382+0000","last_fullsized":"2024-09-10T20:41:50.281382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:24.745124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232330+0000","last_change":"2024-09-10T20:41:49.232330+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232330+0000","last_undegraded":"2024-09-10T20:41:49.232330+0000","last_fullsized":"2024-09-10T20:41:49.232330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242361+0000","last_change":"2024-09-10T20:41:49.236963+0000","last_active":"2024-09-10T20:41:49.242361+0000","last_peered":"2024-09-10T20:41:49.242361+0000","last_clean":"2024-09-10T20:41:49.242361+0000","last_became_active":"2024-09-10T20:41:49.236654+0000","last_became_peered":"2024-09-10T20:41:49.236654+0000","last_unstale":"2024-09-10T20:41:49.242361+0000","last_undegraded":"2024-09-10T20:41:49.242361+0000","last_fullsized":"2024-09-10T20:41:49.242361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:48.203284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278810+0000","last_change":"2024-09-10T20:41:50.277395+0000","last_active":"2024-09-10T20:41:50.278810+0000","last_peered":"2024-09-10T20:41:50.278810+0000","last_clean":"2024-09-10T20:41:50.278810+0000","last_became_active":"2024-09-10T20:41:50.277101+0000","last_became_peered":"2024-09-10T20:41:50.277101+0000","last_unstale":"2024-09-10T20:41:50.278810+0000","last_undegraded":"2024-09-10T20:41:50.278810+0000","last_fullsized":"2024-09-10T20:41:50.278810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:47.799998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282196+0000","last_change":"2024-09-10T20:41:49.238213+0000","last_active":"2024-09-10T20:41:50.282196+0000","last_peered":"2024-09-10T20:41:50.282196+0000","last_clean":"2024-09-10T20:41:50.282196+0000","last_became_active":"2024-09-10T20:41:49.237949+0000","last_became_peered":"2024-09-10T20:41:49.237949+0000","last_unstale":"2024-09-10T20:41:50.282196+0000","last_undegraded":"2024-09-10T20:41:50.282196+0000","last_fullsized":"2024-09-10T20:41:50.282196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:45:22.532346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.266258+0000","last_change":"2024-09-10T20:41:50.262357+0000","last_active":"2024-09-10T20:41:50.266258+0000","last_peered":"2024-09-10T20:41:50.266258+0000","last_clean":"2024-09-10T20:41:50.266258+0000","last_became_active":"2024-09-10T20:41:50.262081+0000","last_became_peered":"2024-09-10T20:41:50.262081+0000","last_unstale":"2024-09-10T20:41:50.266258+0000","last_undegraded":"2024-09-10T20:41:50.266258+0000","last_fullsized":"2024-09-10T20:41:50.266258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:44.966942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.279900+0000","last_change":"2024-09-10T20:41:49.247040+0000","last_active":"2024-09-10T20:41:50.279900+0000","last_peered":"2024-09-10T20:41:50.279900+0000","last_clean":"2024-09-10T20:41:50.279900+0000","last_became_active":"2024-09-10T20:41:49.246731+0000","last_became_peered":"2024-09-10T20:41:49.246731+0000","last_unstale":"2024-09-10T20:41:50.279900+0000","last_undegraded":"2024-09-10T20:41:50.279900+0000","last_fullsized":"2024-09-10T20:41:50.279900+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:37.379165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263903+0000","last_change":"2024-09-10T20:41:50.261513+0000","last_active":"2024-09-10T20:41:50.263903+0000","last_peered":"2024-09-10T20:41:50.263903+0000","last_clean":"2024-09-10T20:41:50.263903+0000","last_became_active":"2024-09-10T20:41:50.261203+0000","last_became_peered":"2024-09-10T20:41:50.261203+0000","last_unstale":"2024-09-10T20:41:50.263903+0000","last_undegraded":"2024-09-10T20:41:50.263903+0000","last_fullsized":"2024-09-10T20:41:50.263903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:39.097791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242512+0000","last_change":"2024-09-10T20:41:49.237350+0000","last_active":"2024-09-10T20:41:49.242512+0000","last_peered":"2024-09-10T20:41:49.242512+0000","last_clean":"2024-09-10T20:41:49.242512+0000","last_became_active":"2024-09-10T20:41:49.237086+0000","last_became_peered":"2024-09-10T20:41:49.237086+0000","last_unstale":"2024-09-10T20:41:49.242512+0000","last_undegraded":"2024-09-10T20:41:49.242512+0000","last_fullsized":"2024-09-10T20:41:49.242512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:13.283338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276471+0000","last_change":"2024-09-10T20:41:50.275485+0000","last_active":"2024-09-10T20:41:50.276471+0000","last_peered":"2024-09-10T20:41:50.276471+0000","last_clean":"2024-09-10T20:41:50.276471+0000","last_became_active":"2024-09-10T20:41:50.275179+0000","last_became_peered":"2024-09-10T20:41:50.275179+0000","last_unstale":"2024-09-10T20:41:50.276471+0000","last_undegraded":"2024-09-10T20:41:50.276471+0000","last_fullsized":"2024-09-10T20:41:50.276471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:34.056711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278185+0000","last_change":"2024-09-10T20:41:49.245982+0000","last_active":"2024-09-10T20:41:50.278185+0000","last_peered":"2024-09-10T20:41:50.278185+0000","last_clean":"2024-09-10T20:41:50.278185+0000","last_became_active":"2024-09-10T20:41:49.245669+0000","last_became_peered":"2024-09-10T20:41:49.245669+0000","last_unstale":"2024-09-10T20:41:50.278185+0000","last_undegraded":"2024-09-10T20:41:50.278185+0000","last_fullsized":"2024-09-10T20:41:50.278185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:21:05.620306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.234051+0000","last_change":"2024-09-10T20:41:49.234051+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.234051+0000","last_undegraded":"2024-09-10T20:41:49.234051+0000","last_fullsized":"2024-09-10T20:41:49.234051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.250481+0000","last_change":"2024-09-10T20:41:49.246041+0000","last_active":"2024-09-10T20:41:49.250481+0000","last_peered":"2024-09-10T20:41:49.250481+0000","last_clean":"2024-09-10T20:41:49.250481+0000","last_became_active":"2024-09-10T20:41:49.245783+0000","last_became_peered":"2024-09-10T20:41:49.245783+0000","last_unstale":"2024-09-10T20:41:49.250481+0000","last_undegraded":"2024-09-10T20:41:49.250481+0000","last_fullsized":"2024-09-10T20:41:49.250481+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:23.096775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229857+0000","last_change":"2024-09-10T20:41:49.229857+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229857+0000","last_undegraded":"2024-09-10T20:41:49.229857+0000","last_fullsized":"2024-09-10T20:41:49.229857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238563+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238256+0000","last_became_peered":"2024-09-10T20:41:49.238256+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:16.403282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278144+0000","last_change":"2024-09-10T20:41:50.273036+0000","last_active":"2024-09-10T20:41:50.278144+0000","last_peered":"2024-09-10T20:41:50.278144+0000","last_clean":"2024-09-10T20:41:50.278144+0000","last_became_active":"2024-09-10T20:41:50.272751+0000","last_became_peered":"2024-09-10T20:41:50.272751+0000","last_unstale":"2024-09-10T20:41:50.278144+0000","last_undegraded":"2024-09-10T20:41:50.278144+0000","last_fullsized":"2024-09-10T20:41:50.278144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:48:00.876771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252771+0000","last_change":"2024-09-10T20:41:49.239816+0000","last_active":"2024-09-10T20:41:50.252771+0000","last_peered":"2024-09-10T20:41:50.252771+0000","last_clean":"2024-09-10T20:41:50.252771+0000","last_became_active":"2024-09-10T20:41:49.239509+0000","last_became_peered":"2024-09-10T20:41:49.239509+0000","last_unstale":"2024-09-10T20:41:50.252771+0000","last_undegraded":"2024-09-10T20:41:50.252771+0000","last_fullsized":"2024-09-10T20:41:50.252771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:30.409719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270329+0000","last_change":"2024-09-10T20:41:50.268159+0000","last_active":"2024-09-10T20:41:50.270329+0000","last_peered":"2024-09-10T20:41:50.270329+0000","last_clean":"2024-09-10T20:41:50.270329+0000","last_became_active":"2024-09-10T20:41:50.267522+0000","last_became_peered":"2024-09-10T20:41:50.267522+0000","last_unstale":"2024-09-10T20:41:50.270329+0000","last_undegraded":"2024-09-10T20:41:50.270329+0000","last_fullsized":"2024-09-10T20:41:50.270329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.176084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247689+0000","last_change":"2024-09-10T20:41:49.243018+0000","last_active":"2024-09-10T20:41:50.247689+0000","last_peered":"2024-09-10T20:41:50.247689+0000","last_clean":"2024-09-10T20:41:50.247689+0000","last_became_active":"2024-09-10T20:41:49.242751+0000","last_became_peered":"2024-09-10T20:41:49.242751+0000","last_unstale":"2024-09-10T20:41:50.247689+0000","last_undegraded":"2024-09-10T20:41:50.247689+0000","last_fullsized":"2024-09-10T20:41:50.247689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:47.281477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274705+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.274705+0000","last_peered":"2024-09-10T20:41:50.274705+0000","last_clean":"2024-09-10T20:41:50.274705+0000","last_became_active":"2024-09-10T20:41:50.272083+0000","last_became_peered":"2024-09-10T20:41:50.272083+0000","last_unstale":"2024-09-10T20:41:50.274705+0000","last_undegraded":"2024-09-10T20:41:50.274705+0000","last_fullsized":"2024-09-10T20:41:50.274705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242314+0000","last_change":"2024-09-10T20:41:49.235825+0000","last_active":"2024-09-10T20:41:49.242314+0000","last_peered":"2024-09-10T20:41:49.242314+0000","last_clean":"2024-09-10T20:41:49.242314+0000","last_became_active":"2024-09-10T20:41:49.235466+0000","last_became_peered":"2024-09-10T20:41:49.235466+0000","last_unstale":"2024-09-10T20:41:49.242314+0000","last_undegraded":"2024-09-10T20:41:49.242314+0000","last_fullsized":"2024-09-10T20:41:49.242314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:08.240386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229403+0000","last_change":"2024-09-10T20:41:49.229403+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229403+0000","last_undegraded":"2024-09-10T20:41:49.229403+0000","last_fullsized":"2024-09-10T20:41:49.229403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245167+0000","last_change":"2024-09-10T20:41:49.241236+0000","last_active":"2024-09-10T20:41:49.245167+0000","last_peered":"2024-09-10T20:41:49.245167+0000","last_clean":"2024-09-10T20:41:49.245167+0000","last_became_active":"2024-09-10T20:41:49.240686+0000","last_became_peered":"2024-09-10T20:41:49.240686+0000","last_unstale":"2024-09-10T20:41:49.245167+0000","last_undegraded":"2024-09-10T20:41:49.245167+0000","last_fullsized":"2024-09-10T20:41:49.245167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:40:43.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267631+0000","last_change":"2024-09-10T20:41:50.262537+0000","last_active":"2024-09-10T20:41:50.267631+0000","last_peered":"2024-09-10T20:41:50.267631+0000","last_clean":"2024-09-10T20:41:50.267631+0000","last_became_active":"2024-09-10T20:41:50.261672+0000","last_became_peered":"2024-09-10T20:41:50.261672+0000","last_unstale":"2024-09-10T20:41:50.267631+0000","last_undegraded":"2024-09-10T20:41:50.267631+0000","last_fullsized":"2024-09-10T20:41:50.267631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:57:22.115806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245187+0000","last_change":"2024-09-10T20:41:49.241419+0000","last_active":"2024-09-10T20:41:49.245187+0000","last_peered":"2024-09-10T20:41:49.245187+0000","last_clean":"2024-09-10T20:41:49.245187+0000","last_became_active":"2024-09-10T20:41:49.241125+0000","last_became_peered":"2024-09-10T20:41:49.241125+0000","last_unstale":"2024-09-10T20:41:49.245187+0000","last_undegraded":"2024-09-10T20:41:49.245187+0000","last_fullsized":"2024-09-10T20:41:49.245187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:45:50.357901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278395+0000","last_change":"2024-09-10T20:41:50.272126+0000","last_active":"2024-09-10T20:41:50.278395+0000","last_peered":"2024-09-10T20:41:50.278395+0000","last_clean":"2024-09-10T20:41:50.278395+0000","last_became_active":"2024-09-10T20:41:50.271842+0000","last_became_peered":"2024-09-10T20:41:50.271842+0000","last_unstale":"2024-09-10T20:41:50.278395+0000","last_undegraded":"2024-09-10T20:41:50.278395+0000","last_fullsized":"2024-09-10T20:41:50.278395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:17:24.548442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254957+0000","last_change":"2024-09-10T20:41:49.248639+0000","last_active":"2024-09-10T20:41:50.254957+0000","last_peered":"2024-09-10T20:41:50.254957+0000","last_clean":"2024-09-10T20:41:50.254957+0000","last_became_active":"2024-09-10T20:41:49.248352+0000","last_became_peered":"2024-09-10T20:41:49.248352+0000","last_unstale":"2024-09-10T20:41:50.254957+0000","last_undegraded":"2024-09-10T20:41:50.254957+0000","last_fullsized":"2024-09-10T20:41:50.254957+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:41.288742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228516+0000","last_change":"2024-09-10T20:41:49.228516+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228516+0000","last_undegraded":"2024-09-10T20:41:49.228516+0000","last_fullsized":"2024-09-10T20:41:49.228516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.251913+0000","last_change":"2024-09-10T20:41:49.249615+0000","last_active":"2024-09-10T20:41:50.251913+0000","last_peered":"2024-09-10T20:41:50.251913+0000","last_clean":"2024-09-10T20:41:50.251913+0000","last_became_active":"2024-09-10T20:41:49.249414+0000","last_became_peered":"2024-09-10T20:41:49.249414+0000","last_unstale":"2024-09-10T20:41:50.251913+0000","last_undegraded":"2024-09-10T20:41:50.251913+0000","last_fullsized":"2024-09-10T20:41:50.251913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:36.540914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231389+0000","last_change":"2024-09-10T20:41:49.231389+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231389+0000","last_undegraded":"2024-09-10T20:41:49.231389+0000","last_fullsized":"2024-09-10T20:41:49.231389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282862+0000","last_change":"2024-09-10T20:41:49.244860+0000","last_active":"2024-09-10T20:41:50.282862+0000","last_peered":"2024-09-10T20:41:50.282862+0000","last_clean":"2024-09-10T20:41:50.282862+0000","last_became_active":"2024-09-10T20:41:49.244532+0000","last_became_peered":"2024-09-10T20:41:49.244532+0000","last_unstale":"2024-09-10T20:41:50.282862+0000","last_undegraded":"2024-09-10T20:41:50.282862+0000","last_fullsized":"2024-09-10T20:41:50.282862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:33.273565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257492+0000","last_change":"2024-09-10T20:41:50.255850+0000","last_active":"2024-09-10T20:41:50.257492+0000","last_peered":"2024-09-10T20:41:50.257492+0000","last_clean":"2024-09-10T20:41:50.257492+0000","last_became_active":"2024-09-10T20:41:50.254971+0000","last_became_peered":"2024-09-10T20:41:50.254971+0000","last_unstale":"2024-09-10T20:41:50.257492+0000","last_undegraded":"2024-09-10T20:41:50.257492+0000","last_fullsized":"2024-09-10T20:41:50.257492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:24:15.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283814+0000","last_change":"2024-09-10T20:41:49.239536+0000","last_active":"2024-09-10T20:41:50.283814+0000","last_peered":"2024-09-10T20:41:50.283814+0000","last_clean":"2024-09-10T20:41:50.283814+0000","last_became_active":"2024-09-10T20:41:49.239253+0000","last_became_peered":"2024-09-10T20:41:49.239253+0000","last_unstale":"2024-09-10T20:41:50.283814+0000","last_undegraded":"2024-09-10T20:41:50.283814+0000","last_fullsized":"2024-09-10T20:41:50.283814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:15.552579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254349+0000","last_change":"2024-09-10T20:41:50.251514+0000","last_active":"2024-09-10T20:41:50.254349+0000","last_peered":"2024-09-10T20:41:50.254349+0000","last_clean":"2024-09-10T20:41:50.254349+0000","last_became_active":"2024-09-10T20:41:50.251241+0000","last_became_peered":"2024-09-10T20:41:50.251241+0000","last_unstale":"2024-09-10T20:41:50.254349+0000","last_undegraded":"2024-09-10T20:41:50.254349+0000","last_fullsized":"2024-09-10T20:41:50.254349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.239713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283432+0000","last_change":"2024-09-10T20:41:49.248393+0000","last_active":"2024-09-10T20:41:50.283432+0000","last_peered":"2024-09-10T20:41:50.283432+0000","last_clean":"2024-09-10T20:41:50.283432+0000","last_became_active":"2024-09-10T20:41:49.248103+0000","last_became_peered":"2024-09-10T20:41:49.248103+0000","last_unstale":"2024-09-10T20:41:50.283432+0000","last_undegraded":"2024-09-10T20:41:50.283432+0000","last_fullsized":"2024-09-10T20:41:50.283432+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:54.196476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231812+0000","last_change":"2024-09-10T20:41:49.231812+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231812+0000","last_undegraded":"2024-09-10T20:41:49.231812+0000","last_fullsized":"2024-09-10T20:41:49.231812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276182+0000","last_change":"2024-09-10T20:41:49.239383+0000","last_active":"2024-09-10T20:41:50.276182+0000","last_peered":"2024-09-10T20:41:50.276182+0000","last_clean":"2024-09-10T20:41:50.276182+0000","last_became_active":"2024-09-10T20:41:49.239097+0000","last_became_peered":"2024-09-10T20:41:49.239097+0000","last_unstale":"2024-09-10T20:41:50.276182+0000","last_undegraded":"2024-09-10T20:41:50.276182+0000","last_fullsized":"2024-09-10T20:41:50.276182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:51.420841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232592+0000","last_change":"2024-09-10T20:41:49.232592+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232592+0000","last_undegraded":"2024-09-10T20:41:49.232592+0000","last_fullsized":"2024-09-10T20:41:49.232592+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257075+0000","last_change":"2024-09-10T20:41:49.243612+0000","last_active":"2024-09-10T20:41:50.257075+0000","last_peered":"2024-09-10T20:41:50.257075+0000","last_clean":"2024-09-10T20:41:50.257075+0000","last_became_active":"2024-09-10T20:41:49.243065+0000","last_became_peered":"2024-09-10T20:41:49.243065+0000","last_unstale":"2024-09-10T20:41:50.257075+0000","last_undegraded":"2024-09-10T20:41:50.257075+0000","last_fullsized":"2024-09-10T20:41:50.257075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:20:31.645185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233707+0000","last_change":"2024-09-10T20:41:49.233707+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233707+0000","last_undegraded":"2024-09-10T20:41:49.233707+0000","last_fullsized":"2024-09-10T20:41:49.233707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252699+0000","last_change":"2024-09-10T20:41:49.241358+0000","last_active":"2024-09-10T20:41:50.252699+0000","last_peered":"2024-09-10T20:41:50.252699+0000","last_clean":"2024-09-10T20:41:50.252699+0000","last_became_active":"2024-09-10T20:41:49.241098+0000","last_became_peered":"2024-09-10T20:41:49.241098+0000","last_unstale":"2024-09-10T20:41:50.252699+0000","last_undegraded":"2024-09-10T20:41:50.252699+0000","last_fullsized":"2024-09-10T20:41:50.252699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:53:49.000467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233626+0000","last_change":"2024-09-10T20:41:49.233626+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233626+0000","last_undegraded":"2024-09-10T20:41:49.233626+0000","last_fullsized":"2024-09-10T20:41:49.233626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247275+0000","last_change":"2024-09-10T20:41:49.234909+0000","last_active":"2024-09-10T20:41:50.247275+0000","last_peered":"2024-09-10T20:41:50.247275+0000","last_clean":"2024-09-10T20:41:50.247275+0000","last_became_active":"2024-09-10T20:41:49.234598+0000","last_became_peered":"2024-09-10T20:41:49.234598+0000","last_unstale":"2024-09-10T20:41:50.247275+0000","last_undegraded":"2024-09-10T20:41:50.247275+0000","last_fullsized":"2024-09-10T20:41:50.247275+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:00.862079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229401+0000","last_change":"2024-09-10T20:41:49.229401+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229401+0000","last_undegraded":"2024-09-10T20:41:49.229401+0000","last_fullsized":"2024-09-10T20:41:49.229401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248123+0000","last_change":"2024-09-10T20:41:49.245102+0000","last_active":"2024-09-10T20:41:49.248123+0000","last_peered":"2024-09-10T20:41:49.248123+0000","last_clean":"2024-09-10T20:41:49.248123+0000","last_became_active":"2024-09-10T20:41:49.244790+0000","last_became_peered":"2024-09-10T20:41:49.244790+0000","last_unstale":"2024-09-10T20:41:49.248123+0000","last_undegraded":"2024-09-10T20:41:49.248123+0000","last_fullsized":"2024-09-10T20:41:49.248123+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:26.633006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230237+0000","last_change":"2024-09-10T20:41:49.230237+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230237+0000","last_undegraded":"2024-09-10T20:41:49.230237+0000","last_fullsized":"2024-09-10T20:41:49.230237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281880+0000","last_change":"2024-09-10T20:41:49.239083+0000","last_active":"2024-09-10T20:41:50.281880+0000","last_peered":"2024-09-10T20:41:50.281880+0000","last_clean":"2024-09-10T20:41:50.281880+0000","last_became_active":"2024-09-10T20:41:49.238741+0000","last_became_peered":"2024-09-10T20:41:49.238741+0000","last_unstale":"2024-09-10T20:41:50.281880+0000","last_undegraded":"2024-09-10T20:41:50.281880+0000","last_fullsized":"2024-09-10T20:41:50.281880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:09:56.852921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.277987+0000","last_change":"2024-09-10T20:41:50.270889+0000","last_active":"2024-09-10T20:41:50.277987+0000","last_peered":"2024-09-10T20:41:50.277987+0000","last_clean":"2024-09-10T20:41:50.277987+0000","last_became_active":"2024-09-10T20:41:50.270603+0000","last_became_peered":"2024-09-10T20:41:50.270603+0000","last_unstale":"2024-09-10T20:41:50.277987+0000","last_undegraded":"2024-09-10T20:41:50.277987+0000","last_fullsized":"2024-09-10T20:41:50.277987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:23.399016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252628+0000","last_change":"2024-09-10T20:41:49.249494+0000","last_active":"2024-09-10T20:41:50.252628+0000","last_peered":"2024-09-10T20:41:50.252628+0000","last_clean":"2024-09-10T20:41:50.252628+0000","last_became_active":"2024-09-10T20:41:49.249207+0000","last_became_peered":"2024-09-10T20:41:49.249207+0000","last_unstale":"2024-09-10T20:41:50.252628+0000","last_undegraded":"2024-09-10T20:41:50.252628+0000","last_fullsized":"2024-09-10T20:41:50.252628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:47.078899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.268605+0000","last_change":"2024-09-10T20:41:50.265954+0000","last_active":"2024-09-10T20:41:50.268605+0000","last_peered":"2024-09-10T20:41:50.268605+0000","last_clean":"2024-09-10T20:41:50.268605+0000","last_became_active":"2024-09-10T20:41:50.265644+0000","last_became_peered":"2024-09-10T20:41:50.265644+0000","last_unstale":"2024-09-10T20:41:50.268605+0000","last_undegraded":"2024-09-10T20:41:50.268605+0000","last_fullsized":"2024-09-10T20:41:50.268605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:22.562059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238772+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238312+0000","last_became_peered":"2024-09-10T20:41:49.238312+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:20.684231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278050+0000","last_change":"2024-09-10T20:41:50.265096+0000","last_active":"2024-09-10T20:41:50.278050+0000","last_peered":"2024-09-10T20:41:50.278050+0000","last_clean":"2024-09-10T20:41:50.278050+0000","last_became_active":"2024-09-10T20:41:50.264830+0000","last_became_peered":"2024-09-10T20:41:50.264830+0000","last_unstale":"2024-09-10T20:41:50.278050+0000","last_undegraded":"2024-09-10T20:41:50.278050+0000","last_fullsized":"2024-09-10T20:41:50.278050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:15.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242477+0000","last_change":"2024-09-10T20:41:49.236972+0000","last_active":"2024-09-10T20:41:49.242477+0000","last_peered":"2024-09-10T20:41:49.242477+0000","last_clean":"2024-09-10T20:41:49.242477+0000","last_became_active":"2024-09-10T20:41:49.236675+0000","last_became_peered":"2024-09-10T20:41:49.236675+0000","last_unstale":"2024-09-10T20:41:49.242477+0000","last_undegraded":"2024-09-10T20:41:49.242477+0000","last_fullsized":"2024-09-10T20:41:49.242477+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:08.249515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229009+0000","last_change":"2024-09-10T20:41:49.229009+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229009+0000","last_undegraded":"2024-09-10T20:41:49.229009+0000","last_fullsized":"2024-09-10T20:41:49.229009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.239295+0000","last_change":"2024-09-10T20:41:49.235872+0000","last_active":"2024-09-10T20:41:49.239295+0000","last_peered":"2024-09-10T20:41:49.239295+0000","last_clean":"2024-09-10T20:41:49.239295+0000","last_became_active":"2024-09-10T20:41:49.235579+0000","last_became_peered":"2024-09-10T20:41:49.235579+0000","last_unstale":"2024-09-10T20:41:49.239295+0000","last_undegraded":"2024-09-10T20:41:49.239295+0000","last_fullsized":"2024-09-10T20:41:49.239295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:32.310476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263923+0000","last_change":"2024-09-10T20:41:50.260226+0000","last_active":"2024-09-10T20:41:50.263923+0000","last_peered":"2024-09-10T20:41:50.263923+0000","last_clean":"2024-09-10T20:41:50.263923+0000","last_became_active":"2024-09-10T20:41:50.259625+0000","last_became_peered":"2024-09-10T20:41:50.259625+0000","last_unstale":"2024-09-10T20:41:50.263923+0000","last_undegraded":"2024-09-10T20:41:50.263923+0000","last_fullsized":"2024-09-10T20:41:50.263923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:38.991086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248086+0000","last_change":"2024-09-10T20:41:49.242417+0000","last_active":"2024-09-10T20:41:49.248086+0000","last_peered":"2024-09-10T20:41:49.248086+0000","last_clean":"2024-09-10T20:41:49.248086+0000","last_became_active":"2024-09-10T20:41:49.242161+0000","last_became_peered":"2024-09-10T20:41:49.242161+0000","last_unstale":"2024-09-10T20:41:49.248086+0000","last_undegraded":"2024-09-10T20:41:49.248086+0000","last_fullsized":"2024-09-10T20:41:49.248086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:13.922975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233118+0000","last_change":"2024-09-10T20:41:49.233118+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233118+0000","last_undegraded":"2024-09-10T20:41:49.233118+0000","last_fullsized":"2024-09-10T20:41:49.233118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247637+0000","last_change":"2024-09-10T20:41:49.240524+0000","last_active":"2024-09-10T20:41:50.247637+0000","last_peered":"2024-09-10T20:41:50.247637+0000","last_clean":"2024-09-10T20:41:50.247637+0000","last_became_active":"2024-09-10T20:41:49.240245+0000","last_became_peered":"2024-09-10T20:41:49.240245+0000","last_unstale":"2024-09-10T20:41:50.247637+0000","last_undegraded":"2024-09-10T20:41:50.247637+0000","last_fullsized":"2024-09-10T20:41:50.247637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:14.912119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269687+0000","last_change":"2024-09-10T20:41:50.267529+0000","last_active":"2024-09-10T20:41:50.269687+0000","last_peered":"2024-09-10T20:41:50.269687+0000","last_clean":"2024-09-10T20:41:50.269687+0000","last_became_active":"2024-09-10T20:41:50.267222+0000","last_became_peered":"2024-09-10T20:41:50.267222+0000","last_unstale":"2024-09-10T20:41:50.269687+0000","last_undegraded":"2024-09-10T20:41:50.269687+0000","last_fullsized":"2024-09-10T20:41:50.269687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:21:30.112985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.253694+0000","last_change":"2024-09-10T20:41:49.236467+0000","last_active":"2024-09-10T20:41:50.253694+0000","last_peered":"2024-09-10T20:41:50.253694+0000","last_clean":"2024-09-10T20:41:50.253694+0000","last_became_active":"2024-09-10T20:41:49.236170+0000","last_became_peered":"2024-09-10T20:41:49.236170+0000","last_unstale":"2024-09-10T20:41:50.253694+0000","last_undegraded":"2024-09-10T20:41:50.253694+0000","last_fullsized":"2024-09-10T20:41:50.253694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:12.514693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229765+0000","last_change":"2024-09-10T20:41:49.229765+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229765+0000","last_undegraded":"2024-09-10T20:41:49.229765+0000","last_fullsized":"2024-09-10T20:41:49.229765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.280647+0000","last_change":"2024-09-10T20:41:49.249357+0000","last_active":"2024-09-10T20:41:50.280647+0000","last_peered":"2024-09-10T20:41:50.280647+0000","last_clean":"2024-09-10T20:41:50.280647+0000","last_became_active":"2024-09-10T20:41:49.248955+0000","last_became_peered":"2024-09-10T20:41:49.248955+0000","last_unstale":"2024-09-10T20:41:50.280647+0000","last_undegraded":"2024-09-10T20:41:50.280647+0000","last_fullsized":"2024-09-10T20:41:50.280647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:26.862222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.250227+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:41:50.250227+0000","last_peered":"2024-09-10T20:41:50.250227+0000","last_clean":"2024-09-10T20:41:50.250227+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:41:50.250227+0000","last_undegraded":"2024-09-10T20:41:50.250227+0000","last_fullsized":"2024-09-10T20:41:50.250227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509490,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":819200,"data_stored":337112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509489,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509489,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1400832,"data_stored":927354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509489,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":811008,"data_stored":337056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542194,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":782336,"data_stored":323944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1388544,"data_stored":920579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:41:54.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:41:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.787+0000 7f3eb2782640 1 -- 172.21.15.28:0/3022108454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac15de70 msgr2=0x7f3eac0d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.787+0000 7f3eb2782640 1 --2- 172.21.15.28:0/3022108454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac15de70 0x7f3eac0d2b90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009830 tx=0x7f3e9c02a4a0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.787+0000 7f3eb2782640 1 -- 172.21.15.28:0/3022108454 shutdown_connections 2024-09-10T20:41:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.787+0000 7f3eb2782640 1 --2- 172.21.15.28:0/3022108454 >> v2:172.21.15.159:3300/0 conn(0x7f3eac0cf780 0x7f3eac0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.787+0000 7f3eb2782640 1 --2- 172.21.15.28:0/3022108454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac15de70 0x7f3eac0d2b90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.787+0000 7f3eb2782640 1 -- 172.21.15.28:0/3022108454 >> 172.21.15.28:0/3022108454 conn(0x7f3eac0d3fb0 msgr2=0x7f3eac0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.787+0000 7f3eb2782640 1 -- 172.21.15.28:0/3022108454 shutdown_connections 2024-09-10T20:41:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.788+0000 7f3eb2782640 1 -- 172.21.15.28:0/3022108454 wait complete. 2024-09-10T20:41:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.789+0000 7f3eb2782640 1 Processor -- start 2024-09-10T20:41:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.789+0000 7f3eb2782640 1 -- start start 2024-09-10T20:41:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb2782640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eac0cf780 0x7f3eac160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb2782640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3eac15de70 0x7f3eac15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb2782640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac161b50 0x7f3eac0a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb2782640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3eac162350 con 0x7f3eac0cf780 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb2782640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3eac1621d0 con 0x7f3eac15de70 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb2782640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3eac1624d0 con 0x7f3eac161b50 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eac0cf780 0x7f3eac160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eac0cf780 0x7f3eac160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58364/0 (socket says 172.21.15.28:58364) 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 -- 172.21.15.28:0/151574701 learned_addr learned my addr 172.21.15.28:0/151574701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eab7fe640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3eac15de70 0x7f3eac15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb0d1f640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac161b50 0x7f3eac0a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 -- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac161b50 msgr2=0x7f3eac0a1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac161b50 0x7f3eac0a1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 -- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3eac15de70 msgr2=0x7f3eac15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3eac15de70 0x7f3eac15eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 -- 172.21.15.28:0/151574701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eac160030 con 0x7f3eac0cf780 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eb0d1f640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac161b50 0x7f3eac0a1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3eabfff640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eac0cf780 0x7f3eac160890 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009950 tx=0x7f3e9c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.790+0000 7f3ea8ff9640 1 -- 172.21.15.28:0/151574701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c005c90 con 0x7f3eac0cf780 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.791+0000 7f3ea8ff9640 1 -- 172.21.15.28:0/151574701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e9c005020 con 0x7f3eac0cf780 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.791+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eac15f790 con 0x7f3eac0cf780 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.791+0000 7f3ea8ff9640 1 -- 172.21.15.28:0/151574701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c005450 con 0x7f3eac0cf780 2024-09-10T20:41:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.791+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eac15fd50 con 0x7f3eac0cf780 2024-09-10T20:41:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.791+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eac0cfbc0 con 0x7f3eac0cf780 2024-09-10T20:41:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.792+0000 7f3ea8ff9640 1 -- 172.21.15.28:0/151574701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f3e9c0317c0 con 0x7f3eac0cf780 2024-09-10T20:41:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.793+0000 7f3ea8ff9640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3e98078300 0x7f3e9807a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.793+0000 7f3ea8ff9640 1 -- 172.21.15.28:0/151574701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f3e9c0bb5e0 con 0x7f3eac0cf780 2024-09-10T20:41:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.793+0000 7f3eab7fe640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3e98078300 0x7f3e9807a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.793+0000 7f3eab7fe640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3e98078300 0x7f3e9807a7c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0cd890 tx=0x7f3ea0005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.797+0000 7f3ea8ff9640 1 -- 172.21.15.28:0/151574701 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3e9c040030 con 0x7f3eac0cf780 2024-09-10T20:41:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.976+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3eac0b9f30 con 0x7f3e98078300 2024-09-10T20:41:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.981+0000 7f3ea8ff9640 1 -- 172.21.15.28:0/151574701 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249747 (secure 0 0 0) 0x7f3eac0b9f30 con 0x7f3e98078300 2024-09-10T20:41:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:54.988 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:41:54.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3e98078300 msgr2=0x7f3e9807a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3e98078300 0x7f3e9807a7c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0cd890 tx=0x7f3ea0005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eac0cf780 msgr2=0x7f3eac160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eac0cf780 0x7f3eac160890 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009950 tx=0x7f3e9c003c10 comp rx=0 tx=0).stop 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 shutdown_connections 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f3e98078300 0x7f3e9807a7c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3eac161b50 0x7f3eac0a1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3eac15de70 0x7f3eac15eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 --2- 172.21.15.28:0/151574701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eac0cf780 0x7f3eac160890 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 >> 172.21.15.28:0/151574701 conn(0x7f3eac0d3fb0 msgr2=0x7f3eac0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 shutdown_connections 2024-09-10T20:41:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:54.984+0000 7f3eb2782640 1 -- 172.21.15.28:0/151574701 wait complete. 2024-09-10T20:41:54.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-10T20:41:54.029282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591042,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":132,"num_write_kb":1631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244464,"kb_used_data":7408,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730256,"statfs":{"total":773094113280,"available":772843782144,"internally_reserved":0,"allocated":7585792,"data_stored":3836678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.805315"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264924+0000","last_change":"2024-09-10T20:41:50.262634+0000","last_active":"2024-09-10T20:41:50.264924+0000","last_peered":"2024-09-10T20:41:50.264924+0000","last_clean":"2024-09-10T20:41:50.264924+0000","last_became_active":"2024-09-10T20:41:50.262341+0000","last_became_peered":"2024-09-10T20:41:50.262341+0000","last_unstale":"2024-09-10T20:41:50.264924+0000","last_undegraded":"2024-09-10T20:41:50.264924+0000","last_fullsized":"2024-09-10T20:41:50.264924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:09:16.358398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270227+0000","last_change":"2024-09-10T20:41:50.268290+0000","last_active":"2024-09-10T20:41:50.270227+0000","last_peered":"2024-09-10T20:41:50.270227+0000","last_clean":"2024-09-10T20:41:50.270227+0000","last_became_active":"2024-09-10T20:41:50.268020+0000","last_became_peered":"2024-09-10T20:41:50.268020+0000","last_unstale":"2024-09-10T20:41:50.270227+0000","last_undegraded":"2024-09-10T20:41:50.270227+0000","last_fullsized":"2024-09-10T20:41:50.270227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:44.595286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232984+0000","last_change":"2024-09-10T20:41:49.232984+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232984+0000","last_undegraded":"2024-09-10T20:41:49.232984+0000","last_fullsized":"2024-09-10T20:41:49.232984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233187+0000","last_change":"2024-09-10T20:41:49.233187+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233187+0000","last_undegraded":"2024-09-10T20:41:49.233187+0000","last_fullsized":"2024-09-10T20:41:49.233187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230680+0000","last_change":"2024-09-10T20:41:49.230680+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230680+0000","last_undegraded":"2024-09-10T20:41:49.230680+0000","last_fullsized":"2024-09-10T20:41:49.230680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267661+0000","last_change":"2024-09-10T20:41:50.262868+0000","last_active":"2024-09-10T20:41:50.267661+0000","last_peered":"2024-09-10T20:41:50.267661+0000","last_clean":"2024-09-10T20:41:50.267661+0000","last_became_active":"2024-09-10T20:41:50.262182+0000","last_became_peered":"2024-09-10T20:41:50.262182+0000","last_unstale":"2024-09-10T20:41:50.267661+0000","last_undegraded":"2024-09-10T20:41:50.267661+0000","last_fullsized":"2024-09-10T20:41:50.267661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:10.294469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278278+0000","last_change":"2024-09-10T20:41:50.269379+0000","last_active":"2024-09-10T20:41:50.278278+0000","last_peered":"2024-09-10T20:41:50.278278+0000","last_clean":"2024-09-10T20:41:50.278278+0000","last_became_active":"2024-09-10T20:41:50.268955+0000","last_became_peered":"2024-09-10T20:41:50.268955+0000","last_unstale":"2024-09-10T20:41:50.278278+0000","last_undegraded":"2024-09-10T20:41:50.278278+0000","last_fullsized":"2024-09-10T20:41:50.278278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:55.335452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264003+0000","last_change":"2024-09-10T20:41:50.261207+0000","last_active":"2024-09-10T20:41:50.264003+0000","last_peered":"2024-09-10T20:41:50.264003+0000","last_clean":"2024-09-10T20:41:50.264003+0000","last_became_active":"2024-09-10T20:41:50.260866+0000","last_became_peered":"2024-09-10T20:41:50.260866+0000","last_unstale":"2024-09-10T20:41:50.264003+0000","last_undegraded":"2024-09-10T20:41:50.264003+0000","last_fullsized":"2024-09-10T20:41:50.264003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:48.716401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232777+0000","last_change":"2024-09-10T20:41:49.232777+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232777+0000","last_undegraded":"2024-09-10T20:41:49.232777+0000","last_fullsized":"2024-09-10T20:41:49.232777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278111+0000","last_change":"2024-09-10T20:41:50.270275+0000","last_active":"2024-09-10T20:41:50.278111+0000","last_peered":"2024-09-10T20:41:50.278111+0000","last_clean":"2024-09-10T20:41:50.278111+0000","last_became_active":"2024-09-10T20:41:50.269980+0000","last_became_peered":"2024-09-10T20:41:50.269980+0000","last_unstale":"2024-09-10T20:41:50.278111+0000","last_undegraded":"2024-09-10T20:41:50.278111+0000","last_fullsized":"2024-09-10T20:41:50.278111+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:12.357126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263972+0000","last_change":"2024-09-10T20:41:50.260076+0000","last_active":"2024-09-10T20:41:50.263972+0000","last_peered":"2024-09-10T20:41:50.263972+0000","last_clean":"2024-09-10T20:41:50.263972+0000","last_became_active":"2024-09-10T20:41:50.259752+0000","last_became_peered":"2024-09-10T20:41:50.259752+0000","last_unstale":"2024-09-10T20:41:50.263972+0000","last_undegraded":"2024-09-10T20:41:50.263972+0000","last_fullsized":"2024-09-10T20:41:50.263972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:08.756456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228409+0000","last_change":"2024-09-10T20:41:49.228409+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228409+0000","last_undegraded":"2024-09-10T20:41:49.228409+0000","last_fullsized":"2024-09-10T20:41:49.228409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232209+0000","last_change":"2024-09-10T20:41:49.232209+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232209+0000","last_undegraded":"2024-09-10T20:41:49.232209+0000","last_fullsized":"2024-09-10T20:41:49.232209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229613+0000","last_change":"2024-09-10T20:41:49.229613+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229613+0000","last_undegraded":"2024-09-10T20:41:49.229613+0000","last_fullsized":"2024-09-10T20:41:49.229613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230334+0000","last_change":"2024-09-10T20:41:49.230334+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230334+0000","last_undegraded":"2024-09-10T20:41:49.230334+0000","last_fullsized":"2024-09-10T20:41:49.230334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.262988+0000","last_change":"2024-09-10T20:41:50.259865+0000","last_active":"2024-09-10T20:41:50.262988+0000","last_peered":"2024-09-10T20:41:50.262988+0000","last_clean":"2024-09-10T20:41:50.262988+0000","last_became_active":"2024-09-10T20:41:50.259492+0000","last_became_peered":"2024-09-10T20:41:50.259492+0000","last_unstale":"2024-09-10T20:41:50.262988+0000","last_undegraded":"2024-09-10T20:41:50.262988+0000","last_fullsized":"2024-09-10T20:41:50.262988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:32:07.918850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228536+0000","last_change":"2024-09-10T20:41:49.228536+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228536+0000","last_undegraded":"2024-09-10T20:41:49.228536+0000","last_fullsized":"2024-09-10T20:41:49.228536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230656+0000","last_change":"2024-09-10T20:41:49.230656+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230656+0000","last_undegraded":"2024-09-10T20:41:49.230656+0000","last_fullsized":"2024-09-10T20:41:49.230656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269422+0000","last_change":"2024-09-10T20:41:50.264629+0000","last_active":"2024-09-10T20:41:50.269422+0000","last_peered":"2024-09-10T20:41:50.269422+0000","last_clean":"2024-09-10T20:41:50.269422+0000","last_became_active":"2024-09-10T20:41:50.264345+0000","last_became_peered":"2024-09-10T20:41:50.264345+0000","last_unstale":"2024-09-10T20:41:50.269422+0000","last_undegraded":"2024-09-10T20:41:50.269422+0000","last_fullsized":"2024-09-10T20:41:50.269422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:33:47.084724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228954+0000","last_change":"2024-09-10T20:41:49.228954+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228954+0000","last_undegraded":"2024-09-10T20:41:49.228954+0000","last_fullsized":"2024-09-10T20:41:49.228954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257652+0000","last_change":"2024-09-10T20:41:50.255889+0000","last_active":"2024-09-10T20:41:50.257652+0000","last_peered":"2024-09-10T20:41:50.257652+0000","last_clean":"2024-09-10T20:41:50.257652+0000","last_became_active":"2024-09-10T20:41:50.255572+0000","last_became_peered":"2024-09-10T20:41:50.255572+0000","last_unstale":"2024-09-10T20:41:50.257652+0000","last_undegraded":"2024-09-10T20:41:50.257652+0000","last_fullsized":"2024-09-10T20:41:50.257652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:38.194068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274208+0000","last_change":"2024-09-10T20:41:50.265660+0000","last_active":"2024-09-10T20:41:50.274208+0000","last_peered":"2024-09-10T20:41:50.274208+0000","last_clean":"2024-09-10T20:41:50.274208+0000","last_became_active":"2024-09-10T20:41:50.265364+0000","last_became_peered":"2024-09-10T20:41:50.265364+0000","last_unstale":"2024-09-10T20:41:50.274208+0000","last_undegraded":"2024-09-10T20:41:50.274208+0000","last_fullsized":"2024-09-10T20:41:50.274208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:52.067493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269174+0000","last_change":"2024-09-10T20:41:50.263991+0000","last_active":"2024-09-10T20:41:50.269174+0000","last_peered":"2024-09-10T20:41:50.269174+0000","last_clean":"2024-09-10T20:41:50.269174+0000","last_became_active":"2024-09-10T20:41:50.263690+0000","last_became_peered":"2024-09-10T20:41:50.263690+0000","last_unstale":"2024-09-10T20:41:50.269174+0000","last_undegraded":"2024-09-10T20:41:50.269174+0000","last_fullsized":"2024-09-10T20:41:50.269174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:08.679923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264022+0000","last_change":"2024-09-10T20:41:50.261825+0000","last_active":"2024-09-10T20:41:50.264022+0000","last_peered":"2024-09-10T20:41:50.264022+0000","last_clean":"2024-09-10T20:41:50.264022+0000","last_became_active":"2024-09-10T20:41:50.261516+0000","last_became_peered":"2024-09-10T20:41:50.261516+0000","last_unstale":"2024-09-10T20:41:50.264022+0000","last_undegraded":"2024-09-10T20:41:50.264022+0000","last_fullsized":"2024-09-10T20:41:50.264022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:26.740959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278057+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.278057+0000","last_peered":"2024-09-10T20:41:50.278057+0000","last_clean":"2024-09-10T20:41:50.278057+0000","last_became_active":"2024-09-10T20:41:50.272144+0000","last_became_peered":"2024-09-10T20:41:50.272144+0000","last_unstale":"2024-09-10T20:41:50.278057+0000","last_undegraded":"2024-09-10T20:41:50.278057+0000","last_fullsized":"2024-09-10T20:41:50.278057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:05.044942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.275708+0000","last_change":"2024-09-10T20:41:50.273908+0000","last_active":"2024-09-10T20:41:50.275708+0000","last_peered":"2024-09-10T20:41:50.275708+0000","last_clean":"2024-09-10T20:41:50.275708+0000","last_became_active":"2024-09-10T20:41:50.273619+0000","last_became_peered":"2024-09-10T20:41:50.273619+0000","last_unstale":"2024-09-10T20:41:50.275708+0000","last_undegraded":"2024-09-10T20:41:50.275708+0000","last_fullsized":"2024-09-10T20:41:50.275708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:57.474072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229786+0000","last_change":"2024-09-10T20:41:49.229786+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229786+0000","last_undegraded":"2024-09-10T20:41:49.229786+0000","last_fullsized":"2024-09-10T20:41:49.229786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269719+0000","last_change":"2024-09-10T20:41:50.267808+0000","last_active":"2024-09-10T20:41:50.269719+0000","last_peered":"2024-09-10T20:41:50.269719+0000","last_clean":"2024-09-10T20:41:50.269719+0000","last_became_active":"2024-09-10T20:41:50.267481+0000","last_became_peered":"2024-09-10T20:41:50.267481+0000","last_unstale":"2024-09-10T20:41:50.269719+0000","last_undegraded":"2024-09-10T20:41:50.269719+0000","last_fullsized":"2024-09-10T20:41:50.269719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:18:49.925720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231774+0000","last_change":"2024-09-10T20:41:49.231774+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231774+0000","last_undegraded":"2024-09-10T20:41:49.231774+0000","last_fullsized":"2024-09-10T20:41:49.231774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.265287+0000","last_change":"2024-09-10T20:41:50.263690+0000","last_active":"2024-09-10T20:41:50.265287+0000","last_peered":"2024-09-10T20:41:50.265287+0000","last_clean":"2024-09-10T20:41:50.265287+0000","last_became_active":"2024-09-10T20:41:50.263407+0000","last_became_peered":"2024-09-10T20:41:50.263407+0000","last_unstale":"2024-09-10T20:41:50.265287+0000","last_undegraded":"2024-09-10T20:41:50.265287+0000","last_fullsized":"2024-09-10T20:41:50.265287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:16.841293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269657+0000","last_change":"2024-09-10T20:41:50.266413+0000","last_active":"2024-09-10T20:41:50.269657+0000","last_peered":"2024-09-10T20:41:50.269657+0000","last_clean":"2024-09-10T20:41:50.269657+0000","last_became_active":"2024-09-10T20:41:50.266150+0000","last_became_peered":"2024-09-10T20:41:50.266150+0000","last_unstale":"2024-09-10T20:41:50.269657+0000","last_undegraded":"2024-09-10T20:41:50.269657+0000","last_fullsized":"2024-09-10T20:41:50.269657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:20.996250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270272+0000","last_change":"2024-09-10T20:41:50.267692+0000","last_active":"2024-09-10T20:41:50.270272+0000","last_peered":"2024-09-10T20:41:50.270272+0000","last_clean":"2024-09-10T20:41:50.270272+0000","last_became_active":"2024-09-10T20:41:50.267181+0000","last_became_peered":"2024-09-10T20:41:50.267181+0000","last_unstale":"2024-09-10T20:41:50.270272+0000","last_undegraded":"2024-09-10T20:41:50.270272+0000","last_fullsized":"2024-09-10T20:41:50.270272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:14.639786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263938+0000","last_change":"2024-09-10T20:41:50.260450+0000","last_active":"2024-09-10T20:41:50.263938+0000","last_peered":"2024-09-10T20:41:50.263938+0000","last_clean":"2024-09-10T20:41:50.263938+0000","last_became_active":"2024-09-10T20:41:50.260093+0000","last_became_peered":"2024-09-10T20:41:50.260093+0000","last_unstale":"2024-09-10T20:41:50.263938+0000","last_undegraded":"2024-09-10T20:41:50.263938+0000","last_fullsized":"2024-09-10T20:41:50.263938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:16.794344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281382+0000","last_change":"2024-09-10T20:41:49.244556+0000","last_active":"2024-09-10T20:41:50.281382+0000","last_peered":"2024-09-10T20:41:50.281382+0000","last_clean":"2024-09-10T20:41:50.281382+0000","last_became_active":"2024-09-10T20:41:49.244277+0000","last_became_peered":"2024-09-10T20:41:49.244277+0000","last_unstale":"2024-09-10T20:41:50.281382+0000","last_undegraded":"2024-09-10T20:41:50.281382+0000","last_fullsized":"2024-09-10T20:41:50.281382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:24.745124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232330+0000","last_change":"2024-09-10T20:41:49.232330+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232330+0000","last_undegraded":"2024-09-10T20:41:49.232330+0000","last_fullsized":"2024-09-10T20:41:49.232330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242361+0000","last_change":"2024-09-10T20:41:49.236963+0000","last_active":"2024-09-10T20:41:49.242361+0000","last_peered":"2024-09-10T20:41:49.242361+0000","last_clean":"2024-09-10T20:41:49.242361+0000","last_became_active":"2024-09-10T20:41:49.236654+0000","last_became_peered":"2024-09-10T20:41:49.236654+0000","last_unstale":"2024-09-10T20:41:49.242361+0000","last_undegraded":"2024-09-10T20:41:49.242361+0000","last_fullsized":"2024-09-10T20:41:49.242361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:48.203284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278810+0000","last_change":"2024-09-10T20:41:50.277395+0000","last_active":"2024-09-10T20:41:50.278810+0000","last_peered":"2024-09-10T20:41:50.278810+0000","last_clean":"2024-09-10T20:41:50.278810+0000","last_became_active":"2024-09-10T20:41:50.277101+0000","last_became_peered":"2024-09-10T20:41:50.277101+0000","last_unstale":"2024-09-10T20:41:50.278810+0000","last_undegraded":"2024-09-10T20:41:50.278810+0000","last_fullsized":"2024-09-10T20:41:50.278810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:47.799998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282196+0000","last_change":"2024-09-10T20:41:49.238213+0000","last_active":"2024-09-10T20:41:50.282196+0000","last_peered":"2024-09-10T20:41:50.282196+0000","last_clean":"2024-09-10T20:41:50.282196+0000","last_became_active":"2024-09-10T20:41:49.237949+0000","last_became_peered":"2024-09-10T20:41:49.237949+0000","last_unstale":"2024-09-10T20:41:50.282196+0000","last_undegraded":"2024-09-10T20:41:50.282196+0000","last_fullsized":"2024-09-10T20:41:50.282196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:45:22.532346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.266258+0000","last_change":"2024-09-10T20:41:50.262357+0000","last_active":"2024-09-10T20:41:50.266258+0000","last_peered":"2024-09-10T20:41:50.266258+0000","last_clean":"2024-09-10T20:41:50.266258+0000","last_became_active":"2024-09-10T20:41:50.262081+0000","last_became_peered":"2024-09-10T20:41:50.262081+0000","last_unstale":"2024-09-10T20:41:50.266258+0000","last_undegraded":"2024-09-10T20:41:50.266258+0000","last_fullsized":"2024-09-10T20:41:50.266258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:44.966942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.279900+0000","last_change":"2024-09-10T20:41:49.247040+0000","last_active":"2024-09-10T20:41:50.279900+0000","last_peered":"2024-09-10T20:41:50.279900+0000","last_clean":"2024-09-10T20:41:50.279900+0000","last_became_active":"2024-09-10T20:41:49.246731+0000","last_became_peered":"2024-09-10T20:41:49.246731+0000","last_unstale":"2024-09-10T20:41:50.279900+0000","last_undegraded":"2024-09-10T20:41:50.279900+0000","last_fullsized":"2024-09-10T20:41:50.279900+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:37.379165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263903+0000","last_change":"2024-09-10T20:41:50.261513+0000","last_active":"2024-09-10T20:41:50.263903+0000","last_peered":"2024-09-10T20:41:50.263903+0000","last_clean":"2024-09-10T20:41:50.263903+0000","last_became_active":"2024-09-10T20:41:50.261203+0000","last_became_peered":"2024-09-10T20:41:50.261203+0000","last_unstale":"2024-09-10T20:41:50.263903+0000","last_undegraded":"2024-09-10T20:41:50.263903+0000","last_fullsized":"2024-09-10T20:41:50.263903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:39.097791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242512+0000","last_change":"2024-09-10T20:41:49.237350+0000","last_active":"2024-09-10T20:41:49.242512+0000","last_peered":"2024-09-10T20:41:49.242512+0000","last_clean":"2024-09-10T20:41:49.242512+0000","last_became_active":"2024-09-10T20:41:49.237086+0000","last_became_peered":"2024-09-10T20:41:49.237086+0000","last_unstale":"2024-09-10T20:41:49.242512+0000","last_undegraded":"2024-09-10T20:41:49.242512+0000","last_fullsized":"2024-09-10T20:41:49.242512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:13.283338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276471+0000","last_change":"2024-09-10T20:41:50.275485+0000","last_active":"2024-09-10T20:41:50.276471+0000","last_peered":"2024-09-10T20:41:50.276471+0000","last_clean":"2024-09-10T20:41:50.276471+0000","last_became_active":"2024-09-10T20:41:50.275179+0000","last_became_peered":"2024-09-10T20:41:50.275179+0000","last_unstale":"2024-09-10T20:41:50.276471+0000","last_undegraded":"2024-09-10T20:41:50.276471+0000","last_fullsized":"2024-09-10T20:41:50.276471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:34.056711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278185+0000","last_change":"2024-09-10T20:41:49.245982+0000","last_active":"2024-09-10T20:41:50.278185+0000","last_peered":"2024-09-10T20:41:50.278185+0000","last_clean":"2024-09-10T20:41:50.278185+0000","last_became_active":"2024-09-10T20:41:49.245669+0000","last_became_peered":"2024-09-10T20:41:49.245669+0000","last_unstale":"2024-09-10T20:41:50.278185+0000","last_undegraded":"2024-09-10T20:41:50.278185+0000","last_fullsized":"2024-09-10T20:41:50.278185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:21:05.620306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.234051+0000","last_change":"2024-09-10T20:41:49.234051+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.234051+0000","last_undegraded":"2024-09-10T20:41:49.234051+0000","last_fullsized":"2024-09-10T20:41:49.234051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.250481+0000","last_change":"2024-09-10T20:41:49.246041+0000","last_active":"2024-09-10T20:41:49.250481+0000","last_peered":"2024-09-10T20:41:49.250481+0000","last_clean":"2024-09-10T20:41:49.250481+0000","last_became_active":"2024-09-10T20:41:49.245783+0000","last_became_peered":"2024-09-10T20:41:49.245783+0000","last_unstale":"2024-09-10T20:41:49.250481+0000","last_undegraded":"2024-09-10T20:41:49.250481+0000","last_fullsized":"2024-09-10T20:41:49.250481+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:23.096775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229857+0000","last_change":"2024-09-10T20:41:49.229857+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229857+0000","last_undegraded":"2024-09-10T20:41:49.229857+0000","last_fullsized":"2024-09-10T20:41:49.229857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238563+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238256+0000","last_became_peered":"2024-09-10T20:41:49.238256+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:16.403282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278144+0000","last_change":"2024-09-10T20:41:50.273036+0000","last_active":"2024-09-10T20:41:50.278144+0000","last_peered":"2024-09-10T20:41:50.278144+0000","last_clean":"2024-09-10T20:41:50.278144+0000","last_became_active":"2024-09-10T20:41:50.272751+0000","last_became_peered":"2024-09-10T20:41:50.272751+0000","last_unstale":"2024-09-10T20:41:50.278144+0000","last_undegraded":"2024-09-10T20:41:50.278144+0000","last_fullsized":"2024-09-10T20:41:50.278144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:48:00.876771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252771+0000","last_change":"2024-09-10T20:41:49.239816+0000","last_active":"2024-09-10T20:41:50.252771+0000","last_peered":"2024-09-10T20:41:50.252771+0000","last_clean":"2024-09-10T20:41:50.252771+0000","last_became_active":"2024-09-10T20:41:49.239509+0000","last_became_peered":"2024-09-10T20:41:49.239509+0000","last_unstale":"2024-09-10T20:41:50.252771+0000","last_undegraded":"2024-09-10T20:41:50.252771+0000","last_fullsized":"2024-09-10T20:41:50.252771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:30.409719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270329+0000","last_change":"2024-09-10T20:41:50.268159+0000","last_active":"2024-09-10T20:41:50.270329+0000","last_peered":"2024-09-10T20:41:50.270329+0000","last_clean":"2024-09-10T20:41:50.270329+0000","last_became_active":"2024-09-10T20:41:50.267522+0000","last_became_peered":"2024-09-10T20:41:50.267522+0000","last_unstale":"2024-09-10T20:41:50.270329+0000","last_undegraded":"2024-09-10T20:41:50.270329+0000","last_fullsized":"2024-09-10T20:41:50.270329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.176084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247689+0000","last_change":"2024-09-10T20:41:49.243018+0000","last_active":"2024-09-10T20:41:50.247689+0000","last_peered":"2024-09-10T20:41:50.247689+0000","last_clean":"2024-09-10T20:41:50.247689+0000","last_became_active":"2024-09-10T20:41:49.242751+0000","last_became_peered":"2024-09-10T20:41:49.242751+0000","last_unstale":"2024-09-10T20:41:50.247689+0000","last_undegraded":"2024-09-10T20:41:50.247689+0000","last_fullsized":"2024-09-10T20:41:50.247689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:47.281477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274705+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.274705+0000","last_peered":"2024-09-10T20:41:50.274705+0000","last_clean":"2024-09-10T20:41:50.274705+0000","last_became_active":"2024-09-10T20:41:50.272083+0000","last_became_peered":"2024-09-10T20:41:50.272083+0000","last_unstale":"2024-09-10T20:41:50.274705+0000","last_undegraded":"2024-09-10T20:41:50.274705+0000","last_fullsized":"2024-09-10T20:41:50.274705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242314+0000","last_change":"2024-09-10T20:41:49.235825+0000","last_active":"2024-09-10T20:41:49.242314+0000","last_peered":"2024-09-10T20:41:49.242314+0000","last_clean":"2024-09-10T20:41:49.242314+0000","last_became_active":"2024-09-10T20:41:49.235466+0000","last_became_peered":"2024-09-10T20:41:49.235466+0000","last_unstale":"2024-09-10T20:41:49.242314+0000","last_undegraded":"2024-09-10T20:41:49.242314+0000","last_fullsized":"2024-09-10T20:41:49.242314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:08.240386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229403+0000","last_change":"2024-09-10T20:41:49.229403+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229403+0000","last_undegraded":"2024-09-10T20:41:49.229403+0000","last_fullsized":"2024-09-10T20:41:49.229403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245167+0000","last_change":"2024-09-10T20:41:49.241236+0000","last_active":"2024-09-10T20:41:49.245167+0000","last_peered":"2024-09-10T20:41:49.245167+0000","last_clean":"2024-09-10T20:41:49.245167+0000","last_became_active":"2024-09-10T20:41:49.240686+0000","last_became_peered":"2024-09-10T20:41:49.240686+0000","last_unstale":"2024-09-10T20:41:49.245167+0000","last_undegraded":"2024-09-10T20:41:49.245167+0000","last_fullsized":"2024-09-10T20:41:49.245167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:40:43.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267631+0000","last_change":"2024-09-10T20:41:50.262537+0000","last_active":"2024-09-10T20:41:50.267631+0000","last_peered":"2024-09-10T20:41:50.267631+0000","last_clean":"2024-09-10T20:41:50.267631+0000","last_became_active":"2024-09-10T20:41:50.261672+0000","last_became_peered":"2024-09-10T20:41:50.261672+0000","last_unstale":"2024-09-10T20:41:50.267631+0000","last_undegraded":"2024-09-10T20:41:50.267631+0000","last_fullsized":"2024-09-10T20:41:50.267631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:57:22.115806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245187+0000","last_change":"2024-09-10T20:41:49.241419+0000","last_active":"2024-09-10T20:41:49.245187+0000","last_peered":"2024-09-10T20:41:49.245187+0000","last_clean":"2024-09-10T20:41:49.245187+0000","last_became_active":"2024-09-10T20:41:49.241125+0000","last_became_peered":"2024-09-10T20:41:49.241125+0000","last_unstale":"2024-09-10T20:41:49.245187+0000","last_undegraded":"2024-09-10T20:41:49.245187+0000","last_fullsized":"2024-09-10T20:41:49.245187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:45:50.357901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278395+0000","last_change":"2024-09-10T20:41:50.272126+0000","last_active":"2024-09-10T20:41:50.278395+0000","last_peered":"2024-09-10T20:41:50.278395+0000","last_clean":"2024-09-10T20:41:50.278395+0000","last_became_active":"2024-09-10T20:41:50.271842+0000","last_became_peered":"2024-09-10T20:41:50.271842+0000","last_unstale":"2024-09-10T20:41:50.278395+0000","last_undegraded":"2024-09-10T20:41:50.278395+0000","last_fullsized":"2024-09-10T20:41:50.278395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:17:24.548442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254957+0000","last_change":"2024-09-10T20:41:49.248639+0000","last_active":"2024-09-10T20:41:50.254957+0000","last_peered":"2024-09-10T20:41:50.254957+0000","last_clean":"2024-09-10T20:41:50.254957+0000","last_became_active":"2024-09-10T20:41:49.248352+0000","last_became_peered":"2024-09-10T20:41:49.248352+0000","last_unstale":"2024-09-10T20:41:50.254957+0000","last_undegraded":"2024-09-10T20:41:50.254957+0000","last_fullsized":"2024-09-10T20:41:50.254957+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:41.288742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228516+0000","last_change":"2024-09-10T20:41:49.228516+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228516+0000","last_undegraded":"2024-09-10T20:41:49.228516+0000","last_fullsized":"2024-09-10T20:41:49.228516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.251913+0000","last_change":"2024-09-10T20:41:49.249615+0000","last_active":"2024-09-10T20:41:50.251913+0000","last_peered":"2024-09-10T20:41:50.251913+0000","last_clean":"2024-09-10T20:41:50.251913+0000","last_became_active":"2024-09-10T20:41:49.249414+0000","last_became_peered":"2024-09-10T20:41:49.249414+0000","last_unstale":"2024-09-10T20:41:50.251913+0000","last_undegraded":"2024-09-10T20:41:50.251913+0000","last_fullsized":"2024-09-10T20:41:50.251913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:36.540914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231389+0000","last_change":"2024-09-10T20:41:49.231389+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231389+0000","last_undegraded":"2024-09-10T20:41:49.231389+0000","last_fullsized":"2024-09-10T20:41:49.231389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282862+0000","last_change":"2024-09-10T20:41:49.244860+0000","last_active":"2024-09-10T20:41:50.282862+0000","last_peered":"2024-09-10T20:41:50.282862+0000","last_clean":"2024-09-10T20:41:50.282862+0000","last_became_active":"2024-09-10T20:41:49.244532+0000","last_became_peered":"2024-09-10T20:41:49.244532+0000","last_unstale":"2024-09-10T20:41:50.282862+0000","last_undegraded":"2024-09-10T20:41:50.282862+0000","last_fullsized":"2024-09-10T20:41:50.282862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:33.273565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257492+0000","last_change":"2024-09-10T20:41:50.255850+0000","last_active":"2024-09-10T20:41:50.257492+0000","last_peered":"2024-09-10T20:41:50.257492+0000","last_clean":"2024-09-10T20:41:50.257492+0000","last_became_active":"2024-09-10T20:41:50.254971+0000","last_became_peered":"2024-09-10T20:41:50.254971+0000","last_unstale":"2024-09-10T20:41:50.257492+0000","last_undegraded":"2024-09-10T20:41:50.257492+0000","last_fullsized":"2024-09-10T20:41:50.257492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:24:15.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283814+0000","last_change":"2024-09-10T20:41:49.239536+0000","last_active":"2024-09-10T20:41:50.283814+0000","last_peered":"2024-09-10T20:41:50.283814+0000","last_clean":"2024-09-10T20:41:50.283814+0000","last_became_active":"2024-09-10T20:41:49.239253+0000","last_became_peered":"2024-09-10T20:41:49.239253+0000","last_unstale":"2024-09-10T20:41:50.283814+0000","last_undegraded":"2024-09-10T20:41:50.283814+0000","last_fullsized":"2024-09-10T20:41:50.283814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:15.552579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254349+0000","last_change":"2024-09-10T20:41:50.251514+0000","last_active":"2024-09-10T20:41:50.254349+0000","last_peered":"2024-09-10T20:41:50.254349+0000","last_clean":"2024-09-10T20:41:50.254349+0000","last_became_active":"2024-09-10T20:41:50.251241+0000","last_became_peered":"2024-09-10T20:41:50.251241+0000","last_unstale":"2024-09-10T20:41:50.254349+0000","last_undegraded":"2024-09-10T20:41:50.254349+0000","last_fullsized":"2024-09-10T20:41:50.254349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.239713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283432+0000","last_change":"2024-09-10T20:41:49.248393+0000","last_active":"2024-09-10T20:41:50.283432+0000","last_peered":"2024-09-10T20:41:50.283432+0000","last_clean":"2024-09-10T20:41:50.283432+0000","last_became_active":"2024-09-10T20:41:49.248103+0000","last_became_peered":"2024-09-10T20:41:49.248103+0000","last_unstale":"2024-09-10T20:41:50.283432+0000","last_undegraded":"2024-09-10T20:41:50.283432+0000","last_fullsized":"2024-09-10T20:41:50.283432+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:54.196476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231812+0000","last_change":"2024-09-10T20:41:49.231812+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231812+0000","last_undegraded":"2024-09-10T20:41:49.231812+0000","last_fullsized":"2024-09-10T20:41:49.231812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276182+0000","last_change":"2024-09-10T20:41:49.239383+0000","last_active":"2024-09-10T20:41:50.276182+0000","last_peered":"2024-09-10T20:41:50.276182+0000","last_clean":"2024-09-10T20:41:50.276182+0000","last_became_active":"2024-09-10T20:41:49.239097+0000","last_became_peered":"2024-09-10T20:41:49.239097+0000","last_unstale":"2024-09-10T20:41:50.276182+0000","last_undegraded":"2024-09-10T20:41:50.276182+0000","last_fullsized":"2024-09-10T20:41:50.276182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:51.420841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232592+0000","last_change":"2024-09-10T20:41:49.232592+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232592+0000","last_undegraded":"2024-09-10T20:41:49.232592+0000","last_fullsized":"2024-09-10T20:41:49.232592+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257075+0000","last_change":"2024-09-10T20:41:49.243612+0000","last_active":"2024-09-10T20:41:50.257075+0000","last_peered":"2024-09-10T20:41:50.257075+0000","last_clean":"2024-09-10T20:41:50.257075+0000","last_became_active":"2024-09-10T20:41:49.243065+0000","last_became_peered":"2024-09-10T20:41:49.243065+0000","last_unstale":"2024-09-10T20:41:50.257075+0000","last_undegraded":"2024-09-10T20:41:50.257075+0000","last_fullsized":"2024-09-10T20:41:50.257075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:20:31.645185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233707+0000","last_change":"2024-09-10T20:41:49.233707+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233707+0000","last_undegraded":"2024-09-10T20:41:49.233707+0000","last_fullsized":"2024-09-10T20:41:49.233707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252699+0000","last_change":"2024-09-10T20:41:49.241358+0000","last_active":"2024-09-10T20:41:50.252699+0000","last_peered":"2024-09-10T20:41:50.252699+0000","last_clean":"2024-09-10T20:41:50.252699+0000","last_became_active":"2024-09-10T20:41:49.241098+0000","last_became_peered":"2024-09-10T20:41:49.241098+0000","last_unstale":"2024-09-10T20:41:50.252699+0000","last_undegraded":"2024-09-10T20:41:50.252699+0000","last_fullsized":"2024-09-10T20:41:50.252699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:53:49.000467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233626+0000","last_change":"2024-09-10T20:41:49.233626+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233626+0000","last_undegraded":"2024-09-10T20:41:49.233626+0000","last_fullsized":"2024-09-10T20:41:49.233626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247275+0000","last_change":"2024-09-10T20:41:49.234909+0000","last_active":"2024-09-10T20:41:50.247275+0000","last_peered":"2024-09-10T20:41:50.247275+0000","last_clean":"2024-09-10T20:41:50.247275+0000","last_became_active":"2024-09-10T20:41:49.234598+0000","last_became_peered":"2024-09-10T20:41:49.234598+0000","last_unstale":"2024-09-10T20:41:50.247275+0000","last_undegraded":"2024-09-10T20:41:50.247275+0000","last_fullsized":"2024-09-10T20:41:50.247275+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:00.862079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229401+0000","last_change":"2024-09-10T20:41:49.229401+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229401+0000","last_undegraded":"2024-09-10T20:41:49.229401+0000","last_fullsized":"2024-09-10T20:41:49.229401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248123+0000","last_change":"2024-09-10T20:41:49.245102+0000","last_active":"2024-09-10T20:41:49.248123+0000","last_peered":"2024-09-10T20:41:49.248123+0000","last_clean":"2024-09-10T20:41:49.248123+0000","last_became_active":"2024-09-10T20:41:49.244790+0000","last_became_peered":"2024-09-10T20:41:49.244790+0000","last_unstale":"2024-09-10T20:41:49.248123+0000","last_undegraded":"2024-09-10T20:41:49.248123+0000","last_fullsized":"2024-09-10T20:41:49.248123+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:26.633006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230237+0000","last_change":"2024-09-10T20:41:49.230237+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230237+0000","last_undegraded":"2024-09-10T20:41:49.230237+0000","last_fullsized":"2024-09-10T20:41:49.230237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281880+0000","last_change":"2024-09-10T20:41:49.239083+0000","last_active":"2024-09-10T20:41:50.281880+0000","last_peered":"2024-09-10T20:41:50.281880+0000","last_clean":"2024-09-10T20:41:50.281880+0000","last_became_active":"2024-09-10T20:41:49.238741+0000","last_became_peered":"2024-09-10T20:41:49.238741+0000","last_unstale":"2024-09-10T20:41:50.281880+0000","last_undegraded":"2024-09-10T20:41:50.281880+0000","last_fullsized":"2024-09-10T20:41:50.281880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:09:56.852921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.277987+0000","last_change":"2024-09-10T20:41:50.270889+0000","last_active":"2024-09-10T20:41:50.277987+0000","last_peered":"2024-09-10T20:41:50.277987+0000","last_clean":"2024-09-10T20:41:50.277987+0000","last_became_active":"2024-09-10T20:41:50.270603+0000","last_became_peered":"2024-09-10T20:41:50.270603+0000","last_unstale":"2024-09-10T20:41:50.277987+0000","last_undegraded":"2024-09-10T20:41:50.277987+0000","last_fullsized":"2024-09-10T20:41:50.277987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:23.399016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252628+0000","last_change":"2024-09-10T20:41:49.249494+0000","last_active":"2024-09-10T20:41:50.252628+0000","last_peered":"2024-09-10T20:41:50.252628+0000","last_clean":"2024-09-10T20:41:50.252628+0000","last_became_active":"2024-09-10T20:41:49.249207+0000","last_became_peered":"2024-09-10T20:41:49.249207+0000","last_unstale":"2024-09-10T20:41:50.252628+0000","last_undegraded":"2024-09-10T20:41:50.252628+0000","last_fullsized":"2024-09-10T20:41:50.252628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:47.078899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.268605+0000","last_change":"2024-09-10T20:41:50.265954+0000","last_active":"2024-09-10T20:41:50.268605+0000","last_peered":"2024-09-10T20:41:50.268605+0000","last_clean":"2024-09-10T20:41:50.268605+0000","last_became_active":"2024-09-10T20:41:50.265644+0000","last_became_peered":"2024-09-10T20:41:50.265644+0000","last_unstale":"2024-09-10T20:41:50.268605+0000","last_undegraded":"2024-09-10T20:41:50.268605+0000","last_fullsized":"2024-09-10T20:41:50.268605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:22.562059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238772+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238312+0000","last_became_peered":"2024-09-10T20:41:49.238312+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:20.684231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278050+0000","last_change":"2024-09-10T20:41:50.265096+0000","last_active":"2024-09-10T20:41:50.278050+0000","last_peered":"2024-09-10T20:41:50.278050+0000","last_clean":"2024-09-10T20:41:50.278050+0000","last_became_active":"2024-09-10T20:41:50.264830+0000","last_became_peered":"2024-09-10T20:41:50.264830+0000","last_unstale":"2024-09-10T20:41:50.278050+0000","last_undegraded":"2024-09-10T20:41:50.278050+0000","last_fullsized":"2024-09-10T20:41:50.278050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:15.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242477+0000","last_change":"2024-09-10T20:41:49.236972+0000","last_active":"2024-09-10T20:41:49.242477+0000","last_peered":"2024-09-10T20:41:49.242477+0000","last_clean":"2024-09-10T20:41:49.242477+0000","last_became_active":"2024-09-10T20:41:49.236675+0000","last_became_peered":"2024-09-10T20:41:49.236675+0000","last_unstale":"2024-09-10T20:41:49.242477+0000","last_undegraded":"2024-09-10T20:41:49.242477+0000","last_fullsized":"2024-09-10T20:41:49.242477+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:08.249515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229009+0000","last_change":"2024-09-10T20:41:49.229009+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229009+0000","last_undegraded":"2024-09-10T20:41:49.229009+0000","last_fullsized":"2024-09-10T20:41:49.229009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.239295+0000","last_change":"2024-09-10T20:41:49.235872+0000","last_active":"2024-09-10T20:41:49.239295+0000","last_peered":"2024-09-10T20:41:49.239295+0000","last_clean":"2024-09-10T20:41:49.239295+0000","last_became_active":"2024-09-10T20:41:49.235579+0000","last_became_peered":"2024-09-10T20:41:49.235579+0000","last_unstale":"2024-09-10T20:41:49.239295+0000","last_undegraded":"2024-09-10T20:41:49.239295+0000","last_fullsized":"2024-09-10T20:41:49.239295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:32.310476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263923+0000","last_change":"2024-09-10T20:41:50.260226+0000","last_active":"2024-09-10T20:41:50.263923+0000","last_peered":"2024-09-10T20:41:50.263923+0000","last_clean":"2024-09-10T20:41:50.263923+0000","last_became_active":"2024-09-10T20:41:50.259625+0000","last_became_peered":"2024-09-10T20:41:50.259625+0000","last_unstale":"2024-09-10T20:41:50.263923+0000","last_undegraded":"2024-09-10T20:41:50.263923+0000","last_fullsized":"2024-09-10T20:41:50.263923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:38.991086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248086+0000","last_change":"2024-09-10T20:41:49.242417+0000","last_active":"2024-09-10T20:41:49.248086+0000","last_peered":"2024-09-10T20:41:49.248086+0000","last_clean":"2024-09-10T20:41:49.248086+0000","last_became_active":"2024-09-10T20:41:49.242161+0000","last_became_peered":"2024-09-10T20:41:49.242161+0000","last_unstale":"2024-09-10T20:41:49.248086+0000","last_undegraded":"2024-09-10T20:41:49.248086+0000","last_fullsized":"2024-09-10T20:41:49.248086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:13.922975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233118+0000","last_change":"2024-09-10T20:41:49.233118+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233118+0000","last_undegraded":"2024-09-10T20:41:49.233118+0000","last_fullsized":"2024-09-10T20:41:49.233118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247637+0000","last_change":"2024-09-10T20:41:49.240524+0000","last_active":"2024-09-10T20:41:50.247637+0000","last_peered":"2024-09-10T20:41:50.247637+0000","last_clean":"2024-09-10T20:41:50.247637+0000","last_became_active":"2024-09-10T20:41:49.240245+0000","last_became_peered":"2024-09-10T20:41:49.240245+0000","last_unstale":"2024-09-10T20:41:50.247637+0000","last_undegraded":"2024-09-10T20:41:50.247637+0000","last_fullsized":"2024-09-10T20:41:50.247637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:14.912119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269687+0000","last_change":"2024-09-10T20:41:50.267529+0000","last_active":"2024-09-10T20:41:50.269687+0000","last_peered":"2024-09-10T20:41:50.269687+0000","last_clean":"2024-09-10T20:41:50.269687+0000","last_became_active":"2024-09-10T20:41:50.267222+0000","last_became_peered":"2024-09-10T20:41:50.267222+0000","last_unstale":"2024-09-10T20:41:50.269687+0000","last_undegraded":"2024-09-10T20:41:50.269687+0000","last_fullsized":"2024-09-10T20:41:50.269687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:21:30.112985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.253694+0000","last_change":"2024-09-10T20:41:49.236467+0000","last_active":"2024-09-10T20:41:50.253694+0000","last_peered":"2024-09-10T20:41:50.253694+0000","last_clean":"2024-09-10T20:41:50.253694+0000","last_became_active":"2024-09-10T20:41:49.236170+0000","last_became_peered":"2024-09-10T20:41:49.236170+0000","last_unstale":"2024-09-10T20:41:50.253694+0000","last_undegraded":"2024-09-10T20:41:50.253694+0000","last_fullsized":"2024-09-10T20:41:50.253694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:12.514693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229765+0000","last_change":"2024-09-10T20:41:49.229765+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229765+0000","last_undegraded":"2024-09-10T20:41:49.229765+0000","last_fullsized":"2024-09-10T20:41:49.229765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.280647+0000","last_change":"2024-09-10T20:41:49.249357+0000","last_active":"2024-09-10T20:41:50.280647+0000","last_peered":"2024-09-10T20:41:50.280647+0000","last_clean":"2024-09-10T20:41:50.280647+0000","last_became_active":"2024-09-10T20:41:49.248955+0000","last_became_peered":"2024-09-10T20:41:49.248955+0000","last_unstale":"2024-09-10T20:41:50.280647+0000","last_undegraded":"2024-09-10T20:41:50.280647+0000","last_fullsized":"2024-09-10T20:41:50.280647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:26.862222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.250227+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:41:50.250227+0000","last_peered":"2024-09-10T20:41:50.250227+0000","last_clean":"2024-09-10T20:41:50.250227+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:41:50.250227+0000","last_undegraded":"2024-09-10T20:41:50.250227+0000","last_fullsized":"2024-09-10T20:41:50.250227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509490,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":819200,"data_stored":337112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509489,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509489,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1400832,"data_stored":927354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509489,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":811008,"data_stored":337056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542194,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":782336,"data_stored":323944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1388544,"data_stored":920579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:41:55.002 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3d is not active+clean 2024-09-10T20:41:55.002 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.232984+0000', 'last_change': '2024-09-10T20:41:49.232984+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.232984+0000', 'last_undegraded': '2024-09-10T20:41:49.232984+0000', 'last_fullsized': '2024-09-10T20:41:49.232984+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.002 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3c is not active+clean 2024-09-10T20:41:55.002 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.233187+0000', 'last_change': '2024-09-10T20:41:49.233187+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.233187+0000', 'last_undegraded': '2024-09-10T20:41:49.233187+0000', 'last_fullsized': '2024-09-10T20:41:49.233187+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3b is not active+clean 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.230680+0000', 'last_change': '2024-09-10T20:41:49.230680+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.230680+0000', 'last_undegraded': '2024-09-10T20:41:49.230680+0000', 'last_fullsized': '2024-09-10T20:41:49.230680+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:PG 11.37 is not active+clean 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.232777+0000', 'last_change': '2024-09-10T20:41:49.232777+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.232777+0000', 'last_undegraded': '2024-09-10T20:41:49.232777+0000', 'last_fullsized': '2024-09-10T20:41:49.232777+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:PG 11.34 is not active+clean 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.228409+0000', 'last_change': '2024-09-10T20:41:49.228409+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.228409+0000', 'last_undegraded': '2024-09-10T20:41:49.228409+0000', 'last_fullsized': '2024-09-10T20:41:49.228409+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:PG 11.33 is not active+clean 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.232209+0000', 'last_change': '2024-09-10T20:41:49.232209+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.232209+0000', 'last_undegraded': '2024-09-10T20:41:49.232209+0000', 'last_fullsized': '2024-09-10T20:41:49.232209+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:PG 11.32 is not active+clean 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.229613+0000', 'last_change': '2024-09-10T20:41:49.229613+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.229613+0000', 'last_undegraded': '2024-09-10T20:41:49.229613+0000', 'last_fullsized': '2024-09-10T20:41:49.229613+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:PG 11.31 is not active+clean 2024-09-10T20:41:55.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.230334+0000', 'last_change': '2024-09-10T20:41:49.230334+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.230334+0000', 'last_undegraded': '2024-09-10T20:41:49.230334+0000', 'last_fullsized': '2024-09-10T20:41:49.230334+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2f is not active+clean 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.228536+0000', 'last_change': '2024-09-10T20:41:49.228536+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.228536+0000', 'last_undegraded': '2024-09-10T20:41:49.228536+0000', 'last_fullsized': '2024-09-10T20:41:49.228536+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2e is not active+clean 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.230656+0000', 'last_change': '2024-09-10T20:41:49.230656+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.230656+0000', 'last_undegraded': '2024-09-10T20:41:49.230656+0000', 'last_fullsized': '2024-09-10T20:41:49.230656+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2c is not active+clean 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.228954+0000', 'last_change': '2024-09-10T20:41:49.228954+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.228954+0000', 'last_undegraded': '2024-09-10T20:41:49.228954+0000', 'last_fullsized': '2024-09-10T20:41:49.228954+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:PG 11.25 is not active+clean 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.229786+0000', 'last_change': '2024-09-10T20:41:49.229786+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.229786+0000', 'last_undegraded': '2024-09-10T20:41:49.229786+0000', 'last_fullsized': '2024-09-10T20:41:49.229786+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:PG 11.23 is not active+clean 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.231774+0000', 'last_change': '2024-09-10T20:41:49.231774+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.231774+0000', 'last_undegraded': '2024-09-10T20:41:49.231774+0000', 'last_fullsized': '2024-09-10T20:41:49.231774+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1f is not active+clean 2024-09-10T20:41:55.004 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.232330+0000', 'last_change': '2024-09-10T20:41:49.232330+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.232330+0000', 'last_undegraded': '2024-09-10T20:41:49.232330+0000', 'last_fullsized': '2024-09-10T20:41:49.232330+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:PG 11.18 is not active+clean 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.234051+0000', 'last_change': '2024-09-10T20:41:49.234051+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.234051+0000', 'last_undegraded': '2024-09-10T20:41:49.234051+0000', 'last_fullsized': '2024-09-10T20:41:49.234051+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:PG 11.19 is not active+clean 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.229857+0000', 'last_change': '2024-09-10T20:41:49.229857+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.229857+0000', 'last_undegraded': '2024-09-10T20:41:49.229857+0000', 'last_fullsized': '2024-09-10T20:41:49.229857+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:PG 11.15 is not active+clean 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.229403+0000', 'last_change': '2024-09-10T20:41:49.229403+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.229403+0000', 'last_undegraded': '2024-09-10T20:41:49.229403+0000', 'last_fullsized': '2024-09-10T20:41:49.229403+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:PG 11.10 is not active+clean 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.228516+0000', 'last_change': '2024-09-10T20:41:49.228516+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.228516+0000', 'last_undegraded': '2024-09-10T20:41:49.228516+0000', 'last_fullsized': '2024-09-10T20:41:49.228516+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:PG 11.11 is not active+clean 2024-09-10T20:41:55.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.231389+0000', 'last_change': '2024-09-10T20:41:49.231389+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.231389+0000', 'last_undegraded': '2024-09-10T20:41:49.231389+0000', 'last_fullsized': '2024-09-10T20:41:49.231389+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:PG 11.c is not active+clean 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.231812+0000', 'last_change': '2024-09-10T20:41:49.231812+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.231812+0000', 'last_undegraded': '2024-09-10T20:41:49.231812+0000', 'last_fullsized': '2024-09-10T20:41:49.231812+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:PG 11.d is not active+clean 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.232592+0000', 'last_change': '2024-09-10T20:41:49.232592+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.232592+0000', 'last_undegraded': '2024-09-10T20:41:49.232592+0000', 'last_fullsized': '2024-09-10T20:41:49.232592+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:PG 11.b is not active+clean 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.233707+0000', 'last_change': '2024-09-10T20:41:49.233707+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.233707+0000', 'last_undegraded': '2024-09-10T20:41:49.233707+0000', 'last_fullsized': '2024-09-10T20:41:49.233707+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:PG 11.8 is not active+clean 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.233626+0000', 'last_change': '2024-09-10T20:41:49.233626+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.233626+0000', 'last_undegraded': '2024-09-10T20:41:49.233626+0000', 'last_fullsized': '2024-09-10T20:41:49.233626+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:PG 11.9 is not active+clean 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.229401+0000', 'last_change': '2024-09-10T20:41:49.229401+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.229401+0000', 'last_undegraded': '2024-09-10T20:41:49.229401+0000', 'last_fullsized': '2024-09-10T20:41:49.229401+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:PG 11.6 is not active+clean 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.230237+0000', 'last_change': '2024-09-10T20:41:49.230237+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.230237+0000', 'last_undegraded': '2024-09-10T20:41:49.230237+0000', 'last_fullsized': '2024-09-10T20:41:49.230237+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.006 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2 is not active+clean 2024-09-10T20:41:55.007 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.229009+0000', 'last_change': '2024-09-10T20:41:49.229009+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.229009+0000', 'last_undegraded': '2024-09-10T20:41:49.229009+0000', 'last_fullsized': '2024-09-10T20:41:49.229009+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.007 INFO:tasks.ceph.ceph_manager.ceph:PG 11.0 is not active+clean 2024-09-10T20:41:55.007 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.233118+0000', 'last_change': '2024-09-10T20:41:49.233118+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.233118+0000', 'last_undegraded': '2024-09-10T20:41:49.233118+0000', 'last_fullsized': '2024-09-10T20:41:49.233118+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-10T20:41:55.007 INFO:tasks.ceph.ceph_manager.ceph:PG 11.a is not active+clean 2024-09-10T20:41:55.007 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-09-10T20:41:49.229765+0000', 'last_change': '2024-09-10T20:41:49.229765+0000', 'last_active': '2024-09-10T20:41:49.221972+0000', 'last_peered': '2024-09-10T20:41:49.221972+0000', 'last_clean': '2024-09-10T20:41:49.221972+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-10T20:41:49.229765+0000', 'last_undegraded': '2024-09-10T20:41:49.229765+0000', 'last_fullsized': '2024-09-10T20:41:49.229765+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'last_clean_scrub_stamp': '2024-09-10T20:41:49.221972+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-10T20:41:55.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T20:41:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.140+0000 7f7ba4de2640 1 -- 172.21.15.28:0/524010183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c004d40 msgr2=0x7f7b8c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.140+0000 7f7ba4de2640 1 --2- 172.21.15.28:0/524010183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c004d40 0x7f7b8c0051c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009ed0 tx=0x7f7b8c005830 comp rx=0 tx=0).stop 2024-09-10T20:41:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.140+0000 7f7b9ed9d640 1 -- 172.21.15.28:0/524010183 reap_dead start 2024-09-10T20:41:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.140+0000 7f7ba4de2640 1 -- 172.21.15.28:0/524010183 shutdown_connections 2024-09-10T20:41:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.140+0000 7f7ba4de2640 1 -- 172.21.15.28:0/524010183 >> 172.21.15.28:0/524010183 conn(0x7f7ba006ccc0 msgr2=0x7f7ba006d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.140+0000 7f7ba4de2640 1 -- 172.21.15.28:0/524010183 shutdown_connections 2024-09-10T20:41:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.140+0000 7f7ba4de2640 1 -- 172.21.15.28:0/524010183 wait complete. 2024-09-10T20:41:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 Processor -- start 2024-09-10T20:41:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 -- start start 2024-09-10T20:41:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ba00aec80 0x7f7ba01f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0150ef0 0x7f7ba006f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba014b9e0 0x7f7ba014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba00b2e80 con 0x7f7ba0150ef0 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba00d07d0 con 0x7f7ba00aec80 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.142+0000 7f7ba4de2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7ba00b3000 con 0x7f7ba014b9e0 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0150ef0 0x7f7ba006f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0150ef0 0x7f7ba006f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58382/0 (socket says 172.21.15.28:58382) 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 -- 172.21.15.28:0/866698634 learned_addr learned my addr 172.21.15.28:0/866698634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b9ed9d640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba014b9e0 0x7f7ba014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b9e59c640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ba00aec80 0x7f7ba01f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 -- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba014b9e0 msgr2=0x7f7ba014be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba014b9e0 0x7f7ba014be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 -- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ba00aec80 msgr2=0x7f7ba01f5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ba00aec80 0x7f7ba01f5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 -- 172.21.15.28:0/866698634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba006eb90 con 0x7f7ba0150ef0 2024-09-10T20:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b9ed9d640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba014b9e0 0x7f7ba014be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b9e59c640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ba00aec80 0x7f7ba01f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b97fff640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0150ef0 0x7f7ba006f950 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800e5c0 tx=0x7f7b8800ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b96ffd640 1 -- 172.21.15.28:0/866698634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88015070 con 0x7f7ba0150ef0 2024-09-10T20:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b96ffd640 1 -- 172.21.15.28:0/866698634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b88010980 con 0x7f7ba0150ef0 2024-09-10T20:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7b96ffd640 1 -- 172.21.15.28:0/866698634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88010dd0 con 0x7f7ba0150ef0 2024-09-10T20:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba006ee70 con 0x7f7ba0150ef0 2024-09-10T20:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.143+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba014c7b0 con 0x7f7ba0150ef0 2024-09-10T20:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.144+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ba0072680 con 0x7f7ba0150ef0 2024-09-10T20:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.145+0000 7f7b96ffd640 1 -- 172.21.15.28:0/866698634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f7b88011460 con 0x7f7ba0150ef0 2024-09-10T20:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.145+0000 7f7b96ffd640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7b800782d0 0x7f7b8007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.145+0000 7f7b96ffd640 1 -- 172.21.15.28:0/866698634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f7b8809aad0 con 0x7f7ba0150ef0 2024-09-10T20:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.146+0000 7f7b9e59c640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7b800782d0 0x7f7b8007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.146+0000 7f7b9e59c640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7b800782d0 0x7f7b8007a790 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7b90000c00 tx=0x7f7b90006b10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.150+0000 7f7b96ffd640 1 -- 172.21.15.28:0/866698634 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7b88063040 con 0x7f7ba0150ef0 2024-09-10T20:41:55.213 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-10T20:41:55.210+0000 7f4702e05640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:41:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.408+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f7ba015bde0 con 0x7f7ba0150ef0 2024-09-10T20:41:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.409+0000 7f7b96ffd640 1 -- 172.21.15.28:0/866698634 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1559 (secure 0 0 0) 0x7f7b88062940 con 0x7f7ba0150ef0 2024-09-10T20:41:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:55.413 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":247,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":68,"num_osds":8,"num_up_osds":8,"osd_up_since":1726000680,"num_in_osds":8,"osd_in_since":1726000670,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":69},{"state_name":"creating+peering","count":28}],"num_pgs":97,"num_pools":3,"num_objects":15,"data_bytes":591042,"bytes_used":250331136,"bytes_avail":772843782144,"bytes_total":773094113280,"inactive_pgs_ratio":0.28865978121757507,"write_bytes_sec":529,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":44,"btime":"2024-09-10T20:41:53:275679+0000","id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"c","status":"up:active","gid":6138}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-10T20:41:38.025727+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T20:41:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7b800782d0 msgr2=0x7f7b8007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7b800782d0 0x7f7b8007a790 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7b90000c00 tx=0x7f7b90006b10 comp rx=0 tx=0).stop 2024-09-10T20:41:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0150ef0 msgr2=0x7f7ba006f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0150ef0 0x7f7ba006f950 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800e5c0 tx=0x7f7b8800ea80 comp rx=0 tx=0).stop 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 shutdown_connections 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f7b800782d0 0x7f7b8007a790 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba014b9e0 0x7f7ba014be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0150ef0 0x7f7ba006f950 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 --2- 172.21.15.28:0/866698634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ba00aec80 0x7f7ba01f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 >> 172.21.15.28:0/866698634 conn(0x7f7ba006ccc0 msgr2=0x7f7ba015d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 shutdown_connections 2024-09-10T20:41:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.411+0000 7f7ba4de2640 1 -- 172.21.15.28:0/866698634 wait complete. 2024-09-10T20:41:55.426 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T20:41:55.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:41:55.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 -- 172.21.15.28:0/1981784182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b4070280 msgr2=0x7f47b4070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 --2- 172.21.15.28:0/1981784182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b4070280 0x7f47b4070680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f47ac005fd0 tx=0x7f47ac02aa90 comp rx=0 tx=0).stop 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 -- 172.21.15.28:0/1981784182 shutdown_connections 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 --2- 172.21.15.28:0/1981784182 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b4070280 0x7f47b4070680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 --2- 172.21.15.28:0/1981784182 >> v2:172.21.15.159:3300/0 conn(0x7f47b40d0c80 0x7f47b415bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 -- 172.21.15.28:0/1981784182 >> 172.21.15.28:0/1981784182 conn(0x7f47b406dbf0 msgr2=0x7f47b406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 -- 172.21.15.28:0/1981784182 shutdown_connections 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.527+0000 7f47b9857640 1 -- 172.21.15.28:0/1981784182 wait complete. 2024-09-10T20:41:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.529+0000 7f47b9857640 1 Processor -- start 2024-09-10T20:41:55.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.529+0000 7f47b9857640 1 -- start start 2024-09-10T20:41:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.529+0000 7f47b9857640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47b4070280 0x7f47b41f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b9857640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b40d0c80 0x7f47b41f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b9857640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b40d1060 0x7f47b4150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b9857640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47b415edf0 con 0x7f47b40d1060 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b9857640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f47b40b54a0 con 0x7f47b4070280 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b9857640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f47b415ef70 con 0x7f47b40d0c80 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47b4070280 0x7f47b41f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b40d1060 0x7f47b4150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b2ffd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47b4070280 0x7f47b41f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:43308/0 (socket says 172.21.15.28:43308) 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b2ffd640 1 -- 172.21.15.28:0/3972672368 learned_addr learned my addr 172.21.15.28:0/3972672368 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b27fc640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b40d0c80 0x7f47b41f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b37fe640 1 -- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b40d0c80 msgr2=0x7f47b41f3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b37fe640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b40d0c80 0x7f47b41f3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b37fe640 1 -- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47b4070280 msgr2=0x7f47b41f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b37fe640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47b4070280 0x7f47b41f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b37fe640 1 -- 172.21.15.28:0/3972672368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b414e140 con 0x7f47b40d1060 2024-09-10T20:41:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b2ffd640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47b4070280 0x7f47b41f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:41:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.530+0000 7f47b37fe640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b40d1060 0x7f47b4150960 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f47ac032000 tx=0x7f47ac003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.531+0000 7f479bfff640 1 -- 172.21.15.28:0/3972672368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac046070 con 0x7f47b40d1060 2024-09-10T20:41:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.531+0000 7f479bfff640 1 -- 172.21.15.28:0/3972672368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47ac003e00 con 0x7f47b40d1060 2024-09-10T20:41:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.531+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b414e420 con 0x7f47b40d1060 2024-09-10T20:41:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.531+0000 7f479bfff640 1 -- 172.21.15.28:0/3972672368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0044b0 con 0x7f47b40d1060 2024-09-10T20:41:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.531+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b41f33e0 con 0x7f47b40d1060 2024-09-10T20:41:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.532+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b4070ed0 con 0x7f47b40d1060 2024-09-10T20:41:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.532+0000 7f479bfff640 1 -- 172.21.15.28:0/3972672368 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f47ac004650 con 0x7f47b40d1060 2024-09-10T20:41:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.533+0000 7f479bfff640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f47a00782d0 0x7f47a007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.533+0000 7f479bfff640 1 -- 172.21.15.28:0/3972672368 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f47ac0bb3d0 con 0x7f47b40d1060 2024-09-10T20:41:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.533+0000 7f47b2ffd640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f47a00782d0 0x7f47a007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.534+0000 7f47b2ffd640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f47a00782d0 0x7f47a007a790 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f47b415d4e0 tx=0x7f47a4005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.537+0000 7f479bfff640 1 -- 172.21.15.28:0/3972672368 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f47ac032030 con 0x7f47b40d1060 2024-09-10T20:41:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.719+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f47b40d0a60 con 0x7f47a00782d0 2024-09-10T20:41:55.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.724+0000 7f479bfff640 1 -- 172.21.15.28:0/3972672368 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249747 (secure 0 0 0) 0x7f47b40d0a60 con 0x7f47a00782d0 2024-09-10T20:41:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f47a00782d0 msgr2=0x7f47a007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f47a00782d0 0x7f47a007a790 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f47b415d4e0 tx=0x7f47a4005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b40d1060 msgr2=0x7f47b4150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b40d1060 0x7f47b4150960 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f47ac032000 tx=0x7f47ac003c10 comp rx=0 tx=0).stop 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 shutdown_connections 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f47a00782d0 0x7f47a007a790 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47b40d1060 0x7f47b4150960 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47b40d0c80 0x7f47b41f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 --2- 172.21.15.28:0/3972672368 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47b4070280 0x7f47b41f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 >> 172.21.15.28:0/3972672368 conn(0x7f47b406dbf0 msgr2=0x7f47b4152210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 shutdown_connections 2024-09-10T20:41:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:55.727+0000 7f47b9857640 1 -- 172.21.15.28:0/3972672368 wait complete. 2024-09-10T20:41:55.742 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-10T20:41:54.029282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591042,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":132,"num_write_kb":1631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244464,"kb_used_data":7408,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730256,"statfs":{"total":773094113280,"available":772843782144,"internally_reserved":0,"allocated":7585792,"data_stored":3836678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.805315"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264924+0000","last_change":"2024-09-10T20:41:50.262634+0000","last_active":"2024-09-10T20:41:50.264924+0000","last_peered":"2024-09-10T20:41:50.264924+0000","last_clean":"2024-09-10T20:41:50.264924+0000","last_became_active":"2024-09-10T20:41:50.262341+0000","last_became_peered":"2024-09-10T20:41:50.262341+0000","last_unstale":"2024-09-10T20:41:50.264924+0000","last_undegraded":"2024-09-10T20:41:50.264924+0000","last_fullsized":"2024-09-10T20:41:50.264924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:09:16.358398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270227+0000","last_change":"2024-09-10T20:41:50.268290+0000","last_active":"2024-09-10T20:41:50.270227+0000","last_peered":"2024-09-10T20:41:50.270227+0000","last_clean":"2024-09-10T20:41:50.270227+0000","last_became_active":"2024-09-10T20:41:50.268020+0000","last_became_peered":"2024-09-10T20:41:50.268020+0000","last_unstale":"2024-09-10T20:41:50.270227+0000","last_undegraded":"2024-09-10T20:41:50.270227+0000","last_fullsized":"2024-09-10T20:41:50.270227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:44.595286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232984+0000","last_change":"2024-09-10T20:41:49.232984+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232984+0000","last_undegraded":"2024-09-10T20:41:49.232984+0000","last_fullsized":"2024-09-10T20:41:49.232984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233187+0000","last_change":"2024-09-10T20:41:49.233187+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233187+0000","last_undegraded":"2024-09-10T20:41:49.233187+0000","last_fullsized":"2024-09-10T20:41:49.233187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230680+0000","last_change":"2024-09-10T20:41:49.230680+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230680+0000","last_undegraded":"2024-09-10T20:41:49.230680+0000","last_fullsized":"2024-09-10T20:41:49.230680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267661+0000","last_change":"2024-09-10T20:41:50.262868+0000","last_active":"2024-09-10T20:41:50.267661+0000","last_peered":"2024-09-10T20:41:50.267661+0000","last_clean":"2024-09-10T20:41:50.267661+0000","last_became_active":"2024-09-10T20:41:50.262182+0000","last_became_peered":"2024-09-10T20:41:50.262182+0000","last_unstale":"2024-09-10T20:41:50.267661+0000","last_undegraded":"2024-09-10T20:41:50.267661+0000","last_fullsized":"2024-09-10T20:41:50.267661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:10.294469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278278+0000","last_change":"2024-09-10T20:41:50.269379+0000","last_active":"2024-09-10T20:41:50.278278+0000","last_peered":"2024-09-10T20:41:50.278278+0000","last_clean":"2024-09-10T20:41:50.278278+0000","last_became_active":"2024-09-10T20:41:50.268955+0000","last_became_peered":"2024-09-10T20:41:50.268955+0000","last_unstale":"2024-09-10T20:41:50.278278+0000","last_undegraded":"2024-09-10T20:41:50.278278+0000","last_fullsized":"2024-09-10T20:41:50.278278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:55.335452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264003+0000","last_change":"2024-09-10T20:41:50.261207+0000","last_active":"2024-09-10T20:41:50.264003+0000","last_peered":"2024-09-10T20:41:50.264003+0000","last_clean":"2024-09-10T20:41:50.264003+0000","last_became_active":"2024-09-10T20:41:50.260866+0000","last_became_peered":"2024-09-10T20:41:50.260866+0000","last_unstale":"2024-09-10T20:41:50.264003+0000","last_undegraded":"2024-09-10T20:41:50.264003+0000","last_fullsized":"2024-09-10T20:41:50.264003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:48.716401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232777+0000","last_change":"2024-09-10T20:41:49.232777+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232777+0000","last_undegraded":"2024-09-10T20:41:49.232777+0000","last_fullsized":"2024-09-10T20:41:49.232777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278111+0000","last_change":"2024-09-10T20:41:50.270275+0000","last_active":"2024-09-10T20:41:50.278111+0000","last_peered":"2024-09-10T20:41:50.278111+0000","last_clean":"2024-09-10T20:41:50.278111+0000","last_became_active":"2024-09-10T20:41:50.269980+0000","last_became_peered":"2024-09-10T20:41:50.269980+0000","last_unstale":"2024-09-10T20:41:50.278111+0000","last_undegraded":"2024-09-10T20:41:50.278111+0000","last_fullsized":"2024-09-10T20:41:50.278111+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:12.357126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263972+0000","last_change":"2024-09-10T20:41:50.260076+0000","last_active":"2024-09-10T20:41:50.263972+0000","last_peered":"2024-09-10T20:41:50.263972+0000","last_clean":"2024-09-10T20:41:50.263972+0000","last_became_active":"2024-09-10T20:41:50.259752+0000","last_became_peered":"2024-09-10T20:41:50.259752+0000","last_unstale":"2024-09-10T20:41:50.263972+0000","last_undegraded":"2024-09-10T20:41:50.263972+0000","last_fullsized":"2024-09-10T20:41:50.263972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:08.756456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228409+0000","last_change":"2024-09-10T20:41:49.228409+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228409+0000","last_undegraded":"2024-09-10T20:41:49.228409+0000","last_fullsized":"2024-09-10T20:41:49.228409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232209+0000","last_change":"2024-09-10T20:41:49.232209+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232209+0000","last_undegraded":"2024-09-10T20:41:49.232209+0000","last_fullsized":"2024-09-10T20:41:49.232209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229613+0000","last_change":"2024-09-10T20:41:49.229613+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229613+0000","last_undegraded":"2024-09-10T20:41:49.229613+0000","last_fullsized":"2024-09-10T20:41:49.229613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230334+0000","last_change":"2024-09-10T20:41:49.230334+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230334+0000","last_undegraded":"2024-09-10T20:41:49.230334+0000","last_fullsized":"2024-09-10T20:41:49.230334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.262988+0000","last_change":"2024-09-10T20:41:50.259865+0000","last_active":"2024-09-10T20:41:50.262988+0000","last_peered":"2024-09-10T20:41:50.262988+0000","last_clean":"2024-09-10T20:41:50.262988+0000","last_became_active":"2024-09-10T20:41:50.259492+0000","last_became_peered":"2024-09-10T20:41:50.259492+0000","last_unstale":"2024-09-10T20:41:50.262988+0000","last_undegraded":"2024-09-10T20:41:50.262988+0000","last_fullsized":"2024-09-10T20:41:50.262988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:32:07.918850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228536+0000","last_change":"2024-09-10T20:41:49.228536+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228536+0000","last_undegraded":"2024-09-10T20:41:49.228536+0000","last_fullsized":"2024-09-10T20:41:49.228536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230656+0000","last_change":"2024-09-10T20:41:49.230656+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230656+0000","last_undegraded":"2024-09-10T20:41:49.230656+0000","last_fullsized":"2024-09-10T20:41:49.230656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269422+0000","last_change":"2024-09-10T20:41:50.264629+0000","last_active":"2024-09-10T20:41:50.269422+0000","last_peered":"2024-09-10T20:41:50.269422+0000","last_clean":"2024-09-10T20:41:50.269422+0000","last_became_active":"2024-09-10T20:41:50.264345+0000","last_became_peered":"2024-09-10T20:41:50.264345+0000","last_unstale":"2024-09-10T20:41:50.269422+0000","last_undegraded":"2024-09-10T20:41:50.269422+0000","last_fullsized":"2024-09-10T20:41:50.269422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:33:47.084724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228954+0000","last_change":"2024-09-10T20:41:49.228954+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228954+0000","last_undegraded":"2024-09-10T20:41:49.228954+0000","last_fullsized":"2024-09-10T20:41:49.228954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257652+0000","last_change":"2024-09-10T20:41:50.255889+0000","last_active":"2024-09-10T20:41:50.257652+0000","last_peered":"2024-09-10T20:41:50.257652+0000","last_clean":"2024-09-10T20:41:50.257652+0000","last_became_active":"2024-09-10T20:41:50.255572+0000","last_became_peered":"2024-09-10T20:41:50.255572+0000","last_unstale":"2024-09-10T20:41:50.257652+0000","last_undegraded":"2024-09-10T20:41:50.257652+0000","last_fullsized":"2024-09-10T20:41:50.257652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:38.194068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274208+0000","last_change":"2024-09-10T20:41:50.265660+0000","last_active":"2024-09-10T20:41:50.274208+0000","last_peered":"2024-09-10T20:41:50.274208+0000","last_clean":"2024-09-10T20:41:50.274208+0000","last_became_active":"2024-09-10T20:41:50.265364+0000","last_became_peered":"2024-09-10T20:41:50.265364+0000","last_unstale":"2024-09-10T20:41:50.274208+0000","last_undegraded":"2024-09-10T20:41:50.274208+0000","last_fullsized":"2024-09-10T20:41:50.274208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:22:52.067493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269174+0000","last_change":"2024-09-10T20:41:50.263991+0000","last_active":"2024-09-10T20:41:50.269174+0000","last_peered":"2024-09-10T20:41:50.269174+0000","last_clean":"2024-09-10T20:41:50.269174+0000","last_became_active":"2024-09-10T20:41:50.263690+0000","last_became_peered":"2024-09-10T20:41:50.263690+0000","last_unstale":"2024-09-10T20:41:50.269174+0000","last_undegraded":"2024-09-10T20:41:50.269174+0000","last_fullsized":"2024-09-10T20:41:50.269174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:08.679923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.264022+0000","last_change":"2024-09-10T20:41:50.261825+0000","last_active":"2024-09-10T20:41:50.264022+0000","last_peered":"2024-09-10T20:41:50.264022+0000","last_clean":"2024-09-10T20:41:50.264022+0000","last_became_active":"2024-09-10T20:41:50.261516+0000","last_became_peered":"2024-09-10T20:41:50.261516+0000","last_unstale":"2024-09-10T20:41:50.264022+0000","last_undegraded":"2024-09-10T20:41:50.264022+0000","last_fullsized":"2024-09-10T20:41:50.264022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:26.740959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278057+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.278057+0000","last_peered":"2024-09-10T20:41:50.278057+0000","last_clean":"2024-09-10T20:41:50.278057+0000","last_became_active":"2024-09-10T20:41:50.272144+0000","last_became_peered":"2024-09-10T20:41:50.272144+0000","last_unstale":"2024-09-10T20:41:50.278057+0000","last_undegraded":"2024-09-10T20:41:50.278057+0000","last_fullsized":"2024-09-10T20:41:50.278057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:05.044942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.275708+0000","last_change":"2024-09-10T20:41:50.273908+0000","last_active":"2024-09-10T20:41:50.275708+0000","last_peered":"2024-09-10T20:41:50.275708+0000","last_clean":"2024-09-10T20:41:50.275708+0000","last_became_active":"2024-09-10T20:41:50.273619+0000","last_became_peered":"2024-09-10T20:41:50.273619+0000","last_unstale":"2024-09-10T20:41:50.275708+0000","last_undegraded":"2024-09-10T20:41:50.275708+0000","last_fullsized":"2024-09-10T20:41:50.275708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:57.474072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229786+0000","last_change":"2024-09-10T20:41:49.229786+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229786+0000","last_undegraded":"2024-09-10T20:41:49.229786+0000","last_fullsized":"2024-09-10T20:41:49.229786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269719+0000","last_change":"2024-09-10T20:41:50.267808+0000","last_active":"2024-09-10T20:41:50.269719+0000","last_peered":"2024-09-10T20:41:50.269719+0000","last_clean":"2024-09-10T20:41:50.269719+0000","last_became_active":"2024-09-10T20:41:50.267481+0000","last_became_peered":"2024-09-10T20:41:50.267481+0000","last_unstale":"2024-09-10T20:41:50.269719+0000","last_undegraded":"2024-09-10T20:41:50.269719+0000","last_fullsized":"2024-09-10T20:41:50.269719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:18:49.925720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231774+0000","last_change":"2024-09-10T20:41:49.231774+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231774+0000","last_undegraded":"2024-09-10T20:41:49.231774+0000","last_fullsized":"2024-09-10T20:41:49.231774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.265287+0000","last_change":"2024-09-10T20:41:50.263690+0000","last_active":"2024-09-10T20:41:50.265287+0000","last_peered":"2024-09-10T20:41:50.265287+0000","last_clean":"2024-09-10T20:41:50.265287+0000","last_became_active":"2024-09-10T20:41:50.263407+0000","last_became_peered":"2024-09-10T20:41:50.263407+0000","last_unstale":"2024-09-10T20:41:50.265287+0000","last_undegraded":"2024-09-10T20:41:50.265287+0000","last_fullsized":"2024-09-10T20:41:50.265287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:16.841293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269657+0000","last_change":"2024-09-10T20:41:50.266413+0000","last_active":"2024-09-10T20:41:50.269657+0000","last_peered":"2024-09-10T20:41:50.269657+0000","last_clean":"2024-09-10T20:41:50.269657+0000","last_became_active":"2024-09-10T20:41:50.266150+0000","last_became_peered":"2024-09-10T20:41:50.266150+0000","last_unstale":"2024-09-10T20:41:50.269657+0000","last_undegraded":"2024-09-10T20:41:50.269657+0000","last_fullsized":"2024-09-10T20:41:50.269657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:20.996250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270272+0000","last_change":"2024-09-10T20:41:50.267692+0000","last_active":"2024-09-10T20:41:50.270272+0000","last_peered":"2024-09-10T20:41:50.270272+0000","last_clean":"2024-09-10T20:41:50.270272+0000","last_became_active":"2024-09-10T20:41:50.267181+0000","last_became_peered":"2024-09-10T20:41:50.267181+0000","last_unstale":"2024-09-10T20:41:50.270272+0000","last_undegraded":"2024-09-10T20:41:50.270272+0000","last_fullsized":"2024-09-10T20:41:50.270272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:14.639786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263938+0000","last_change":"2024-09-10T20:41:50.260450+0000","last_active":"2024-09-10T20:41:50.263938+0000","last_peered":"2024-09-10T20:41:50.263938+0000","last_clean":"2024-09-10T20:41:50.263938+0000","last_became_active":"2024-09-10T20:41:50.260093+0000","last_became_peered":"2024-09-10T20:41:50.260093+0000","last_unstale":"2024-09-10T20:41:50.263938+0000","last_undegraded":"2024-09-10T20:41:50.263938+0000","last_fullsized":"2024-09-10T20:41:50.263938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:16.794344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281382+0000","last_change":"2024-09-10T20:41:49.244556+0000","last_active":"2024-09-10T20:41:50.281382+0000","last_peered":"2024-09-10T20:41:50.281382+0000","last_clean":"2024-09-10T20:41:50.281382+0000","last_became_active":"2024-09-10T20:41:49.244277+0000","last_became_peered":"2024-09-10T20:41:49.244277+0000","last_unstale":"2024-09-10T20:41:50.281382+0000","last_undegraded":"2024-09-10T20:41:50.281382+0000","last_fullsized":"2024-09-10T20:41:50.281382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:24.745124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232330+0000","last_change":"2024-09-10T20:41:49.232330+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232330+0000","last_undegraded":"2024-09-10T20:41:49.232330+0000","last_fullsized":"2024-09-10T20:41:49.232330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242361+0000","last_change":"2024-09-10T20:41:49.236963+0000","last_active":"2024-09-10T20:41:49.242361+0000","last_peered":"2024-09-10T20:41:49.242361+0000","last_clean":"2024-09-10T20:41:49.242361+0000","last_became_active":"2024-09-10T20:41:49.236654+0000","last_became_peered":"2024-09-10T20:41:49.236654+0000","last_unstale":"2024-09-10T20:41:49.242361+0000","last_undegraded":"2024-09-10T20:41:49.242361+0000","last_fullsized":"2024-09-10T20:41:49.242361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:48.203284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278810+0000","last_change":"2024-09-10T20:41:50.277395+0000","last_active":"2024-09-10T20:41:50.278810+0000","last_peered":"2024-09-10T20:41:50.278810+0000","last_clean":"2024-09-10T20:41:50.278810+0000","last_became_active":"2024-09-10T20:41:50.277101+0000","last_became_peered":"2024-09-10T20:41:50.277101+0000","last_unstale":"2024-09-10T20:41:50.278810+0000","last_undegraded":"2024-09-10T20:41:50.278810+0000","last_fullsized":"2024-09-10T20:41:50.278810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:47.799998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282196+0000","last_change":"2024-09-10T20:41:49.238213+0000","last_active":"2024-09-10T20:41:50.282196+0000","last_peered":"2024-09-10T20:41:50.282196+0000","last_clean":"2024-09-10T20:41:50.282196+0000","last_became_active":"2024-09-10T20:41:49.237949+0000","last_became_peered":"2024-09-10T20:41:49.237949+0000","last_unstale":"2024-09-10T20:41:50.282196+0000","last_undegraded":"2024-09-10T20:41:50.282196+0000","last_fullsized":"2024-09-10T20:41:50.282196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:45:22.532346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.266258+0000","last_change":"2024-09-10T20:41:50.262357+0000","last_active":"2024-09-10T20:41:50.266258+0000","last_peered":"2024-09-10T20:41:50.266258+0000","last_clean":"2024-09-10T20:41:50.266258+0000","last_became_active":"2024-09-10T20:41:50.262081+0000","last_became_peered":"2024-09-10T20:41:50.262081+0000","last_unstale":"2024-09-10T20:41:50.266258+0000","last_undegraded":"2024-09-10T20:41:50.266258+0000","last_fullsized":"2024-09-10T20:41:50.266258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:44.966942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.279900+0000","last_change":"2024-09-10T20:41:49.247040+0000","last_active":"2024-09-10T20:41:50.279900+0000","last_peered":"2024-09-10T20:41:50.279900+0000","last_clean":"2024-09-10T20:41:50.279900+0000","last_became_active":"2024-09-10T20:41:49.246731+0000","last_became_peered":"2024-09-10T20:41:49.246731+0000","last_unstale":"2024-09-10T20:41:50.279900+0000","last_undegraded":"2024-09-10T20:41:50.279900+0000","last_fullsized":"2024-09-10T20:41:50.279900+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:37.379165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263903+0000","last_change":"2024-09-10T20:41:50.261513+0000","last_active":"2024-09-10T20:41:50.263903+0000","last_peered":"2024-09-10T20:41:50.263903+0000","last_clean":"2024-09-10T20:41:50.263903+0000","last_became_active":"2024-09-10T20:41:50.261203+0000","last_became_peered":"2024-09-10T20:41:50.261203+0000","last_unstale":"2024-09-10T20:41:50.263903+0000","last_undegraded":"2024-09-10T20:41:50.263903+0000","last_fullsized":"2024-09-10T20:41:50.263903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:39.097791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242512+0000","last_change":"2024-09-10T20:41:49.237350+0000","last_active":"2024-09-10T20:41:49.242512+0000","last_peered":"2024-09-10T20:41:49.242512+0000","last_clean":"2024-09-10T20:41:49.242512+0000","last_became_active":"2024-09-10T20:41:49.237086+0000","last_became_peered":"2024-09-10T20:41:49.237086+0000","last_unstale":"2024-09-10T20:41:49.242512+0000","last_undegraded":"2024-09-10T20:41:49.242512+0000","last_fullsized":"2024-09-10T20:41:49.242512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:13.283338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276471+0000","last_change":"2024-09-10T20:41:50.275485+0000","last_active":"2024-09-10T20:41:50.276471+0000","last_peered":"2024-09-10T20:41:50.276471+0000","last_clean":"2024-09-10T20:41:50.276471+0000","last_became_active":"2024-09-10T20:41:50.275179+0000","last_became_peered":"2024-09-10T20:41:50.275179+0000","last_unstale":"2024-09-10T20:41:50.276471+0000","last_undegraded":"2024-09-10T20:41:50.276471+0000","last_fullsized":"2024-09-10T20:41:50.276471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:34.056711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278185+0000","last_change":"2024-09-10T20:41:49.245982+0000","last_active":"2024-09-10T20:41:50.278185+0000","last_peered":"2024-09-10T20:41:50.278185+0000","last_clean":"2024-09-10T20:41:50.278185+0000","last_became_active":"2024-09-10T20:41:49.245669+0000","last_became_peered":"2024-09-10T20:41:49.245669+0000","last_unstale":"2024-09-10T20:41:50.278185+0000","last_undegraded":"2024-09-10T20:41:50.278185+0000","last_fullsized":"2024-09-10T20:41:50.278185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:21:05.620306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.234051+0000","last_change":"2024-09-10T20:41:49.234051+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.234051+0000","last_undegraded":"2024-09-10T20:41:49.234051+0000","last_fullsized":"2024-09-10T20:41:49.234051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.250481+0000","last_change":"2024-09-10T20:41:49.246041+0000","last_active":"2024-09-10T20:41:49.250481+0000","last_peered":"2024-09-10T20:41:49.250481+0000","last_clean":"2024-09-10T20:41:49.250481+0000","last_became_active":"2024-09-10T20:41:49.245783+0000","last_became_peered":"2024-09-10T20:41:49.245783+0000","last_unstale":"2024-09-10T20:41:49.250481+0000","last_undegraded":"2024-09-10T20:41:49.250481+0000","last_fullsized":"2024-09-10T20:41:49.250481+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:23.096775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229857+0000","last_change":"2024-09-10T20:41:49.229857+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229857+0000","last_undegraded":"2024-09-10T20:41:49.229857+0000","last_fullsized":"2024-09-10T20:41:49.229857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238563+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238256+0000","last_became_peered":"2024-09-10T20:41:49.238256+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:16.403282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278144+0000","last_change":"2024-09-10T20:41:50.273036+0000","last_active":"2024-09-10T20:41:50.278144+0000","last_peered":"2024-09-10T20:41:50.278144+0000","last_clean":"2024-09-10T20:41:50.278144+0000","last_became_active":"2024-09-10T20:41:50.272751+0000","last_became_peered":"2024-09-10T20:41:50.272751+0000","last_unstale":"2024-09-10T20:41:50.278144+0000","last_undegraded":"2024-09-10T20:41:50.278144+0000","last_fullsized":"2024-09-10T20:41:50.278144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:48:00.876771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252771+0000","last_change":"2024-09-10T20:41:49.239816+0000","last_active":"2024-09-10T20:41:50.252771+0000","last_peered":"2024-09-10T20:41:50.252771+0000","last_clean":"2024-09-10T20:41:50.252771+0000","last_became_active":"2024-09-10T20:41:49.239509+0000","last_became_peered":"2024-09-10T20:41:49.239509+0000","last_unstale":"2024-09-10T20:41:50.252771+0000","last_undegraded":"2024-09-10T20:41:50.252771+0000","last_fullsized":"2024-09-10T20:41:50.252771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:30.409719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.270329+0000","last_change":"2024-09-10T20:41:50.268159+0000","last_active":"2024-09-10T20:41:50.270329+0000","last_peered":"2024-09-10T20:41:50.270329+0000","last_clean":"2024-09-10T20:41:50.270329+0000","last_became_active":"2024-09-10T20:41:50.267522+0000","last_became_peered":"2024-09-10T20:41:50.267522+0000","last_unstale":"2024-09-10T20:41:50.270329+0000","last_undegraded":"2024-09-10T20:41:50.270329+0000","last_fullsized":"2024-09-10T20:41:50.270329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.176084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247689+0000","last_change":"2024-09-10T20:41:49.243018+0000","last_active":"2024-09-10T20:41:50.247689+0000","last_peered":"2024-09-10T20:41:50.247689+0000","last_clean":"2024-09-10T20:41:50.247689+0000","last_became_active":"2024-09-10T20:41:49.242751+0000","last_became_peered":"2024-09-10T20:41:49.242751+0000","last_unstale":"2024-09-10T20:41:50.247689+0000","last_undegraded":"2024-09-10T20:41:50.247689+0000","last_fullsized":"2024-09-10T20:41:50.247689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:47.281477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.274705+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:50.274705+0000","last_peered":"2024-09-10T20:41:50.274705+0000","last_clean":"2024-09-10T20:41:50.274705+0000","last_became_active":"2024-09-10T20:41:50.272083+0000","last_became_peered":"2024-09-10T20:41:50.272083+0000","last_unstale":"2024-09-10T20:41:50.274705+0000","last_undegraded":"2024-09-10T20:41:50.274705+0000","last_fullsized":"2024-09-10T20:41:50.274705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242314+0000","last_change":"2024-09-10T20:41:49.235825+0000","last_active":"2024-09-10T20:41:49.242314+0000","last_peered":"2024-09-10T20:41:49.242314+0000","last_clean":"2024-09-10T20:41:49.242314+0000","last_became_active":"2024-09-10T20:41:49.235466+0000","last_became_peered":"2024-09-10T20:41:49.235466+0000","last_unstale":"2024-09-10T20:41:49.242314+0000","last_undegraded":"2024-09-10T20:41:49.242314+0000","last_fullsized":"2024-09-10T20:41:49.242314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:08.240386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229403+0000","last_change":"2024-09-10T20:41:49.229403+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229403+0000","last_undegraded":"2024-09-10T20:41:49.229403+0000","last_fullsized":"2024-09-10T20:41:49.229403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245167+0000","last_change":"2024-09-10T20:41:49.241236+0000","last_active":"2024-09-10T20:41:49.245167+0000","last_peered":"2024-09-10T20:41:49.245167+0000","last_clean":"2024-09-10T20:41:49.245167+0000","last_became_active":"2024-09-10T20:41:49.240686+0000","last_became_peered":"2024-09-10T20:41:49.240686+0000","last_unstale":"2024-09-10T20:41:49.245167+0000","last_undegraded":"2024-09-10T20:41:49.245167+0000","last_fullsized":"2024-09-10T20:41:49.245167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:40:43.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.267631+0000","last_change":"2024-09-10T20:41:50.262537+0000","last_active":"2024-09-10T20:41:50.267631+0000","last_peered":"2024-09-10T20:41:50.267631+0000","last_clean":"2024-09-10T20:41:50.267631+0000","last_became_active":"2024-09-10T20:41:50.261672+0000","last_became_peered":"2024-09-10T20:41:50.261672+0000","last_unstale":"2024-09-10T20:41:50.267631+0000","last_undegraded":"2024-09-10T20:41:50.267631+0000","last_fullsized":"2024-09-10T20:41:50.267631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:57:22.115806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.245187+0000","last_change":"2024-09-10T20:41:49.241419+0000","last_active":"2024-09-10T20:41:49.245187+0000","last_peered":"2024-09-10T20:41:49.245187+0000","last_clean":"2024-09-10T20:41:49.245187+0000","last_became_active":"2024-09-10T20:41:49.241125+0000","last_became_peered":"2024-09-10T20:41:49.241125+0000","last_unstale":"2024-09-10T20:41:49.245187+0000","last_undegraded":"2024-09-10T20:41:49.245187+0000","last_fullsized":"2024-09-10T20:41:49.245187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:45:50.357901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278395+0000","last_change":"2024-09-10T20:41:50.272126+0000","last_active":"2024-09-10T20:41:50.278395+0000","last_peered":"2024-09-10T20:41:50.278395+0000","last_clean":"2024-09-10T20:41:50.278395+0000","last_became_active":"2024-09-10T20:41:50.271842+0000","last_became_peered":"2024-09-10T20:41:50.271842+0000","last_unstale":"2024-09-10T20:41:50.278395+0000","last_undegraded":"2024-09-10T20:41:50.278395+0000","last_fullsized":"2024-09-10T20:41:50.278395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:17:24.548442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254957+0000","last_change":"2024-09-10T20:41:49.248639+0000","last_active":"2024-09-10T20:41:50.254957+0000","last_peered":"2024-09-10T20:41:50.254957+0000","last_clean":"2024-09-10T20:41:50.254957+0000","last_became_active":"2024-09-10T20:41:49.248352+0000","last_became_peered":"2024-09-10T20:41:49.248352+0000","last_unstale":"2024-09-10T20:41:50.254957+0000","last_undegraded":"2024-09-10T20:41:50.254957+0000","last_fullsized":"2024-09-10T20:41:50.254957+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:41.288742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.228516+0000","last_change":"2024-09-10T20:41:49.228516+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.228516+0000","last_undegraded":"2024-09-10T20:41:49.228516+0000","last_fullsized":"2024-09-10T20:41:49.228516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.251913+0000","last_change":"2024-09-10T20:41:49.249615+0000","last_active":"2024-09-10T20:41:50.251913+0000","last_peered":"2024-09-10T20:41:50.251913+0000","last_clean":"2024-09-10T20:41:50.251913+0000","last_became_active":"2024-09-10T20:41:49.249414+0000","last_became_peered":"2024-09-10T20:41:49.249414+0000","last_unstale":"2024-09-10T20:41:50.251913+0000","last_undegraded":"2024-09-10T20:41:50.251913+0000","last_fullsized":"2024-09-10T20:41:50.251913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:36.540914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231389+0000","last_change":"2024-09-10T20:41:49.231389+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231389+0000","last_undegraded":"2024-09-10T20:41:49.231389+0000","last_fullsized":"2024-09-10T20:41:49.231389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.282862+0000","last_change":"2024-09-10T20:41:49.244860+0000","last_active":"2024-09-10T20:41:50.282862+0000","last_peered":"2024-09-10T20:41:50.282862+0000","last_clean":"2024-09-10T20:41:50.282862+0000","last_became_active":"2024-09-10T20:41:49.244532+0000","last_became_peered":"2024-09-10T20:41:49.244532+0000","last_unstale":"2024-09-10T20:41:50.282862+0000","last_undegraded":"2024-09-10T20:41:50.282862+0000","last_fullsized":"2024-09-10T20:41:50.282862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:33.273565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257492+0000","last_change":"2024-09-10T20:41:50.255850+0000","last_active":"2024-09-10T20:41:50.257492+0000","last_peered":"2024-09-10T20:41:50.257492+0000","last_clean":"2024-09-10T20:41:50.257492+0000","last_became_active":"2024-09-10T20:41:50.254971+0000","last_became_peered":"2024-09-10T20:41:50.254971+0000","last_unstale":"2024-09-10T20:41:50.257492+0000","last_undegraded":"2024-09-10T20:41:50.257492+0000","last_fullsized":"2024-09-10T20:41:50.257492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:24:15.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283814+0000","last_change":"2024-09-10T20:41:49.239536+0000","last_active":"2024-09-10T20:41:50.283814+0000","last_peered":"2024-09-10T20:41:50.283814+0000","last_clean":"2024-09-10T20:41:50.283814+0000","last_became_active":"2024-09-10T20:41:49.239253+0000","last_became_peered":"2024-09-10T20:41:49.239253+0000","last_unstale":"2024-09-10T20:41:50.283814+0000","last_undegraded":"2024-09-10T20:41:50.283814+0000","last_fullsized":"2024-09-10T20:41:50.283814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:15.552579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.254349+0000","last_change":"2024-09-10T20:41:50.251514+0000","last_active":"2024-09-10T20:41:50.254349+0000","last_peered":"2024-09-10T20:41:50.254349+0000","last_clean":"2024-09-10T20:41:50.254349+0000","last_became_active":"2024-09-10T20:41:50.251241+0000","last_became_peered":"2024-09-10T20:41:50.251241+0000","last_unstale":"2024-09-10T20:41:50.254349+0000","last_undegraded":"2024-09-10T20:41:50.254349+0000","last_fullsized":"2024-09-10T20:41:50.254349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:59.239713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.283432+0000","last_change":"2024-09-10T20:41:49.248393+0000","last_active":"2024-09-10T20:41:50.283432+0000","last_peered":"2024-09-10T20:41:50.283432+0000","last_clean":"2024-09-10T20:41:50.283432+0000","last_became_active":"2024-09-10T20:41:49.248103+0000","last_became_peered":"2024-09-10T20:41:49.248103+0000","last_unstale":"2024-09-10T20:41:50.283432+0000","last_undegraded":"2024-09-10T20:41:50.283432+0000","last_fullsized":"2024-09-10T20:41:50.283432+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:54.196476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.231812+0000","last_change":"2024-09-10T20:41:49.231812+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.231812+0000","last_undegraded":"2024-09-10T20:41:49.231812+0000","last_fullsized":"2024-09-10T20:41:49.231812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.276182+0000","last_change":"2024-09-10T20:41:49.239383+0000","last_active":"2024-09-10T20:41:50.276182+0000","last_peered":"2024-09-10T20:41:50.276182+0000","last_clean":"2024-09-10T20:41:50.276182+0000","last_became_active":"2024-09-10T20:41:49.239097+0000","last_became_peered":"2024-09-10T20:41:49.239097+0000","last_unstale":"2024-09-10T20:41:50.276182+0000","last_undegraded":"2024-09-10T20:41:50.276182+0000","last_fullsized":"2024-09-10T20:41:50.276182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:51.420841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.232592+0000","last_change":"2024-09-10T20:41:49.232592+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.232592+0000","last_undegraded":"2024-09-10T20:41:49.232592+0000","last_fullsized":"2024-09-10T20:41:49.232592+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.257075+0000","last_change":"2024-09-10T20:41:49.243612+0000","last_active":"2024-09-10T20:41:50.257075+0000","last_peered":"2024-09-10T20:41:50.257075+0000","last_clean":"2024-09-10T20:41:50.257075+0000","last_became_active":"2024-09-10T20:41:49.243065+0000","last_became_peered":"2024-09-10T20:41:49.243065+0000","last_unstale":"2024-09-10T20:41:50.257075+0000","last_undegraded":"2024-09-10T20:41:50.257075+0000","last_fullsized":"2024-09-10T20:41:50.257075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:20:31.645185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233707+0000","last_change":"2024-09-10T20:41:49.233707+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233707+0000","last_undegraded":"2024-09-10T20:41:49.233707+0000","last_fullsized":"2024-09-10T20:41:49.233707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252699+0000","last_change":"2024-09-10T20:41:49.241358+0000","last_active":"2024-09-10T20:41:50.252699+0000","last_peered":"2024-09-10T20:41:50.252699+0000","last_clean":"2024-09-10T20:41:50.252699+0000","last_became_active":"2024-09-10T20:41:49.241098+0000","last_became_peered":"2024-09-10T20:41:49.241098+0000","last_unstale":"2024-09-10T20:41:50.252699+0000","last_undegraded":"2024-09-10T20:41:50.252699+0000","last_fullsized":"2024-09-10T20:41:50.252699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:53:49.000467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233626+0000","last_change":"2024-09-10T20:41:49.233626+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233626+0000","last_undegraded":"2024-09-10T20:41:49.233626+0000","last_fullsized":"2024-09-10T20:41:49.233626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247275+0000","last_change":"2024-09-10T20:41:49.234909+0000","last_active":"2024-09-10T20:41:50.247275+0000","last_peered":"2024-09-10T20:41:50.247275+0000","last_clean":"2024-09-10T20:41:50.247275+0000","last_became_active":"2024-09-10T20:41:49.234598+0000","last_became_peered":"2024-09-10T20:41:49.234598+0000","last_unstale":"2024-09-10T20:41:50.247275+0000","last_undegraded":"2024-09-10T20:41:50.247275+0000","last_fullsized":"2024-09-10T20:41:50.247275+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:00.862079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229401+0000","last_change":"2024-09-10T20:41:49.229401+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229401+0000","last_undegraded":"2024-09-10T20:41:49.229401+0000","last_fullsized":"2024-09-10T20:41:49.229401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248123+0000","last_change":"2024-09-10T20:41:49.245102+0000","last_active":"2024-09-10T20:41:49.248123+0000","last_peered":"2024-09-10T20:41:49.248123+0000","last_clean":"2024-09-10T20:41:49.248123+0000","last_became_active":"2024-09-10T20:41:49.244790+0000","last_became_peered":"2024-09-10T20:41:49.244790+0000","last_unstale":"2024-09-10T20:41:49.248123+0000","last_undegraded":"2024-09-10T20:41:49.248123+0000","last_fullsized":"2024-09-10T20:41:49.248123+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:26.633006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.230237+0000","last_change":"2024-09-10T20:41:49.230237+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.230237+0000","last_undegraded":"2024-09-10T20:41:49.230237+0000","last_fullsized":"2024-09-10T20:41:49.230237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.281880+0000","last_change":"2024-09-10T20:41:49.239083+0000","last_active":"2024-09-10T20:41:50.281880+0000","last_peered":"2024-09-10T20:41:50.281880+0000","last_clean":"2024-09-10T20:41:50.281880+0000","last_became_active":"2024-09-10T20:41:49.238741+0000","last_became_peered":"2024-09-10T20:41:49.238741+0000","last_unstale":"2024-09-10T20:41:50.281880+0000","last_undegraded":"2024-09-10T20:41:50.281880+0000","last_fullsized":"2024-09-10T20:41:50.281880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:09:56.852921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.277987+0000","last_change":"2024-09-10T20:41:50.270889+0000","last_active":"2024-09-10T20:41:50.277987+0000","last_peered":"2024-09-10T20:41:50.277987+0000","last_clean":"2024-09-10T20:41:50.277987+0000","last_became_active":"2024-09-10T20:41:50.270603+0000","last_became_peered":"2024-09-10T20:41:50.270603+0000","last_unstale":"2024-09-10T20:41:50.277987+0000","last_undegraded":"2024-09-10T20:41:50.277987+0000","last_fullsized":"2024-09-10T20:41:50.277987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:23.399016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.252628+0000","last_change":"2024-09-10T20:41:49.249494+0000","last_active":"2024-09-10T20:41:50.252628+0000","last_peered":"2024-09-10T20:41:50.252628+0000","last_clean":"2024-09-10T20:41:50.252628+0000","last_became_active":"2024-09-10T20:41:49.249207+0000","last_became_peered":"2024-09-10T20:41:49.249207+0000","last_unstale":"2024-09-10T20:41:50.252628+0000","last_undegraded":"2024-09-10T20:41:50.252628+0000","last_fullsized":"2024-09-10T20:41:50.252628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:47.078899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.268605+0000","last_change":"2024-09-10T20:41:50.265954+0000","last_active":"2024-09-10T20:41:50.268605+0000","last_peered":"2024-09-10T20:41:50.268605+0000","last_clean":"2024-09-10T20:41:50.268605+0000","last_became_active":"2024-09-10T20:41:50.265644+0000","last_became_peered":"2024-09-10T20:41:50.265644+0000","last_unstale":"2024-09-10T20:41:50.268605+0000","last_undegraded":"2024-09-10T20:41:50.268605+0000","last_fullsized":"2024-09-10T20:41:50.268605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:22.562059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242530+0000","last_change":"2024-09-10T20:41:49.238772+0000","last_active":"2024-09-10T20:41:49.242530+0000","last_peered":"2024-09-10T20:41:49.242530+0000","last_clean":"2024-09-10T20:41:49.242530+0000","last_became_active":"2024-09-10T20:41:49.238312+0000","last_became_peered":"2024-09-10T20:41:49.238312+0000","last_unstale":"2024-09-10T20:41:49.242530+0000","last_undegraded":"2024-09-10T20:41:49.242530+0000","last_fullsized":"2024-09-10T20:41:49.242530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:20.684231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.278050+0000","last_change":"2024-09-10T20:41:50.265096+0000","last_active":"2024-09-10T20:41:50.278050+0000","last_peered":"2024-09-10T20:41:50.278050+0000","last_clean":"2024-09-10T20:41:50.278050+0000","last_became_active":"2024-09-10T20:41:50.264830+0000","last_became_peered":"2024-09-10T20:41:50.264830+0000","last_unstale":"2024-09-10T20:41:50.278050+0000","last_undegraded":"2024-09-10T20:41:50.278050+0000","last_fullsized":"2024-09-10T20:41:50.278050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:15.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.242477+0000","last_change":"2024-09-10T20:41:49.236972+0000","last_active":"2024-09-10T20:41:49.242477+0000","last_peered":"2024-09-10T20:41:49.242477+0000","last_clean":"2024-09-10T20:41:49.242477+0000","last_became_active":"2024-09-10T20:41:49.236675+0000","last_became_peered":"2024-09-10T20:41:49.236675+0000","last_unstale":"2024-09-10T20:41:49.242477+0000","last_undegraded":"2024-09-10T20:41:49.242477+0000","last_fullsized":"2024-09-10T20:41:49.242477+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:08.249515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229009+0000","last_change":"2024-09-10T20:41:49.229009+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229009+0000","last_undegraded":"2024-09-10T20:41:49.229009+0000","last_fullsized":"2024-09-10T20:41:49.229009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.239295+0000","last_change":"2024-09-10T20:41:49.235872+0000","last_active":"2024-09-10T20:41:49.239295+0000","last_peered":"2024-09-10T20:41:49.239295+0000","last_clean":"2024-09-10T20:41:49.239295+0000","last_became_active":"2024-09-10T20:41:49.235579+0000","last_became_peered":"2024-09-10T20:41:49.235579+0000","last_unstale":"2024-09-10T20:41:49.239295+0000","last_undegraded":"2024-09-10T20:41:49.239295+0000","last_fullsized":"2024-09-10T20:41:49.239295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:32.310476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.263923+0000","last_change":"2024-09-10T20:41:50.260226+0000","last_active":"2024-09-10T20:41:50.263923+0000","last_peered":"2024-09-10T20:41:50.263923+0000","last_clean":"2024-09-10T20:41:50.263923+0000","last_became_active":"2024-09-10T20:41:50.259625+0000","last_became_peered":"2024-09-10T20:41:50.259625+0000","last_unstale":"2024-09-10T20:41:50.263923+0000","last_undegraded":"2024-09-10T20:41:50.263923+0000","last_fullsized":"2024-09-10T20:41:50.263923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:38.991086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-10T20:41:49.248086+0000","last_change":"2024-09-10T20:41:49.242417+0000","last_active":"2024-09-10T20:41:49.248086+0000","last_peered":"2024-09-10T20:41:49.248086+0000","last_clean":"2024-09-10T20:41:49.248086+0000","last_became_active":"2024-09-10T20:41:49.242161+0000","last_became_peered":"2024-09-10T20:41:49.242161+0000","last_unstale":"2024-09-10T20:41:49.248086+0000","last_undegraded":"2024-09-10T20:41:49.248086+0000","last_fullsized":"2024-09-10T20:41:49.248086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:13.922975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.233118+0000","last_change":"2024-09-10T20:41:49.233118+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.233118+0000","last_undegraded":"2024-09-10T20:41:49.233118+0000","last_fullsized":"2024-09-10T20:41:49.233118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.247637+0000","last_change":"2024-09-10T20:41:49.240524+0000","last_active":"2024-09-10T20:41:50.247637+0000","last_peered":"2024-09-10T20:41:50.247637+0000","last_clean":"2024-09-10T20:41:50.247637+0000","last_became_active":"2024-09-10T20:41:49.240245+0000","last_became_peered":"2024-09-10T20:41:49.240245+0000","last_unstale":"2024-09-10T20:41:50.247637+0000","last_undegraded":"2024-09-10T20:41:50.247637+0000","last_fullsized":"2024-09-10T20:41:50.247637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:14.912119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.269687+0000","last_change":"2024-09-10T20:41:50.267529+0000","last_active":"2024-09-10T20:41:50.269687+0000","last_peered":"2024-09-10T20:41:50.269687+0000","last_clean":"2024-09-10T20:41:50.269687+0000","last_became_active":"2024-09-10T20:41:50.267222+0000","last_became_peered":"2024-09-10T20:41:50.267222+0000","last_unstale":"2024-09-10T20:41:50.269687+0000","last_undegraded":"2024-09-10T20:41:50.269687+0000","last_fullsized":"2024-09-10T20:41:50.269687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:21:30.112985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.253694+0000","last_change":"2024-09-10T20:41:49.236467+0000","last_active":"2024-09-10T20:41:50.253694+0000","last_peered":"2024-09-10T20:41:50.253694+0000","last_clean":"2024-09-10T20:41:50.253694+0000","last_became_active":"2024-09-10T20:41:49.236170+0000","last_became_peered":"2024-09-10T20:41:49.236170+0000","last_unstale":"2024-09-10T20:41:50.253694+0000","last_undegraded":"2024-09-10T20:41:50.253694+0000","last_fullsized":"2024-09-10T20:41:50.253694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:12.514693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-10T20:41:49.229765+0000","last_change":"2024-09-10T20:41:49.229765+0000","last_active":"2024-09-10T20:41:49.221972+0000","last_peered":"2024-09-10T20:41:49.221972+0000","last_clean":"2024-09-10T20:41:49.221972+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:41:49.229765+0000","last_undegraded":"2024-09-10T20:41:49.229765+0000","last_fullsized":"2024-09-10T20:41:49.229765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.280647+0000","last_change":"2024-09-10T20:41:49.249357+0000","last_active":"2024-09-10T20:41:50.280647+0000","last_peered":"2024-09-10T20:41:50.280647+0000","last_clean":"2024-09-10T20:41:50.280647+0000","last_became_active":"2024-09-10T20:41:49.248955+0000","last_became_peered":"2024-09-10T20:41:49.248955+0000","last_unstale":"2024-09-10T20:41:50.280647+0000","last_undegraded":"2024-09-10T20:41:50.280647+0000","last_fullsized":"2024-09-10T20:41:50.280647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:26.862222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-10T20:41:50.250227+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:41:50.250227+0000","last_peered":"2024-09-10T20:41:50.250227+0000","last_clean":"2024-09-10T20:41:50.250227+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:41:50.250227+0000","last_undegraded":"2024-09-10T20:41:50.250227+0000","last_fullsized":"2024-09-10T20:41:50.250227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":674,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509490,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":819200,"data_stored":337112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509489,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509489,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1400832,"data_stored":927354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509489,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":811008,"data_stored":337056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542194,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":782336,"data_stored":323944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1388544,"data_stored":920579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":330211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:41:58.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:41:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 -- 172.21.15.28:0/3268721433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ee70 msgr2=0x7f8fbc0d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 --2- 172.21.15.28:0/3268721433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ee70 0x7f8fbc0d0140 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4005fd0 tx=0x7f8fb402a8b0 comp rx=0 tx=0).stop 2024-09-10T20:41:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 -- 172.21.15.28:0/3268721433 shutdown_connections 2024-09-10T20:41:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 --2- 172.21.15.28:0/3268721433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ee70 0x7f8fbc0d0140 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 --2- 172.21.15.28:0/3268721433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8fbc1554e0 0x7f8fbc15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 -- 172.21.15.28:0/3268721433 >> 172.21.15.28:0/3268721433 conn(0x7f8fbc06cf00 msgr2=0x7f8fbc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 -- 172.21.15.28:0/3268721433 shutdown_connections 2024-09-10T20:41:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.861+0000 7f8fc448c640 1 -- 172.21.15.28:0/3268721433 wait complete. 2024-09-10T20:41:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.862+0000 7f8fc448c640 1 Processor -- start 2024-09-10T20:41:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.862+0000 7f8fc448c640 1 -- start start 2024-09-10T20:41:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc448c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc155100 0x7f8fbc09d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc448c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8fbc1554e0 0x7f8fbc0aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc448c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8fbc15ee70 0x7f8fbc0af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc448c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc072f60 con 0x7f8fbc155100 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc448c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc072de0 con 0x7f8fbc1554e0 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc448c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8fbc0730e0 con 0x7f8fbc15ee70 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc2228640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc155100 0x7f8fbc09d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc2228640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc155100 0x7f8fbc09d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58398/0 (socket says 172.21.15.28:58398) 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.863+0000 7f8fc2228640 1 -- 172.21.15.28:0/4229042138 learned_addr learned my addr 172.21.15.28:0/4229042138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc2a29640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8fbc15ee70 0x7f8fbc0af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc1a27640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8fbc1554e0 0x7f8fbc0aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc2228640 1 -- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8fbc15ee70 msgr2=0x7f8fbc0af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc2228640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8fbc15ee70 0x7f8fbc0af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc2228640 1 -- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8fbc1554e0 msgr2=0x7f8fbc0aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc2228640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8fbc1554e0 0x7f8fbc0aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc2228640 1 -- 172.21.15.28:0/4229042138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc1f4700 con 0x7f8fbc155100 2024-09-10T20:41:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc1a27640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8fbc1554e0 0x7f8fbc0aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc2228640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc155100 0x7f8fbc09d670 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c3a0 tx=0x7f8fb800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/4229042138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8011790 con 0x7f8fbc155100 2024-09-10T20:41:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/4229042138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fb8017070 con 0x7f8fbc155100 2024-09-10T20:41:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/4229042138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8011cc0 con 0x7f8fbc155100 2024-09-10T20:41:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc1f49e0 con 0x7f8fbc155100 2024-09-10T20:41:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.864+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc1f43f0 con 0x7f8fbc155100 2024-09-10T20:41:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.865+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fbc073590 con 0x7f8fbc155100 2024-09-10T20:41:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.866+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/4229042138 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f8fb801c460 con 0x7f8fbc155100 2024-09-10T20:41:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.866+0000 7f8fb2ffd640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8fa00782d0 0x7f8fa007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.867+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/4229042138 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f8fb809a230 con 0x7f8fbc155100 2024-09-10T20:41:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.867+0000 7f8fc1a27640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8fa00782d0 0x7f8fa007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.867+0000 7f8fc1a27640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8fa00782d0 0x7f8fa007a790 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0aea10 tx=0x7f8fac005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:58.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:58.871+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/4229042138 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8fb8062750 con 0x7f8fbc155100 2024-09-10T20:41:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.054+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8fbc069a80 con 0x7f8fa00782d0 2024-09-10T20:41:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.058+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/4229042138 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252037 (secure 0 0 0) 0x7f8fbc069a80 con 0x7f8fa00782d0 2024-09-10T20:41:59.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:59.066 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:41:59.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8fa00782d0 msgr2=0x7f8fa007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8fa00782d0 0x7f8fa007a790 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0aea10 tx=0x7f8fac005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:59.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc155100 msgr2=0x7f8fbc09d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc155100 0x7f8fbc09d670 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c3a0 tx=0x7f8fb800c860 comp rx=0 tx=0).stop 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 shutdown_connections 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8fa00782d0 0x7f8fa007a790 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8fbc15ee70 0x7f8fbc0af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8fbc1554e0 0x7f8fbc0aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 --2- 172.21.15.28:0/4229042138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc155100 0x7f8fbc09d670 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 >> 172.21.15.28:0/4229042138 conn(0x7f8fbc06cf00 msgr2=0x7f8fbc15bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 shutdown_connections 2024-09-10T20:41:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.061+0000 7f8fc448c640 1 -- 172.21.15.28:0/4229042138 wait complete. 2024-09-10T20:41:59.075 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-10T20:41:58.030388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244664,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730056,"statfs":{"total":773094113280,"available":772843577344,"internally_reserved":0,"allocated":7806976,"data_stored":3931844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002226"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268461+0000","last_change":"2024-09-10T20:41:50.262634+0000","last_active":"2024-09-10T20:41:51.268461+0000","last_peered":"2024-09-10T20:41:51.268461+0000","last_clean":"2024-09-10T20:41:51.268461+0000","last_became_active":"2024-09-10T20:41:50.262341+0000","last_became_peered":"2024-09-10T20:41:50.262341+0000","last_unstale":"2024-09-10T20:41:51.268461+0000","last_undegraded":"2024-09-10T20:41:51.268461+0000","last_fullsized":"2024-09-10T20:41:51.268461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:24.601010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273982+0000","last_change":"2024-09-10T20:41:50.268290+0000","last_active":"2024-09-10T20:41:52.273982+0000","last_peered":"2024-09-10T20:41:52.273982+0000","last_clean":"2024-09-10T20:41:52.273982+0000","last_became_active":"2024-09-10T20:41:50.268020+0000","last_became_peered":"2024-09-10T20:41:50.268020+0000","last_unstale":"2024-09-10T20:41:52.273982+0000","last_undegraded":"2024-09-10T20:41:52.273982+0000","last_fullsized":"2024-09-10T20:41:52.273982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:27:26.341495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266933+0000","last_change":"2024-09-10T20:41:50.272610+0000","last_active":"2024-09-10T20:41:51.266933+0000","last_peered":"2024-09-10T20:41:51.266933+0000","last_clean":"2024-09-10T20:41:51.266933+0000","last_became_active":"2024-09-10T20:41:50.272193+0000","last_became_peered":"2024-09-10T20:41:50.272193+0000","last_unstale":"2024-09-10T20:41:51.266933+0000","last_undegraded":"2024-09-10T20:41:51.266933+0000","last_fullsized":"2024-09-10T20:41:51.266933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:07.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266860+0000","last_change":"2024-09-10T20:41:50.272486+0000","last_active":"2024-09-10T20:41:51.266860+0000","last_peered":"2024-09-10T20:41:51.266860+0000","last_clean":"2024-09-10T20:41:51.266860+0000","last_became_active":"2024-09-10T20:41:50.272188+0000","last_became_peered":"2024-09-10T20:41:50.272188+0000","last_unstale":"2024-09-10T20:41:51.266860+0000","last_undegraded":"2024-09-10T20:41:51.266860+0000","last_fullsized":"2024-09-10T20:41:51.266860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:21:55.664334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273822+0000","last_change":"2024-09-10T20:41:50.255343+0000","last_active":"2024-09-10T20:41:52.273822+0000","last_peered":"2024-09-10T20:41:52.273822+0000","last_clean":"2024-09-10T20:41:52.273822+0000","last_became_active":"2024-09-10T20:41:50.255049+0000","last_became_peered":"2024-09-10T20:41:50.255049+0000","last_unstale":"2024-09-10T20:41:52.273822+0000","last_undegraded":"2024-09-10T20:41:52.273822+0000","last_fullsized":"2024-09-10T20:41:52.273822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:42.265270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274094+0000","last_change":"2024-09-10T20:41:50.262868+0000","last_active":"2024-09-10T20:41:52.274094+0000","last_peered":"2024-09-10T20:41:52.274094+0000","last_clean":"2024-09-10T20:41:52.274094+0000","last_became_active":"2024-09-10T20:41:50.262182+0000","last_became_peered":"2024-09-10T20:41:50.262182+0000","last_unstale":"2024-09-10T20:41:52.274094+0000","last_undegraded":"2024-09-10T20:41:52.274094+0000","last_fullsized":"2024-09-10T20:41:52.274094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:53.037486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.260942+0000","last_change":"2024-09-10T20:41:50.269379+0000","last_active":"2024-09-10T20:41:51.260942+0000","last_peered":"2024-09-10T20:41:51.260942+0000","last_clean":"2024-09-10T20:41:51.260942+0000","last_became_active":"2024-09-10T20:41:50.268955+0000","last_became_peered":"2024-09-10T20:41:50.268955+0000","last_unstale":"2024-09-10T20:41:51.260942+0000","last_undegraded":"2024-09-10T20:41:51.260942+0000","last_fullsized":"2024-09-10T20:41:51.260942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:06.496683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267643+0000","last_change":"2024-09-10T20:41:50.261207+0000","last_active":"2024-09-10T20:41:51.267643+0000","last_peered":"2024-09-10T20:41:51.267643+0000","last_clean":"2024-09-10T20:41:51.267643+0000","last_became_active":"2024-09-10T20:41:50.260866+0000","last_became_peered":"2024-09-10T20:41:50.260866+0000","last_unstale":"2024-09-10T20:41:51.267643+0000","last_undegraded":"2024-09-10T20:41:51.267643+0000","last_fullsized":"2024-09-10T20:41:51.267643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:49.637883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267177+0000","last_change":"2024-09-10T20:41:50.264082+0000","last_active":"2024-09-10T20:41:51.267177+0000","last_peered":"2024-09-10T20:41:51.267177+0000","last_clean":"2024-09-10T20:41:51.267177+0000","last_became_active":"2024-09-10T20:41:50.263788+0000","last_became_peered":"2024-09-10T20:41:50.263788+0000","last_unstale":"2024-09-10T20:41:51.267177+0000","last_undegraded":"2024-09-10T20:41:51.267177+0000","last_fullsized":"2024-09-10T20:41:51.267177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:25:24.909038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261549+0000","last_change":"2024-09-10T20:41:50.270275+0000","last_active":"2024-09-10T20:41:51.261549+0000","last_peered":"2024-09-10T20:41:51.261549+0000","last_clean":"2024-09-10T20:41:51.261549+0000","last_became_active":"2024-09-10T20:41:50.269980+0000","last_became_peered":"2024-09-10T20:41:50.269980+0000","last_unstale":"2024-09-10T20:41:51.261549+0000","last_undegraded":"2024-09-10T20:41:51.261549+0000","last_fullsized":"2024-09-10T20:41:51.261549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:12:35.375914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268270+0000","last_change":"2024-09-10T20:41:50.260076+0000","last_active":"2024-09-10T20:41:51.268270+0000","last_peered":"2024-09-10T20:41:51.268270+0000","last_clean":"2024-09-10T20:41:51.268270+0000","last_became_active":"2024-09-10T20:41:50.259752+0000","last_became_peered":"2024-09-10T20:41:50.259752+0000","last_unstale":"2024-09-10T20:41:51.268270+0000","last_undegraded":"2024-09-10T20:41:51.268270+0000","last_fullsized":"2024-09-10T20:41:51.268270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:22.523975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266549+0000","last_change":"2024-09-10T20:41:50.264962+0000","last_active":"2024-09-10T20:41:51.266549+0000","last_peered":"2024-09-10T20:41:51.266549+0000","last_clean":"2024-09-10T20:41:51.266549+0000","last_became_active":"2024-09-10T20:41:50.264679+0000","last_became_peered":"2024-09-10T20:41:50.264679+0000","last_unstale":"2024-09-10T20:41:51.266549+0000","last_undegraded":"2024-09-10T20:41:51.266549+0000","last_fullsized":"2024-09-10T20:41:51.266549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:29:27.807214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266576+0000","last_change":"2024-09-10T20:41:50.272533+0000","last_active":"2024-09-10T20:41:51.266576+0000","last_peered":"2024-09-10T20:41:51.266576+0000","last_clean":"2024-09-10T20:41:51.266576+0000","last_became_active":"2024-09-10T20:41:50.272273+0000","last_became_peered":"2024-09-10T20:41:50.272273+0000","last_unstale":"2024-09-10T20:41:51.266576+0000","last_undegraded":"2024-09-10T20:41:51.266576+0000","last_fullsized":"2024-09-10T20:41:51.266576+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:56.038042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267369+0000","last_change":"2024-09-10T20:41:50.264650+0000","last_active":"2024-09-10T20:41:51.267369+0000","last_peered":"2024-09-10T20:41:51.267369+0000","last_clean":"2024-09-10T20:41:51.267369+0000","last_became_active":"2024-09-10T20:41:50.264378+0000","last_became_peered":"2024-09-10T20:41:50.264378+0000","last_unstale":"2024-09-10T20:41:51.267369+0000","last_undegraded":"2024-09-10T20:41:51.267369+0000","last_fullsized":"2024-09-10T20:41:51.267369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:18.939715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266810+0000","last_change":"2024-09-10T20:41:50.254764+0000","last_active":"2024-09-10T20:41:51.266810+0000","last_peered":"2024-09-10T20:41:51.266810+0000","last_clean":"2024-09-10T20:41:51.266810+0000","last_became_active":"2024-09-10T20:41:50.254405+0000","last_became_peered":"2024-09-10T20:41:50.254405+0000","last_unstale":"2024-09-10T20:41:51.266810+0000","last_undegraded":"2024-09-10T20:41:51.266810+0000","last_fullsized":"2024-09-10T20:41:51.266810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:49:56.137817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267649+0000","last_change":"2024-09-10T20:41:50.259865+0000","last_active":"2024-09-10T20:41:51.267649+0000","last_peered":"2024-09-10T20:41:51.267649+0000","last_clean":"2024-09-10T20:41:51.267649+0000","last_became_active":"2024-09-10T20:41:50.259492+0000","last_became_peered":"2024-09-10T20:41:50.259492+0000","last_unstale":"2024-09-10T20:41:51.267649+0000","last_undegraded":"2024-09-10T20:41:51.267649+0000","last_fullsized":"2024-09-10T20:41:51.267649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:39.667663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274375+0000","last_change":"2024-09-10T20:41:50.256654+0000","last_active":"2024-09-10T20:41:52.274375+0000","last_peered":"2024-09-10T20:41:52.274375+0000","last_clean":"2024-09-10T20:41:52.274375+0000","last_became_active":"2024-09-10T20:41:50.256420+0000","last_became_peered":"2024-09-10T20:41:50.256420+0000","last_unstale":"2024-09-10T20:41:52.274375+0000","last_undegraded":"2024-09-10T20:41:52.274375+0000","last_fullsized":"2024-09-10T20:41:52.274375+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:05.552173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274417+0000","last_change":"2024-09-10T20:41:50.263513+0000","last_active":"2024-09-10T20:41:52.274417+0000","last_peered":"2024-09-10T20:41:52.274417+0000","last_clean":"2024-09-10T20:41:52.274417+0000","last_became_active":"2024-09-10T20:41:50.263228+0000","last_became_peered":"2024-09-10T20:41:50.263228+0000","last_unstale":"2024-09-10T20:41:52.274417+0000","last_undegraded":"2024-09-10T20:41:52.274417+0000","last_fullsized":"2024-09-10T20:41:52.274417+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:07.623838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268102+0000","last_change":"2024-09-10T20:41:50.264629+0000","last_active":"2024-09-10T20:41:51.268102+0000","last_peered":"2024-09-10T20:41:51.268102+0000","last_clean":"2024-09-10T20:41:51.268102+0000","last_became_active":"2024-09-10T20:41:50.264345+0000","last_became_peered":"2024-09-10T20:41:50.264345+0000","last_unstale":"2024-09-10T20:41:51.268102+0000","last_undegraded":"2024-09-10T20:41:51.268102+0000","last_fullsized":"2024-09-10T20:41:51.268102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:06:28.680157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273868+0000","last_change":"2024-09-10T20:41:50.265875+0000","last_active":"2024-09-10T20:41:52.273868+0000","last_peered":"2024-09-10T20:41:52.273868+0000","last_clean":"2024-09-10T20:41:52.273868+0000","last_became_active":"2024-09-10T20:41:50.265580+0000","last_became_peered":"2024-09-10T20:41:50.265580+0000","last_unstale":"2024-09-10T20:41:52.273868+0000","last_undegraded":"2024-09-10T20:41:52.273868+0000","last_fullsized":"2024-09-10T20:41:52.273868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:52:36.275643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274148+0000","last_change":"2024-09-10T20:41:50.255889+0000","last_active":"2024-09-10T20:41:52.274148+0000","last_peered":"2024-09-10T20:41:52.274148+0000","last_clean":"2024-09-10T20:41:52.274148+0000","last_became_active":"2024-09-10T20:41:50.255572+0000","last_became_peered":"2024-09-10T20:41:50.255572+0000","last_unstale":"2024-09-10T20:41:52.274148+0000","last_undegraded":"2024-09-10T20:41:52.274148+0000","last_fullsized":"2024-09-10T20:41:52.274148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:53.037486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274318+0000","last_change":"2024-09-10T20:41:50.265660+0000","last_active":"2024-09-10T20:41:52.274318+0000","last_peered":"2024-09-10T20:41:52.274318+0000","last_clean":"2024-09-10T20:41:52.274318+0000","last_became_active":"2024-09-10T20:41:50.265364+0000","last_became_peered":"2024-09-10T20:41:50.265364+0000","last_unstale":"2024-09-10T20:41:52.274318+0000","last_undegraded":"2024-09-10T20:41:52.274318+0000","last_fullsized":"2024-09-10T20:41:52.274318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:43.142674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.260988+0000","last_change":"2024-09-10T20:41:50.263991+0000","last_active":"2024-09-10T20:41:51.260988+0000","last_peered":"2024-09-10T20:41:51.260988+0000","last_clean":"2024-09-10T20:41:51.260988+0000","last_became_active":"2024-09-10T20:41:50.263690+0000","last_became_peered":"2024-09-10T20:41:50.263690+0000","last_unstale":"2024-09-10T20:41:51.260988+0000","last_undegraded":"2024-09-10T20:41:51.260988+0000","last_fullsized":"2024-09-10T20:41:51.260988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:23.522789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267882+0000","last_change":"2024-09-10T20:41:50.261825+0000","last_active":"2024-09-10T20:41:51.267882+0000","last_peered":"2024-09-10T20:41:51.267882+0000","last_clean":"2024-09-10T20:41:51.267882+0000","last_became_active":"2024-09-10T20:41:50.261516+0000","last_became_peered":"2024-09-10T20:41:50.261516+0000","last_unstale":"2024-09-10T20:41:51.267882+0000","last_undegraded":"2024-09-10T20:41:51.267882+0000","last_fullsized":"2024-09-10T20:41:51.267882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:24:10.017612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261322+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:51.261322+0000","last_peered":"2024-09-10T20:41:51.261322+0000","last_clean":"2024-09-10T20:41:51.261322+0000","last_became_active":"2024-09-10T20:41:50.272144+0000","last_became_peered":"2024-09-10T20:41:50.272144+0000","last_unstale":"2024-09-10T20:41:51.261322+0000","last_undegraded":"2024-09-10T20:41:51.261322+0000","last_fullsized":"2024-09-10T20:41:51.261322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:45.499279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268013+0000","last_change":"2024-09-10T20:41:50.273908+0000","last_active":"2024-09-10T20:41:51.268013+0000","last_peered":"2024-09-10T20:41:51.268013+0000","last_clean":"2024-09-10T20:41:51.268013+0000","last_became_active":"2024-09-10T20:41:50.273619+0000","last_became_peered":"2024-09-10T20:41:50.273619+0000","last_unstale":"2024-09-10T20:41:51.268013+0000","last_undegraded":"2024-09-10T20:41:51.268013+0000","last_fullsized":"2024-09-10T20:41:51.268013+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:43:09.098364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274226+0000","last_change":"2024-09-10T20:41:50.262644+0000","last_active":"2024-09-10T20:41:52.274226+0000","last_peered":"2024-09-10T20:41:52.274226+0000","last_clean":"2024-09-10T20:41:52.274226+0000","last_became_active":"2024-09-10T20:41:50.262373+0000","last_became_peered":"2024-09-10T20:41:50.262373+0000","last_unstale":"2024-09-10T20:41:52.274226+0000","last_undegraded":"2024-09-10T20:41:52.274226+0000","last_fullsized":"2024-09-10T20:41:52.274226+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:50:41.231413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274339+0000","last_change":"2024-09-10T20:41:50.267808+0000","last_active":"2024-09-10T20:41:52.274339+0000","last_peered":"2024-09-10T20:41:52.274339+0000","last_clean":"2024-09-10T20:41:52.274339+0000","last_became_active":"2024-09-10T20:41:50.267481+0000","last_became_peered":"2024-09-10T20:41:50.267481+0000","last_unstale":"2024-09-10T20:41:52.274339+0000","last_undegraded":"2024-09-10T20:41:52.274339+0000","last_fullsized":"2024-09-10T20:41:52.274339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:49.209128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266595+0000","last_change":"2024-09-10T20:41:50.261583+0000","last_active":"2024-09-10T20:41:51.266595+0000","last_peered":"2024-09-10T20:41:51.266595+0000","last_clean":"2024-09-10T20:41:51.266595+0000","last_became_active":"2024-09-10T20:41:50.261310+0000","last_became_peered":"2024-09-10T20:41:50.261310+0000","last_unstale":"2024-09-10T20:41:51.266595+0000","last_undegraded":"2024-09-10T20:41:51.266595+0000","last_fullsized":"2024-09-10T20:41:51.266595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:27:12.778590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267882+0000","last_change":"2024-09-10T20:41:50.263690+0000","last_active":"2024-09-10T20:41:51.267882+0000","last_peered":"2024-09-10T20:41:51.267882+0000","last_clean":"2024-09-10T20:41:51.267882+0000","last_became_active":"2024-09-10T20:41:50.263407+0000","last_became_peered":"2024-09-10T20:41:50.263407+0000","last_unstale":"2024-09-10T20:41:51.267882+0000","last_undegraded":"2024-09-10T20:41:51.267882+0000","last_fullsized":"2024-09-10T20:41:51.267882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:42.942447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274424+0000","last_change":"2024-09-10T20:41:50.266413+0000","last_active":"2024-09-10T20:41:52.274424+0000","last_peered":"2024-09-10T20:41:52.274424+0000","last_clean":"2024-09-10T20:41:52.274424+0000","last_became_active":"2024-09-10T20:41:50.266150+0000","last_became_peered":"2024-09-10T20:41:50.266150+0000","last_unstale":"2024-09-10T20:41:52.274424+0000","last_undegraded":"2024-09-10T20:41:52.274424+0000","last_fullsized":"2024-09-10T20:41:52.274424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:14.928819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273746+0000","last_change":"2024-09-10T20:41:50.267692+0000","last_active":"2024-09-10T20:41:52.273746+0000","last_peered":"2024-09-10T20:41:52.273746+0000","last_clean":"2024-09-10T20:41:52.273746+0000","last_became_active":"2024-09-10T20:41:50.267181+0000","last_became_peered":"2024-09-10T20:41:50.267181+0000","last_unstale":"2024-09-10T20:41:52.273746+0000","last_undegraded":"2024-09-10T20:41:52.273746+0000","last_fullsized":"2024-09-10T20:41:52.273746+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:58:03.570884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268217+0000","last_change":"2024-09-10T20:41:50.260450+0000","last_active":"2024-09-10T20:41:51.268217+0000","last_peered":"2024-09-10T20:41:51.268217+0000","last_clean":"2024-09-10T20:41:51.268217+0000","last_became_active":"2024-09-10T20:41:50.260093+0000","last_became_peered":"2024-09-10T20:41:50.260093+0000","last_unstale":"2024-09-10T20:41:51.268217+0000","last_undegraded":"2024-09-10T20:41:51.268217+0000","last_fullsized":"2024-09-10T20:41:51.268217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:22:28.739089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274198+0000","last_change":"2024-09-10T20:41:49.244556+0000","last_active":"2024-09-10T20:41:52.274198+0000","last_peered":"2024-09-10T20:41:52.274198+0000","last_clean":"2024-09-10T20:41:52.274198+0000","last_became_active":"2024-09-10T20:41:49.244277+0000","last_became_peered":"2024-09-10T20:41:49.244277+0000","last_unstale":"2024-09-10T20:41:52.274198+0000","last_undegraded":"2024-09-10T20:41:52.274198+0000","last_fullsized":"2024-09-10T20:41:52.274198+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:24.745124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267409+0000","last_change":"2024-09-10T20:41:50.262712+0000","last_active":"2024-09-10T20:41:51.267409+0000","last_peered":"2024-09-10T20:41:51.267409+0000","last_clean":"2024-09-10T20:41:51.267409+0000","last_became_active":"2024-09-10T20:41:50.262402+0000","last_became_peered":"2024-09-10T20:41:50.262402+0000","last_unstale":"2024-09-10T20:41:51.267409+0000","last_undegraded":"2024-09-10T20:41:51.267409+0000","last_fullsized":"2024-09-10T20:41:51.267409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:58.091089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274563+0000","last_change":"2024-09-10T20:41:49.236963+0000","last_active":"2024-09-10T20:41:52.274563+0000","last_peered":"2024-09-10T20:41:52.274563+0000","last_clean":"2024-09-10T20:41:52.274563+0000","last_became_active":"2024-09-10T20:41:49.236654+0000","last_became_peered":"2024-09-10T20:41:49.236654+0000","last_unstale":"2024-09-10T20:41:52.274563+0000","last_undegraded":"2024-09-10T20:41:52.274563+0000","last_fullsized":"2024-09-10T20:41:52.274563+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:15:31.822564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274674+0000","last_change":"2024-09-10T20:41:50.277395+0000","last_active":"2024-09-10T20:41:52.274674+0000","last_peered":"2024-09-10T20:41:52.274674+0000","last_clean":"2024-09-10T20:41:52.274674+0000","last_became_active":"2024-09-10T20:41:50.277101+0000","last_became_peered":"2024-09-10T20:41:50.277101+0000","last_unstale":"2024-09-10T20:41:52.274674+0000","last_undegraded":"2024-09-10T20:41:52.274674+0000","last_fullsized":"2024-09-10T20:41:52.274674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:49.209128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268254+0000","last_change":"2024-09-10T20:41:49.238213+0000","last_active":"2024-09-10T20:41:51.268254+0000","last_peered":"2024-09-10T20:41:51.268254+0000","last_clean":"2024-09-10T20:41:51.268254+0000","last_became_active":"2024-09-10T20:41:49.237949+0000","last_became_peered":"2024-09-10T20:41:49.237949+0000","last_unstale":"2024-09-10T20:41:51.268254+0000","last_undegraded":"2024-09-10T20:41:51.268254+0000","last_fullsized":"2024-09-10T20:41:51.268254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:45:22.532346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261252+0000","last_change":"2024-09-10T20:41:50.262357+0000","last_active":"2024-09-10T20:41:51.261252+0000","last_peered":"2024-09-10T20:41:51.261252+0000","last_clean":"2024-09-10T20:41:51.261252+0000","last_became_active":"2024-09-10T20:41:50.262081+0000","last_became_peered":"2024-09-10T20:41:50.262081+0000","last_unstale":"2024-09-10T20:41:51.261252+0000","last_undegraded":"2024-09-10T20:41:51.261252+0000","last_fullsized":"2024-09-10T20:41:51.261252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:57.612873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274809+0000","last_change":"2024-09-10T20:41:49.247040+0000","last_active":"2024-09-10T20:41:52.274809+0000","last_peered":"2024-09-10T20:41:52.274809+0000","last_clean":"2024-09-10T20:41:52.274809+0000","last_became_active":"2024-09-10T20:41:49.246731+0000","last_became_peered":"2024-09-10T20:41:49.246731+0000","last_unstale":"2024-09-10T20:41:52.274809+0000","last_undegraded":"2024-09-10T20:41:52.274809+0000","last_fullsized":"2024-09-10T20:41:52.274809+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:37.379165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267791+0000","last_change":"2024-09-10T20:41:50.261513+0000","last_active":"2024-09-10T20:41:51.267791+0000","last_peered":"2024-09-10T20:41:51.267791+0000","last_clean":"2024-09-10T20:41:51.267791+0000","last_became_active":"2024-09-10T20:41:50.261203+0000","last_became_peered":"2024-09-10T20:41:50.261203+0000","last_unstale":"2024-09-10T20:41:51.267791+0000","last_undegraded":"2024-09-10T20:41:51.267791+0000","last_fullsized":"2024-09-10T20:41:51.267791+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:21.848410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266565+0000","last_change":"2024-09-10T20:41:49.237350+0000","last_active":"2024-09-10T20:41:51.266565+0000","last_peered":"2024-09-10T20:41:51.266565+0000","last_clean":"2024-09-10T20:41:51.266565+0000","last_became_active":"2024-09-10T20:41:49.237086+0000","last_became_peered":"2024-09-10T20:41:49.237086+0000","last_unstale":"2024-09-10T20:41:51.266565+0000","last_undegraded":"2024-09-10T20:41:51.266565+0000","last_fullsized":"2024-09-10T20:41:51.266565+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:47:00.464349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273770+0000","last_change":"2024-09-10T20:41:50.275485+0000","last_active":"2024-09-10T20:41:52.273770+0000","last_peered":"2024-09-10T20:41:52.273770+0000","last_clean":"2024-09-10T20:41:52.273770+0000","last_became_active":"2024-09-10T20:41:50.275179+0000","last_became_peered":"2024-09-10T20:41:50.275179+0000","last_unstale":"2024-09-10T20:41:52.273770+0000","last_undegraded":"2024-09-10T20:41:52.273770+0000","last_fullsized":"2024-09-10T20:41:52.273770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:55.179983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.260887+0000","last_change":"2024-09-10T20:41:49.245982+0000","last_active":"2024-09-10T20:41:51.260887+0000","last_peered":"2024-09-10T20:41:51.260887+0000","last_clean":"2024-09-10T20:41:51.260887+0000","last_became_active":"2024-09-10T20:41:49.245669+0000","last_became_peered":"2024-09-10T20:41:49.245669+0000","last_unstale":"2024-09-10T20:41:51.260887+0000","last_undegraded":"2024-09-10T20:41:51.260887+0000","last_fullsized":"2024-09-10T20:41:51.260887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:21:05.620306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.724176+0000","last_change":"2024-09-10T20:41:50.274850+0000","last_active":"2024-09-10T20:41:51.724176+0000","last_peered":"2024-09-10T20:41:51.724176+0000","last_clean":"2024-09-10T20:41:51.724176+0000","last_became_active":"2024-09-10T20:41:50.274600+0000","last_became_peered":"2024-09-10T20:41:50.274600+0000","last_unstale":"2024-09-10T20:41:51.724176+0000","last_undegraded":"2024-09-10T20:41:51.724176+0000","last_fullsized":"2024-09-10T20:41:51.724176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:22.562059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.724253+0000","last_change":"2024-09-10T20:41:49.246041+0000","last_active":"2024-09-10T20:41:51.724253+0000","last_peered":"2024-09-10T20:41:51.724253+0000","last_clean":"2024-09-10T20:41:51.724253+0000","last_became_active":"2024-09-10T20:41:49.245783+0000","last_became_peered":"2024-09-10T20:41:49.245783+0000","last_unstale":"2024-09-10T20:41:51.724253+0000","last_undegraded":"2024-09-10T20:41:51.724253+0000","last_fullsized":"2024-09-10T20:41:51.724253+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:40.500932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266864+0000","last_change":"2024-09-10T20:41:50.265406+0000","last_active":"2024-09-10T20:41:51.266864+0000","last_peered":"2024-09-10T20:41:51.266864+0000","last_clean":"2024-09-10T20:41:51.266864+0000","last_became_active":"2024-09-10T20:41:50.264997+0000","last_became_peered":"2024-09-10T20:41:50.264997+0000","last_unstale":"2024-09-10T20:41:51.266864+0000","last_undegraded":"2024-09-10T20:41:51.266864+0000","last_fullsized":"2024-09-10T20:41:51.266864+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:59:18.291683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266384+0000","last_change":"2024-09-10T20:41:49.238563+0000","last_active":"2024-09-10T20:41:51.266384+0000","last_peered":"2024-09-10T20:41:51.266384+0000","last_clean":"2024-09-10T20:41:51.266384+0000","last_became_active":"2024-09-10T20:41:49.238256+0000","last_became_peered":"2024-09-10T20:41:49.238256+0000","last_unstale":"2024-09-10T20:41:51.266384+0000","last_undegraded":"2024-09-10T20:41:51.266384+0000","last_fullsized":"2024-09-10T20:41:51.266384+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:08.603112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261584+0000","last_change":"2024-09-10T20:41:50.273036+0000","last_active":"2024-09-10T20:41:51.261584+0000","last_peered":"2024-09-10T20:41:51.261584+0000","last_clean":"2024-09-10T20:41:51.261584+0000","last_became_active":"2024-09-10T20:41:50.272751+0000","last_became_peered":"2024-09-10T20:41:50.272751+0000","last_unstale":"2024-09-10T20:41:51.261584+0000","last_undegraded":"2024-09-10T20:41:51.261584+0000","last_fullsized":"2024-09-10T20:41:51.261584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:54.400821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267771+0000","last_change":"2024-09-10T20:41:49.239816+0000","last_active":"2024-09-10T20:41:51.267771+0000","last_peered":"2024-09-10T20:41:51.267771+0000","last_clean":"2024-09-10T20:41:51.267771+0000","last_became_active":"2024-09-10T20:41:49.239509+0000","last_became_peered":"2024-09-10T20:41:49.239509+0000","last_unstale":"2024-09-10T20:41:51.267771+0000","last_undegraded":"2024-09-10T20:41:51.267771+0000","last_fullsized":"2024-09-10T20:41:51.267771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:30.409719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274245+0000","last_change":"2024-09-10T20:41:50.268159+0000","last_active":"2024-09-10T20:41:52.274245+0000","last_peered":"2024-09-10T20:41:52.274245+0000","last_clean":"2024-09-10T20:41:52.274245+0000","last_became_active":"2024-09-10T20:41:50.267522+0000","last_became_peered":"2024-09-10T20:41:50.267522+0000","last_unstale":"2024-09-10T20:41:52.274245+0000","last_undegraded":"2024-09-10T20:41:52.274245+0000","last_fullsized":"2024-09-10T20:41:52.274245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:26:09.551252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274203+0000","last_change":"2024-09-10T20:41:49.243018+0000","last_active":"2024-09-10T20:41:52.274203+0000","last_peered":"2024-09-10T20:41:52.274203+0000","last_clean":"2024-09-10T20:41:52.274203+0000","last_became_active":"2024-09-10T20:41:49.242751+0000","last_became_peered":"2024-09-10T20:41:49.242751+0000","last_unstale":"2024-09-10T20:41:52.274203+0000","last_undegraded":"2024-09-10T20:41:52.274203+0000","last_fullsized":"2024-09-10T20:41:52.274203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:47.281477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268073+0000","last_change":"2024-09-10T20:41:50.272420+0000","last_active":"2024-09-10T20:41:51.268073+0000","last_peered":"2024-09-10T20:41:51.268073+0000","last_clean":"2024-09-10T20:41:51.268073+0000","last_became_active":"2024-09-10T20:41:50.272083+0000","last_became_peered":"2024-09-10T20:41:50.272083+0000","last_unstale":"2024-09-10T20:41:51.268073+0000","last_undegraded":"2024-09-10T20:41:51.268073+0000","last_fullsized":"2024-09-10T20:41:51.268073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:09:16.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267636+0000","last_change":"2024-09-10T20:41:49.235825+0000","last_active":"2024-09-10T20:41:51.267636+0000","last_peered":"2024-09-10T20:41:51.267636+0000","last_clean":"2024-09-10T20:41:51.267636+0000","last_became_active":"2024-09-10T20:41:49.235466+0000","last_became_peered":"2024-09-10T20:41:49.235466+0000","last_unstale":"2024-09-10T20:41:51.267636+0000","last_undegraded":"2024-09-10T20:41:51.267636+0000","last_fullsized":"2024-09-10T20:41:51.267636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:26:29.300608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274357+0000","last_change":"2024-09-10T20:41:50.261225+0000","last_active":"2024-09-10T20:41:52.274357+0000","last_peered":"2024-09-10T20:41:52.274357+0000","last_clean":"2024-09-10T20:41:52.274357+0000","last_became_active":"2024-09-10T20:41:50.260910+0000","last_became_peered":"2024-09-10T20:41:50.260910+0000","last_unstale":"2024-09-10T20:41:52.274357+0000","last_undegraded":"2024-09-10T20:41:52.274357+0000","last_fullsized":"2024-09-10T20:41:52.274357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:31.114649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":21,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.724571+0000","last_change":"2024-09-10T20:41:49.241236+0000","last_active":"2024-09-10T20:41:51.724571+0000","last_peered":"2024-09-10T20:41:51.724571+0000","last_clean":"2024-09-10T20:41:51.724571+0000","last_became_active":"2024-09-10T20:41:49.240686+0000","last_became_peered":"2024-09-10T20:41:49.240686+0000","last_unstale":"2024-09-10T20:41:51.724571+0000","last_undegraded":"2024-09-10T20:41:51.724571+0000","last_fullsized":"2024-09-10T20:41:51.724571+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:56.607029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274412+0000","last_change":"2024-09-10T20:41:50.262537+0000","last_active":"2024-09-10T20:41:52.274412+0000","last_peered":"2024-09-10T20:41:52.274412+0000","last_clean":"2024-09-10T20:41:52.274412+0000","last_became_active":"2024-09-10T20:41:50.261672+0000","last_became_peered":"2024-09-10T20:41:50.261672+0000","last_unstale":"2024-09-10T20:41:52.274412+0000","last_undegraded":"2024-09-10T20:41:52.274412+0000","last_fullsized":"2024-09-10T20:41:52.274412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:26:09.551252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.724272+0000","last_change":"2024-09-10T20:41:49.241419+0000","last_active":"2024-09-10T20:41:51.724272+0000","last_peered":"2024-09-10T20:41:51.724272+0000","last_clean":"2024-09-10T20:41:51.724272+0000","last_became_active":"2024-09-10T20:41:49.241125+0000","last_became_peered":"2024-09-10T20:41:49.241125+0000","last_unstale":"2024-09-10T20:41:51.724272+0000","last_undegraded":"2024-09-10T20:41:51.724272+0000","last_fullsized":"2024-09-10T20:41:51.724272+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:44.493434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261080+0000","last_change":"2024-09-10T20:41:50.272126+0000","last_active":"2024-09-10T20:41:51.261080+0000","last_peered":"2024-09-10T20:41:51.261080+0000","last_clean":"2024-09-10T20:41:51.261080+0000","last_became_active":"2024-09-10T20:41:50.271842+0000","last_became_peered":"2024-09-10T20:41:50.271842+0000","last_unstale":"2024-09-10T20:41:51.261080+0000","last_undegraded":"2024-09-10T20:41:51.261080+0000","last_fullsized":"2024-09-10T20:41:51.261080+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:26.282250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.260987+0000","last_change":"2024-09-10T20:41:49.248639+0000","last_active":"2024-09-10T20:41:51.260987+0000","last_peered":"2024-09-10T20:41:51.260987+0000","last_clean":"2024-09-10T20:41:51.260987+0000","last_became_active":"2024-09-10T20:41:49.248352+0000","last_became_peered":"2024-09-10T20:41:49.248352+0000","last_unstale":"2024-09-10T20:41:51.260987+0000","last_undegraded":"2024-09-10T20:41:51.260987+0000","last_fullsized":"2024-09-10T20:41:51.260987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:41.288742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273748+0000","last_change":"2024-09-10T20:41:50.263368+0000","last_active":"2024-09-10T20:41:52.273748+0000","last_peered":"2024-09-10T20:41:52.273748+0000","last_clean":"2024-09-10T20:41:52.273748+0000","last_became_active":"2024-09-10T20:41:50.263091+0000","last_became_peered":"2024-09-10T20:41:50.263091+0000","last_unstale":"2024-09-10T20:41:52.273748+0000","last_undegraded":"2024-09-10T20:41:52.273748+0000","last_fullsized":"2024-09-10T20:41:52.273748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:12:07.021900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261097+0000","last_change":"2024-09-10T20:41:49.249615+0000","last_active":"2024-09-10T20:41:51.261097+0000","last_peered":"2024-09-10T20:41:51.261097+0000","last_clean":"2024-09-10T20:41:51.261097+0000","last_became_active":"2024-09-10T20:41:49.249414+0000","last_became_peered":"2024-09-10T20:41:49.249414+0000","last_unstale":"2024-09-10T20:41:51.261097+0000","last_undegraded":"2024-09-10T20:41:51.261097+0000","last_fullsized":"2024-09-10T20:41:51.261097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:36.540914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266331+0000","last_change":"2024-09-10T20:41:50.263751+0000","last_active":"2024-09-10T20:41:51.266331+0000","last_peered":"2024-09-10T20:41:51.266331+0000","last_clean":"2024-09-10T20:41:51.266331+0000","last_became_active":"2024-09-10T20:41:50.263496+0000","last_became_peered":"2024-09-10T20:41:50.263496+0000","last_unstale":"2024-09-10T20:41:51.266331+0000","last_undegraded":"2024-09-10T20:41:51.266331+0000","last_fullsized":"2024-09-10T20:41:51.266331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:44:33.443686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.260771+0000","last_change":"2024-09-10T20:41:49.244860+0000","last_active":"2024-09-10T20:41:51.260771+0000","last_peered":"2024-09-10T20:41:51.260771+0000","last_clean":"2024-09-10T20:41:51.260771+0000","last_became_active":"2024-09-10T20:41:49.244532+0000","last_became_peered":"2024-09-10T20:41:49.244532+0000","last_unstale":"2024-09-10T20:41:51.260771+0000","last_undegraded":"2024-09-10T20:41:51.260771+0000","last_fullsized":"2024-09-10T20:41:51.260771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:33.273565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274025+0000","last_change":"2024-09-10T20:41:50.255850+0000","last_active":"2024-09-10T20:41:52.274025+0000","last_peered":"2024-09-10T20:41:52.274025+0000","last_clean":"2024-09-10T20:41:52.274025+0000","last_became_active":"2024-09-10T20:41:50.254971+0000","last_became_peered":"2024-09-10T20:41:50.254971+0000","last_unstale":"2024-09-10T20:41:52.274025+0000","last_undegraded":"2024-09-10T20:41:52.274025+0000","last_fullsized":"2024-09-10T20:41:52.274025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:37:44.077305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268386+0000","last_change":"2024-09-10T20:41:49.239536+0000","last_active":"2024-09-10T20:41:51.268386+0000","last_peered":"2024-09-10T20:41:51.268386+0000","last_clean":"2024-09-10T20:41:51.268386+0000","last_became_active":"2024-09-10T20:41:49.239253+0000","last_became_peered":"2024-09-10T20:41:49.239253+0000","last_unstale":"2024-09-10T20:41:51.268386+0000","last_undegraded":"2024-09-10T20:41:51.268386+0000","last_fullsized":"2024-09-10T20:41:51.268386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:15.552579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274508+0000","last_change":"2024-09-10T20:41:50.251514+0000","last_active":"2024-09-10T20:41:52.274508+0000","last_peered":"2024-09-10T20:41:52.274508+0000","last_clean":"2024-09-10T20:41:52.274508+0000","last_became_active":"2024-09-10T20:41:50.251241+0000","last_became_peered":"2024-09-10T20:41:50.251241+0000","last_unstale":"2024-09-10T20:41:52.274508+0000","last_undegraded":"2024-09-10T20:41:52.274508+0000","last_fullsized":"2024-09-10T20:41:52.274508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:27.638850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274322+0000","last_change":"2024-09-10T20:41:49.248393+0000","last_active":"2024-09-10T20:41:52.274322+0000","last_peered":"2024-09-10T20:41:52.274322+0000","last_clean":"2024-09-10T20:41:52.274322+0000","last_became_active":"2024-09-10T20:41:49.248103+0000","last_became_peered":"2024-09-10T20:41:49.248103+0000","last_unstale":"2024-09-10T20:41:52.274322+0000","last_undegraded":"2024-09-10T20:41:52.274322+0000","last_fullsized":"2024-09-10T20:41:52.274322+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:54.196476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267105+0000","last_change":"2024-09-10T20:41:50.261231+0000","last_active":"2024-09-10T20:41:51.267105+0000","last_peered":"2024-09-10T20:41:51.267105+0000","last_clean":"2024-09-10T20:41:51.267105+0000","last_became_active":"2024-09-10T20:41:50.260958+0000","last_became_peered":"2024-09-10T20:41:50.260958+0000","last_unstale":"2024-09-10T20:41:51.267105+0000","last_undegraded":"2024-09-10T20:41:51.267105+0000","last_fullsized":"2024-09-10T20:41:51.267105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:01:27.069259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273746+0000","last_change":"2024-09-10T20:41:49.239383+0000","last_active":"2024-09-10T20:41:52.273746+0000","last_peered":"2024-09-10T20:41:52.273746+0000","last_clean":"2024-09-10T20:41:52.273746+0000","last_became_active":"2024-09-10T20:41:49.239097+0000","last_became_peered":"2024-09-10T20:41:49.239097+0000","last_unstale":"2024-09-10T20:41:52.273746+0000","last_undegraded":"2024-09-10T20:41:52.273746+0000","last_fullsized":"2024-09-10T20:41:52.273746+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:51.420841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267203+0000","last_change":"2024-09-10T20:41:50.272314+0000","last_active":"2024-09-10T20:41:51.267203+0000","last_peered":"2024-09-10T20:41:51.267203+0000","last_clean":"2024-09-10T20:41:51.267203+0000","last_became_active":"2024-09-10T20:41:50.272021+0000","last_became_peered":"2024-09-10T20:41:50.272021+0000","last_unstale":"2024-09-10T20:41:51.267203+0000","last_undegraded":"2024-09-10T20:41:51.267203+0000","last_fullsized":"2024-09-10T20:41:51.267203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:24:57.838349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274809+0000","last_change":"2024-09-10T20:41:49.243612+0000","last_active":"2024-09-10T20:41:52.274809+0000","last_peered":"2024-09-10T20:41:52.274809+0000","last_clean":"2024-09-10T20:41:52.274809+0000","last_became_active":"2024-09-10T20:41:49.243065+0000","last_became_peered":"2024-09-10T20:41:49.243065+0000","last_unstale":"2024-09-10T20:41:52.274809+0000","last_undegraded":"2024-09-10T20:41:52.274809+0000","last_fullsized":"2024-09-10T20:41:52.274809+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:20:31.645185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.724282+0000","last_change":"2024-09-10T20:41:50.260036+0000","last_active":"2024-09-10T20:41:51.724282+0000","last_peered":"2024-09-10T20:41:51.724282+0000","last_clean":"2024-09-10T20:41:51.724282+0000","last_became_active":"2024-09-10T20:41:50.259745+0000","last_became_peered":"2024-09-10T20:41:50.259745+0000","last_unstale":"2024-09-10T20:41:51.724282+0000","last_undegraded":"2024-09-10T20:41:51.724282+0000","last_fullsized":"2024-09-10T20:41:51.724282+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:20.996250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274489+0000","last_change":"2024-09-10T20:41:49.241358+0000","last_active":"2024-09-10T20:41:52.274489+0000","last_peered":"2024-09-10T20:41:52.274489+0000","last_clean":"2024-09-10T20:41:52.274489+0000","last_became_active":"2024-09-10T20:41:49.241098+0000","last_became_peered":"2024-09-10T20:41:49.241098+0000","last_unstale":"2024-09-10T20:41:52.274489+0000","last_undegraded":"2024-09-10T20:41:52.274489+0000","last_fullsized":"2024-09-10T20:41:52.274489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:53:49.000467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.724365+0000","last_change":"2024-09-10T20:41:50.261560+0000","last_active":"2024-09-10T20:41:51.724365+0000","last_peered":"2024-09-10T20:41:51.724365+0000","last_clean":"2024-09-10T20:41:51.724365+0000","last_became_active":"2024-09-10T20:41:50.261218+0000","last_became_peered":"2024-09-10T20:41:50.261218+0000","last_unstale":"2024-09-10T20:41:51.724365+0000","last_undegraded":"2024-09-10T20:41:51.724365+0000","last_fullsized":"2024-09-10T20:41:51.724365+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:21:30.112985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274608+0000","last_change":"2024-09-10T20:41:49.234909+0000","last_active":"2024-09-10T20:41:52.274608+0000","last_peered":"2024-09-10T20:41:52.274608+0000","last_clean":"2024-09-10T20:41:52.274608+0000","last_became_active":"2024-09-10T20:41:49.234598+0000","last_became_peered":"2024-09-10T20:41:49.234598+0000","last_unstale":"2024-09-10T20:41:52.274608+0000","last_undegraded":"2024-09-10T20:41:52.274608+0000","last_fullsized":"2024-09-10T20:41:52.274608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:00.862079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267020+0000","last_change":"2024-09-10T20:41:50.259211+0000","last_active":"2024-09-10T20:41:51.267020+0000","last_peered":"2024-09-10T20:41:51.267020+0000","last_clean":"2024-09-10T20:41:51.267020+0000","last_became_active":"2024-09-10T20:41:50.258916+0000","last_became_peered":"2024-09-10T20:41:50.258916+0000","last_unstale":"2024-09-10T20:41:51.267020+0000","last_undegraded":"2024-09-10T20:41:51.267020+0000","last_fullsized":"2024-09-10T20:41:51.267020+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:16:38.334681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273601+0000","last_change":"2024-09-10T20:41:49.245102+0000","last_active":"2024-09-10T20:41:52.273601+0000","last_peered":"2024-09-10T20:41:52.273601+0000","last_clean":"2024-09-10T20:41:52.273601+0000","last_became_active":"2024-09-10T20:41:49.244790+0000","last_became_peered":"2024-09-10T20:41:49.244790+0000","last_unstale":"2024-09-10T20:41:52.273601+0000","last_undegraded":"2024-09-10T20:41:52.273601+0000","last_fullsized":"2024-09-10T20:41:52.273601+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:05:58.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274432+0000","last_change":"2024-09-10T20:41:50.261767+0000","last_active":"2024-09-10T20:41:52.274432+0000","last_peered":"2024-09-10T20:41:52.274432+0000","last_clean":"2024-09-10T20:41:52.274432+0000","last_became_active":"2024-09-10T20:41:50.260854+0000","last_became_peered":"2024-09-10T20:41:50.260854+0000","last_unstale":"2024-09-10T20:41:52.274432+0000","last_undegraded":"2024-09-10T20:41:52.274432+0000","last_fullsized":"2024-09-10T20:41:52.274432+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:27:54.914228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268447+0000","last_change":"2024-09-10T20:41:49.239083+0000","last_active":"2024-09-10T20:41:51.268447+0000","last_peered":"2024-09-10T20:41:51.268447+0000","last_clean":"2024-09-10T20:41:51.268447+0000","last_became_active":"2024-09-10T20:41:49.238741+0000","last_became_peered":"2024-09-10T20:41:49.238741+0000","last_unstale":"2024-09-10T20:41:51.268447+0000","last_undegraded":"2024-09-10T20:41:51.268447+0000","last_fullsized":"2024-09-10T20:41:51.268447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:09:56.852921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261320+0000","last_change":"2024-09-10T20:41:50.270889+0000","last_active":"2024-09-10T20:41:51.261320+0000","last_peered":"2024-09-10T20:41:51.261320+0000","last_clean":"2024-09-10T20:41:51.261320+0000","last_became_active":"2024-09-10T20:41:50.270603+0000","last_became_peered":"2024-09-10T20:41:50.270603+0000","last_unstale":"2024-09-10T20:41:51.261320+0000","last_undegraded":"2024-09-10T20:41:51.261320+0000","last_fullsized":"2024-09-10T20:41:51.261320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:41.506776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"68'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.268276+0000","last_change":"2024-09-10T20:41:49.249494+0000","last_active":"2024-09-10T20:41:51.268276+0000","last_peered":"2024-09-10T20:41:51.268276+0000","last_clean":"2024-09-10T20:41:51.268276+0000","last_became_active":"2024-09-10T20:41:49.249207+0000","last_became_peered":"2024-09-10T20:41:49.249207+0000","last_unstale":"2024-09-10T20:41:51.268276+0000","last_undegraded":"2024-09-10T20:41:51.268276+0000","last_fullsized":"2024-09-10T20:41:51.268276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:47.078899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274429+0000","last_change":"2024-09-10T20:41:50.265954+0000","last_active":"2024-09-10T20:41:52.274429+0000","last_peered":"2024-09-10T20:41:52.274429+0000","last_clean":"2024-09-10T20:41:52.274429+0000","last_became_active":"2024-09-10T20:41:50.265644+0000","last_became_peered":"2024-09-10T20:41:50.265644+0000","last_unstale":"2024-09-10T20:41:52.274429+0000","last_undegraded":"2024-09-10T20:41:52.274429+0000","last_fullsized":"2024-09-10T20:41:52.274429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:05:59.078190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267698+0000","last_change":"2024-09-10T20:41:49.238772+0000","last_active":"2024-09-10T20:41:51.267698+0000","last_peered":"2024-09-10T20:41:51.267698+0000","last_clean":"2024-09-10T20:41:51.267698+0000","last_became_active":"2024-09-10T20:41:49.238312+0000","last_became_peered":"2024-09-10T20:41:49.238312+0000","last_unstale":"2024-09-10T20:41:51.267698+0000","last_undegraded":"2024-09-10T20:41:51.267698+0000","last_fullsized":"2024-09-10T20:41:51.267698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:19:16.972667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261312+0000","last_change":"2024-09-10T20:41:50.265096+0000","last_active":"2024-09-10T20:41:51.261312+0000","last_peered":"2024-09-10T20:41:51.261312+0000","last_clean":"2024-09-10T20:41:51.261312+0000","last_became_active":"2024-09-10T20:41:50.264830+0000","last_became_peered":"2024-09-10T20:41:50.264830+0000","last_unstale":"2024-09-10T20:41:51.261312+0000","last_undegraded":"2024-09-10T20:41:51.261312+0000","last_fullsized":"2024-09-10T20:41:51.261312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:24.102619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267494+0000","last_change":"2024-09-10T20:41:49.236972+0000","last_active":"2024-09-10T20:41:51.267494+0000","last_peered":"2024-09-10T20:41:51.267494+0000","last_clean":"2024-09-10T20:41:51.267494+0000","last_became_active":"2024-09-10T20:41:49.236675+0000","last_became_peered":"2024-09-10T20:41:49.236675+0000","last_unstale":"2024-09-10T20:41:51.267494+0000","last_undegraded":"2024-09-10T20:41:51.267494+0000","last_fullsized":"2024-09-10T20:41:51.267494+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:09:53.261442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274068+0000","last_change":"2024-09-10T20:41:50.256387+0000","last_active":"2024-09-10T20:41:52.274068+0000","last_peered":"2024-09-10T20:41:52.274068+0000","last_clean":"2024-09-10T20:41:52.274068+0000","last_became_active":"2024-09-10T20:41:50.256098+0000","last_became_peered":"2024-09-10T20:41:50.256098+0000","last_unstale":"2024-09-10T20:41:52.274068+0000","last_undegraded":"2024-09-10T20:41:52.274068+0000","last_fullsized":"2024-09-10T20:41:52.274068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:22.597501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.266661+0000","last_change":"2024-09-10T20:41:49.235872+0000","last_active":"2024-09-10T20:41:51.266661+0000","last_peered":"2024-09-10T20:41:51.266661+0000","last_clean":"2024-09-10T20:41:51.266661+0000","last_became_active":"2024-09-10T20:41:49.235579+0000","last_became_peered":"2024-09-10T20:41:49.235579+0000","last_unstale":"2024-09-10T20:41:51.266661+0000","last_undegraded":"2024-09-10T20:41:51.266661+0000","last_fullsized":"2024-09-10T20:41:51.266661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:48:47.620557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.267959+0000","last_change":"2024-09-10T20:41:50.260226+0000","last_active":"2024-09-10T20:41:51.267959+0000","last_peered":"2024-09-10T20:41:51.267959+0000","last_clean":"2024-09-10T20:41:51.267959+0000","last_became_active":"2024-09-10T20:41:50.259625+0000","last_became_peered":"2024-09-10T20:41:50.259625+0000","last_unstale":"2024-09-10T20:41:51.267959+0000","last_undegraded":"2024-09-10T20:41:51.267959+0000","last_fullsized":"2024-09-10T20:41:51.267959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:25:53.173333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274117+0000","last_change":"2024-09-10T20:41:49.242417+0000","last_active":"2024-09-10T20:41:52.274117+0000","last_peered":"2024-09-10T20:41:52.274117+0000","last_clean":"2024-09-10T20:41:52.274117+0000","last_became_active":"2024-09-10T20:41:49.242161+0000","last_became_peered":"2024-09-10T20:41:49.242161+0000","last_unstale":"2024-09-10T20:41:52.274117+0000","last_undegraded":"2024-09-10T20:41:52.274117+0000","last_fullsized":"2024-09-10T20:41:52.274117+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:23:46.481592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.724536+0000","last_change":"2024-09-10T20:41:50.260915+0000","last_active":"2024-09-10T20:41:51.724536+0000","last_peered":"2024-09-10T20:41:51.724536+0000","last_clean":"2024-09-10T20:41:51.724536+0000","last_became_active":"2024-09-10T20:41:50.260571+0000","last_became_peered":"2024-09-10T20:41:50.260571+0000","last_unstale":"2024-09-10T20:41:51.724536+0000","last_undegraded":"2024-09-10T20:41:51.724536+0000","last_fullsized":"2024-09-10T20:41:51.724536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:14.639786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.274713+0000","last_change":"2024-09-10T20:41:49.240524+0000","last_active":"2024-09-10T20:41:52.274713+0000","last_peered":"2024-09-10T20:41:52.274713+0000","last_clean":"2024-09-10T20:41:52.274713+0000","last_became_active":"2024-09-10T20:41:49.240245+0000","last_became_peered":"2024-09-10T20:41:49.240245+0000","last_unstale":"2024-09-10T20:41:52.274713+0000","last_undegraded":"2024-09-10T20:41:52.274713+0000","last_fullsized":"2024-09-10T20:41:52.274713+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:14.912119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273748+0000","last_change":"2024-09-10T20:41:50.267529+0000","last_active":"2024-09-10T20:41:52.273748+0000","last_peered":"2024-09-10T20:41:52.273748+0000","last_clean":"2024-09-10T20:41:52.273748+0000","last_became_active":"2024-09-10T20:41:50.267222+0000","last_became_peered":"2024-09-10T20:41:50.267222+0000","last_unstale":"2024-09-10T20:41:52.273748+0000","last_undegraded":"2024-09-10T20:41:52.273748+0000","last_fullsized":"2024-09-10T20:41:52.273748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:42.534052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273659+0000","last_change":"2024-09-10T20:41:49.236467+0000","last_active":"2024-09-10T20:41:52.273659+0000","last_peered":"2024-09-10T20:41:52.273659+0000","last_clean":"2024-09-10T20:41:52.273659+0000","last_became_active":"2024-09-10T20:41:49.236170+0000","last_became_peered":"2024-09-10T20:41:49.236170+0000","last_unstale":"2024-09-10T20:41:52.273659+0000","last_undegraded":"2024-09-10T20:41:52.273659+0000","last_fullsized":"2024-09-10T20:41:52.273659+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:12.514693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:52.273954+0000","last_change":"2024-09-10T20:41:50.266388+0000","last_active":"2024-09-10T20:41:52.273954+0000","last_peered":"2024-09-10T20:41:52.273954+0000","last_clean":"2024-09-10T20:41:52.273954+0000","last_became_active":"2024-09-10T20:41:50.266109+0000","last_became_peered":"2024-09-10T20:41:50.266109+0000","last_unstale":"2024-09-10T20:41:52.273954+0000","last_undegraded":"2024-09-10T20:41:52.273954+0000","last_fullsized":"2024-09-10T20:41:52.273954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:49.221972+0000","last_clean_scrub_stamp":"2024-09-10T20:41:49.221972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:17.775089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.261230+0000","last_change":"2024-09-10T20:41:49.249357+0000","last_active":"2024-09-10T20:41:51.261230+0000","last_peered":"2024-09-10T20:41:51.261230+0000","last_clean":"2024-09-10T20:41:51.261230+0000","last_became_active":"2024-09-10T20:41:49.248955+0000","last_became_peered":"2024-09-10T20:41:49.248955+0000","last_unstale":"2024-09-10T20:41:51.261230+0000","last_undegraded":"2024-09-10T20:41:51.261230+0000","last_fullsized":"2024-09-10T20:41:51.261230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:41:48.216128+0000","last_clean_scrub_stamp":"2024-09-10T20:41:48.216128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:49:26.862222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":197,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-10T20:41:51.260851+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:41:51.260851+0000","last_peered":"2024-09-10T20:41:51.260851+0000","last_clean":"2024-09-10T20:41:51.260851+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:41:51.260851+0000","last_undegraded":"2024-09-10T20:41:51.260851+0000","last_fullsized":"2024-09-10T20:41:51.260851+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509491,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":831488,"data_stored":343412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509490,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":823296,"data_stored":343868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509490,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1413120,"data_stored":933654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509490,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":823296,"data_stored":343356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542194,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":831488,"data_stored":344898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542195,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27848,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343992,"statfs":{"total":96636764160,"available":96608247808,"internally_reserved":0,"allocated":835584,"data_stored":344932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542193,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":933744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542193,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27848,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343992,"statfs":{"total":96636764160,"available":96608247808,"internally_reserved":0,"allocated":831488,"data_stored":343980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:41:59.079 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:41:59.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:41:59.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/772171941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad015e1b0 msgr2=0x7f4ad00d1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/772171941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad015e1b0 0x7f4ad00d1c00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009830 tx=0x7f4ac002a480 comp rx=0 tx=0).stop 2024-09-10T20:41:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/772171941 shutdown_connections 2024-09-10T20:41:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/772171941 >> v2:172.21.15.159:3300/0 conn(0x7f4ad00b3010 0x7f4ad00d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/772171941 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad015e1b0 0x7f4ad00d1c00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/772171941 >> 172.21.15.28:0/772171941 conn(0x7f4ad009d240 msgr2=0x7f4ad00d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/772171941 shutdown_connections 2024-09-10T20:41:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.224+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/772171941 wait complete. 2024-09-10T20:41:59.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.226+0000 7f4ad5cb6640 1 Processor -- start 2024-09-10T20:41:59.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.226+0000 7f4ad5cb6640 1 -- start start 2024-09-10T20:41:59.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.226+0000 7f4ad5cb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad00b3010 0x7f4ad0151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.226+0000 7f4ad5cb6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad00d0f80 0x7f4ad01f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.226+0000 7f4ad5cb6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad015e1b0 0x7f4ad01f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.226+0000 7f4ad5cb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad00cf360 con 0x7f4ad00b3010 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4ad5cb6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad00cf1e0 con 0x7f4ad015e1b0 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4ad5cb6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4ad00cf4e0 con 0x7f4ad00d0f80 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad00b3010 0x7f4ad0151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad00b3010 0x7f4ad0151350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58402/0 (socket says 172.21.15.28:58402) 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 -- 172.21.15.28:0/4252086206 learned_addr learned my addr 172.21.15.28:0/4252086206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acffff640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad015e1b0 0x7f4ad01f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4aceffd640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad00d0f80 0x7f4ad01f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 -- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad00d0f80 msgr2=0x7f4ad01f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad00d0f80 0x7f4ad01f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 -- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad015e1b0 msgr2=0x7f4ad01f87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad015e1b0 0x7f4ad01f87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 -- 172.21.15.28:0/4252086206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad01f53c0 con 0x7f4ad00b3010 2024-09-10T20:41:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acffff640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad015e1b0 0x7f4ad01f87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:41:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4acf7fe640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad00b3010 0x7f4ad0151350 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4ac002aa70 tx=0x7f4ac0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4ab7fff640 1 -- 172.21.15.28:0/4252086206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0005c90 con 0x7f4ad00b3010 2024-09-10T20:41:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad00b8030 con 0x7f4ad00b3010 2024-09-10T20:41:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4ab7fff640 1 -- 172.21.15.28:0/4252086206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ac0031e50 con 0x7f4ad00b3010 2024-09-10T20:41:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.227+0000 7f4ab7fff640 1 -- 172.21.15.28:0/4252086206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0005260 con 0x7f4ad00b3010 2024-09-10T20:41:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.228+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad00ca1a0 con 0x7f4ad00b3010 2024-09-10T20:41:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.228+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ad00b2b90 con 0x7f4ad00b3010 2024-09-10T20:41:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.229+0000 7f4ab7fff640 1 -- 172.21.15.28:0/4252086206 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f4ac0005400 con 0x7f4ad00b3010 2024-09-10T20:41:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.230+0000 7f4ab7fff640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4abc0782d0 0x7f4abc07a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.230+0000 7f4ab7fff640 1 -- 172.21.15.28:0/4252086206 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f4ac00bb3a0 con 0x7f4ad00b3010 2024-09-10T20:41:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.230+0000 7f4aceffd640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4abc0782d0 0x7f4abc07a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.231+0000 7f4aceffd640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4abc0782d0 0x7f4abc07a790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00d4c50 tx=0x7f4ac4005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.234+0000 7f4ab7fff640 1 -- 172.21.15.28:0/4252086206 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4ac003f030 con 0x7f4ad00b3010 2024-09-10T20:41:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.495+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f4ad0076240 con 0x7f4ad00b3010 2024-09-10T20:41:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.497+0000 7f4ab7fff640 1 -- 172.21.15.28:0/4252086206 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v30) ==== 190+0+147 (secure 0 0 0) 0x7f4ac00bb1c0 con 0x7f4ad00b3010 2024-09-10T20:41:59.501 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:41:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:41:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:41:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:41:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:41:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:41:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4abc0782d0 msgr2=0x7f4abc07a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4abc0782d0 0x7f4abc07a790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00d4c50 tx=0x7f4ac4005f90 comp rx=0 tx=0).stop 2024-09-10T20:41:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad00b3010 msgr2=0x7f4ad0151350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad00b3010 0x7f4ad0151350 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4ac002aa70 tx=0x7f4ac0033c20 comp rx=0 tx=0).stop 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 shutdown_connections 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f4abc0782d0 0x7f4abc07a790 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ad015e1b0 0x7f4ad01f87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ad00d0f80 0x7f4ad01f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 --2- 172.21.15.28:0/4252086206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad00b3010 0x7f4ad0151350 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 >> 172.21.15.28:0/4252086206 conn(0x7f4ad009d240 msgr2=0x7f4ad00b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 shutdown_connections 2024-09-10T20:41:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.499+0000 7f4ad5cb6640 1 -- 172.21.15.28:0/4252086206 wait complete. 2024-09-10T20:41:59.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.615+0000 7f6110c33640 1 -- 172.21.15.28:0/3673545782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c15ed20 msgr2=0x7f610c0cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.615+0000 7f6110c33640 1 --2- 172.21.15.28:0/3673545782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c15ed20 0x7f610c0cb210 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f60fc005fd0 tx=0x7f60fc02a530 comp rx=0 tx=0).stop 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.616+0000 7f6110c33640 1 -- 172.21.15.28:0/3673545782 shutdown_connections 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.616+0000 7f6110c33640 1 --2- 172.21.15.28:0/3673545782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c15ed20 0x7f610c0cb210 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.616+0000 7f6110c33640 1 --2- 172.21.15.28:0/3673545782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f610c0cf0f0 0x7f610c15e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.616+0000 7f6110c33640 1 -- 172.21.15.28:0/3673545782 >> 172.21.15.28:0/3673545782 conn(0x7f610c0d0420 msgr2=0x7f610c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.616+0000 7f6110c33640 1 -- 172.21.15.28:0/3673545782 shutdown_connections 2024-09-10T20:41:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.616+0000 7f6110c33640 1 -- 172.21.15.28:0/3673545782 wait complete. 2024-09-10T20:41:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.617+0000 7f6110c33640 1 Processor -- start 2024-09-10T20:41:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.617+0000 7f6110c33640 1 -- start start 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6110c33640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f610c0cf0f0 0x7f610c0b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6110c33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c151cc0 0x7f610c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6110c33640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f610c15ed20 0x7f610c0b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6110c33640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f610c15f5b0 con 0x7f610c151cc0 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6110c33640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f610c15f430 con 0x7f610c15ed20 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6110c33640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f610c0cb750 con 0x7f610c0cf0f0 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c151cc0 0x7f610c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c151cc0 0x7f610c154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58428/0 (socket says 172.21.15.28:58428) 2024-09-10T20:41:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 -- 172.21.15.28:0/1396427171 learned_addr learned my addr 172.21.15.28:0/1396427171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f610ad9d640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f610c15ed20 0x7f610c0b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f610a59c640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f610c0cf0f0 0x7f610c0b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 -- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f610c0cf0f0 msgr2=0x7f610c0b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f610c0cf0f0 0x7f610c0b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 -- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f610c15ed20 msgr2=0x7f610c0b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f610c15ed20 0x7f610c0b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 -- 172.21.15.28:0/1396427171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c0b07b0 con 0x7f610c151cc0 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f610ad9d640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f610c15ed20 0x7f610c0b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f610a59c640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f610c0cf0f0 0x7f610c0b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6109d9b640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c151cc0 0x7f610c154a60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f60f400c970 tx=0x7f60f400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6102ffd640 1 -- 172.21.15.28:0/1396427171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4004220 con 0x7f610c151cc0 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6102ffd640 1 -- 172.21.15.28:0/1396427171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60f40079b0 con 0x7f610c151cc0 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610c0b0a90 con 0x7f610c151cc0 2024-09-10T20:41:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.618+0000 7f6102ffd640 1 -- 172.21.15.28:0/1396427171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4007e00 con 0x7f610c151cc0 2024-09-10T20:41:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.619+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f610c1e8a30 con 0x7f610c151cc0 2024-09-10T20:41:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.619+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610c0cbb70 con 0x7f610c151cc0 2024-09-10T20:41:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.620+0000 7f6102ffd640 1 -- 172.21.15.28:0/1396427171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f60f4020060 con 0x7f610c151cc0 2024-09-10T20:41:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.621+0000 7f6102ffd640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f60ec0782d0 0x7f60ec07a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:41:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.621+0000 7f6102ffd640 1 -- 172.21.15.28:0/1396427171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f60f409a280 con 0x7f610c151cc0 2024-09-10T20:41:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.621+0000 7f610a59c640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f60ec0782d0 0x7f60ec07a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:41:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.621+0000 7f610a59c640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f60ec0782d0 0x7f60ec07a790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6104004520 tx=0x7f6104009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:41:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.625+0000 7f6102ffd640 1 -- 172.21.15.28:0/1396427171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f60f4014030 con 0x7f610c151cc0 2024-09-10T20:41:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.844+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f610c151070 con 0x7f610c151cc0 2024-09-10T20:41:59.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.845+0000 7f6102ffd640 1 -- 172.21.15.28:0/1396427171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5228 (secure 0 0 0) 0x7f60f40627f0 con 0x7f610c151cc0 2024-09-10T20:41:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:41:59.849 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":44,"btime":"2024-09-10T20:41:53:275679+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:41:59.849 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 44 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f60ec0782d0 msgr2=0x7f60ec07a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f60ec0782d0 0x7f60ec07a790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6104004520 tx=0x7f6104009040 comp rx=0 tx=0).stop 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c151cc0 msgr2=0x7f610c154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c151cc0 0x7f610c154a60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f60f400c970 tx=0x7f60f400ce30 comp rx=0 tx=0).stop 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 shutdown_connections 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f60ec0782d0 0x7f60ec07a790 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f610c15ed20 0x7f610c0b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c151cc0 0x7f610c154a60 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 --2- 172.21.15.28:0/1396427171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f610c0cf0f0 0x7f610c0b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:41:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 >> 172.21.15.28:0/1396427171 conn(0x7f610c0d0420 msgr2=0x7f610c0cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:41:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 shutdown_connections 2024-09-10T20:41:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:41:59.847+0000 7f6110c33640 1 -- 172.21.15.28:0/1396427171 wait complete. 2024-09-10T20:42:00.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:41:49.602862+0000', 'modified': '2024-09-10T20:41:52.641867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6138}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6138': {'gid': 6138, 'name': 'c', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6834/3447265222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3447265222}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 3447265222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6138, 'qdb_cluster': [6138]} 2024-09-10T20:42:00.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:42:00.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-10T20:42:00.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 -- 172.21.15.28:0/3825510346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 msgr2=0x7f57bc162780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 --2- 172.21.15.28:0/3825510346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc162780 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f57a4005fd0 tx=0x7f57a402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 -- 172.21.15.28:0/3825510346 shutdown_connections 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 --2- 172.21.15.28:0/3825510346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc162780 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 --2- 172.21.15.28:0/3825510346 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57bc0d2960 0x7f57bc0d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 -- 172.21.15.28:0/3825510346 >> 172.21.15.28:0/3825510346 conn(0x7f57bc0d4130 msgr2=0x7f57bc0d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 -- 172.21.15.28:0/3825510346 shutdown_connections 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.371+0000 7f57c18e2640 1 -- 172.21.15.28:0/3825510346 wait complete. 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.372+0000 7f57c18e2640 1 Processor -- start 2024-09-10T20:42:00.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.373+0000 7f57c18e2640 1 -- start start 2024-09-10T20:42:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.373+0000 7f57c18e2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57bc0d1730 0x7f57bc153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.373+0000 7f57c18e2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57bc0d2960 0x7f57bc0b9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.373+0000 7f57c18e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.373+0000 7f57c18e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc0cf3e0 con 0x7f57bc0d2d40 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57c18e2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc15f610 con 0x7f57bc0d2960 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57c18e2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f57bc0cf560 con 0x7f57bc0d1730 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc160580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45308/0 (socket says 172.21.15.28:45308) 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2755107873 learned_addr learned my addr 172.21.15.28:0/2755107873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57baffd640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57bc0d1730 0x7f57bc153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57bc0d2960 0x7f57bc0b9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57bc0d1730 msgr2=0x7f57bc153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57bc0d1730 0x7f57bc153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57bc0d2960 msgr2=0x7f57bc0b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57bc0d2960 0x7f57bc0b9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2755107873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57bc0b8170 con 0x7f57bc0d2d40 2024-09-10T20:42:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc160580 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00c430 tx=0x7f57ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57b3fff640 1 -- 172.21.15.28:0/2755107873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac017070 con 0x7f57bc0d2d40 2024-09-10T20:42:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57b3fff640 1 -- 172.21.15.28:0/2755107873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57ac0053c0 con 0x7f57bc0d2d40 2024-09-10T20:42:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57bc0b8450 con 0x7f57bc0d2d40 2024-09-10T20:42:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57b3fff640 1 -- 172.21.15.28:0/2755107873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac003850 con 0x7f57bc0d2d40 2024-09-10T20:42:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.374+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57bc0b8a10 con 0x7f57bc0d2d40 2024-09-10T20:42:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.377+0000 7f57b3fff640 1 -- 172.21.15.28:0/2755107873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f57ac0039f0 con 0x7f57bc0d2d40 2024-09-10T20:42:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.377+0000 7f57b3fff640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f579c078300 0x7f579c07a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.377+0000 7f57b3fff640 1 -- 172.21.15.28:0/2755107873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f57ac099930 con 0x7f57bc0d2d40 2024-09-10T20:42:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.377+0000 7f57baffd640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f579c078300 0x7f579c07a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.378+0000 7f57c46b1740 1 Processor -- start 2024-09-10T20:42:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.378+0000 7f57c46b1740 1 -- start start 2024-09-10T20:42:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.378+0000 7f57baffd640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f579c078300 0x7f579c07a7c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005fd0 tx=0x7f57b4009680 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.378+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474e2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b94752b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b9474e05f0 0x55b9474e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57c46b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b947457820 con 0x55b94750b1b0 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b947446c50 con 0x55b94752adf0 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b94744aa60 con 0x55b9474e05f0 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474e2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474e2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45316/0 (socket says 172.21.15.28:45316) 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 -- 172.21.15.28:0/3063721228 learned_addr learned my addr 172.21.15.28:0/3063721228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b94752b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b9474e05f0 0x55b9474e0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 -- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b9474e05f0 msgr2=0x55b9474e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b9474e05f0 0x55b9474e0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 -- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 msgr2=0x55b94752b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b94752b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 -- 172.21.15.28:0/3063721228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9471a8850 con 0x55b94750b1b0 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57baffd640 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474e2b20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f57b40127a0 tx=0x7f57b4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1861d1311a52e5d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57b17fa640 1 -- 172.21.15.28:0/3063721228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4023070 con 0x55b94750b1b0 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57b17fa640 1 -- 172.21.15.28:0/3063721228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57b4011960 con 0x55b94750b1b0 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.379+0000 7f57b17fa640 1 -- 172.21.15.28:0/3063721228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4011d50 con 0x55b94750b1b0 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 -- 172.21.15.28:0/3063721228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 msgr2=0x55b9474e2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474e2b20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f57b40127a0 tx=0x7f57b4013000 comp rx=0 tx=0).stop 2024-09-10T20:42:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 -- 172.21.15.28:0/3063721228 shutdown_connections 2024-09-10T20:42:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b9474e05f0 0x55b9474e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b94752b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 --2- 172.21.15.28:0/3063721228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474e2b20 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 -- 172.21.15.28:0/3063721228 >> 172.21.15.28:0/3063721228 conn(0x55b9474e16a0 msgr2=0x55b9474e50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 -- 172.21.15.28:0/3063721228 shutdown_connections 2024-09-10T20:42:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.380+0000 7f57c46b1740 1 -- 172.21.15.28:0/3063721228 wait complete. 2024-09-10T20:42:00.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.381+0000 7f57c46b1740 1 -- start start 2024-09-10T20:42:00.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.381+0000 7f57c46b1740 1 Processor -- start 2024-09-10T20:42:00.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b94750d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b9474f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9474f8650 0x55b9474f8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b9471b7620 con 0x55b9474f8650 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b9473ec5f0 con 0x55b94752adf0 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b9474dfb30 con 0x55b94750b1b0 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 10 client.0 _ll_register_callbacks cb 0x55b9473f00a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9474f8650 0x55b9474f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9474f8650 0x55b9474f8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45324/0 (socket says 172.21.15.28:45324) 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2335371743 learned_addr learned my addr 172.21.15.28:0/2335371743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57baffd640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b94750d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b9474f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 msgr2=0x55b94750d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b94750d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 msgr2=0x55b9474f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b9474f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 -- 172.21.15.28:0/2335371743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b94721f6b0 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b9474f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9474f8650 0x55b9474f8000 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f57ac097900 tx=0x7f57ac061e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5974178e43da607 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f579a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f579a7fc640 1 -- 172.21.15.28:0/2335371743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0610c0 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f579a7fc640 1 -- 172.21.15.28:0/2335371743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f57ac05d2a0 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b4011960 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.382+0000 7f579a7fc640 1 -- 172.21.15.28:0/2335371743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f57ac061460 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 10 client.6303 fetch_fsmap learned FSMap version 44 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x55b947506b40 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f579a7fc640 1 -- 172.21.15.28:0/2335371743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57b4011960 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f579a7fc640 1 -- 172.21.15.28:0/2335371743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 44) ==== 3502+0+0 (secure 0 0 0) 0x7f57ac05d4a0 con 0x55b9474f8650 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 10 client.6303 fetch_fsmap finished waiting for FSMap version 44 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 10 client.6303 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-10T20:42:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 20 client.6303 populate_metadata read hostname 'smithi028' 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b947506d70 0x55b94753d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 4 client.6303 mds_command: new command op to 6138 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 --> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b947459770 con 0x55b947506d70 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57baffd640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b947506d70 0x55b94753d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f579a7fc640 10 client.6303 ms_handle_connect on v2:172.21.15.28:6832/3447265222 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.383+0000 7f57baffd640 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b947506d70 0x55b94753d500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50466e9dbef82c9d server_cookie=14a60c1cf032ae37 in_seq=0 out_seq=0 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f579a7fc640 1 -- 172.21.15.28:0/2335371743 <== mds.0 v2:172.21.15.28:6832/3447265222 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b947459770 con 0x55b947506d70 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f579a7fc640 10 client.6303 handle_command_reply: tid=0 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 client.6303 shutdown 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9474f8650 msgr2=0x55b9474f8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9474f8650 0x55b9474f8000 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f57ac097900 tx=0x7f57ac061e60 comp rx=0 tx=0).stop 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 shutdown_connections 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b947506d70 0x55b94753d500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9474f8650 0x55b9474f8000 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b94752adf0 0x55b9474f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 --2- 172.21.15.28:0/2335371743 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b94750d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 >> 172.21.15.28:0/2335371743 conn(0x55b9474e05f0 msgr2=0x55b9474e4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f579a7fc640 0 client.6303 ms_handle_reset on v2:172.21.15.28:6832/3447265222 2024-09-10T20:42:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 shutdown_connections 2024-09-10T20:42:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 1 -- 172.21.15.28:0/2335371743 wait complete. 2024-09-10T20:42:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.384+0000 7f57c46b1740 20 client.6303 trim_cache size 0 max 16384 2024-09-10T20:42:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.403+0000 7f57c46b1740 1 Processor -- start 2024-09-10T20:42:00.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.403+0000 7f57c46b1740 1 -- start start 2024-09-10T20:42:00.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94752b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474de5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94752adf0 0x55b9474deb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57c46b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1f9040 con 0x55b94750b1b0 2024-09-10T20:42:00.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc06d7b0 con 0x55b9474f8650 2024-09-10T20:42:00.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f57bc1f89f0 con 0x55b94752adf0 2024-09-10T20:42:00.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474de5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474de5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45334/0 (socket says 172.21.15.28:45334) 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 -- 172.21.15.28:0/1400049868 learned_addr learned my addr 172.21.15.28:0/1400049868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57baffd640 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94752b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 -- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94752adf0 msgr2=0x55b9474deb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94752adf0 0x55b9474deb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94752adf0 0x55b9474deb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 -- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 msgr2=0x55b94752b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94752b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 -- 172.21.15.28:0/1400049868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b947503420 con 0x55b94750b1b0 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57baffd640 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94752b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94752adf0 0x55b9474deb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474de5d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f57a4009230 tx=0x7f57a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81b125e723d6b09 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.404+0000 7f57b0ff9640 1 -- 172.21.15.28:0/1400049868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a4035040 con 0x55b94750b1b0 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57b0ff9640 1 -- 172.21.15.28:0/1400049868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57a4005310 con 0x55b94750b1b0 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57b0ff9640 1 -- 172.21.15.28:0/1400049868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a40047a0 con 0x55b94750b1b0 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 -- 172.21.15.28:0/1400049868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 msgr2=0x55b9474de5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474de5d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f57a4009230 tx=0x7f57a40046c0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 -- 172.21.15.28:0/1400049868 shutdown_connections 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94752adf0 0x55b9474deb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94750b1b0 0x55b9474de5d0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 --2- 172.21.15.28:0/1400049868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94752b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 -- 172.21.15.28:0/1400049868 >> 172.21.15.28:0/1400049868 conn(0x55b9474e05f0 msgr2=0x55b9475039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 -- 172.21.15.28:0/1400049868 shutdown_connections 2024-09-10T20:42:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.405+0000 7f57c46b1740 1 -- 172.21.15.28:0/1400049868 wait complete. 2024-09-10T20:42:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.406+0000 7f57c46b1740 1 -- start start 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.406+0000 7f57c46b1740 1 Processor -- start 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94754ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b9475a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57c46b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94752adf0 0x55b9475aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57c46b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b947523a70 con 0x55b94752adf0 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55b947523670 con 0x55b9474f8650 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57c46b1740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55b947507180 con 0x55b94750b1b0 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57c46b1740 10 client.0 _ll_register_callbacks cb 0x7f57b4009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94752adf0 0x55b9475aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94752adf0 0x55b9475aa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45338/0 (socket says 172.21.15.28:45338) 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 -- 172.21.15.28:0/44623056 learned_addr learned my addr 172.21.15.28:0/44623056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57ba7fc640 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b9475a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57baffd640 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94754ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 -- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 msgr2=0x55b9475a9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b9475a9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 -- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 msgr2=0x55b94754ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94754ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 -- 172.21.15.28:0/44623056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9474dedf0 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f57bb7fe640 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94752adf0 0x55b9475aa4c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0617d0 tx=0x7f57ac060f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94201c4d40023c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.407+0000 7f579a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f579a7fc640 1 -- 172.21.15.28:0/44623056 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0610c0 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f579a7fc640 1 -- 172.21.15.28:0/44623056 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f57ac05d2a0 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a4005310 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f579a7fc640 1 -- 172.21.15.28:0/44623056 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f57ac097930 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 10 client.6309 fetch_fsmap learned FSMap version 44 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x55b94750d780 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f579a7fc640 1 -- 172.21.15.28:0/44623056 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57a4005310 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f579a7fc640 1 -- 172.21.15.28:0/44623056 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 44) ==== 3502+0+0 (secure 0 0 0) 0x7f57ac05d4a0 con 0x55b94752adf0 2024-09-10T20:42:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 10 client.6309 fetch_fsmap finished waiting for FSMap version 44 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 10 client.6309 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 20 client.6309 populate_metadata read hostname 'smithi028' 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b9474f7020 0x55b9475ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 4 client.6309 mds_command: new command op to 6138 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 --> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] -- command(tid 0: {"prefix": "status"}) -- 0x55b947455220 con 0x55b9474f7020 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.408+0000 7f57baffd640 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b9474f7020 0x55b9475ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f579a7fc640 10 client.6309 ms_handle_connect on v2:172.21.15.28:6832/3447265222 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57baffd640 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b9474f7020 0x55b9475ae170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f0f49afed211072 server_cookie=1994c15e83f39cb5 in_seq=0 out_seq=0 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f579a7fc640 1 -- 172.21.15.28:0/44623056 <== mds.0 v2:172.21.15.28:6832/3447265222 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x55b947455220 con 0x55b9474f7020 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f579a7fc640 10 client.6309 handle_command_reply: tid=0 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 client.6309 shutdown 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94752adf0 msgr2=0x55b9475aa4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94752adf0 0x55b9475aa4c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0617d0 tx=0x7f57ac060f80 comp rx=0 tx=0).stop 2024-09-10T20:42:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 shutdown_connections 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:6832/3447265222,v1:172.21.15.28:6834/3447265222] conn(0x55b9474f7020 0x55b9475ae170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b94752adf0 0x55b9475aa4c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55b94750b1b0 0x55b9475a9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f579a7fc640 0 client.6309 ms_handle_reset on v2:172.21.15.28:6832/3447265222 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 --2- 172.21.15.28:0/44623056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55b9474f8650 0x55b94754ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 >> 172.21.15.28:0/44623056 conn(0x55b9474e05f0 msgr2=0x55b9474e50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 shutdown_connections 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 1 -- 172.21.15.28:0/44623056 wait complete. 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.409+0000 7f57c46b1740 20 client.6309 trim_cache size 0 max 16384 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:42:00.418 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6138, 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 10.151824816, 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 44, 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 68, 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 68, 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 15.854485173 2024-09-10T20:42:00.419 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f579c078300 msgr2=0x7f579c07a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f579c078300 0x7f579c07a7c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005fd0 tx=0x7f57b4009680 comp rx=0 tx=0).stop 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 msgr2=0x7f57bc160580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc160580 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00c430 tx=0x7f57ac00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 shutdown_connections 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f579c078300 0x7f579c07a7c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57bc0d2d40 0x7f57bc160580 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57bc0d2960 0x7f57bc0b9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 --2- 172.21.15.28:0/2755107873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57bc0d1730 0x7f57bc153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 >> 172.21.15.28:0/2755107873 conn(0x7f57bc0d4130 msgr2=0x7f57bc0b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 shutdown_connections 2024-09-10T20:42:00.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.410+0000 7f57c18e2640 1 -- 172.21.15.28:0/2755107873 wait complete. 2024-09-10T20:42:00.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:42:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 -- 172.21.15.28:0/2594869654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 msgr2=0x7fb70415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 --2- 172.21.15.28:0/2594869654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 0x7fb70415bd50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4005fd0 tx=0x7fb6f402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 -- 172.21.15.28:0/2594869654 shutdown_connections 2024-09-10T20:42:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 --2- 172.21.15.28:0/2594869654 >> v2:172.21.15.159:3300/0 conn(0x7fb7041554e0 0x7fb70415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 --2- 172.21.15.28:0/2594869654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 0x7fb70415bd50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 -- 172.21.15.28:0/2594869654 >> 172.21.15.28:0/2594869654 conn(0x7fb70406cf00 msgr2=0x7fb70406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 -- 172.21.15.28:0/2594869654 shutdown_connections 2024-09-10T20:42:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.525+0000 7fb7090ef640 1 -- 172.21.15.28:0/2594869654 wait complete. 2024-09-10T20:42:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 Processor -- start 2024-09-10T20:42:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 -- start start 2024-09-10T20:42:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 0x7fb7041588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7041554e0 0x7fb7041f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb70415ee70 0x7fb7040cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb704072f60 con 0x7fb7041554e0 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb704072de0 con 0x7fb70415ee70 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.527+0000 7fb7090ef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb7040730e0 con 0x7fb704155100 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7041554e0 0x7fb7041f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7041554e0 0x7fb7041f8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45354/0 (socket says 172.21.15.28:45354) 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 -- 172.21.15.28:0/3646359427 learned_addr learned my addr 172.21.15.28:0/3646359427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb702d9d640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 0x7fb7041588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70359e640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb70415ee70 0x7fb7040cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 -- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 msgr2=0x7fb7041588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 0x7fb7041588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 -- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb70415ee70 msgr2=0x7fb7040cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb70415ee70 0x7fb7040cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 -- 172.21.15.28:0/3646359427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7040cbb30 con 0x7fb7041554e0 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb70259c640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7041554e0 0x7fb7041f8860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800e9c0 tx=0x7fb6f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb6eb7fe640 1 -- 172.21.15.28:0/3646359427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f800cd80 con 0x7fb7041554e0 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb6eb7fe640 1 -- 172.21.15.28:0/3646359427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6f8004020 con 0x7fb7041554e0 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb6eb7fe640 1 -- 172.21.15.28:0/3646359427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8004450 con 0x7fb7041554e0 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb704158f50 con 0x7fb7041554e0 2024-09-10T20:42:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.528+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7041f4970 con 0x7fb7041554e0 2024-09-10T20:42:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.530+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb704073590 con 0x7fb7041554e0 2024-09-10T20:42:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.530+0000 7fb6eb7fe640 1 -- 172.21.15.28:0/3646359427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fb6f80035f0 con 0x7fb7041554e0 2024-09-10T20:42:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.531+0000 7fb6eb7fe640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6f00782d0 0x7fb6f007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.531+0000 7fb6eb7fe640 1 -- 172.21.15.28:0/3646359427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fb6f809ae50 con 0x7fb7041554e0 2024-09-10T20:42:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.531+0000 7fb702d9d640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6f00782d0 0x7fb6f007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.532+0000 7fb702d9d640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6f00782d0 0x7fb6f007a790 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4033000 tx=0x7fb6f4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:00.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.535+0000 7fb6eb7fe640 1 -- 172.21.15.28:0/3646359427 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb7040cbb30 con 0x7fb7041554e0 2024-09-10T20:42:00.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.759+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb70406d4f0 con 0x7fb7041554e0 2024-09-10T20:42:00.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.760+0000 7fb6eb7fe640 1 -- 172.21.15.28:0/3646359427 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7fb6f8062cb0 con 0x7fb7041554e0 2024-09-10T20:42:00.764 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:42:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6f00782d0 msgr2=0x7fb6f007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6f00782d0 0x7fb6f007a790 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4033000 tx=0x7fb6f4004730 comp rx=0 tx=0).stop 2024-09-10T20:42:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7041554e0 msgr2=0x7fb7041f8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7041554e0 0x7fb7041f8860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800e9c0 tx=0x7fb6f800ee80 comp rx=0 tx=0).stop 2024-09-10T20:42:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 shutdown_connections 2024-09-10T20:42:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6f00782d0 0x7fb6f007a790 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb70415ee70 0x7fb7040cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7041554e0 0x7fb7041f8860 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 --2- 172.21.15.28:0/3646359427 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb704155100 0x7fb7041588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 >> 172.21.15.28:0/3646359427 conn(0x7fb70406cf00 msgr2=0x7fb70415e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 shutdown_connections 2024-09-10T20:42:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:00.762+0000 7fb7090ef640 1 -- 172.21.15.28:0/3646359427 wait complete. 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:42:00.777 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:42:00.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:00.778 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:42:00.794 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:42:00.794 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:42:00.794 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:42:00.794 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:00.794 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:42:00.794 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:00.794 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42510sec preferred_lft 42510sec 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:00.795 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:42:00.796 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:00.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:00.797 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:42:00.859 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:42:00.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:00.860 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:42:00.917 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:42:00.917 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:42:00.917 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:42:00.917 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:42:00.917 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:42:00.918 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:42:00.918 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:42:00.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:42:00.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:42:00.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:42:00.918 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:42:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:42:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:01.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:01.046 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:42:01.063 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:42:01.063 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:42:01.063 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:42:01.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:01.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:42:01.128 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:01.128 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:01.184 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:42:01.185 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:01.185 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:01.240 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:42:01.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:01.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:01.324 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:01.325 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:42:01.325 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:42:01.333 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:01.333 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:42:01.334 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:42:01.334 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:01.334 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:42:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:42:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:42:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:42:01.340 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:42:01.340 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:42:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:42:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:42:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:42:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:42:01.342 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:42:01.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:01.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:01.368 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:42:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:42:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:01.474 INFO:teuthology.orchestra.run.smithi028.stdout:client5543 2024-09-10T20:42:01.474 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:01.474 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:01.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:42:01.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:42:01.534 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:42:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 -- 172.21.15.28:0/1532645546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 msgr2=0x7f427415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 --2- 172.21.15.28:0/1532645546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f427415bd50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4264005fd0 tx=0x7f426402a530 comp rx=0 tx=0).stop 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 -- 172.21.15.28:0/1532645546 shutdown_connections 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 --2- 172.21.15.28:0/1532645546 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42741554e0 0x7f427415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 --2- 172.21.15.28:0/1532645546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f427415bd50 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 -- 172.21.15.28:0/1532645546 >> 172.21.15.28:0/1532645546 conn(0x7f427406cf00 msgr2=0x7f427406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 -- 172.21.15.28:0/1532645546 shutdown_connections 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.633+0000 7f427a517640 1 -- 172.21.15.28:0/1532645546 wait complete. 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.634+0000 7f427a517640 1 Processor -- start 2024-09-10T20:42:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f427a517640 1 -- start start 2024-09-10T20:42:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f427a517640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f42741f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f427a517640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42741554e0 0x7f427409d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f427a517640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f427415ee70 0x7f4274145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f427a517640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4274072f60 con 0x7f4274155100 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f427a517640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4274072de0 con 0x7f427415ee70 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f427a517640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f42740730e0 con 0x7f42741554e0 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f4273fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f42741f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f4273fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f42741f7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45364/0 (socket says 172.21.15.28:45364) 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f4273fff640 1 -- 172.21.15.28:0/37877550 learned_addr learned my addr 172.21.15.28:0/37877550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f42737fe640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42741554e0 0x7f427409d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.635+0000 7f4278ab4640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f427415ee70 0x7f4274145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4273fff640 1 -- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42741554e0 msgr2=0x7f427409d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4273fff640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42741554e0 0x7f427409d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4273fff640 1 -- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f427415ee70 msgr2=0x7f4274145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4273fff640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f427415ee70 0x7f4274145ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4273fff640 1 -- 172.21.15.28:0/37877550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42741f84e0 con 0x7f4274155100 2024-09-10T20:42:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4273fff640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f42741f7d90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4264034040 tx=0x7f42640039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4270ff9640 1 -- 172.21.15.28:0/37877550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426403e030 con 0x7f4274155100 2024-09-10T20:42:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4270ff9640 1 -- 172.21.15.28:0/37877550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42640047a0 con 0x7f4274155100 2024-09-10T20:42:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42741f87c0 con 0x7f4274155100 2024-09-10T20:42:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f4270ff9640 1 -- 172.21.15.28:0/37877550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264035040 con 0x7f4274155100 2024-09-10T20:42:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.636+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427406e5e0 con 0x7f4274155100 2024-09-10T20:42:01.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.637+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42741f8ce0 con 0x7f4274155100 2024-09-10T20:42:01.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.638+0000 7f4270ff9640 1 -- 172.21.15.28:0/37877550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f4264004b90 con 0x7f4274155100 2024-09-10T20:42:01.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.638+0000 7f4270ff9640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f42540782d0 0x7f425407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:01.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.638+0000 7f4270ff9640 1 -- 172.21.15.28:0/37877550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f42640bac50 con 0x7f4274155100 2024-09-10T20:42:01.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.639+0000 7f42737fe640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f42540782d0 0x7f425407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:01.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.639+0000 7f42737fe640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f42540782d0 0x7f425407a790 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f427406fb20 tx=0x7f425c009110 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:01.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.643+0000 7f4270ff9640 1 -- 172.21.15.28:0/37877550 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f42741f8ce0 con 0x7f4274155100 2024-09-10T20:42:01.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.869+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4274074790 con 0x7f4274155100 2024-09-10T20:42:01.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.869+0000 7f4270ff9640 1 -- 172.21.15.28:0/37877550 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7f427415c290 con 0x7f4274155100 2024-09-10T20:42:01.873 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:42:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f42540782d0 msgr2=0x7f425407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f42540782d0 0x7f425407a790 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f427406fb20 tx=0x7f425c009110 comp rx=0 tx=0).stop 2024-09-10T20:42:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 msgr2=0x7f42741f7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f42741f7d90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4264034040 tx=0x7f42640039e0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 shutdown_connections 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f42540782d0 0x7f425407a790 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f427415ee70 0x7f4274145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42741554e0 0x7f427409d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 --2- 172.21.15.28:0/37877550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4274155100 0x7f42741f7d90 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 >> 172.21.15.28:0/37877550 conn(0x7f427406cf00 msgr2=0x7f427415d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 shutdown_connections 2024-09-10T20:42:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:01.871+0000 7f427a517640 1 -- 172.21.15.28:0/37877550 wait complete. 2024-09-10T20:42:01.886 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:42:01.886 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:42:01.886 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:42:01.886 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:42:01.886 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:01.887 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:42:01.887 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:42:01.887 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:42:01.887 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:42:01.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:01.887 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:42:01.907 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42527sec preferred_lft 42527sec 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:42:01.908 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:01.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:01.910 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:42:01.974 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:42:01.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:01.976 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:42:02.036 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:42:02.036 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:42:02.036 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:42:02.036 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:42:02.037 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:42:02.037 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:42:02.037 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:42:02.037 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:42:02.037 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:42:02.037 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:42:02.037 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:42:02.132 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:42:02.166 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:02.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:02.173 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:42:02.189 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:42:02.189 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:42:02.189 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:42:02.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:02.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:42:02.253 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:02.254 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:02.310 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:42:02.310 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:02.310 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:02.365 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:42:02.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:02.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:02.448 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:02.448 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:42:02.449 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:42:02.457 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:02.457 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:42:02.457 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:42:02.457 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:02.457 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:42:02.465 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:42:02.466 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:42:02.466 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:42:02.466 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:42:02.466 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:42:02.466 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:42:02.467 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:42:02.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:02.467 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:02.532 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:42:02.620 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:42:02.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:02.628 INFO:teuthology.orchestra.run.smithi159.stdout:client5549 2024-09-10T20:42:02.628 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:02.628 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:02.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 -- 172.21.15.28:0/3085544251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4006fc40 msgr2=0x7f6a4015bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3085544251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4006fc40 0x7f6a4015bf80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f6a40076d90 tx=0x7f6a3802a980 comp rx=0 tx=0).stop 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 -- 172.21.15.28:0/3085544251 shutdown_connections 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3085544251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4006fc40 0x7f6a4015bf80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3085544251 >> v2:172.21.15.159:3300/0 conn(0x7f6a400b1dd0 0x7f6a400af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 -- 172.21.15.28:0/3085544251 >> 172.21.15.28:0/3085544251 conn(0x7f6a400ccb60 msgr2=0x7f6a400ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 -- 172.21.15.28:0/3085544251 shutdown_connections 2024-09-10T20:42:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.763+0000 7f6a4547f640 1 -- 172.21.15.28:0/3085544251 wait complete. 2024-09-10T20:42:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.764+0000 7f6a4547f640 1 Processor -- start 2024-09-10T20:42:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a4547f640 1 -- start start 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a4547f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a4006fc40 0x7f6a4009b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a4547f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a400b1dd0 0x7f6a4015b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a4547f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a400b21b0 0x7f6a400cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a4547f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4015c960 con 0x7f6a400b1dd0 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a4547f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4015c7e0 con 0x7f6a4006fc40 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a4547f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6a4015cae0 con 0x7f6a400b21b0 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.765+0000 7f6a3e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a400b1dd0 0x7f6a4015b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a400b1dd0 0x7f6a4015b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45388/0 (socket says 172.21.15.28:45388) 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 -- 172.21.15.28:0/3560165501 learned_addr learned my addr 172.21.15.28:0/3560165501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3effd640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a4006fc40 0x7f6a4009b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3f7fe640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a400b21b0 0x7f6a400cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 -- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a400b21b0 msgr2=0x7f6a400cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a400b21b0 0x7f6a400cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 -- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a4006fc40 msgr2=0x7f6a4009b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a4006fc40 0x7f6a4009b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 -- 172.21.15.28:0/3560165501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a401f86c0 con 0x7f6a400b1dd0 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3effd640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a4006fc40 0x7f6a4009b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a3e7fc640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a400b1dd0 0x7f6a4015b610 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400b420 tx=0x7f6a3400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a277fe640 1 -- 172.21.15.28:0/3560165501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34011830 con 0x7f6a400b1dd0 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a277fe640 1 -- 172.21.15.28:0/3560165501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a34016070 con 0x7f6a400b1dd0 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a401f89a0 con 0x7f6a400b1dd0 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a277fe640 1 -- 172.21.15.28:0/3560165501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34011d60 con 0x7f6a400b1dd0 2024-09-10T20:42:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.766+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a401f8f60 con 0x7f6a400b1dd0 2024-09-10T20:42:02.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.767+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a401f9160 con 0x7f6a400b1dd0 2024-09-10T20:42:02.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.768+0000 7f6a277fe640 1 -- 172.21.15.28:0/3560165501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f6a3401b460 con 0x7f6a400b1dd0 2024-09-10T20:42:02.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.769+0000 7f6a277fe640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6a2c0782d0 0x7f6a2c07a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:02.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.769+0000 7f6a277fe640 1 -- 172.21.15.28:0/3560165501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f6a34099e60 con 0x7f6a400b1dd0 2024-09-10T20:42:02.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.769+0000 7f6a3effd640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6a2c0782d0 0x7f6a2c07a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.769+0000 7f6a3effd640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6a2c0782d0 0x7f6a2c07a790 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6a401f8200 tx=0x7f6a30005e70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.773+0000 7f6a277fe640 1 -- 172.21.15.28:0/3560165501 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6a34020020 con 0x7f6a400b1dd0 2024-09-10T20:42:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:02.994+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f6a400cc7a0 con 0x7f6a400b1dd0 2024-09-10T20:42:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.505+0000 7f6a277fe640 1 -- 172.21.15.28:0/3560165501 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v45) ==== 136+0+0 (secure 0 0 0) 0x7f6a340624d0 con 0x7f6a400b1dd0 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6a2c0782d0 msgr2=0x7f6a2c07a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6a2c0782d0 0x7f6a2c07a790 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6a401f8200 tx=0x7f6a30005e70 comp rx=0 tx=0).stop 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a400b1dd0 msgr2=0x7f6a4015b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a400b1dd0 0x7f6a4015b610 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400b420 tx=0x7f6a3400b8e0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 shutdown_connections 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6a2c0782d0 0x7f6a2c07a790 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a400b21b0 0x7f6a400cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a400b1dd0 0x7f6a4015b610 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 --2- 172.21.15.28:0/3560165501 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a4006fc40 0x7f6a4009b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 >> 172.21.15.28:0/3560165501 conn(0x7f6a400ccb60 msgr2=0x7f6a400ccf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 shutdown_connections 2024-09-10T20:42:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.509+0000 7f6a4547f640 1 -- 172.21.15.28:0/3560165501 wait complete. 2024-09-10T20:42:03.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:42:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 -- 172.21.15.28:0/1711213454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c161b50 msgr2=0x7f016c161fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 --2- 172.21.15.28:0/1711213454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c161b50 0x7f016c161fc0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f016c08c020 tx=0x7f015c02aa30 comp rx=0 tx=0).stop 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 -- 172.21.15.28:0/1711213454 shutdown_connections 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 --2- 172.21.15.28:0/1711213454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c161b50 0x7f016c161fc0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 --2- 172.21.15.28:0/1711213454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f016c0cf780 0x7f016c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 --2- 172.21.15.28:0/1711213454 >> v2:172.21.15.159:3300/0 conn(0x7f016c15de70 0x7f016c0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 -- 172.21.15.28:0/1711213454 >> 172.21.15.28:0/1711213454 conn(0x7f016c0d3fb0 msgr2=0x7f016c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 -- 172.21.15.28:0/1711213454 shutdown_connections 2024-09-10T20:42:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.633+0000 7f017395a640 1 -- 172.21.15.28:0/1711213454 wait complete. 2024-09-10T20:42:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 Processor -- start 2024-09-10T20:42:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 -- start start 2024-09-10T20:42:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c0cf780 0x7f016c160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f016c15de70 0x7f016c1f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f016c161b50 0x7f016c1f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f016c0b6e00 con 0x7f016c0cf780 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f016c15e460 con 0x7f016c161b50 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f017395a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f016c0b6f80 con 0x7f016c15de70 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f01716f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c0cf780 0x7f016c160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f01716f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c0cf780 0x7f016c160070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45408/0 (socket says 172.21.15.28:45408) 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.635+0000 7f01716f6640 1 -- 172.21.15.28:0/1450992780 learned_addr learned my addr 172.21.15.28:0/1450992780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f0170ef5640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f016c15de70 0x7f016c1f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f0171ef7640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f016c161b50 0x7f016c1f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f01716f6640 1 -- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f016c15de70 msgr2=0x7f016c1f7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f01716f6640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f016c15de70 0x7f016c1f7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f01716f6640 1 -- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f016c161b50 msgr2=0x7f016c1f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f01716f6640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f016c161b50 0x7f016c1f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f01716f6640 1 -- 172.21.15.28:0/1450992780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016c160b10 con 0x7f016c0cf780 2024-09-10T20:42:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f0170ef5640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f016c15de70 0x7f016c1f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f01716f6640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c0cf780 0x7f016c160070 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f016400d950 tx=0x7f016400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f0169ffb640 1 -- 172.21.15.28:0/1450992780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0164017070 con 0x7f016c0cf780 2024-09-10T20:42:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f0169ffb640 1 -- 172.21.15.28:0/1450992780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0164009d20 con 0x7f016c0cf780 2024-09-10T20:42:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f0169ffb640 1 -- 172.21.15.28:0/1450992780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640106c0 con 0x7f016c0cf780 2024-09-10T20:42:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016c160df0 con 0x7f016c0cf780 2024-09-10T20:42:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.636+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f016c1f89c0 con 0x7f016c0cf780 2024-09-10T20:42:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.637+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f016c1f5830 con 0x7f016c0cf780 2024-09-10T20:42:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.638+0000 7f0169ffb640 1 -- 172.21.15.28:0/1450992780 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f0164010860 con 0x7f016c0cf780 2024-09-10T20:42:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.638+0000 7f0169ffb640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01580782d0 0x7f015807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.638+0000 7f0169ffb640 1 -- 172.21.15.28:0/1450992780 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f016409a880 con 0x7f016c0cf780 2024-09-10T20:42:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.639+0000 7f0170ef5640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01580782d0 0x7f015807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.639+0000 7f0170ef5640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01580782d0 0x7f015807a790 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f016c0a1350 tx=0x7f0160005f10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:03.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.643+0000 7f0169ffb640 1 -- 172.21.15.28:0/1450992780 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f01640626d0 con 0x7f016c0cf780 2024-09-10T20:42:03.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.853+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f016c0d2b90 con 0x7f01580782d0 2024-09-10T20:42:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.855+0000 7f0169ffb640 1 -- 172.21.15.28:0/1450992780 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f016c0d2b90 con 0x7f01580782d0 2024-09-10T20:42:03.859 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:42:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:42:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:03.859 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:42:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01580782d0 msgr2=0x7f015807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01580782d0 0x7f015807a790 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f016c0a1350 tx=0x7f0160005f10 comp rx=0 tx=0).stop 2024-09-10T20:42:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c0cf780 msgr2=0x7f016c160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c0cf780 0x7f016c160070 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f016400d950 tx=0x7f016400de10 comp rx=0 tx=0).stop 2024-09-10T20:42:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 shutdown_connections 2024-09-10T20:42:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f01580782d0 0x7f015807a790 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f016c161b50 0x7f016c1f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f016c15de70 0x7f016c1f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 --2- 172.21.15.28:0/1450992780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c0cf780 0x7f016c160070 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 >> 172.21.15.28:0/1450992780 conn(0x7f016c0d3fb0 msgr2=0x7f016c075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 shutdown_connections 2024-09-10T20:42:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.857+0000 7f017395a640 1 -- 172.21.15.28:0/1450992780 wait complete. 2024-09-10T20:42:03.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:42:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 -- 172.21.15.28:0/2277446211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 msgr2=0x7fc1b00ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/2277446211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00ce750 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0073dd0 tx=0x7fc1a002a560 comp rx=0 tx=0).stop 2024-09-10T20:42:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 -- 172.21.15.28:0/2277446211 shutdown_connections 2024-09-10T20:42:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/2277446211 >> v2:172.21.15.159:3300/0 conn(0x7fc1b00cdb30 0x7fc1b00cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/2277446211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00ce750 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 -- 172.21.15.28:0/2277446211 >> 172.21.15.28:0/2277446211 conn(0x7fc1b00d1f30 msgr2=0x7fc1b00d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 -- 172.21.15.28:0/2277446211 shutdown_connections 2024-09-10T20:42:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.977+0000 7fc1b6bec640 1 -- 172.21.15.28:0/2277446211 wait complete. 2024-09-10T20:42:03.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.978+0000 7fc1b6bec640 1 Processor -- start 2024-09-10T20:42:03.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.978+0000 7fc1b6bec640 1 -- start start 2024-09-10T20:42:03.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b6bec640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1b00b3110 0x7fc1b0153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b6bec640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1b00cdb30 0x7fc1b00c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b6bec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b6bec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b00d1030 con 0x7fc1b015f580 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b6bec640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b00d0eb0 con 0x7fc1b00cdb30 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b6bec640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc1b00b5470 con 0x7fc1b00b3110 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b5189640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b5189640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45422/0 (socket says 172.21.15.28:45422) 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.979+0000 7fc1b5189640 1 -- 172.21.15.28:0/1097107937 learned_addr learned my addr 172.21.15.28:0/1097107937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1affff640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1b00cdb30 0x7fc1b00c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b4988640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1b00b3110 0x7fc1b0153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b5189640 1 -- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1b00b3110 msgr2=0x7fc1b0153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b5189640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1b00b3110 0x7fc1b0153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b5189640 1 -- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1b00cdb30 msgr2=0x7fc1b00c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b5189640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1b00cdb30 0x7fc1b00c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b5189640 1 -- 172.21.15.28:0/1097107937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b014b170 con 0x7fc1b015f580 2024-09-10T20:42:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b5189640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00c86d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800cad0 tx=0x7fc1a800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1ad7fa640 1 -- 172.21.15.28:0/1097107937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800ae20 con 0x7fc1b015f580 2024-09-10T20:42:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1ad7fa640 1 -- 172.21.15.28:0/1097107937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1a8004b20 con 0x7fc1b015f580 2024-09-10T20:42:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b0154400 con 0x7fc1b015f580 2024-09-10T20:42:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1ad7fa640 1 -- 172.21.15.28:0/1097107937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8005020 con 0x7fc1b015f580 2024-09-10T20:42:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.980+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b0156a90 con 0x7fc1b015f580 2024-09-10T20:42:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.981+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b00b5a00 con 0x7fc1b015f580 2024-09-10T20:42:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.982+0000 7fc1ad7fa640 1 -- 172.21.15.28:0/1097107937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fc1a80051c0 con 0x7fc1b015f580 2024-09-10T20:42:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.982+0000 7fc1ad7fa640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc19c0782d0 0x7fc19c07a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.983+0000 7fc1ad7fa640 1 -- 172.21.15.28:0/1097107937 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fc1a809acb0 con 0x7fc1b015f580 2024-09-10T20:42:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.983+0000 7fc1b4988640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc19c0782d0 0x7fc19c07a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.983+0000 7fc1b4988640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc19c0782d0 0x7fc19c07a790 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0033000 tx=0x7fc1a0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:03.987+0000 7fc1ad7fa640 1 -- 172.21.15.28:0/1097107937 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc1b014b170 con 0x7fc1b015f580 2024-09-10T20:42:04.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.171+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc1b00ce290 con 0x7fc1b015f580 2024-09-10T20:42:04.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.171+0000 7fc1ad7fa640 1 -- 172.21.15.28:0/1097107937 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7fc1a8014030 con 0x7fc1b015f580 2024-09-10T20:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc19c0782d0 msgr2=0x7fc19c07a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc19c0782d0 0x7fc19c07a790 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0033000 tx=0x7fc1a0004730 comp rx=0 tx=0).stop 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 msgr2=0x7fc1b00c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00c86d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800cad0 tx=0x7fc1a800cf90 comp rx=0 tx=0).stop 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 shutdown_connections 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fc19c0782d0 0x7fc19c07a790 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1b015f580 0x7fc1b00c86d0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc1b00cdb30 0x7fc1b00c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 --2- 172.21.15.28:0/1097107937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc1b00b3110 0x7fc1b0153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.173+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 >> 172.21.15.28:0/1097107937 conn(0x7fc1b00d1f30 msgr2=0x7fc1b00b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.174+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 shutdown_connections 2024-09-10T20:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.174+0000 7fc1b6bec640 1 -- 172.21.15.28:0/1097107937 wait complete. 2024-09-10T20:42:04.189 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:04.189 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:04.205 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:04.206 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:04.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 -- 172.21.15.28:0/1742925734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415bc40 msgr2=0x7fb6c415e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 --2- 172.21.15.28:0/1742925734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415bc40 0x7fb6c415e6e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac005fd0 tx=0x7fb6ac02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 -- 172.21.15.28:0/1742925734 shutdown_connections 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 --2- 172.21.15.28:0/1742925734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6c415ec20 0x7fb6c4070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 --2- 172.21.15.28:0/1742925734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415bc40 0x7fb6c415e6e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 -- 172.21.15.28:0/1742925734 >> 172.21.15.28:0/1742925734 conn(0x7fb6c406dbf0 msgr2=0x7fb6c406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 -- 172.21.15.28:0/1742925734 shutdown_connections 2024-09-10T20:42:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.405+0000 7fb6c8885640 1 -- 172.21.15.28:0/1742925734 wait complete. 2024-09-10T20:42:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.407+0000 7fb6c8885640 1 Processor -- start 2024-09-10T20:42:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.407+0000 7fb6c8885640 1 -- start start 2024-09-10T20:42:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.407+0000 7fb6c8885640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6c40d0c80 0x7fb6c4150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c8885640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6c415bc40 0x7fb6c409cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c8885640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415ec20 0x7fb6c409d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c8885640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4070940 con 0x7fb6c415ec20 2024-09-10T20:42:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c8885640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c40707c0 con 0x7fb6c415bc40 2024-09-10T20:42:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c8885640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb6c4070ac0 con 0x7fb6c40d0c80 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415ec20 0x7fb6c409d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415ec20 0x7fb6c409d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45438/0 (socket says 172.21.15.28:45438) 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 -- 172.21.15.28:0/622499973 learned_addr learned my addr 172.21.15.28:0/622499973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c1d9b640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6c415bc40 0x7fb6c409cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c259c640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6c40d0c80 0x7fb6c4150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 -- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6c40d0c80 msgr2=0x7fb6c4150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6c40d0c80 0x7fb6c4150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 -- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6c415bc40 msgr2=0x7fb6c409cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6c415bc40 0x7fb6c409cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 -- 172.21.15.28:0/622499973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c40d0500 con 0x7fb6c415ec20 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c1d9b640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6c415bc40 0x7fb6c409cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c259c640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6c40d0c80 0x7fb6c4150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c2d9d640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415ec20 0x7fb6c409d220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00b410 tx=0x7fb6bc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6baffd640 1 -- 172.21.15.28:0/622499973 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc015020 con 0x7fb6c415ec20 2024-09-10T20:42:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6baffd640 1 -- 172.21.15.28:0/622499973 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6bc011940 con 0x7fb6c415ec20 2024-09-10T20:42:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c415a480 con 0x7fb6c415ec20 2024-09-10T20:42:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6baffd640 1 -- 172.21.15.28:0/622499973 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc011d90 con 0x7fb6c415ec20 2024-09-10T20:42:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.408+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c415aa40 con 0x7fb6c415ec20 2024-09-10T20:42:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.409+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c415e6e0 con 0x7fb6c415ec20 2024-09-10T20:42:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.410+0000 7fb6baffd640 1 -- 172.21.15.28:0/622499973 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7fb6bc01a400 con 0x7fb6c415ec20 2024-09-10T20:42:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.411+0000 7fb6baffd640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6a40782d0 0x7fb6a407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.411+0000 7fb6baffd640 1 -- 172.21.15.28:0/622499973 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fb6bc09a610 con 0x7fb6c415ec20 2024-09-10T20:42:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.411+0000 7fb6c259c640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6a40782d0 0x7fb6a407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.411+0000 7fb6c259c640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6a40782d0 0x7fb6a407a790 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4001db0 tx=0x7fb6b4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.415+0000 7fb6baffd640 1 -- 172.21.15.28:0/622499973 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb6bc01f020 con 0x7fb6c415ec20 2024-09-10T20:42:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.616+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb6c40d0a60 con 0x7fb6c415ec20 2024-09-10T20:42:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.616+0000 7fb6baffd640 1 -- 172.21.15.28:0/622499973 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v68) ==== 80+0+0 (secure 0 0 0) 0x7fb6bc062bf0 con 0x7fb6c415ec20 2024-09-10T20:42:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6a40782d0 msgr2=0x7fb6a407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6a40782d0 0x7fb6a407a790 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4001db0 tx=0x7fb6b4009040 comp rx=0 tx=0).stop 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415ec20 msgr2=0x7fb6c409d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415ec20 0x7fb6c409d220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00b410 tx=0x7fb6bc00b8d0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 shutdown_connections 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7fb6a40782d0 0x7fb6a407a790 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c415ec20 0x7fb6c409d220 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6c415bc40 0x7fb6c409cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 --2- 172.21.15.28:0/622499973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6c40d0c80 0x7fb6c4150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 >> 172.21.15.28:0/622499973 conn(0x7fb6c406dbf0 msgr2=0x7fb6c41532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 shutdown_connections 2024-09-10T20:42:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.619+0000 7fb6c8885640 1 -- 172.21.15.28:0/622499973 wait complete. 2024-09-10T20:42:04.633 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:42:04.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:04.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:04.663 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:42:04.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:04.663 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:42:04.720 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:04.720 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:04.740 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:04.740 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:04.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:42:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.902+0000 7ffbf4a87640 1 -- 172.21.15.28:0/1718598163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc0051c0 msgr2=0x7ffbdc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.902+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/1718598163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc0051c0 0x7ffbdc00b9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0006fd0 tx=0x7ffbdc005830 comp rx=0 tx=0).stop 2024-09-10T20:42:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.902+0000 7ffbf4a87640 1 -- 172.21.15.28:0/1718598163 shutdown_connections 2024-09-10T20:42:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.902+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/1718598163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc0051c0 0x7ffbdc00b9c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.902+0000 7ffbf4a87640 1 -- 172.21.15.28:0/1718598163 >> 172.21.15.28:0/1718598163 conn(0x7ffbf00d3fb0 msgr2=0x7ffbf00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.902+0000 7ffbf4a87640 1 -- 172.21.15.28:0/1718598163 shutdown_connections 2024-09-10T20:42:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.902+0000 7ffbf4a87640 1 -- 172.21.15.28:0/1718598163 wait complete. 2024-09-10T20:42:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.903+0000 7ffbf4a87640 1 Processor -- start 2024-09-10T20:42:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.903+0000 7ffbf4a87640 1 -- start start 2024-09-10T20:42:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbf4a87640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffbdc0051c0 0x7ffbf015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbf4a87640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf015fb30 0x7ffbf01f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbf4a87640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffbf009e9a0 0x7ffbf00cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbf4a87640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf0161b50 con 0x7ffbf015fb30 2024-09-10T20:42:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbf4a87640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf015e460 con 0x7ffbf009e9a0 2024-09-10T20:42:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbf4a87640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffbf0161cd0 con 0x7ffbdc0051c0 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbedd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf015fb30 0x7ffbf01f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbedd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf015fb30 0x7ffbf01f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45458/0 (socket says 172.21.15.28:45458) 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.904+0000 7ffbedd9b640 1 -- 172.21.15.28:0/260605754 learned_addr learned my addr 172.21.15.28:0/260605754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbeed9d640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffbf009e9a0 0x7ffbf00cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbee59c640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffbdc0051c0 0x7ffbf015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbedd9b640 1 -- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffbdc0051c0 msgr2=0x7ffbf015ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbedd9b640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffbdc0051c0 0x7ffbf015ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbedd9b640 1 -- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffbf009e9a0 msgr2=0x7ffbf00cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbedd9b640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffbf009e9a0 0x7ffbf00cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbedd9b640 1 -- 172.21.15.28:0/260605754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbf01f5570 con 0x7ffbf015fb30 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbeed9d640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffbf009e9a0 0x7ffbf00cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbee59c640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffbdc0051c0 0x7ffbf015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbedd9b640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf015fb30 0x7ffbf01f4bf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8011ab0 tx=0x7ffbd8011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/260605754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd801e070 con 0x7ffbf015fb30 2024-09-10T20:42:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/260605754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffbd800fe70 con 0x7ffbf015fb30 2024-09-10T20:42:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbf01f5850 con 0x7ffbf015fb30 2024-09-10T20:42:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/260605754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80042b0 con 0x7ffbf015fb30 2024-09-10T20:42:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.905+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbf015e9f0 con 0x7ffbf015fb30 2024-09-10T20:42:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.906+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbf00cf860 con 0x7ffbf015fb30 2024-09-10T20:42:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.907+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/260605754 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7ffbd80035f0 con 0x7ffbf015fb30 2024-09-10T20:42:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.907+0000 7ffbe6ffd640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffbd00782d0 0x7ffbd007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.907+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/260605754 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7ffbd809b0b0 con 0x7ffbf015fb30 2024-09-10T20:42:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.907+0000 7ffbee59c640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffbd00782d0 0x7ffbd007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.908+0000 7ffbee59c640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffbd00782d0 0x7ffbd007a790 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8004fd0 tx=0x7ffbe8006e30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:04.911+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/260605754 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffbd8062fd0 con 0x7ffbf015fb30 2024-09-10T20:42:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.107+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ffbf00b9f30 con 0x7ffbf015fb30 2024-09-10T20:42:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.108+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/260605754 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v68) ==== 80+0+0 (secure 0 0 0) 0x7ffbd8067a00 con 0x7ffbf015fb30 2024-09-10T20:42:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:42:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffbd00782d0 msgr2=0x7ffbd007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffbd00782d0 0x7ffbd007a790 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8004fd0 tx=0x7ffbe8006e30 comp rx=0 tx=0).stop 2024-09-10T20:42:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf015fb30 msgr2=0x7ffbf01f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf015fb30 0x7ffbf01f4bf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8011ab0 tx=0x7ffbd8011f70 comp rx=0 tx=0).stop 2024-09-10T20:42:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 shutdown_connections 2024-09-10T20:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7ffbd00782d0 0x7ffbd007a790 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffbf009e9a0 0x7ffbf00cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf015fb30 0x7ffbf01f4bf0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 --2- 172.21.15.28:0/260605754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffbdc0051c0 0x7ffbf015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 >> 172.21.15.28:0/260605754 conn(0x7ffbf00d3fb0 msgr2=0x7ffbf0075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 shutdown_connections 2024-09-10T20:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.110+0000 7ffbf4a87640 1 -- 172.21.15.28:0/260605754 wait complete. 2024-09-10T20:42:05.125 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:42:05.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:05.125 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:05.159 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:42:05.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:05.160 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:42:05.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.322+0000 7f973cadd640 1 -- 172.21.15.28:0/2272233652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724004800 msgr2=0x7f9724004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.322+0000 7f973cadd640 1 --2- 172.21.15.28:0/2272233652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724004800 0x7f9724004c80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9720009b80 tx=0x7f9724005830 comp rx=0 tx=0).stop 2024-09-10T20:42:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.322+0000 7f9736d9d640 1 -- 172.21.15.28:0/2272233652 reap_dead start 2024-09-10T20:42:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.322+0000 7f973cadd640 1 -- 172.21.15.28:0/2272233652 shutdown_connections 2024-09-10T20:42:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.322+0000 7f973cadd640 1 -- 172.21.15.28:0/2272233652 >> 172.21.15.28:0/2272233652 conn(0x7f973806cf00 msgr2=0x7f973806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.322+0000 7f973cadd640 1 -- 172.21.15.28:0/2272233652 shutdown_connections 2024-09-10T20:42:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.322+0000 7f973cadd640 1 -- 172.21.15.28:0/2272233652 wait complete. 2024-09-10T20:42:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.324+0000 7f973cadd640 1 Processor -- start 2024-09-10T20:42:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.324+0000 7f973cadd640 1 -- start start 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.324+0000 7f973cadd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9724004800 0x7f97380c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.324+0000 7f973cadd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97381f8410 0x7f973809ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f973cadd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97381f3890 0x7f97381f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f973cadd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97380b2e80 con 0x7f97381f3890 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f973cadd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f97380b4b60 con 0x7f9724004800 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f973cadd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f97380b3000 con 0x7f97381f8410 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97381f3890 0x7f97381f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97381f3890 0x7f97381f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45468/0 (socket says 172.21.15.28:45468) 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 -- 172.21.15.28:0/880894494 learned_addr learned my addr 172.21.15.28:0/880894494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f973659c640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9724004800 0x7f97380c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f972ffff640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97381f8410 0x7f973809ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 -- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97381f8410 msgr2=0x7f973809ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97381f8410 0x7f973809ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 -- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9724004800 msgr2=0x7f97380c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9724004800 0x7f97380c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 -- 172.21.15.28:0/880894494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97380cb830 con 0x7f97381f3890 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f9736d9d640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97381f3890 0x7f97381f3d10 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f97380b3df0 tx=0x7f9730010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f972effd640 1 -- 172.21.15.28:0/880894494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730003970 con 0x7f97381f3890 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f972effd640 1 -- 172.21.15.28:0/880894494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9730015070 con 0x7f97381f3890 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97380cbb10 con 0x7f97381f3890 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.325+0000 7f972effd640 1 -- 172.21.15.28:0/880894494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730004030 con 0x7f97381f3890 2024-09-10T20:42:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.326+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97380b1590 con 0x7f97381f3890 2024-09-10T20:42:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.327+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97380d0090 con 0x7f97381f3890 2024-09-10T20:42:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.327+0000 7f972effd640 1 -- 172.21.15.28:0/880894494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f97300041d0 con 0x7f97381f3890 2024-09-10T20:42:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.328+0000 7f972effd640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f97180782d0 0x7f971807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.328+0000 7f972effd640 1 -- 172.21.15.28:0/880894494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f973009af70 con 0x7f97381f3890 2024-09-10T20:42:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.328+0000 7f973659c640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f97180782d0 0x7f971807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.329+0000 7f973659c640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f97180782d0 0x7f971807a790 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9728000ff0 tx=0x7f9728001b90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.332+0000 7f972effd640 1 -- 172.21.15.28:0/880894494 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9730063450 con 0x7f97381f3890 2024-09-10T20:42:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.553+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f973806d560 con 0x7f97381f3890 2024-09-10T20:42:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.554+0000 7f972effd640 1 -- 172.21.15.28:0/880894494 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5228 (secure 0 0 0) 0x7f9730062d50 con 0x7f97381f3890 2024-09-10T20:42:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:05.558 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"btime":"2024-09-10T20:42:03:504689+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:42:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 45 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f97180782d0 msgr2=0x7f971807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f97180782d0 0x7f971807a790 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9728000ff0 tx=0x7f9728001b90 comp rx=0 tx=0).stop 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97381f3890 msgr2=0x7f97381f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97381f3890 0x7f97381f3d10 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f97380b3df0 tx=0x7f9730010f40 comp rx=0 tx=0).stop 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 shutdown_connections 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f97180782d0 0x7f971807a790 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97381f3890 0x7f97381f3d10 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97381f8410 0x7f973809ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 --2- 172.21.15.28:0/880894494 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9724004800 0x7f97380c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 >> 172.21.15.28:0/880894494 conn(0x7f973806cf00 msgr2=0x7f97380cf4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 shutdown_connections 2024-09-10T20:42:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.556+0000 7f973cadd640 1 -- 172.21.15.28:0/880894494 wait complete. 2024-09-10T20:42:05.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-10T20:42:05.692 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-10T20:42:05.692 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-10T20:42:05.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol78990 2024-09-10T20:42:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1339554751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e08161b50 msgr2=0x7f8e080d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1339554751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e08161b50 0x7f8e080d2620 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f8e0808c610 tx=0x7f8df802a4d0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1339554751 shutdown_connections 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1339554751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e080d4b90 0x7f8e080b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1339554751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e08161b50 0x7f8e080d2620 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1339554751 >> v2:172.21.15.159:3300/0 conn(0x7f8e08161770 0x7f8e080d20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1339554751 >> 172.21.15.28:0/1339554751 conn(0x7f8e080d4070 msgr2=0x7f8e080d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1339554751 shutdown_connections 2024-09-10T20:42:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.796+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1339554751 wait complete. 2024-09-10T20:42:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 Processor -- start 2024-09-10T20:42:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 -- start start 2024-09-10T20:42:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e080d4b90 0x7f8e080c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e08161770 0x7f8e080a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e08161b50 0x7f8e080ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e080b3f30 con 0x7f8e080d4b90 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8e080b3db0 con 0x7f8e08161b50 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.798+0000 7f8e0fd52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8e080b40b0 con 0x7f8e08161770 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e080d4b90 0x7f8e080c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e080d4b90 0x7f8e080c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45494/0 (socket says 172.21.15.28:45494) 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 -- 172.21.15.28:0/1489517381 learned_addr learned my addr 172.21.15.28:0/1489517381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0e2ef640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e08161b50 0x7f8e080ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0d2ed640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e08161770 0x7f8e080a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 -- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e08161770 msgr2=0x7f8e080a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e08161770 0x7f8e080a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 -- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e08161b50 msgr2=0x7f8e080ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e08161b50 0x7f8e080ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 -- 172.21.15.28:0/1489517381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e080c7cd0 con 0x7f8e080d4b90 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0daee640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e080d4b90 0x7f8e080c8d30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400c890 tx=0x7f8e0400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8dfe7fc640 1 -- 172.21.15.28:0/1489517381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04004220 con 0x7f8e080d4b90 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8dfe7fc640 1 -- 172.21.15.28:0/1489517381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e040079b0 con 0x7f8e080d4b90 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e080c7f70 con 0x7f8e080d4b90 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8dfe7fc640 1 -- 172.21.15.28:0/1489517381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04007de0 con 0x7f8e080d4b90 2024-09-10T20:42:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.799+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e080c8530 con 0x7f8e080d4b90 2024-09-10T20:42:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.800+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e080b3870 con 0x7f8e080d4b90 2024-09-10T20:42:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.801+0000 7f8dfe7fc640 1 -- 172.21.15.28:0/1489517381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f8e04005020 con 0x7f8e080d4b90 2024-09-10T20:42:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.801+0000 7f8dfe7fc640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8dec0782d0 0x7f8dec07a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.802+0000 7f8dfe7fc640 1 -- 172.21.15.28:0/1489517381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f8e0409a8f0 con 0x7f8e080d4b90 2024-09-10T20:42:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.802+0000 7f8e0d2ed640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8dec0782d0 0x7f8dec07a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.802+0000 7f8e0d2ed640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8dec0782d0 0x7f8dec07a790 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8df8033000 tx=0x7f8df8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:05.806+0000 7f8dfe7fc640 1 -- 172.21.15.28:0/1489517381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8e04014030 con 0x7f8e080d4b90 2024-09-10T20:42:06.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.021+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 --> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol78990", "target": ["mon-mgr", ""]}) -- 0x7f8e0806d810 con 0x7f8dec0782d0 2024-09-10T20:42:06.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:42:06.027+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:06.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:42:06.027+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:06.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:42:06.027+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:06.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:42:06.027+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:06.031 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-10T20:42:06.027+0000 7f97d2bcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.061+0000 7f8dfe7fc640 1 -- 172.21.15.28:0/1489517381 <== mgr.4115 v2:172.21.15.159:6800/2898596254 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8e0806d810 con 0x7f8dec0782d0 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8dec0782d0 msgr2=0x7f8dec07a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8dec0782d0 0x7f8dec07a790 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8df8033000 tx=0x7f8df8004730 comp rx=0 tx=0).stop 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e080d4b90 msgr2=0x7f8e080c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e080d4b90 0x7f8e080c8d30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400c890 tx=0x7f8e0400cd50 comp rx=0 tx=0).stop 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 shutdown_connections 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f8dec0782d0 0x7f8dec07a790 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e08161b50 0x7f8e080ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e08161770 0x7f8e080a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 --2- 172.21.15.28:0/1489517381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e080d4b90 0x7f8e080c8d30 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 >> 172.21.15.28:0/1489517381 conn(0x7f8e080d4070 msgr2=0x7f8e0806ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 shutdown_connections 2024-09-10T20:42:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.063+0000 7f8e0fd52640 1 -- 172.21.15.28:0/1489517381 wait complete. 2024-09-10T20:42:06.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 -- 172.21.15.28:0/320436660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 msgr2=0x7f46e40d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 --2- 172.21.15.28:0/320436660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e40d2b90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f46e0009830 tx=0x7f46e002a510 comp rx=0 tx=0).stop 2024-09-10T20:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 -- 172.21.15.28:0/320436660 shutdown_connections 2024-09-10T20:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 --2- 172.21.15.28:0/320436660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46e4162260 0x7f46e40cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 --2- 172.21.15.28:0/320436660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e40d2b90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 -- 172.21.15.28:0/320436660 >> 172.21.15.28:0/320436660 conn(0x7f46e40d3fb0 msgr2=0x7f46e40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 -- 172.21.15.28:0/320436660 shutdown_connections 2024-09-10T20:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.178+0000 7f46ec08e640 1 -- 172.21.15.28:0/320436660 wait complete. 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 Processor -- start 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 -- start start 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46e40cf780 0x7f46e40a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e414ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46e4162260 0x7f46e414f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40b7190 con 0x7f46e415de70 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40b7010 con 0x7f46e40cf780 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.180+0000 7f46ec08e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f46e40b7310 con 0x7f46e4162260 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e414ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e414ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45514/0 (socket says 172.21.15.28:45514) 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 -- 172.21.15.28:0/1979877747 learned_addr learned my addr 172.21.15.28:0/1979877747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46ea62b640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46e4162260 0x7f46e414f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9e2a640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46e40cf780 0x7f46e40a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 -- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46e4162260 msgr2=0x7f46e414f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46e4162260 0x7f46e414f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 -- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46e40cf780 msgr2=0x7f46e40a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46e40cf780 0x7f46e40a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 -- 172.21.15.28:0/1979877747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e414fac0 con 0x7f46e415de70 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9e2a640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46e40cf780 0x7f46e40a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46e9629640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e414ee80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f46d400c500 tx=0x7f46d400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46da7fc640 1 -- 172.21.15.28:0/1979877747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d4016020 con 0x7f46e415de70 2024-09-10T20:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46da7fc640 1 -- 172.21.15.28:0/1979877747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46d4012920 con 0x7f46e415de70 2024-09-10T20:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e414fda0 con 0x7f46e415de70 2024-09-10T20:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46da7fc640 1 -- 172.21.15.28:0/1979877747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d4012d90 con 0x7f46e415de70 2024-09-10T20:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.181+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e40cc450 con 0x7f46e415de70 2024-09-10T20:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.182+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e40b7830 con 0x7f46e415de70 2024-09-10T20:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.183+0000 7f46da7fc640 1 -- 172.21.15.28:0/1979877747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 100907+0+0 (secure 0 0 0) 0x7f46d401a3b0 con 0x7f46e415de70 2024-09-10T20:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.184+0000 7f46da7fc640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f46c80782d0 0x7f46c807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.184+0000 7f46da7fc640 1 -- 172.21.15.28:0/1979877747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f46d409a310 con 0x7f46e415de70 2024-09-10T20:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.184+0000 7f46e9e2a640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f46c80782d0 0x7f46c807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.184+0000 7f46e9e2a640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f46c80782d0 0x7f46c807a790 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f46e0034000 tx=0x7f46e0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:06.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.188+0000 7f46da7fc640 1 -- 172.21.15.28:0/1979877747 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f46d4020030 con 0x7f46e415de70 2024-09-10T20:42:06.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.410+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46e40b9f30 con 0x7f46e415de70 2024-09-10T20:42:06.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.411+0000 7f46da7fc640 1 -- 172.21.15.28:0/1979877747 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5228 (secure 0 0 0) 0x7f46d4062900 con 0x7f46e415de70 2024-09-10T20:42:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:06.415 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"btime":"2024-09-10T20:42:03:504689+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:42:06.415 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 45 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f46c80782d0 msgr2=0x7f46c807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f46c80782d0 0x7f46c807a790 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f46e0034000 tx=0x7f46e0004730 comp rx=0 tx=0).stop 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 msgr2=0x7f46e414ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e414ee80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f46d400c500 tx=0x7f46d400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 shutdown_connections 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f46c80782d0 0x7f46c807a790 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46e4162260 0x7f46e414f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46e415de70 0x7f46e414ee80 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 --2- 172.21.15.28:0/1979877747 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46e40cf780 0x7f46e40a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 >> 172.21.15.28:0/1979877747 conn(0x7f46e40d3fb0 msgr2=0x7f46e40b8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:06.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 shutdown_connections 2024-09-10T20:42:06.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.413+0000 7f46ec08e640 1 -- 172.21.15.28:0/1979877747 wait complete. 2024-09-10T20:42:06.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-10T20:42:06.549 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":5576,"entity":{"name":{"type":"client","num":5576},"addr":{"type":"any","addr":"172.21.15.159:0","nonce":3218025516}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.50598549900000001,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.974916174168197,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5576 172.21.15.159:0/3218025516","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"x","hostname":"smithi159","pid":"36017","root":"/"}}] 2024-09-10T20:42:06.550 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi159", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36017", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.159:0", "nonce": 3218025516, "type": "any" }, "name": { "num": 5576, "type": "client" } }, "id": 5576, "inst": "client.5576 172.21.15.159:0/3218025516", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.974916174168197 }, "state": "open", "uptime": 0.505985499 } ] 2024-09-10T20:42:06.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-10T20:42:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.652+0000 7f8045533640 1 -- 172.21.15.28:0/4012702965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8028004000 msgr2=0x7f8028004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.652+0000 7f8045533640 1 --2- 172.21.15.28:0/4012702965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8028004000 0x7f8028004800 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f8034009b50 tx=0x7f8028005830 comp rx=0 tx=0).stop 2024-09-10T20:42:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.652+0000 7f803f7fe640 1 -- 172.21.15.28:0/4012702965 reap_dead start 2024-09-10T20:42:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.653+0000 7f8045533640 1 -- 172.21.15.28:0/4012702965 shutdown_connections 2024-09-10T20:42:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.653+0000 7f8045533640 1 -- 172.21.15.28:0/4012702965 >> 172.21.15.28:0/4012702965 conn(0x7f804006cf00 msgr2=0x7f804006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.653+0000 7f8045533640 1 -- 172.21.15.28:0/4012702965 shutdown_connections 2024-09-10T20:42:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.653+0000 7f8045533640 1 -- 172.21.15.28:0/4012702965 wait complete. 2024-09-10T20:42:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.654+0000 7f8045533640 1 Processor -- start 2024-09-10T20:42:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.654+0000 7f8045533640 1 -- start start 2024-09-10T20:42:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f8045533640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8028004000 0x7f80400b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f8045533640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80401f59f0 0x7f80401f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f8045533640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80400b10f0 0x7f8040159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f8045533640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80400d0090 con 0x7f80400b10f0 2024-09-10T20:42:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f8045533640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f80400cd7b0 con 0x7f8028004000 2024-09-10T20:42:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f8045533640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f80400d0210 con 0x7f80401f59f0 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80400b10f0 0x7f8040159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80400b10f0 0x7f8040159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45544/0 (socket says 172.21.15.28:45544) 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 -- 172.21.15.28:0/923700567 learned_addr learned my addr 172.21.15.28:0/923700567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803e7fc640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80401f59f0 0x7f80401f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803effd640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8028004000 0x7f80400b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 -- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80401f59f0 msgr2=0x7f80401f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80401f59f0 0x7f80401f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 -- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8028004000 msgr2=0x7f80400b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8028004000 0x7f80400b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 -- 172.21.15.28:0/923700567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80401f2d00 con 0x7f80400b10f0 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.655+0000 7f803f7fe640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80400b10f0 0x7f8040159f00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f804015f830 tx=0x7f8038011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.656+0000 7f80277fe640 1 -- 172.21.15.28:0/923700567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038004b20 con 0x7f80400b10f0 2024-09-10T20:42:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.656+0000 7f80277fe640 1 -- 172.21.15.28:0/923700567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8038004cc0 con 0x7f80400b10f0 2024-09-10T20:42:06.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.656+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80401f2fe0 con 0x7f80400b10f0 2024-09-10T20:42:06.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.656+0000 7f80277fe640 1 -- 172.21.15.28:0/923700567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038003bd0 con 0x7f80400b10f0 2024-09-10T20:42:06.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.656+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80400af8b0 con 0x7f80400b10f0 2024-09-10T20:42:06.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.657+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f804006d4b0 con 0x7f80400b10f0 2024-09-10T20:42:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.657+0000 7f80277fe640 1 -- 172.21.15.28:0/923700567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 100958+0+0 (secure 0 0 0) 0x7f8038018070 con 0x7f80400b10f0 2024-09-10T20:42:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.658+0000 7f80277fe640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f802c078390 0x7f802c07a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.658+0000 7f80277fe640 1 -- 172.21.15.28:0/923700567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f803809ad80 con 0x7f80400b10f0 2024-09-10T20:42:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.658+0000 7f803effd640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f802c078390 0x7f802c07a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.659+0000 7f803effd640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f802c078390 0x7f802c07a850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8030001860 tx=0x7f8030001650 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:06.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.662+0000 7f80277fe640 1 -- 172.21.15.28:0/923700567 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f803801f060 con 0x7f80400b10f0 2024-09-10T20:42:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.896+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f8040069a80 con 0x7f80400b10f0 2024-09-10T20:42:06.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.901+0000 7f80277fe640 1 -- 172.21.15.28:0/923700567 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v12) ==== 81+0+527614 (secure 0 0 0) 0x7f8038063190 con 0x7f80400b10f0 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 12, 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 4115, 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:6800", 2024-09-10T20:42:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2898596254 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:6801", 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2898596254 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.159:6801/2898596254", 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-09-10T20:37:51.965862+0000", 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540719139924082687, 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-09-10T20:42:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 4156, 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540719139924082687, 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-10T20:42:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-10T20:42:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-10T20:42:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-10T20:42:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nginx": { 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nginx", 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nginx:1.26.1", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nginx container image", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-10T20:42:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_oauth2_proxy": { 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_oauth2_proxy", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "oauth2-proxy container image", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-10T20:42:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba_metrics": { 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba_metrics", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-10T20:42:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-10T20:42:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-10T20:42:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-10T20:42:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-10T20:42:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-10T20:42:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-10T20:42:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-10T20:42:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-10T20:42:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-10T20:42:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-10T20:42:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-10T20:42:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-10T20:42:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-10T20:42:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-10T20:42:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-10T20:42:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-10T20:42:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-10T20:42:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-10T20:42:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.178 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.183 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.186 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.201 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.207 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.211 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.217 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.219 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.222 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.235 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-10T20:42:07.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-10T20:42:07.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-10T20:42:07.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.247 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-10T20:42:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max_requests": { 2024-09-10T20:42:07.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_requests", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-10T20:42:07.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.272 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-10T20:42:07.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-10T20:42:07.276 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.277 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.285 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.299 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-10T20:42:07.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-10T20:42:07.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smb", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "internal_store_backend": { 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "internal_store_backend", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.306 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "update_orchestration": { 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "update_orchestration", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.311 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.317 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.320 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.321 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.322 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.325 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.327 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-10T20:42:07.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-10T20:42:07.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.338 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-10T20:42:07.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-10T20:42:07.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-10T20:42:07.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.355 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.359 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.361 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-10T20:42:07.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.364 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.373 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-10T20:42:07.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-10T20:42:07.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: "restful" 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-10T20:42:07.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.404 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-10T20:42:07.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-10T20:42:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-10T20:42:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-10T20:42:07.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.416 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-10T20:42:07.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-10T20:42:07.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-10T20:42:07.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-10T20:42:07.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-10T20:42:07.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-10T20:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-10T20:42:07.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.433 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-10T20:42:07.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-10T20:42:07.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-10T20:42:07.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.438 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-10T20:42:07.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-10T20:42:07.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.445 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-10T20:42:07.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-10T20:42:07.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-10T20:42:07.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-10T20:42:07.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:07.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-10T20:42:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-10T20:42:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-10T20:42:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-10T20:42:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-10T20:42:07.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-10T20:42:07.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-10T20:42:07.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-10T20:42:07.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.454 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-10T20:42:07.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-10T20:42:07.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-09-10T20:42:07.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:07.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.456 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-10T20:42:07.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-10T20:42:07.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-10T20:42:07.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-10T20:42:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-10T20:42:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-10T20:42:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-10T20:42:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-10T20:42:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-10T20:42:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-10T20:42:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-10T20:42:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-10T20:42:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-10T20:42:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-10T20:42:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-10T20:42:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-10T20:42:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-10T20:42:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-10T20:42:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-10T20:42:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-10T20:42:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-10T20:42:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-10T20:42:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-10T20:42:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-10T20:42:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nginx": { 2024-09-10T20:42:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nginx", 2024-09-10T20:42:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nginx:1.26.1", 2024-09-10T20:42:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nginx container image", 2024-09-10T20:42:07.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.473 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-10T20:42:07.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-10T20:42:07.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-10T20:42:07.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:07.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.476 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-10T20:42:07.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-10T20:42:07.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-10T20:42:07.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-10T20:42:07.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.478 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_oauth2_proxy": { 2024-09-10T20:42:07.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_oauth2_proxy", 2024-09-10T20:42:07.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-10T20:42:07.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "oauth2-proxy container image", 2024-09-10T20:42:07.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.481 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-10T20:42:07.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-10T20:42:07.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-10T20:42:07.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-10T20:42:07.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.483 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-10T20:42:07.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-10T20:42:07.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-10T20:42:07.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-10T20:42:07.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.486 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-10T20:42:07.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-10T20:42:07.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-10T20:42:07.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-10T20:42:07.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.488 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba_metrics": { 2024-09-10T20:42:07.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba_metrics", 2024-09-10T20:42:07.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-10T20:42:07.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-10T20:42:07.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.491 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-10T20:42:07.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-10T20:42:07.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-10T20:42:07.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-10T20:42:07.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.493 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-10T20:42:07.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-10T20:42:07.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-10T20:42:07.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.495 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-10T20:42:07.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-10T20:42:07.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:07.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-10T20:42:07.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-10T20:42:07.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-10T20:42:07.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-10T20:42:07.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-10T20:42:07.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-10T20:42:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.504 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-10T20:42:07.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-10T20:42:07.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:07.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-10T20:42:07.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.506 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-10T20:42:07.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-10T20:42:07.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-10T20:42:07.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-10T20:42:07.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.508 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-10T20:42:07.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-10T20:42:07.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:07.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.524 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.530 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-10T20:42:07.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-10T20:42:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.539 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.540 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-10T20:42:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.548 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-10T20:42:07.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-10T20:42:07.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-10T20:42:07.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-10T20:42:07.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-10T20:42:07.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-10T20:42:07.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-10T20:42:07.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.576 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-10T20:42:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-10T20:42:07.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-10T20:42:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.616 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-10T20:42:07.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-10T20:42:07.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-10T20:42:07.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.627 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.629 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-10T20:42:07.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.636 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.637 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.640 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-10T20:42:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.652 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.661 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.664 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-10T20:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.672 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.673 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.675 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.684 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.692 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.693 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.696 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.712 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.715 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.716 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.719 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.721 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-10T20:42:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-10T20:42:07.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.729 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.732 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-10T20:42:07.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-10T20:42:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.739 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.754 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.756 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-10T20:42:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-10T20:42:07.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-10T20:42:07.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.765 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max_requests": { 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_requests", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.779 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-10T20:42:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-10T20:42:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-10T20:42:07.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-10T20:42:07.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smb", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "internal_store_backend": { 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "internal_store_backend", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.811 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.814 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "update_orchestration": { 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "update_orchestration", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.818 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.819 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.821 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.828 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.829 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.830 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-10T20:42:07.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.844 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.851 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.864 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.870 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.871 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.877 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-10T20:42:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-10T20:42:07.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-10T20:42:07.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-10T20:42:07.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-10T20:42:07.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-10T20:42:07.893 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-10T20:42:07.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-10T20:42:07.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-10T20:42:07.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-10T20:42:07.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-10T20:42:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "services": {}, 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-09-10T20:42:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-09-10T20:42:07.900 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-10T20:42:07.901 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-10T20:42:07.902 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-10T20:42:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "force_disabled_modules": {}, 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 0, 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2568415012 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3526126208 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.905 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1402110084 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1101500009 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3593980694 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2389577643 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2920078177 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 105414844 2024-09-10T20:42:07.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4280913810 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2416835213 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.159:0", 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3218025516 2024-09-10T20:42:07.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.912 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:07.912 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:07.912 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:42:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f802c078390 msgr2=0x7f802c07a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f802c078390 0x7f802c07a850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8030001860 tx=0x7f8030001650 comp rx=0 tx=0).stop 2024-09-10T20:42:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80400b10f0 msgr2=0x7f8040159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80400b10f0 0x7f8040159f00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f804015f830 tx=0x7f8038011ef0 comp rx=0 tx=0).stop 2024-09-10T20:42:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 shutdown_connections 2024-09-10T20:42:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f802c078390 0x7f802c07a850 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80400b10f0 0x7f8040159f00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80401f59f0 0x7f80401f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 --2- 172.21.15.28:0/923700567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8028004000 0x7f80400b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 >> 172.21.15.28:0/923700567 conn(0x7f804006cf00 msgr2=0x7f804015d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 shutdown_connections 2024-09-10T20:42:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:06.906+0000 7f8045533640 1 -- 172.21.15.28:0/923700567 wait complete. 2024-09-10T20:42:07.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr fail x 2024-09-10T20:42:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.034+0000 7f6d49ba0640 1 -- 172.21.15.28:0/599306014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d4415df70 msgr2=0x7f6d440b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.034+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/599306014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d4415df70 0x7f6d440b2ba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009830 tx=0x7f6d3802a480 comp rx=0 tx=0).stop 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.034+0000 7f6d49ba0640 1 -- 172.21.15.28:0/599306014 shutdown_connections 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.034+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/599306014 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d440b4d10 0x7f6d440ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.034+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/599306014 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d4415df70 0x7f6d440b2ba0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.034+0000 7f6d49ba0640 1 -- 172.21.15.28:0/599306014 >> 172.21.15.28:0/599306014 conn(0x7f6d4406cf00 msgr2=0x7f6d4406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.034+0000 7f6d49ba0640 1 -- 172.21.15.28:0/599306014 shutdown_connections 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.035+0000 7f6d49ba0640 1 -- 172.21.15.28:0/599306014 wait complete. 2024-09-10T20:42:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.036+0000 7f6d49ba0640 1 Processor -- start 2024-09-10T20:42:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.036+0000 7f6d49ba0640 1 -- start start 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d49ba0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d440b4d10 0x7f6d440c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d49ba0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4415a190 0x7f6d440c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d49ba0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d4415df70 0x7f6d4415ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d49ba0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4415a6f0 con 0x7f6d4415a190 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d49ba0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4415a570 con 0x7f6d440b4d10 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d49ba0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6d4415a870 con 0x7f6d4415df70 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4415a190 0x7f6d440c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4415a190 0x7f6d440c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45550/0 (socket says 172.21.15.28:45550) 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 -- 172.21.15.28:0/2677215021 learned_addr learned my addr 172.21.15.28:0/2677215021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d43fff640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d4415df70 0x7f6d4415ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d437fe640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d440b4d10 0x7f6d440c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 -- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d4415df70 msgr2=0x7f6d4415ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d4415df70 0x7f6d4415ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 -- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d440b4d10 msgr2=0x7f6d440c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d440b4d10 0x7f6d440c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 -- 172.21.15.28:0/2677215021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d441f49b0 con 0x7f6d4415a190 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.037+0000 7f6d42ffd640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4415a190 0x7f6d440c8f20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6d3802aa70 tx=0x7f6d38033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.038+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38005c90 con 0x7f6d4415a190 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.038+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d38031e50 con 0x7f6d4415a190 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.038+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d441f83d0 con 0x7f6d4415a190 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.038+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38005260 con 0x7f6d4415a190 2024-09-10T20:42:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.038+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d441f8990 con 0x7f6d4415a190 2024-09-10T20:42:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.039+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d440ce030 con 0x7f6d4415a190 2024-09-10T20:42:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.039+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 100958+0+0 (secure 0 0 0) 0x7f6d38005400 con 0x7f6d4415a190 2024-09-10T20:42:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.040+0000 7f6d2bfff640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6d30078390 0x7f6d3007a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.040+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f6d380bb380 con 0x7f6d4415a190 2024-09-10T20:42:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.040+0000 7f6d437fe640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6d30078390 0x7f6d3007a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.041+0000 7f6d437fe640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6d30078390 0x7f6d3007a850 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6d340017c0 tx=0x7f6d34007e90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.044+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6d3803f030 con 0x7f6d4415a190 2024-09-10T20:42:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.277+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr fail", "who": "x"} v 0) -- 0x7f6d44073510 con 0x7f6d4415a190 2024-09-10T20:42:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.280+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr fail", "who": "x"}]=0 v13) ==== 68+0+0 (secure 0 0 0) 0x7f6d38083150 con 0x7f6d4415a190 2024-09-10T20:42:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.282+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 13) ==== 50324+0+0 (secure 0 0 0) 0x7f6d3807f3c0 con 0x7f6d4415a190 2024-09-10T20:42:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.282+0000 7f6d2bfff640 1 -- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6d30078390 msgr2=0x7f6d3007a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.282+0000 7f6d2bfff640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6d30078390 0x7f6d3007a850 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6d340017c0 tx=0x7f6d34007e90 comp rx=0 tx=0).stop 2024-09-10T20:42:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4415a190 msgr2=0x7f6d440c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4415a190 0x7f6d440c8f20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6d3802aa70 tx=0x7f6d38033c20 comp rx=0 tx=0).stop 2024-09-10T20:42:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 shutdown_connections 2024-09-10T20:42:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:6800/2898596254,v1:172.21.15.159:6801/2898596254] conn(0x7f6d30078390 0x7f6d3007a850 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d4415df70 0x7f6d4415ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4415a190 0x7f6d440c8f20 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 --2- 172.21.15.28:0/2677215021 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d440b4d10 0x7f6d440c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 >> 172.21.15.28:0/2677215021 conn(0x7f6d4406cf00 msgr2=0x7f6d440b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 shutdown_connections 2024-09-10T20:42:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.283+0000 7f6d49ba0640 1 -- 172.21.15.28:0/2677215021 wait complete. 2024-09-10T20:42:08.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:08.377 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.378 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:42:08.373+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:42:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 -- 172.21.15.28:0/1922146191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37806f940 msgr2=0x7fd37806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 --2- 172.21.15.28:0/1922146191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37806f940 0x7fd37806fdf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd370005fd0 tx=0x7fd37002aa90 comp rx=0 tx=0).stop 2024-09-10T20:42:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 -- 172.21.15.28:0/1922146191 shutdown_connections 2024-09-10T20:42:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 --2- 172.21.15.28:0/1922146191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37806f940 0x7fd37806fdf0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 --2- 172.21.15.28:0/1922146191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd37806d560 0x7fd37806ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 -- 172.21.15.28:0/1922146191 >> 172.21.15.28:0/1922146191 conn(0x7fd3780d0420 msgr2=0x7fd3780d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 -- 172.21.15.28:0/1922146191 shutdown_connections 2024-09-10T20:42:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.439+0000 7fd37d194640 1 -- 172.21.15.28:0/1922146191 wait complete. 2024-09-10T20:42:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.441+0000 7fd37d194640 1 Processor -- start 2024-09-10T20:42:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.441+0000 7fd37d194640 1 -- start start 2024-09-10T20:42:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.441+0000 7fd37d194640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd37806f940 0x7fd3781f80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37d194640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37815e4c0 0x7fd37815b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37d194640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3780b58f0 0x7fd3780b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37d194640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3780c9fe0 con 0x7fd37815e4c0 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37d194640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd3780c9e60 con 0x7fd3780b58f0 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37d194640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd3780ca160 con 0x7fd37806f940 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37815e4c0 0x7fd37815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37815e4c0 0x7fd37815b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45664/0 (socket says 172.21.15.28:45664) 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 -- 172.21.15.28:0/431441398 learned_addr learned my addr 172.21.15.28:0/431441398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd376d9d640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd37806f940 0x7fd3781f80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37759e640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3780b58f0 0x7fd3780b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 -- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd37806f940 msgr2=0x7fd3781f80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd37806f940 0x7fd3781f80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 -- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3780b58f0 msgr2=0x7fd3780b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3780b58f0 0x7fd3780b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 -- 172.21.15.28:0/431441398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3781f8fd0 con 0x7fd37815e4c0 2024-09-10T20:42:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd376d9d640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd37806f940 0x7fd3781f80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:42:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37659c640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37815e4c0 0x7fd37815b470 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c370 tx=0x7fd36000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd36f7fe640 1 -- 172.21.15.28:0/431441398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360017070 con 0x7fd37815e4c0 2024-09-10T20:42:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd36f7fe640 1 -- 172.21.15.28:0/431441398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3600053c0 con 0x7fd37815e4c0 2024-09-10T20:42:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd37815b9b0 con 0x7fd37815e4c0 2024-09-10T20:42:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd36f7fe640 1 -- 172.21.15.28:0/431441398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360003850 con 0x7fd37815e4c0 2024-09-10T20:42:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.442+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3781f8a10 con 0x7fd37815e4c0 2024-09-10T20:42:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.443+0000 7fd36f7fe640 1 -- 172.21.15.28:0/431441398 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 50324+0+0 (secure 0 0 0) 0x7fd3600039f0 con 0x7fd37815e4c0 2024-09-10T20:42:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.444+0000 7fd36f7fe640 1 -- 172.21.15.28:0/431441398 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7fd3600524c0 con 0x7fd37815e4c0 2024-09-10T20:42:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.444+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3780ce400 con 0x7fd37815e4c0 2024-09-10T20:42:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.448+0000 7fd36f7fe640 1 -- 172.21.15.28:0/431441398 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd3780ce400 con 0x7fd37815e4c0 2024-09-10T20:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.670+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3780ca680 con 0x7fd37815e4c0 2024-09-10T20:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.671+0000 7fd36f7fe640 1 -- 172.21.15.28:0/431441398 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5228 (secure 0 0 0) 0x7fd360011cf0 con 0x7fd37815e4c0 2024-09-10T20:42:08.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:08.675 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"btime":"2024-09-10T20:42:03:504689+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 45 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37815e4c0 msgr2=0x7fd37815b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37815e4c0 0x7fd37815b470 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c370 tx=0x7fd36000c830 comp rx=0 tx=0).stop 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 shutdown_connections 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3780b58f0 0x7fd3780b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd37815e4c0 0x7fd37815b470 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 --2- 172.21.15.28:0/431441398 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd37806f940 0x7fd3781f80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 >> 172.21.15.28:0/431441398 conn(0x7fd3780d0420 msgr2=0x7fd378160e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 shutdown_connections 2024-09-10T20:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:08.673+0000 7fd37d194640 1 -- 172.21.15.28:0/431441398 wait complete. 2024-09-10T20:42:08.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-10T20:42:08.808 INFO:teuthology.orchestra.run.smithi028.stdout:[{"id":6426,"entity":{"name":{"type":"client","num":6426},"addr":{"type":"any","addr":"172.21.15.28:0","nonce":2865708807}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.418987991,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.9980878832478701,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6426 172.21.15.28:0/2865708807","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"y","hostname":"smithi028","pid":"36395","root":"/"}},{"id":5893,"entity":{"name":{"type":"client","num":5893},"addr":{"type":"any","addr":"172.21.15.28:0","nonce":2369088157}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.41998796300000002,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.998085575030506,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5893 172.21.15.28:0/2369088157","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"8f219c476cb4f26fc5719c661c0323719a45cb30","ceph_version":"ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)","entity_id":"y","hostname":"smithi028","pid":"36395","root":"/"}}] 2024-09-10T20:42:08.809 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi028", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36395", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.28:0", "nonce": 2865708807, "type": "any" }, "name": { "num": 6426, "type": "client" } }, "id": 6426, "inst": "client.6426 172.21.15.28:0/2865708807", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.9980878832478701 }, "state": "open", "uptime": 0.418987991 }, { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi028", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "36395", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.28:0", "nonce": 2369088157, "type": "any" }, "name": { "num": 5893, "type": "client" } }, "id": 5893, "inst": "client.5893 172.21.15.28:0/2369088157", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.998085575030506 }, "state": "open", "uptime": 0.419987963 } ] 2024-09-10T20:42:08.809 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-10T20:42:08.809 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:42:08.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:08.843 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:08.879 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:42:08.879 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:08.879 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:08.879 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:08.899 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:42:08.900 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:08.900 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:08.900 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:08.934 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:42:08.934 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:08.935 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:08.935 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:08.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:42:08.956 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:08.956 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:08.956 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:08.990 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:42:08.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.097+0000 7f15f6fe2640 1 -- 172.21.15.28:0/782293875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00d07a0 msgr2=0x7f15f00b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.097+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/782293875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00d07a0 0x7f15f00b2e80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009be0 tx=0x7f15e802a960 comp rx=0 tx=0).stop 2024-09-10T20:42:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.097+0000 7f15f6fe2640 1 -- 172.21.15.28:0/782293875 shutdown_connections 2024-09-10T20:42:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.097+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/782293875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00d07a0 0x7f15f00b2e80 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.097+0000 7f15f6fe2640 1 -- 172.21.15.28:0/782293875 >> 172.21.15.28:0/782293875 conn(0x7f15f006cf00 msgr2=0x7f15f006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.097+0000 7f15f6fe2640 1 -- 172.21.15.28:0/782293875 shutdown_connections 2024-09-10T20:42:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.097+0000 7f15f6fe2640 1 -- 172.21.15.28:0/782293875 wait complete. 2024-09-10T20:42:09.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.098+0000 7f15f6fe2640 1 Processor -- start 2024-09-10T20:42:09.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.098+0000 7f15f6fe2640 1 -- start start 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f6fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00cc570 0x7f15f01f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f6fe2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15f00cd3d0 0x7f15f00cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f6fe2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15f00d07a0 0x7f15f0070050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f6fe2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15f00b3540 con 0x7f15f00cc570 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f6fe2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f15f00b33c0 con 0x7f15f00cd3d0 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f6fe2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f15f00b36c0 con 0x7f15f00d07a0 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00cc570 0x7f15f01f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00cc570 0x7f15f01f4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45688/0 (socket says 172.21.15.28:45688) 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 -- 172.21.15.28:0/1915773472 learned_addr learned my addr 172.21.15.28:0/1915773472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f557f640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15f00d07a0 0x7f15f0070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15effff640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15f00cd3d0 0x7f15f00cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 -- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15f00d07a0 msgr2=0x7f15f0070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15f00d07a0 0x7f15f0070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 -- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15f00cd3d0 msgr2=0x7f15f00cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15f00cd3d0 0x7f15f00cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.099+0000 7f15f4d7e640 1 -- 172.21.15.28:0/1915773472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f0072910 con 0x7f15f00cc570 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.100+0000 7f15f4d7e640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00cc570 0x7f15f01f4840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c500 tx=0x7f15e400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.100+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4016020 con 0x7f15f00cc570 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.100+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f0072bf0 con 0x7f15f00cc570 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.100+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15e4012920 con 0x7f15f00cc570 2024-09-10T20:42:09.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.100+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4012d90 con 0x7f15f00cc570 2024-09-10T20:42:09.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.100+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f00731b0 con 0x7f15f00cc570 2024-09-10T20:42:09.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.101+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 50324+0+0 (secure 0 0 0) 0x7f15e401b400 con 0x7f15f00cc570 2024-09-10T20:42:09.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.101+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f00b3ae0 con 0x7f15f00cc570 2024-09-10T20:42:09.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.101+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7f15e40531b0 con 0x7f15f00cc570 2024-09-10T20:42:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.105+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f15e4020020 con 0x7f15f00cc570 2024-09-10T20:42:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.285+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 14) ==== 50451+0+0 (secure 0 0 0) 0x7f15e401a3b0 con 0x7f15f00cc570 2024-09-10T20:42:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.285+0000 7f15ed7fa640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f15d4043c20 0x7f15d4046010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.285+0000 7f15effff640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f15d4043c20 0x7f15d4046010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.287+0000 7f15effff640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f15d4043c20 0x7f15d4046010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f15f014f170 tx=0x7f15dc009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.334+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15f006d480 con 0x7f15f00cc570 2024-09-10T20:42:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.335+0000 7f15ed7fa640 1 -- 172.21.15.28:0/1915773472 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5228 (secure 0 0 0) 0x7f15e4020200 con 0x7f15f00cc570 2024-09-10T20:42:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:09.339 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"btime":"2024-09-10T20:42:03:504689+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:42:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 45 2024-09-10T20:42:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f15d4043c20 msgr2=0x7f15d4046010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f15d4043c20 0x7f15d4046010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f15f014f170 tx=0x7f15dc009040 comp rx=0 tx=0).stop 2024-09-10T20:42:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00cc570 msgr2=0x7f15f01f4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00cc570 0x7f15f01f4840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c500 tx=0x7f15e400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 shutdown_connections 2024-09-10T20:42:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f15d4043c20 0x7f15d4046010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15f00d07a0 0x7f15f0070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15f00cd3d0 0x7f15f00cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 --2- 172.21.15.28:0/1915773472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15f00cc570 0x7f15f01f4840 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 >> 172.21.15.28:0/1915773472 conn(0x7f15f006cf00 msgr2=0x7f15f015ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 shutdown_connections 2024-09-10T20:42:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.337+0000 7f15f6fe2640 1 -- 172.21.15.28:0/1915773472 wait complete. 2024-09-10T20:42:09.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.453+0000 7fb9f6b75640 1 -- 172.21.15.28:0/4128519552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f00b4d10 msgr2=0x7fb9f00ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.453+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/4128519552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f00b4d10 0x7fb9f00ce030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009be0 tx=0x7fb9e802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.453+0000 7fb9f6b75640 1 -- 172.21.15.28:0/4128519552 shutdown_connections 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.453+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/4128519552 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f00b4d10 0x7fb9f00ce030 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.453+0000 7fb9f6b75640 1 -- 172.21.15.28:0/4128519552 >> 172.21.15.28:0/4128519552 conn(0x7fb9f006cf00 msgr2=0x7fb9f006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.453+0000 7fb9f6b75640 1 -- 172.21.15.28:0/4128519552 shutdown_connections 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.453+0000 7fb9f6b75640 1 -- 172.21.15.28:0/4128519552 wait complete. 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 Processor -- start 2024-09-10T20:42:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 -- start start 2024-09-10T20:42:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00b2ba0 0x7fb9f014f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9f00b4d10 0x7fb9f014e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f015df70 0x7fb9f014e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f00cea40 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f00ce8c0 con 0x7fb9f00b4d10 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.455+0000 7fb9f6b75640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f0166100 con 0x7fb9f015df70 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00b2ba0 0x7fb9f014f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00b2ba0 0x7fb9f014f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45694/0 (socket says 172.21.15.28:45694) 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 -- 172.21.15.28:0/2122179820 learned_addr learned my addr 172.21.15.28:0/2122179820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f5112640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f015df70 0x7fb9f014e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9effff640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9f00b4d10 0x7fb9f014e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 -- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f015df70 msgr2=0x7fb9f014e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f015df70 0x7fb9f014e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 -- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9f00b4d10 msgr2=0x7fb9f014e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9f00b4d10 0x7fb9f014e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 -- 172.21.15.28:0/2122179820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f014ff60 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f4911640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00b2ba0 0x7fb9f014f660 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c4d0 tx=0x7fb9e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9ed7fa640 1 -- 172.21.15.28:0/2122179820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0017070 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9ed7fa640 1 -- 172.21.15.28:0/2122179820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9e00035f0 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f00b11e0 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9ed7fa640 1 -- 172.21.15.28:0/2122179820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0003a00 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.456+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f00b17a0 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.457+0000 7fb9ed7fa640 1 -- 172.21.15.28:0/2122179820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 14) ==== 50451+0+0 (secure 0 0 0) 0x7fb9e0003ba0 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.457+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f00ce030 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.458+0000 7fb9ed7fa640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb9dc03e060 0x7fb9dc040520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.458+0000 7fb9ed7fa640 1 -- 172.21.15.28:0/2122179820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7fb9e00525c0 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.458+0000 7fb9effff640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb9dc03e060 0x7fb9dc040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.458+0000 7fb9effff640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb9dc03e060 0x7fb9dc040520 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00d07d0 tx=0x7fb9e4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.462+0000 7fb9ed7fa640 1 -- 172.21.15.28:0/2122179820 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb9e001bd50 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.682+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9f00d0470 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.683+0000 7fb9ed7fa640 1 -- 172.21.15.28:0/2122179820 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5228 (secure 0 0 0) 0x7fb9e0010300 con 0x7fb9f00b2ba0 2024-09-10T20:42:09.687 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:09.687 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"btime":"2024-09-10T20:42:03:504689+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:42:09.687 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 45 2024-09-10T20:42:09.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb9dc03e060 msgr2=0x7fb9dc040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb9dc03e060 0x7fb9dc040520 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00d07d0 tx=0x7fb9e4009040 comp rx=0 tx=0).stop 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00b2ba0 msgr2=0x7fb9f014f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00b2ba0 0x7fb9f014f660 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c4d0 tx=0x7fb9e000c990 comp rx=0 tx=0).stop 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 shutdown_connections 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb9dc03e060 0x7fb9dc040520 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9f015df70 0x7fb9f014e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9f00b4d10 0x7fb9f014e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 --2- 172.21.15.28:0/2122179820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00b2ba0 0x7fb9f014f660 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 >> 172.21.15.28:0/2122179820 conn(0x7fb9f006cf00 msgr2=0x7fb9f015c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 shutdown_connections 2024-09-10T20:42:09.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.685+0000 7fb9f6b75640 1 -- 172.21.15.28:0/2122179820 wait complete. 2024-09-10T20:42:09.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 -- 172.21.15.28:0/1721045679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 msgr2=0x7f9e600b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 --2- 172.21.15.28:0/1721045679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 0x7f9e600b5a50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005fd0 tx=0x7f9e4802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 -- 172.21.15.28:0/1721045679 shutdown_connections 2024-09-10T20:42:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 --2- 172.21.15.28:0/1721045679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e6006cae0 0x7f9e6006cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 --2- 172.21.15.28:0/1721045679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 0x7f9e600b5a50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 -- 172.21.15.28:0/1721045679 >> 172.21.15.28:0/1721045679 conn(0x7f9e6006dbf0 msgr2=0x7f9e6006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 -- 172.21.15.28:0/1721045679 shutdown_connections 2024-09-10T20:42:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.801+0000 7f9e659bc640 1 -- 172.21.15.28:0/1721045679 wait complete. 2024-09-10T20:42:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.802+0000 7f9e659bc640 1 Processor -- start 2024-09-10T20:42:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.802+0000 7f9e659bc640 1 -- start start 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e659bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e6006cae0 0x7f9e601f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e659bc640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e600747a0 0x7f9e6015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e659bc640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 0x7f9e601f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e659bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600aff90 con 0x7f9e6006cae0 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e659bc640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600afe10 con 0x7f9e6015c160 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e659bc640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9e600b0110 con 0x7f9e600747a0 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e6006cae0 0x7f9e601f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e6006cae0 0x7f9e601f83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45710/0 (socket says 172.21.15.28:45710) 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e5effd640 1 -- 172.21.15.28:0/985877317 learned_addr learned my addr 172.21.15.28:0/985877317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e5f7fe640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 0x7f9e601f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.803+0000 7f9e567fc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e600747a0 0x7f9e6015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5effd640 1 -- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e600747a0 msgr2=0x7f9e6015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5effd640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e600747a0 0x7f9e6015b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5effd640 1 -- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 msgr2=0x7f9e601f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5effd640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 0x7f9e601f4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5effd640 1 -- 172.21.15.28:0/985877317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e601f47d0 con 0x7f9e6006cae0 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5effd640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e6006cae0 0x7f9e601f83e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c500 tx=0x7f9e5000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5cff9640 1 -- 172.21.15.28:0/985877317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50017070 con 0x7f9e6006cae0 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5cff9640 1 -- 172.21.15.28:0/985877317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e500035f0 con 0x7f9e6006cae0 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e601f4a80 con 0x7f9e6006cae0 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e5cff9640 1 -- 172.21.15.28:0/985877317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50003a40 con 0x7f9e6006cae0 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.804+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e601f5040 con 0x7f9e6006cae0 2024-09-10T20:42:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.805+0000 7f9e5cff9640 1 -- 172.21.15.28:0/985877317 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 14) ==== 50451+0+0 (secure 0 0 0) 0x7f9e50003be0 con 0x7f9e6006cae0 2024-09-10T20:42:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.805+0000 7f9e5cff9640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9e4003e060 0x7f9e40040520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.805+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e600b0630 con 0x7f9e6006cae0 2024-09-10T20:42:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.805+0000 7f9e5cff9640 1 -- 172.21.15.28:0/985877317 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7f9e500524d0 con 0x7f9e6006cae0 2024-09-10T20:42:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.805+0000 7f9e567fc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9e4003e060 0x7f9e40040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.806+0000 7f9e567fc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9e4003e060 0x7f9e40040520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9e48033000 tx=0x7f9e48004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.810+0000 7f9e5cff9640 1 -- 172.21.15.28:0/985877317 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9e50012bb0 con 0x7f9e6006cae0 2024-09-10T20:42:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.993+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e600b3dc0 con 0x7f9e6006cae0 2024-09-10T20:42:09.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.994+0000 7f9e5cff9640 1 -- 172.21.15.28:0/985877317 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+17491 (secure 0 0 0) 0x7f9e50010300 con 0x7f9e6006cae0 2024-09-10T20:42:09.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:09.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":69,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:08.278586+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:47.307952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:48.566196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/105414844":"2024-09-11T20:42:08.278542+0000","172.21.15.159:6801/2898596254":"2024-09-11T20:42:08.278542+0000","172.21.15.159:6800/2898596254":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2416835213":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2389577643":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/1101500009":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2920078177":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3526126208":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3218025516":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2568415012":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/4280913810":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/1402110084":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3593980694":"2024-09-11T20:42:08.278542+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.996+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9e4003e060 msgr2=0x7f9e40040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9e4003e060 0x7f9e40040520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9e48033000 tx=0x7f9e48004730 comp rx=0 tx=0).stop 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e6006cae0 msgr2=0x7f9e601f83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e6006cae0 0x7f9e601f83e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c500 tx=0x7f9e5000c900 comp rx=0 tx=0).stop 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 shutdown_connections 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9e4003e060 0x7f9e40040520 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9e6015c160 0x7f9e601f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9e600747a0 0x7f9e6015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 --2- 172.21.15.28:0/985877317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e6006cae0 0x7f9e601f83e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 >> 172.21.15.28:0/985877317 conn(0x7f9e6006dbf0 msgr2=0x7f9e600d1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 shutdown_connections 2024-09-10T20:42:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:09.997+0000 7f9e659bc640 1 -- 172.21.15.28:0/985877317 wait complete. 2024-09-10T20:42:10.011 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:42:10.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 -- 172.21.15.28:0/1160751505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37281554e0 msgr2=0x7f372815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 --2- 172.21.15.28:0/1160751505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37281554e0 0x7f372815e930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f371c009830 tx=0x7f371c02a510 comp rx=0 tx=0).stop 2024-09-10T20:42:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 -- 172.21.15.28:0/1160751505 shutdown_connections 2024-09-10T20:42:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 --2- 172.21.15.28:0/1160751505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372815ee70 0x7f37280d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 --2- 172.21.15.28:0/1160751505 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37281554e0 0x7f372815e930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 -- 172.21.15.28:0/1160751505 >> 172.21.15.28:0/1160751505 conn(0x7f372806cf00 msgr2=0x7f372806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 -- 172.21.15.28:0/1160751505 shutdown_connections 2024-09-10T20:42:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.113+0000 7f372f52b640 1 -- 172.21.15.28:0/1160751505 wait complete. 2024-09-10T20:42:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.114+0000 7f372f52b640 1 Processor -- start 2024-09-10T20:42:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.114+0000 7f372f52b640 1 -- start start 2024-09-10T20:42:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372f52b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3728155100 0x7f37280b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372f52b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37281554e0 0x7f3728157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372f52b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372815ee70 0x7f37280afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372f52b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3728072de0 con 0x7f3728155100 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372f52b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f37280d0af0 con 0x7f372815ee70 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372f52b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3728072f60 con 0x7f37281554e0 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3728155100 0x7f37280b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3728155100 0x7f37280b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45712/0 (socket says 172.21.15.28:45712) 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372cac6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37281554e0 0x7f3728157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 -- 172.21.15.28:0/3510400185 learned_addr learned my addr 172.21.15.28:0/3510400185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372dac8640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372815ee70 0x7f37280afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 -- 172.21.15.28:0/3510400185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37281554e0 msgr2=0x7f3728157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37281554e0 0x7f3728157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 -- 172.21.15.28:0/3510400185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372815ee70 msgr2=0x7f37280afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372815ee70 0x7f37280afa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 -- 172.21.15.28:0/3510400185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3728157fd0 con 0x7f3728155100 2024-09-10T20:42:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.115+0000 7f372d2c7640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3728155100 0x7f37280b1070 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f371800cbf0 tx=0x7f371800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.116+0000 7f3725ffb640 1 -- 172.21.15.28:0/3510400185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3718016020 con 0x7f3728155100 2024-09-10T20:42:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.116+0000 7f3725ffb640 1 -- 172.21.15.28:0/3510400185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f371800acd0 con 0x7f3728155100 2024-09-10T20:42:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.116+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37280b1690 con 0x7f3728155100 2024-09-10T20:42:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.116+0000 7f3725ffb640 1 -- 172.21.15.28:0/3510400185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3718012680 con 0x7f3728155100 2024-09-10T20:42:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.116+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3728155e40 con 0x7f3728155100 2024-09-10T20:42:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.117+0000 7f3725ffb640 1 -- 172.21.15.28:0/3510400185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 14) ==== 50451+0+0 (secure 0 0 0) 0x7f371801b400 con 0x7f3728155100 2024-09-10T20:42:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.117+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f372815e930 con 0x7f3728155100 2024-09-10T20:42:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.117+0000 7f3725ffb640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f371403e060 0x7f3714040520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.117+0000 7f3725ffb640 1 -- 172.21.15.28:0/3510400185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7f3718053890 con 0x7f3728155100 2024-09-10T20:42:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.117+0000 7f372cac6640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f371403e060 0x7f3714040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.118+0000 7f372cac6640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f371403e060 0x7f3714040520 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f371c034000 tx=0x7f371c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:10.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.122+0000 7f3725ffb640 1 -- 172.21.15.28:0/3510400185 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3718020020 con 0x7f3728155100 2024-09-10T20:42:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.344+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f372809d8e0 con 0x7f3728155100 2024-09-10T20:42:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.344+0000 7f3725ffb640 1 -- 172.21.15.28:0/3510400185 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5228 (secure 0 0 0) 0x7f3718010cf0 con 0x7f3728155100 2024-09-10T20:42:10.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:10.349 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"btime":"2024-09-10T20:42:03:504689+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:42:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 45 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f371403e060 msgr2=0x7f3714040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f371403e060 0x7f3714040520 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f371c034000 tx=0x7f371c004730 comp rx=0 tx=0).stop 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3728155100 msgr2=0x7f37280b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3728155100 0x7f37280b1070 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f371800cbf0 tx=0x7f371800a060 comp rx=0 tx=0).stop 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 shutdown_connections 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f371403e060 0x7f3714040520 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f372815ee70 0x7f37280afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37281554e0 0x7f3728157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 --2- 172.21.15.28:0/3510400185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3728155100 0x7f37280b1070 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 >> 172.21.15.28:0/3510400185 conn(0x7f372806cf00 msgr2=0x7f37280ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 shutdown_connections 2024-09-10T20:42:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.347+0000 7f372f52b640 1 -- 172.21.15.28:0/3510400185 wait complete. 2024-09-10T20:42:10.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.467+0000 7f1ab72f7640 1 -- 172.21.15.28:0/1633432473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab00cf780 msgr2=0x7f1ab00cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.467+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/1633432473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab00cf780 0x7f1ab00cfbc0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00698a0 tx=0x7f1aa402a560 comp rx=0 tx=0).stop 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.467+0000 7f1ab72f7640 1 -- 172.21.15.28:0/1633432473 shutdown_connections 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.467+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/1633432473 >> v2:172.21.15.159:3300/0 conn(0x7f1ab0161b50 0x7f1ab0161fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.467+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/1633432473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab00cf780 0x7f1ab00cfbc0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.467+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/1633432473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ab015de70 0x7f1ab00d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.467+0000 7f1ab72f7640 1 -- 172.21.15.28:0/1633432473 >> 172.21.15.28:0/1633432473 conn(0x7f1ab00d3fb0 msgr2=0x7f1ab00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.468+0000 7f1ab72f7640 1 -- 172.21.15.28:0/1633432473 shutdown_connections 2024-09-10T20:42:10.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.468+0000 7f1ab72f7640 1 -- 172.21.15.28:0/1633432473 wait complete. 2024-09-10T20:42:10.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.469+0000 7f1ab72f7640 1 Processor -- start 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.469+0000 7f1ab72f7640 1 -- start start 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.469+0000 7f1ab72f7640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ab00cf780 0x7f1ab014d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.469+0000 7f1ab72f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab015de70 0x7f1ab00b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.469+0000 7f1ab72f7640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ab0161b50 0x7f1ab00a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.469+0000 7f1ab72f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab00b6e00 con 0x7f1ab015de70 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.469+0000 7f1ab72f7640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab015e460 con 0x7f1ab0161b50 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab72f7640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab00b6f80 con 0x7f1ab00cf780 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab015de70 0x7f1ab00b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab015de70 0x7f1ab00b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45878/0 (socket says 172.21.15.28:45878) 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 -- 172.21.15.28:0/3197477940 learned_addr learned my addr 172.21.15.28:0/3197477940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab5093640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ab00cf780 0x7f1ab014d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab5894640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ab0161b50 0x7f1ab00a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 -- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ab00cf780 msgr2=0x7f1ab014d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ab00cf780 0x7f1ab014d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 -- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ab0161b50 msgr2=0x7f1ab00a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ab0161b50 0x7f1ab00a03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 -- 172.21.15.28:0/3197477940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab015f7e0 con 0x7f1ab015de70 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab4892640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab015de70 0x7f1ab00b4740 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1aa402ab50 tx=0x7f1aa40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1aadffb640 1 -- 172.21.15.28:0/3197477940 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa403f070 con 0x7f1ab015de70 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1aadffb640 1 -- 172.21.15.28:0/3197477940 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1aa40047a0 con 0x7f1ab015de70 2024-09-10T20:42:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab014ffe0 con 0x7f1ab015de70 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.470+0000 7f1aadffb640 1 -- 172.21.15.28:0/3197477940 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4035030 con 0x7f1ab015de70 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.471+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab015fd80 con 0x7f1ab015de70 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.472+0000 7f1aadffb640 1 -- 172.21.15.28:0/3197477940 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 14) ==== 50451+0+0 (secure 0 0 0) 0x7f1aa4004bf0 con 0x7f1ab015de70 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.472+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab00d2b90 con 0x7f1ab015de70 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.472+0000 7f1aadffb640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a9c03e060 0x7f1a9c040520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.472+0000 7f1aadffb640 1 -- 172.21.15.28:0/3197477940 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7f1aa4073f00 con 0x7f1ab015de70 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.472+0000 7f1ab5093640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a9c03e060 0x7f1a9c040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.472+0000 7f1ab5093640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a9c03e060 0x7f1a9c040520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009b50 tx=0x7f1aa8005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:10.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.476+0000 7f1aadffb640 1 -- 172.21.15.28:0/3197477940 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1aa403b460 con 0x7f1ab015de70 2024-09-10T20:42:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.697+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ab015ff50 con 0x7f1ab015de70 2024-09-10T20:42:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.698+0000 7f1aadffb640 1 -- 172.21.15.28:0/3197477940 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5228 (secure 0 0 0) 0x7f1aa4042360 con 0x7f1ab015de70 2024-09-10T20:42:10.702 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:10.702 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"btime":"2024-09-10T20:42:03:504689+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:41:49.602862+0000","modified":"2024-09-10T20:41:52.641867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6138},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6138":{"gid":6138,"name":"c","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3447265222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3447265222},{"type":"v1","addr":"172.21.15.28:6834","nonce":3447265222}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6138,"qdb_cluster":[6138]},"id":5}]} 2024-09-10T20:42:10.702 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 45 2024-09-10T20:42:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a9c03e060 msgr2=0x7f1a9c040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a9c03e060 0x7f1a9c040520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009b50 tx=0x7f1aa8005f90 comp rx=0 tx=0).stop 2024-09-10T20:42:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab015de70 msgr2=0x7f1ab00b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab015de70 0x7f1ab00b4740 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1aa402ab50 tx=0x7f1aa40038a0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 shutdown_connections 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a9c03e060 0x7f1a9c040520 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ab0161b50 0x7f1ab00a03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab015de70 0x7f1ab00b4740 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 --2- 172.21.15.28:0/3197477940 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ab00cf780 0x7f1ab014d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 >> 172.21.15.28:0/3197477940 conn(0x7f1ab00d3fb0 msgr2=0x7f1ab00b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 shutdown_connections 2024-09-10T20:42:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.700+0000 7f1ab72f7640 1 -- 172.21.15.28:0/3197477940 wait complete. 2024-09-10T20:42:10.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 -- 172.21.15.28:0/150516319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac8162090 msgr2=0x7f0ac80b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/150516319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac8162090 0x7f0ac80b6e00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80d4d80 tx=0x7f0ac002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 -- 172.21.15.28:0/150516319 shutdown_connections 2024-09-10T20:42:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/150516319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac8162090 0x7f0ac80b6e00 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/150516319 >> v2:172.21.15.159:3300/0 conn(0x7f0ac80d2b90 0x7f0ac8161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 -- 172.21.15.28:0/150516319 >> 172.21.15.28:0/150516319 conn(0x7f0ac80d3fb0 msgr2=0x7f0ac80d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 -- 172.21.15.28:0/150516319 shutdown_connections 2024-09-10T20:42:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.817+0000 7f0acd2ef640 1 -- 172.21.15.28:0/150516319 wait complete. 2024-09-10T20:42:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.818+0000 7f0acd2ef640 1 Processor -- start 2024-09-10T20:42:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.818+0000 7f0acd2ef640 1 -- start start 2024-09-10T20:42:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0acd2ef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac80d2b90 0x7f0ac81f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0acd2ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac815de70 0x7f0ac80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0acd2ef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac8162090 0x7f0ac815c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0acd2ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac815e3d0 con 0x7f0ac815de70 2024-09-10T20:42:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0acd2ef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac815e250 con 0x7f0ac80d2b90 2024-09-10T20:42:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0acd2ef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac80b7340 con 0x7f0ac8162090 2024-09-10T20:42:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac815de70 0x7f0ac80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac815de70 0x7f0ac80b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45898/0 (socket says 172.21.15.28:45898) 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 -- 172.21.15.28:0/2009140167 learned_addr learned my addr 172.21.15.28:0/2009140167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac77fe640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac8162090 0x7f0ac815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac6ffd640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac80d2b90 0x7f0ac81f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 -- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac8162090 msgr2=0x7f0ac815c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac8162090 0x7f0ac815c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 -- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac80d2b90 msgr2=0x7f0ac81f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac80d2b90 0x7f0ac81f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.819+0000 7f0ac67fc640 1 -- 172.21.15.28:0/2009140167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac80b2090 con 0x7f0ac815de70 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.820+0000 7f0ac67fc640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac815de70 0x7f0ac80b3200 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00d950 tx=0x7f0abc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.820+0000 7f0aaf7fe640 1 -- 172.21.15.28:0/2009140167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc017070 con 0x7f0ac815de70 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.820+0000 7f0aaf7fe640 1 -- 172.21.15.28:0/2009140167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0abc009d20 con 0x7f0ac815de70 2024-09-10T20:42:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.820+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac815f640 con 0x7f0ac815de70 2024-09-10T20:42:10.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.820+0000 7f0aaf7fe640 1 -- 172.21.15.28:0/2009140167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc004280 con 0x7f0ac815de70 2024-09-10T20:42:10.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.820+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac815fc00 con 0x7f0ac815de70 2024-09-10T20:42:10.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.821+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac80b7810 con 0x7f0ac815de70 2024-09-10T20:42:10.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.821+0000 7f0aaf7fe640 1 -- 172.21.15.28:0/2009140167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 14) ==== 50451+0+0 (secure 0 0 0) 0x7f0abc0035f0 con 0x7f0ac815de70 2024-09-10T20:42:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.821+0000 7f0aaf7fe640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0ab403e060 0x7f0ab4040520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.821+0000 7f0aaf7fe640 1 -- 172.21.15.28:0/2009140167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7f0abc0530b0 con 0x7f0ac815de70 2024-09-10T20:42:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.821+0000 7f0ac6ffd640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0ab403e060 0x7f0ab4040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.822+0000 7f0ac6ffd640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0ab403e060 0x7f0ab4040520 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8007fd0 tx=0x7f0ab8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:10.826+0000 7f0aaf7fe640 1 -- 172.21.15.28:0/2009140167 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0abc004420 con 0x7f0ac815de70 2024-09-10T20:42:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.011+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ac80b9f30 con 0x7f0ac815de70 2024-09-10T20:42:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.012+0000 7f0aaf7fe640 1 -- 172.21.15.28:0/2009140167 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+17491 (secure 0 0 0) 0x7f0abc019d20 con 0x7f0ac815de70 2024-09-10T20:42:11.016 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:11.016 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":69,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:08.278586+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:41:47.307952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-10T20:41:48.566196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:0/105414844":"2024-09-11T20:42:08.278542+0000","172.21.15.159:6801/2898596254":"2024-09-11T20:42:08.278542+0000","172.21.15.159:6800/2898596254":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2416835213":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2389577643":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/1101500009":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2920078177":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3526126208":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3218025516":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2568415012":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/4280913810":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/1402110084":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3593980694":"2024-09-11T20:42:08.278542+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0ab403e060 msgr2=0x7f0ab4040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0ab403e060 0x7f0ab4040520 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8007fd0 tx=0x7f0ab8009040 comp rx=0 tx=0).stop 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac815de70 msgr2=0x7f0ac80b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac815de70 0x7f0ac80b3200 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00d950 tx=0x7f0abc00de10 comp rx=0 tx=0).stop 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 shutdown_connections 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0ab403e060 0x7f0ab4040520 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0ac8162090 0x7f0ac815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac815de70 0x7f0ac80b3200 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 --2- 172.21.15.28:0/2009140167 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0ac80d2b90 0x7f0ac81f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 >> 172.21.15.28:0/2009140167 conn(0x7f0ac80d3fb0 msgr2=0x7f0ac80cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 shutdown_connections 2024-09-10T20:42:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.014+0000 7f0acd2ef640 1 -- 172.21.15.28:0/2009140167 wait complete. 2024-09-10T20:42:11.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 -- 172.21.15.28:0/1662935478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 msgr2=0x7fd78c15e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 --2- 172.21.15.28:0/1662935478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c15e7e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009830 tx=0x7fd77c02a510 comp rx=0 tx=0).stop 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 -- 172.21.15.28:0/1662935478 shutdown_connections 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 --2- 172.21.15.28:0/1662935478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd78c15ed20 0x7fd78c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 --2- 172.21.15.28:0/1662935478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c15e7e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 -- 172.21.15.28:0/1662935478 >> 172.21.15.28:0/1662935478 conn(0x7fd78c0d0420 msgr2=0x7fd78c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 -- 172.21.15.28:0/1662935478 shutdown_connections 2024-09-10T20:42:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.131+0000 7fd792ff3640 1 -- 172.21.15.28:0/1662935478 wait complete. 2024-09-10T20:42:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.132+0000 7fd792ff3640 1 Processor -- start 2024-09-10T20:42:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.132+0000 7fd792ff3640 1 -- start start 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd792ff3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd792ff3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd78c151cc0 0x7fd78c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd792ff3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd78c15ed20 0x7fd78c144460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd792ff3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c15f5b0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd792ff3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c15f430 con 0x7fd78c15ed20 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd792ff3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c0cb750 con 0x7fd78c151cc0 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c070910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45906/0 (socket says 172.21.15.28:45906) 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 -- 172.21.15.28:0/3314019597 learned_addr learned my addr 172.21.15.28:0/3314019597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd791590640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd78c15ed20 0x7fd78c144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd783fff640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd78c151cc0 0x7fd78c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 -- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd78c151cc0 msgr2=0x7fd78c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd78c151cc0 0x7fd78c0b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 -- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd78c15ed20 msgr2=0x7fd78c144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd78c15ed20 0x7fd78c144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 -- 172.21.15.28:0/3314019597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c15dbf0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.133+0000 7fd790d8f640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c070910 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd78800c500 tx=0x7fd78800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.134+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788017070 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.134+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7880035f0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.134+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788003a40 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.134+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c15ded0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.134+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c1461f0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.135+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 14) ==== 50451+0+0 (secure 0 0 0) 0x7fd788003be0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.135+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c0cbbe0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.135+0000 7fd7817fa640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd77803e060 0x7fd778040520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.135+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6268+0+0 (secure 0 0 0) 0x7fd7880526f0 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.135+0000 7fd783fff640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd77803e060 0x7fd778040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.136+0000 7fd783fff640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd77803e060 0x7fd778040520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd77c034000 tx=0x7fd77c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.140+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd78801bd40 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.292+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 15) ==== 50659+0+0 (secure 0 0 0) 0x7fd788010300 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.364+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd78c15e110 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.368 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:42:11.364+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:42:11.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.366+0000 7fd7817fa640 1 -- 172.21.15.28:0/3314019597 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7fd788021090 con 0x7fd78c0cf0f0 2024-09-10T20:42:11.369 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd77803e060 msgr2=0x7fd778040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd77803e060 0x7fd778040520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd77c034000 tx=0x7fd77c004730 comp rx=0 tx=0).stop 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 msgr2=0x7fd78c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c070910 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd78800c500 tx=0x7fd78800c9c0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 shutdown_connections 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd77803e060 0x7fd778040520 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd78c15ed20 0x7fd78c144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd78c151cc0 0x7fd78c0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 --2- 172.21.15.28:0/3314019597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd78c0cf0f0 0x7fd78c070910 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 >> 172.21.15.28:0/3314019597 conn(0x7fd78c0d0420 msgr2=0x7fd78c0c9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:11.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 shutdown_connections 2024-09-10T20:42:11.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.369+0000 7fd792ff3640 1 -- 172.21.15.28:0/3314019597 wait complete. 2024-09-10T20:42:11.376 INFO:tasks.ceph.mds.c.smithi028.stderr: -8125> 2024-09-10T20:41:55.210+0000 7f4702e05640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:42:11.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:42:11.410 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:42:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 -- 172.21.15.28:0/2551875994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d41604e0 msgr2=0x7f84d4160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/2551875994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d41604e0 0x7f84d4160990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f84cc005fd0 tx=0x7f84cc02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 -- 172.21.15.28:0/2551875994 shutdown_connections 2024-09-10T20:42:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/2551875994 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 0x7f84d406ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/2551875994 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d41604e0 0x7f84d4160990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 -- 172.21.15.28:0/2551875994 >> 172.21.15.28:0/2551875994 conn(0x7f84d40d1e60 msgr2=0x7f84d40d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 -- 172.21.15.28:0/2551875994 shutdown_connections 2024-09-10T20:42:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.489+0000 7f84d9ad6640 1 -- 172.21.15.28:0/2551875994 wait complete. 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.490+0000 7f84d9ad6640 1 Processor -- start 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.490+0000 7f84d9ad6640 1 -- start start 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d9ad6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 0x7f84d40c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d9ad6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d415e100 0x7f84d40cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d9ad6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84d41604e0 0x7f84d40ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d9ad6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84d4161260 con 0x7f84d41604e0 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d9ad6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f84d41610e0 con 0x7f84d415e100 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d9ad6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f84d41613e0 con 0x7f84d406e600 2024-09-10T20:42:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d37fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 0x7f84d40c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84d41604e0 0x7f84d40ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d37fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 0x7f84d40c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:60082/0 (socket says 172.21.15.28:60082) 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84d37fe640 1 -- 172.21.15.28:0/3329254127 learned_addr learned my addr 172.21.15.28:0/3329254127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.491+0000 7f84cbfff640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d415e100 0x7f84d40cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d37fe640 1 -- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d415e100 msgr2=0x7f84d40cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d37fe640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d415e100 0x7f84d40cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d37fe640 1 -- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84d41604e0 msgr2=0x7f84d40ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d37fe640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84d41604e0 0x7f84d40ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d37fe640 1 -- 172.21.15.28:0/3329254127 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d40c8f30 con 0x7f84d406e600 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d37fe640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 0x7f84d40c9cb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c480 tx=0x7f84c400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d0ff9640 1 -- 172.21.15.28:0/3329254127 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c40052d0 con 0x7f84d406e600 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d0ff9640 1 -- 172.21.15.28:0/3329254127 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84c4005470 con 0x7f84d406e600 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d40cb4b0 con 0x7f84d406e600 2024-09-10T20:42:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d0ff9640 1 -- 172.21.15.28:0/3329254127 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4012870 con 0x7f84d406e600 2024-09-10T20:42:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.492+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d40c94b0 con 0x7f84d406e600 2024-09-10T20:42:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.493+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d406ea40 con 0x7f84d406e600 2024-09-10T20:42:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.493+0000 7f84d0ff9640 1 -- 172.21.15.28:0/3329254127 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 15) ==== 50659+0+0 (secure 0 0 0) 0x7f84c40035f0 con 0x7f84d406e600 2024-09-10T20:42:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.493+0000 7f84d0ff9640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f84b403e2a0 0x7f84b4040760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.493+0000 7f84d0ff9640 1 -- 172.21.15.28:0/3329254127 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(70..70 src has 1..70) ==== 6354+0+0 (secure 0 0 0) 0x7f84c4052f80 con 0x7f84d406e600 2024-09-10T20:42:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.494+0000 7f84cbfff640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f84b403e2a0 0x7f84b4040760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.494+0000 7f84cbfff640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f84b403e2a0 0x7f84b4040760 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f84d415d260 tx=0x7f84bc009260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:11.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.498+0000 7f84d0ff9640 1 -- 172.21.15.28:0/3329254127 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f84c4011dc0 con 0x7f84d406e600 2024-09-10T20:42:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:11.720+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f84d40cbf70 con 0x7f84d406e600 2024-09-10T20:42:12.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.369+0000 7f84d0ff9640 1 -- 172.21.15.28:0/3329254127 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) ==== 104+0+0 (secure 0 0 0) 0x7f84c4019e20 con 0x7f84d406e600 2024-09-10T20:42:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f84b403e2a0 msgr2=0x7f84b4040760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f84b403e2a0 0x7f84b4040760 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f84d415d260 tx=0x7f84bc009260 comp rx=0 tx=0).stop 2024-09-10T20:42:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 msgr2=0x7f84d40c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 0x7f84d40c9cb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c480 tx=0x7f84c400c940 comp rx=0 tx=0).stop 2024-09-10T20:42:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 shutdown_connections 2024-09-10T20:42:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f84b403e2a0 0x7f84b4040760 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84d41604e0 0x7f84d40ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84d415e100 0x7f84d40cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 --2- 172.21.15.28:0/3329254127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84d406e600 0x7f84d40c9cb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 >> 172.21.15.28:0/3329254127 conn(0x7f84d40d1e60 msgr2=0x7f84d40b6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 shutdown_connections 2024-09-10T20:42:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.372+0000 7f84d9ad6640 1 -- 172.21.15.28:0/3329254127 wait complete. 2024-09-10T20:42:12.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.548+0000 7ffa86439640 1 -- 172.21.15.28:0/759511988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 msgr2=0x7ffa800ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.548+0000 7ffa86439640 1 --2- 172.21.15.28:0/759511988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 0x7ffa800ccb90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ffa800736d0 tx=0x7ffa7402a480 comp rx=0 tx=0).stop 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.548+0000 7ffa86439640 1 -- 172.21.15.28:0/759511988 shutdown_connections 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.548+0000 7ffa86439640 1 --2- 172.21.15.28:0/759511988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa80073d90 0x7ffa8015f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.548+0000 7ffa86439640 1 --2- 172.21.15.28:0/759511988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 0x7ffa800ccb90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.548+0000 7ffa86439640 1 -- 172.21.15.28:0/759511988 >> 172.21.15.28:0/759511988 conn(0x7ffa800d0d50 msgr2=0x7ffa800d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.549+0000 7ffa86439640 1 -- 172.21.15.28:0/759511988 shutdown_connections 2024-09-10T20:42:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.549+0000 7ffa86439640 1 -- 172.21.15.28:0/759511988 wait complete. 2024-09-10T20:42:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.550+0000 7ffa86439640 1 Processor -- start 2024-09-10T20:42:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.550+0000 7ffa86439640 1 -- start start 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa86439640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa800739b0 0x7ffa80154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa86439640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa80073d90 0x7ffa8015ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa86439640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 0x7ffa8015d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa86439640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa800cf650 con 0x7ffa80073d90 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa86439640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffa800cf4d0 con 0x7ffa80074480 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa86439640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffa800cf7d0 con 0x7ffa800739b0 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa80073d90 0x7ffa8015ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa80073d90 0x7ffa8015ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45940/0 (socket says 172.21.15.28:45940) 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 -- 172.21.15.28:0/1916671173 learned_addr learned my addr 172.21.15.28:0/1916671173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa849d6640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 0x7ffa8015d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7ffff640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa800739b0 0x7ffa80154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 -- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa800739b0 msgr2=0x7ffa80154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa800739b0 0x7ffa80154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 -- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 msgr2=0x7ffa8015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 0x7ffa8015d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 -- 172.21.15.28:0/1916671173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa800cb0f0 con 0x7ffa80073d90 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.551+0000 7ffa7f7fe640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa80073d90 0x7ffa8015ceb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ffa74034040 tx=0x7ffa74003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.552+0000 7ffa7cff9640 1 -- 172.21.15.28:0/1916671173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7403e030 con 0x7ffa80073d90 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.552+0000 7ffa7cff9640 1 -- 172.21.15.28:0/1916671173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa74003e00 con 0x7ffa80073d90 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.552+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa801f7c70 con 0x7ffa80073d90 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.552+0000 7ffa7cff9640 1 -- 172.21.15.28:0/1916671173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74035040 con 0x7ffa80073d90 2024-09-10T20:42:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.552+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa801f8230 con 0x7ffa80073d90 2024-09-10T20:42:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.553+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa800ccb90 con 0x7ffa80073d90 2024-09-10T20:42:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.553+0000 7ffa7cff9640 1 -- 172.21.15.28:0/1916671173 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7ffa740049c0 con 0x7ffa80073d90 2024-09-10T20:42:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.554+0000 7ffa7cff9640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa6c078090 0x7ffa6c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.554+0000 7ffa7cff9640 1 -- 172.21.15.28:0/1916671173 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6354+0+0 (secure 0 0 0) 0x7ffa740bac40 con 0x7ffa80073d90 2024-09-10T20:42:12.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.554+0000 7ffa7ffff640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa6c078090 0x7ffa6c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:12.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.554+0000 7ffa7ffff640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa6c078090 0x7ffa6c07a550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffa70004f60 tx=0x7ffa70007e30 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.558+0000 7ffa7cff9640 1 -- 172.21.15.28:0/1916671173 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffa74082bc0 con 0x7ffa80073d90 2024-09-10T20:42:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:12.744+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffa800cc4f0 con 0x7ffa80073d90 2024-09-10T20:42:13.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.546+0000 7ffa7cff9640 1 -- 172.21.15.28:0/1916671173 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v71) ==== 181+0+0 (secure 0 0 0) 0x7ffa740875f0 con 0x7ffa80073d90 2024-09-10T20:42:13.551 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:42:13.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.549+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa6c078090 msgr2=0x7ffa6c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.549+0000 7ffa86439640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa6c078090 0x7ffa6c07a550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffa70004f60 tx=0x7ffa70007e30 comp rx=0 tx=0).stop 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.549+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa80073d90 msgr2=0x7ffa8015ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.549+0000 7ffa86439640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa80073d90 0x7ffa8015ceb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ffa74034040 tx=0x7ffa74003910 comp rx=0 tx=0).stop 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.549+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 shutdown_connections 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.549+0000 7ffa86439640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa6c078090 0x7ffa6c07a550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.550+0000 7ffa86439640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa80074480 0x7ffa8015d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.550+0000 7ffa86439640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa80073d90 0x7ffa8015ceb0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.550+0000 7ffa86439640 1 --2- 172.21.15.28:0/1916671173 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa800739b0 0x7ffa80154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.550+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 >> 172.21.15.28:0/1916671173 conn(0x7ffa800d0d50 msgr2=0x7ffa800b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.550+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 shutdown_connections 2024-09-10T20:42:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.550+0000 7ffa86439640 1 -- 172.21.15.28:0/1916671173 wait complete. 2024-09-10T20:42:13.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:42:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 -- 172.21.15.28:0/1032680313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c06e390 msgr2=0x7f9d7c15cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1032680313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c06e390 0x7f9d7c15cab0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009830 tx=0x7f9d6402a510 comp rx=0 tx=0).stop 2024-09-10T20:42:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 -- 172.21.15.28:0/1032680313 shutdown_connections 2024-09-10T20:42:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1032680313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c06e390 0x7f9d7c15cab0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1032680313 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d7c0b5030 0x7f9d7c15f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 -- 172.21.15.28:0/1032680313 >> 172.21.15.28:0/1032680313 conn(0x7f9d7c0cfc80 msgr2=0x7f9d7c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 -- 172.21.15.28:0/1032680313 shutdown_connections 2024-09-10T20:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.680+0000 7f9d8137d640 1 -- 172.21.15.28:0/1032680313 wait complete. 2024-09-10T20:42:13.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.681+0000 7f9d8137d640 1 Processor -- start 2024-09-10T20:42:13.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.681+0000 7f9d8137d640 1 -- start start 2024-09-10T20:42:13.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d8137d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d7c06e390 0x7f9d7c1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:13.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d8137d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c073d40 0x7f9d7c0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:13.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d8137d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d7c0b5030 0x7f9d7c15ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:13.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d8137d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c0ca9e0 con 0x7f9d7c073d40 2024-09-10T20:42:13.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d8137d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c0cab60 con 0x7f9d7c06e390 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d8137d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9d7c0cace0 con 0x7f9d7c0b5030 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d7a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c073d40 0x7f9d7c0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d7a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c073d40 0x7f9d7c0b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45946/0 (socket says 172.21.15.28:45946) 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d7a7fc640 1 -- 172.21.15.28:0/1827092298 learned_addr learned my addr 172.21.15.28:0/1827092298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d7b7fe640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d7c0b5030 0x7f9d7c15ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.682+0000 7f9d7affd640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d7c06e390 0x7f9d7c1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d7a7fc640 1 -- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d7c0b5030 msgr2=0x7f9d7c15ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d7a7fc640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d7c0b5030 0x7f9d7c15ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d7a7fc640 1 -- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d7c06e390 msgr2=0x7f9d7c1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d7a7fc640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d7c06e390 0x7f9d7c1531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d7a7fc640 1 -- 172.21.15.28:0/1827092298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c153700 con 0x7f9d7c073d40 2024-09-10T20:42:13.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d7a7fc640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c073d40 0x7f9d7c0b0fa0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9d64000c00 tx=0x7f9d64032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:13.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d737fe640 1 -- 172.21.15.28:0/1827092298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64040070 con 0x7f9d7c073d40 2024-09-10T20:42:13.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d737fe640 1 -- 172.21.15.28:0/1827092298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d64003af0 con 0x7f9d7c073d40 2024-09-10T20:42:13.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7c1f81d0 con 0x7f9d7c073d40 2024-09-10T20:42:13.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d737fe640 1 -- 172.21.15.28:0/1827092298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64045460 con 0x7f9d7c073d40 2024-09-10T20:42:13.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.683+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7c1f8790 con 0x7f9d7c073d40 2024-09-10T20:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.684+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d7c074480 con 0x7f9d7c073d40 2024-09-10T20:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.685+0000 7f9d737fe640 1 -- 172.21.15.28:0/1827092298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f9d6404d030 con 0x7f9d7c073d40 2024-09-10T20:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.685+0000 7f9d737fe640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9d5c078090 0x7f9d5c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.685+0000 7f9d737fe640 1 -- 172.21.15.28:0/1827092298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5916+0+0 (secure 0 0 0) 0x7f9d640bb710 con 0x7f9d7c073d40 2024-09-10T20:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.685+0000 7f9d7affd640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9d5c078090 0x7f9d5c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.686+0000 7f9d7affd640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9d5c078090 0x7f9d5c07a550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400a2e0 tx=0x7f9d7400a270 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:13.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.690+0000 7f9d737fe640 1 -- 172.21.15.28:0/1827092298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9d7c153700 con 0x7f9d7c073d40 2024-09-10T20:42:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:13.874+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9d7c0b3e30 con 0x7f9d7c073d40 2024-09-10T20:42:14.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.550+0000 7f9d737fe640 1 -- 172.21.15.28:0/1827092298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v72) ==== 169+0+0 (secure 0 0 0) 0x7f9d64083f20 con 0x7f9d7c073d40 2024-09-10T20:42:14.555 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:42:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.554+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9d5c078090 msgr2=0x7f9d5c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.554+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9d5c078090 0x7f9d5c07a550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400a2e0 tx=0x7f9d7400a270 comp rx=0 tx=0).stop 2024-09-10T20:42:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.554+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c073d40 msgr2=0x7f9d7c0b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.554+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c073d40 0x7f9d7c0b0fa0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9d64000c00 tx=0x7f9d64032d50 comp rx=0 tx=0).stop 2024-09-10T20:42:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 shutdown_connections 2024-09-10T20:42:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9d5c078090 0x7f9d5c07a550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d7c0b5030 0x7f9d7c15ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c073d40 0x7f9d7c0b0fa0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 --2- 172.21.15.28:0/1827092298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d7c06e390 0x7f9d7c1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 >> 172.21.15.28:0/1827092298 conn(0x7f9d7c0cfc80 msgr2=0x7f9d7c15a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 shutdown_connections 2024-09-10T20:42:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.555+0000 7f9d8137d640 1 -- 172.21.15.28:0/1827092298 wait complete. 2024-09-10T20:42:14.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:42:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 -- 172.21.15.28:0/1668302070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c540cfec0 msgr2=0x7f3c540d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1668302070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c540cfec0 0x7f3c540d0330 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3c54073cb0 tx=0x7f3c3c02a560 comp rx=0 tx=0).stop 2024-09-10T20:42:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 -- 172.21.15.28:0/1668302070 shutdown_connections 2024-09-10T20:42:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1668302070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c540ce5f0 0x7f3c540cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1668302070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c540cfec0 0x7f3c540d0330 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1668302070 >> v2:172.21.15.159:3300/0 conn(0x7f3c5415bf30 0x7f3c540d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 -- 172.21.15.28:0/1668302070 >> 172.21.15.28:0/1668302070 conn(0x7f3c540d1e40 msgr2=0x7f3c540d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 -- 172.21.15.28:0/1668302070 shutdown_connections 2024-09-10T20:42:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.685+0000 7f3c591d3640 1 -- 172.21.15.28:0/1668302070 wait complete. 2024-09-10T20:42:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 Processor -- start 2024-09-10T20:42:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 -- start start 2024-09-10T20:42:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c540ce5f0 0x7f3c5409f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c540cfec0 0x7f3c5409c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5415bf30 0x7f3c541f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c540d0cc0 con 0x7f3c5415bf30 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3c540d0b40 con 0x7f3c540cfec0 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c591d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3c540cefa0 con 0x7f3c540ce5f0 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.687+0000 7f3c5359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5415bf30 0x7f3c541f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5415bf30 0x7f3c541f3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45964/0 (socket says 172.21.15.28:45964) 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 -- 172.21.15.28:0/1689878605 learned_addr learned my addr 172.21.15.28:0/1689878605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5259c640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c540cfec0 0x7f3c5409c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c52d9d640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c540ce5f0 0x7f3c5409f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 -- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c540ce5f0 msgr2=0x7f3c5409f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c540ce5f0 0x7f3c5409f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 -- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c540cfec0 msgr2=0x7f3c5409c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c540cfec0 0x7f3c5409c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 -- 172.21.15.28:0/1689878605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5409d100 con 0x7f3c5415bf30 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c52d9d640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c540ce5f0 0x7f3c5409f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c5359e640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5415bf30 0x7f3c541f3af0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009fd0 tx=0x7f3c4c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c4b7fe640 1 -- 172.21.15.28:0/1689878605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c017070 con 0x7f3c5415bf30 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c4b7fe640 1 -- 172.21.15.28:0/1689878605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c4c003970 con 0x7f3c5415bf30 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5409fb00 con 0x7f3c5415bf30 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c4b7fe640 1 -- 172.21.15.28:0/1689878605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c003dc0 con 0x7f3c5415bf30 2024-09-10T20:42:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.688+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5406df70 con 0x7f3c5415bf30 2024-09-10T20:42:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.689+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c540cbf90 con 0x7f3c5415bf30 2024-09-10T20:42:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.690+0000 7f3c4b7fe640 1 -- 172.21.15.28:0/1689878605 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f3c4c003f60 con 0x7f3c5415bf30 2024-09-10T20:42:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.691+0000 7f3c4b7fe640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c34078090 0x7f3c3407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.691+0000 7f3c4b7fe640 1 -- 172.21.15.28:0/1689878605 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7f3c4c09a3e0 con 0x7f3c5415bf30 2024-09-10T20:42:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.691+0000 7f3c52d9d640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c34078090 0x7f3c3407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.691+0000 7f3c52d9d640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c34078090 0x7f3c3407a550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009a30 tx=0x7f3c44005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.695+0000 7f3c4b7fe640 1 -- 172.21.15.28:0/1689878605 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3c540cbf90 con 0x7f3c5415bf30 2024-09-10T20:42:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.880+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f3c540cf3c0 con 0x7f3c5415bf30 2024-09-10T20:42:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.883+0000 7f3c4b7fe640 1 -- 172.21.15.28:0/1689878605 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v35) ==== 71+0+0 (secure 0 0 0) 0x7f3c4c062e20 con 0x7f3c5415bf30 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c34078090 msgr2=0x7f3c3407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c34078090 0x7f3c3407a550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009a30 tx=0x7f3c44005f90 comp rx=0 tx=0).stop 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5415bf30 msgr2=0x7f3c541f3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5415bf30 0x7f3c541f3af0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009fd0 tx=0x7f3c4c00b260 comp rx=0 tx=0).stop 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 shutdown_connections 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c34078090 0x7f3c3407a550 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5415bf30 0x7f3c541f3af0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3c540cfec0 0x7f3c5409c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 --2- 172.21.15.28:0/1689878605 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3c540ce5f0 0x7f3c5409f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 >> 172.21.15.28:0/1689878605 conn(0x7f3c540d1e40 msgr2=0x7f3c540721c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 shutdown_connections 2024-09-10T20:42:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:14.886+0000 7f3c591d3640 1 -- 172.21.15.28:0/1689878605 wait complete. 2024-09-10T20:42:14.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-09-10T20:42:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.002+0000 7fb0c5aef640 1 -- 172.21.15.28:0/413205623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c015bb40 msgr2=0x7fb0c00af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.002+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/413205623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c015bb40 0x7fb0c00af660 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb0c015c8d0 tx=0x7fb0b002a510 comp rx=0 tx=0).stop 2024-09-10T20:42:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.002+0000 7fb0c5aef640 1 -- 172.21.15.28:0/413205623 shutdown_connections 2024-09-10T20:42:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.002+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/413205623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c015bb40 0x7fb0c00af660 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.002+0000 7fb0c5aef640 1 -- 172.21.15.28:0/413205623 >> 172.21.15.28:0/413205623 conn(0x7fb0c006dbf0 msgr2=0x7fb0c006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.002+0000 7fb0c5aef640 1 -- 172.21.15.28:0/413205623 shutdown_connections 2024-09-10T20:42:15.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.002+0000 7fb0c5aef640 1 -- 172.21.15.28:0/413205623 wait complete. 2024-09-10T20:42:15.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.003+0000 7fb0c5aef640 1 Processor -- start 2024-09-10T20:42:15.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.003+0000 7fb0c5aef640 1 -- start start 2024-09-10T20:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0c5aef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c006d1f0 0x7fb0c01442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0c5aef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0c015bb40 0x7fb0c0155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0c5aef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0c015bf20 0x7fb0c01f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0c5aef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c015d680 con 0x7fb0c006d1f0 2024-09-10T20:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0c5aef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c015d500 con 0x7fb0c015bf20 2024-09-10T20:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0c5aef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb0c015d800 con 0x7fb0c015bb40 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c006d1f0 0x7fb0c01442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c006d1f0 0x7fb0c01442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45970/0 (socket says 172.21.15.28:45970) 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 -- 172.21.15.28:0/2837341132 learned_addr learned my addr 172.21.15.28:0/2837341132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bffff640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0c015bf20 0x7fb0c01f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0beffd640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0c015bb40 0x7fb0c0155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 -- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0c015bb40 msgr2=0x7fb0c0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0c015bb40 0x7fb0c0155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 -- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0c015bf20 msgr2=0x7fb0c01f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0c015bf20 0x7fb0c01f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0bf7fe640 1 -- 172.21.15.28:0/2837341132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c0154a80 con 0x7fb0c006d1f0 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.004+0000 7fb0beffd640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0c015bb40 0x7fb0c0155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.005+0000 7fb0bf7fe640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c006d1f0 0x7fb0c01442f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb0b002ab00 tx=0x7fb0b0032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.005+0000 7fb0a7fff640 1 -- 172.21.15.28:0/2837341132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0040070 con 0x7fb0c006d1f0 2024-09-10T20:42:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.005+0000 7fb0a7fff640 1 -- 172.21.15.28:0/2837341132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0b0005310 con 0x7fb0c006d1f0 2024-09-10T20:42:15.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.005+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c0154d60 con 0x7fb0c006d1f0 2024-09-10T20:42:15.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.005+0000 7fb0a7fff640 1 -- 172.21.15.28:0/2837341132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00047a0 con 0x7fb0c006d1f0 2024-09-10T20:42:15.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.005+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c0155320 con 0x7fb0c006d1f0 2024-09-10T20:42:15.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.006+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c015dd20 con 0x7fb0c006d1f0 2024-09-10T20:42:15.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.007+0000 7fb0a7fff640 1 -- 172.21.15.28:0/2837341132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fb0b004c030 con 0x7fb0c006d1f0 2024-09-10T20:42:15.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.007+0000 7fb0a7fff640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0ac078090 0x7fb0ac07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.007+0000 7fb0a7fff640 1 -- 172.21.15.28:0/2837341132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7fb0b00bb070 con 0x7fb0c006d1f0 2024-09-10T20:42:15.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.007+0000 7fb0beffd640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0ac078090 0x7fb0ac07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.008+0000 7fb0beffd640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0ac078090 0x7fb0ac07a550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0c014b090 tx=0x7fb0b4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.011+0000 7fb0a7fff640 1 -- 172.21.15.28:0/2837341132 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb0b0083ab0 con 0x7fb0c006d1f0 2024-09-10T20:42:15.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.268+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7fb0c006cd30 con 0x7fb0c006d1f0 2024-09-10T20:42:15.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.552+0000 7fb0a7fff640 1 -- 172.21.15.28:0/2837341132 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v225) ==== 131+0+0 (secure 0 0 0) 0x7fb0b00833b0 con 0x7fb0c006d1f0 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0ac078090 msgr2=0x7fb0ac07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0ac078090 0x7fb0ac07a550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0c014b090 tx=0x7fb0b4009040 comp rx=0 tx=0).stop 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c006d1f0 msgr2=0x7fb0c01442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c006d1f0 0x7fb0c01442f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb0b002ab00 tx=0x7fb0b0032c10 comp rx=0 tx=0).stop 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 shutdown_connections 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0ac078090 0x7fb0ac07a550 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb0c015bf20 0x7fb0c01f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb0c015bb40 0x7fb0c0155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 --2- 172.21.15.28:0/2837341132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c006d1f0 0x7fb0c01442f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 >> 172.21.15.28:0/2837341132 conn(0x7fb0c006dbf0 msgr2=0x7fb0c015abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 shutdown_connections 2024-09-10T20:42:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.554+0000 7fb0c5aef640 1 -- 172.21.15.28:0/2837341132 wait complete. 2024-09-10T20:42:15.570 INFO:tasks.cephfs_test_runner:test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-10T20:42:15.571 INFO:tasks.cephfs_test_runner:Starting test: test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) Test the creating vols, subvols subvolgroups fails when their names uses 2024-09-10T20:42:15.571 INFO:tasks.cephfs_test_runner:test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:42:15.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 -- 172.21.15.28:0/2774572886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 msgr2=0x7f93d80d2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 --2- 172.21.15.28:0/2774572886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d80d2740 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f93d80698c0 tx=0x7f93cc02a530 comp rx=0 tx=0).stop 2024-09-10T20:42:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 -- 172.21.15.28:0/2774572886 shutdown_connections 2024-09-10T20:42:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 --2- 172.21.15.28:0/2774572886 >> v2:172.21.15.159:3300/0 conn(0x7f93d80b77f0 0x7f93d80b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 --2- 172.21.15.28:0/2774572886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d80d2740 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 -- 172.21.15.28:0/2774572886 >> 172.21.15.28:0/2774572886 conn(0x7f93d80d4040 msgr2=0x7f93d80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 -- 172.21.15.28:0/2774572886 shutdown_connections 2024-09-10T20:42:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.672+0000 7f93def2b640 1 -- 172.21.15.28:0/2774572886 wait complete. 2024-09-10T20:42:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.673+0000 7f93def2b640 1 Processor -- start 2024-09-10T20:42:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.673+0000 7f93def2b640 1 -- start start 2024-09-10T20:42:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93def2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93d80b77f0 0x7f93d81f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93def2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d81607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93def2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d815e0f0 0x7f93d8160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93def2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80b8180 con 0x7f93d80d4c40 2024-09-10T20:42:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93def2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80d2e90 con 0x7f93d80b77f0 2024-09-10T20:42:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93def2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f93d80b8300 con 0x7f93d815e0f0 2024-09-10T20:42:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d81607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d81607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45976/0 (socket says 172.21.15.28:45976) 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93d7fff640 1 -- 172.21.15.28:0/2604505170 learned_addr learned my addr 172.21.15.28:0/2604505170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93dccc7640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93d80b77f0 0x7f93d81f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93dd4c8640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d815e0f0 0x7f93d8160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93d7fff640 1 -- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d815e0f0 msgr2=0x7f93d8160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.674+0000 7f93d7fff640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d815e0f0 0x7f93d8160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93d7fff640 1 -- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93d80b77f0 msgr2=0x7f93d81f4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93d7fff640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93d80b77f0 0x7f93d81f4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93d7fff640 1 -- 172.21.15.28:0/2604505170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d80b94d0 con 0x7f93d80d4c40 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93dccc7640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93d80b77f0 0x7f93d81f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93dd4c8640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d815e0f0 0x7f93d8160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93d7fff640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d81607a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f93cc02ab20 tx=0x7f93cc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93d57fa640 1 -- 172.21.15.28:0/2604505170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc040070 con 0x7f93d80d4c40 2024-09-10T20:42:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93d57fa640 1 -- 172.21.15.28:0/2604505170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93cc005310 con 0x7f93d80d4c40 2024-09-10T20:42:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d80b97b0 con 0x7f93d80d4c40 2024-09-10T20:42:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93d57fa640 1 -- 172.21.15.28:0/2604505170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0047a0 con 0x7f93d80d4c40 2024-09-10T20:42:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.675+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d8153710 con 0x7f93d80d4c40 2024-09-10T20:42:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.676+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d808c150 con 0x7f93d80d4c40 2024-09-10T20:42:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.677+0000 7f93d57fa640 1 -- 172.21.15.28:0/2604505170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f93cc004940 con 0x7f93d80d4c40 2024-09-10T20:42:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.677+0000 7f93d57fa640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c4078090 0x7f93c407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.677+0000 7f93d57fa640 1 -- 172.21.15.28:0/2604505170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7f93cc0bb300 con 0x7f93d80d4c40 2024-09-10T20:42:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.677+0000 7f93dccc7640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c4078090 0x7f93c407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.677+0000 7f93dccc7640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c4078090 0x7f93c407a550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000f60 tx=0x7f93c8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.681+0000 7f93d57fa640 1 -- 172.21.15.28:0/2604505170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f93d808c150 con 0x7f93d80d4c40 2024-09-10T20:42:15.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.867+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93d80b87f0 con 0x7f93d80d4c40 2024-09-10T20:42:15.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.868+0000 7f93d57fa640 1 -- 172.21.15.28:0/2604505170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+13857 (secure 0 0 0) 0x7f93cc083d40 con 0x7f93d80d4c40 2024-09-10T20:42:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:15.873 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":72,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:14.548037+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/3447265222":"2024-09-11T20:42:11.365378+0000","172.21.15.28:6832/3447265222":"2024-09-11T20:42:11.365378+0000","172.21.15.159:0/3593980694":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/1402110084":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/4280913810":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2568415012":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3218025516":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2920078177":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/3526126208":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/1101500009":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2389577643":"2024-09-11T20:42:08.278542+0000","172.21.15.159:6800/2898596254":"2024-09-11T20:42:08.278542+0000","172.21.15.159:6801/2898596254":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/2416835213":"2024-09-11T20:42:08.278542+0000","172.21.15.159:0/105414844":"2024-09-11T20:42:08.278542+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c4078090 msgr2=0x7f93c407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c4078090 0x7f93c407a550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000f60 tx=0x7f93c8009040 comp rx=0 tx=0).stop 2024-09-10T20:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 msgr2=0x7f93d81607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d81607a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f93cc02ab20 tx=0x7f93cc032c10 comp rx=0 tx=0).stop 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 shutdown_connections 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c4078090 0x7f93c407a550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93d815e0f0 0x7f93d8160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93d80d4c40 0x7f93d81607a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 --2- 172.21.15.28:0/2604505170 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93d80b77f0 0x7f93d81f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 >> 172.21.15.28:0/2604505170 conn(0x7f93d80d4040 msgr2=0x7f93d80d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 shutdown_connections 2024-09-10T20:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.870+0000 7f93def2b640 1 -- 172.21.15.28:0/2604505170 wait complete. 2024-09-10T20:42:15.886 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:42:15.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:42:15.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.990+0000 7f03e76cf640 1 -- 172.21.15.28:0/429066283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e015bd80 msgr2=0x7f03e0073790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.990+0000 7f03e76cf640 1 --2- 172.21.15.28:0/429066283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e015bd80 0x7f03e0073790 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009830 tx=0x7f03d402a470 comp rx=0 tx=0).stop 2024-09-10T20:42:15.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.990+0000 7f03e76cf640 1 -- 172.21.15.28:0/429066283 shutdown_connections 2024-09-10T20:42:15.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.990+0000 7f03e76cf640 1 --2- 172.21.15.28:0/429066283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e015bd80 0x7f03e0073790 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.990+0000 7f03e76cf640 1 --2- 172.21.15.28:0/429066283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e0151db0 0x7f03e00706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:15.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.990+0000 7f03e76cf640 1 -- 172.21.15.28:0/429066283 >> 172.21.15.28:0/429066283 conn(0x7f03e00d0d50 msgr2=0x7f03e00d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:15.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.991+0000 7f03e76cf640 1 -- 172.21.15.28:0/429066283 shutdown_connections 2024-09-10T20:42:15.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.991+0000 7f03e76cf640 1 -- 172.21.15.28:0/429066283 wait complete. 2024-09-10T20:42:15.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 Processor -- start 2024-09-10T20:42:15.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 -- start start 2024-09-10T20:42:15.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e0073cd0 0x7f03e014cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e0151db0 0x7f03e00c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e015bd80 0x7f03e00b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03e015f390 con 0x7f03e0151db0 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f03e015f210 con 0x7f03e0073cd0 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.992+0000 7f03e76cf640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f03e015f510 con 0x7f03e015bd80 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e0151db0 0x7f03e00c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e0151db0 0x7f03e00c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45986/0 (socket says 172.21.15.28:45986) 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 -- 172.21.15.28:0/3693678431 learned_addr learned my addr 172.21.15.28:0/3693678431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e546b640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e0073cd0 0x7f03e014cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e5c6c640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e015bd80 0x7f03e00b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 -- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e015bd80 msgr2=0x7f03e00b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e015bd80 0x7f03e00b5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 -- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e0073cd0 msgr2=0x7f03e014cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e0073cd0 0x7f03e014cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 -- 172.21.15.28:0/3693678431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e00cab10 con 0x7f03e0151db0 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e546b640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e0073cd0 0x7f03e014cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e4c6a640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e0151db0 0x7f03e00c7e20 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009950 tx=0x7f03d4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03ddffb640 1 -- 172.21.15.28:0/3693678431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4031d60 con 0x7f03e0151db0 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03ddffb640 1 -- 172.21.15.28:0/3693678431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03d4033e10 con 0x7f03e0151db0 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03ddffb640 1 -- 172.21.15.28:0/3693678431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4005260 con 0x7f03e0151db0 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.993+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e00c9a70 con 0x7f03e0151db0 2024-09-10T20:42:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.994+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e00ca030 con 0x7f03e0151db0 2024-09-10T20:42:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.995+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e00706e0 con 0x7f03e0151db0 2024-09-10T20:42:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.995+0000 7f03ddffb640 1 -- 172.21.15.28:0/3693678431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f03d4005400 con 0x7f03e0151db0 2024-09-10T20:42:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.996+0000 7f03ddffb640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f03cc078090 0x7f03cc07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.996+0000 7f03ddffb640 1 -- 172.21.15.28:0/3693678431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7f03d40bb000 con 0x7f03e0151db0 2024-09-10T20:42:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.996+0000 7f03e546b640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f03cc078090 0x7f03cc07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:15.996+0000 7f03e546b640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f03cc078090 0x7f03cc07a550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03d0005fd0 tx=0x7f03d0009500 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.000+0000 7f03ddffb640 1 -- 172.21.15.28:0/3693678431 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f03d403f030 con 0x7f03e0151db0 2024-09-10T20:42:16.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.184+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f03e00749a0 con 0x7f03e0151db0 2024-09-10T20:42:16.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.185+0000 7f03ddffb640 1 -- 172.21.15.28:0/3693678431 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f03d40839b0 con 0x7f03e0151db0 2024-09-10T20:42:16.190 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:42:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f03cc078090 msgr2=0x7f03cc07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f03cc078090 0x7f03cc07a550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03d0005fd0 tx=0x7f03d0009500 comp rx=0 tx=0).stop 2024-09-10T20:42:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e0151db0 msgr2=0x7f03e00c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e0151db0 0x7f03e00c7e20 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009950 tx=0x7f03d4033c20 comp rx=0 tx=0).stop 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 shutdown_connections 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f03cc078090 0x7f03cc07a550 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f03e015bd80 0x7f03e00b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e0151db0 0x7f03e00c7e20 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 --2- 172.21.15.28:0/3693678431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f03e0073cd0 0x7f03e014cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 >> 172.21.15.28:0/3693678431 conn(0x7f03e00d0d50 msgr2=0x7f03e00b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 shutdown_connections 2024-09-10T20:42:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.187+0000 7f03e76cf640 1 -- 172.21.15.28:0/3693678431 wait complete. 2024-09-10T20:42:16.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 -- 172.21.15.28:0/2506748116 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d60162090 msgr2=0x7f1d600b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/2506748116 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d60162090 0x7f1d600b6e00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005fd0 tx=0x7f1d5802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 -- 172.21.15.28:0/2506748116 shutdown_connections 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/2506748116 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d600d2b90 0x7f1d60161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/2506748116 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d60162090 0x7f1d600b6e00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 -- 172.21.15.28:0/2506748116 >> 172.21.15.28:0/2506748116 conn(0x7f1d600d3fb0 msgr2=0x7f1d600d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 -- 172.21.15.28:0/2506748116 shutdown_connections 2024-09-10T20:42:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.304+0000 7f1d67e7f640 1 -- 172.21.15.28:0/2506748116 wait complete. 2024-09-10T20:42:16.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.305+0000 7f1d67e7f640 1 Processor -- start 2024-09-10T20:42:16.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.305+0000 7f1d67e7f640 1 -- start start 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d67e7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d600d2b90 0x7f1d6014d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d67e7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d6015de70 0x7f1d600a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d67e7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d60162090 0x7f1d600a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d67e7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6015e3d0 con 0x7f1d60162090 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d67e7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6015e250 con 0x7f1d600d2b90 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d67e7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1d600b7340 con 0x7f1d6015de70 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d60162090 0x7f1d600a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d60162090 0x7f1d600a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46002/0 (socket says 172.21.15.28:46002) 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 -- 172.21.15.28:0/1980994185 learned_addr learned my addr 172.21.15.28:0/1980994185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d65c1b640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d600d2b90 0x7f1d6014d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6541a640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d6015de70 0x7f1d600a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 -- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d6015de70 msgr2=0x7f1d600a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d6015de70 0x7f1d600a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 -- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d600d2b90 msgr2=0x7f1d6014d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d600d2b90 0x7f1d6014d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 -- 172.21.15.28:0/1980994185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d60151080 con 0x7f1d60162090 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.306+0000 7f1d6641c640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d60162090 0x7f1d600a1210 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009cd0 tx=0x7f1d58033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.307+0000 7f1d567fc640 1 -- 172.21.15.28:0/1980994185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58031990 con 0x7f1d60162090 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.307+0000 7f1d567fc640 1 -- 172.21.15.28:0/1980994185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d58031c20 con 0x7f1d60162090 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.307+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d60160520 con 0x7f1d60162090 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.307+0000 7f1d567fc640 1 -- 172.21.15.28:0/1980994185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58033330 con 0x7f1d60162090 2024-09-10T20:42:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.307+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d60160ae0 con 0x7f1d60162090 2024-09-10T20:42:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.308+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d600b7860 con 0x7f1d60162090 2024-09-10T20:42:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.308+0000 7f1d567fc640 1 -- 172.21.15.28:0/1980994185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f1d5804b030 con 0x7f1d60162090 2024-09-10T20:42:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.309+0000 7f1d567fc640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d44078090 0x7f1d4407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.309+0000 7f1d567fc640 1 -- 172.21.15.28:0/1980994185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7f1d580bad50 con 0x7f1d60162090 2024-09-10T20:42:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.309+0000 7f1d65c1b640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d44078090 0x7f1d4407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.309+0000 7f1d65c1b640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d44078090 0x7f1d4407a550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0017c0 tx=0x7f1d5c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.313+0000 7f1d567fc640 1 -- 172.21.15.28:0/1980994185 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1d580be030 con 0x7f1d60162090 2024-09-10T20:42:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:16.575+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7f1d60151250 con 0x7f1d60162090 2024-09-10T20:42:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.558+0000 7f1d567fc640 1 -- 172.21.15.28:0/1980994185 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v227) ==== 149+0+0 (secure 0 0 0) 0x7f1d5803f030 con 0x7f1d60162090 2024-09-10T20:42:17.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d44078090 msgr2=0x7f1d4407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d44078090 0x7f1d4407a550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0017c0 tx=0x7f1d5c009040 comp rx=0 tx=0).stop 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d60162090 msgr2=0x7f1d600a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d60162090 0x7f1d600a1210 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009cd0 tx=0x7f1d58033f70 comp rx=0 tx=0).stop 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 shutdown_connections 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d44078090 0x7f1d4407a550 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d60162090 0x7f1d600a1210 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d6015de70 0x7f1d600a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.561+0000 7f1d67e7f640 1 --2- 172.21.15.28:0/1980994185 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d600d2b90 0x7f1d6014d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.562+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 >> 172.21.15.28:0/1980994185 conn(0x7f1d600d3fb0 msgr2=0x7f1d601618d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.562+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 shutdown_connections 2024-09-10T20:42:17.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.562+0000 7f1d67e7f640 1 -- 172.21.15.28:0/1980994185 wait complete. 2024-09-10T20:42:17.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:42:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 -- 172.21.15.28:0/3662455441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 msgr2=0x7fee980d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 --2- 172.21.15.28:0/3662455441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee980d0330 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fee98073cb0 tx=0x7fee8002a560 comp rx=0 tx=0).stop 2024-09-10T20:42:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 -- 172.21.15.28:0/3662455441 shutdown_connections 2024-09-10T20:42:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 --2- 172.21.15.28:0/3662455441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee980d0330 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 --2- 172.21.15.28:0/3662455441 >> v2:172.21.15.159:3300/0 conn(0x7fee9815bf30 0x7fee980d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 -- 172.21.15.28:0/3662455441 >> 172.21.15.28:0/3662455441 conn(0x7fee980d1e40 msgr2=0x7fee980d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 -- 172.21.15.28:0/3662455441 shutdown_connections 2024-09-10T20:42:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.687+0000 7fee9d085640 1 -- 172.21.15.28:0/3662455441 wait complete. 2024-09-10T20:42:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 Processor -- start 2024-09-10T20:42:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 -- start start 2024-09-10T20:42:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee980ce5f0 0x7fee9806f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee98147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9815bf30 0x7fee9815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee980d0cc0 con 0x7fee980cfec0 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fee980d0b40 con 0x7fee980ce5f0 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9d085640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fee980b6620 con 0x7fee9815bf30 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee98147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee98147cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46030/0 (socket says 172.21.15.28:46030) 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.689+0000 7fee9659c640 1 -- 172.21.15.28:0/2550518383 learned_addr learned my addr 172.21.15.28:0/2550518383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee96d9d640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee980ce5f0 0x7fee9806f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9759e640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9815bf30 0x7fee9815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9659c640 1 -- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9815bf30 msgr2=0x7fee9815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9659c640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9815bf30 0x7fee9815a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9659c640 1 -- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee980ce5f0 msgr2=0x7fee9806f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9659c640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee980ce5f0 0x7fee9806f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9659c640 1 -- 172.21.15.28:0/2550518383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9814c050 con 0x7fee980cfec0 2024-09-10T20:42:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9659c640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee98147cf0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fee800099c0 tx=0x7fee800039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee8f7fe640 1 -- 172.21.15.28:0/2550518383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8003f070 con 0x7fee980cfec0 2024-09-10T20:42:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee8f7fe640 1 -- 172.21.15.28:0/2550518383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee80004890 con 0x7fee980cfec0 2024-09-10T20:42:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee98148380 con 0x7fee980cfec0 2024-09-10T20:42:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee8f7fe640 1 -- 172.21.15.28:0/2550518383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80035030 con 0x7fee980cfec0 2024-09-10T20:42:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.690+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee98148940 con 0x7fee980cfec0 2024-09-10T20:42:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.691+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee980d0330 con 0x7fee980cfec0 2024-09-10T20:42:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.692+0000 7fee8f7fe640 1 -- 172.21.15.28:0/2550518383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fee80004ce0 con 0x7fee980cfec0 2024-09-10T20:42:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.692+0000 7fee8f7fe640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fee78078090 0x7fee7807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.693+0000 7fee8f7fe640 1 -- 172.21.15.28:0/2550518383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7fee800bb0d0 con 0x7fee980cfec0 2024-09-10T20:42:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.693+0000 7fee96d9d640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fee78078090 0x7fee7807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.693+0000 7fee96d9d640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fee78078090 0x7fee7807a550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee8800a400 tx=0x7fee8800d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.697+0000 7fee8f7fe640 1 -- 172.21.15.28:0/2550518383 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fee9814c050 con 0x7fee980cfec0 2024-09-10T20:42:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.883+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fee98076360 con 0x7fee980cfec0 2024-09-10T20:42:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.883+0000 7fee8f7fe640 1 -- 172.21.15.28:0/2550518383 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v35) ==== 59+0+1998 (secure 0 0 0) 0x7fee80083440 con 0x7fee980cfec0 2024-09-10T20:42:17.888 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:42:17.888 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:42:17.889 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:42:17.889 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:42:17.889 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:42:17.889 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:42:17.889 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:42:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:42:17.891 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:42:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fee78078090 msgr2=0x7fee7807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fee78078090 0x7fee7807a550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee8800a400 tx=0x7fee8800d040 comp rx=0 tx=0).stop 2024-09-10T20:42:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 msgr2=0x7fee98147cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee98147cf0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fee800099c0 tx=0x7fee800039e0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 shutdown_connections 2024-09-10T20:42:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fee78078090 0x7fee7807a550 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fee9815bf30 0x7fee9815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee980cfec0 0x7fee98147cf0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 --2- 172.21.15.28:0/2550518383 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fee980ce5f0 0x7fee9806f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 >> 172.21.15.28:0/2550518383 conn(0x7fee980d1e40 msgr2=0x7fee98070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 shutdown_connections 2024-09-10T20:42:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:17.885+0000 7fee9d085640 1 -- 172.21.15.28:0/2550518383 wait complete. 2024-09-10T20:42:17.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:42:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.005+0000 7fae3eaa2640 1 -- 172.21.15.28:0/2442782365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 msgr2=0x7fae20003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.005+0000 7fae3eaa2640 1 --2- 172.21.15.28:0/2442782365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 0x7fae20003dc0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fae280017c0 tx=0x7fae20005830 comp rx=0 tx=0).stop 2024-09-10T20:42:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.005+0000 7fae3d03f640 1 -- 172.21.15.28:0/2442782365 reap_dead start 2024-09-10T20:42:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.005+0000 7fae3eaa2640 1 -- 172.21.15.28:0/2442782365 shutdown_connections 2024-09-10T20:42:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.005+0000 7fae3eaa2640 1 -- 172.21.15.28:0/2442782365 >> 172.21.15.28:0/2442782365 conn(0x7fae3806d110 msgr2=0x7fae380b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.006+0000 7fae3eaa2640 1 -- 172.21.15.28:0/2442782365 shutdown_connections 2024-09-10T20:42:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.006+0000 7fae3eaa2640 1 -- 172.21.15.28:0/2442782365 wait complete. 2024-09-10T20:42:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.007+0000 7fae3eaa2640 1 Processor -- start 2024-09-10T20:42:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.007+0000 7fae3eaa2640 1 -- start start 2024-09-10T20:42:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3eaa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 0x7fae3814d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3eaa2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae38152470 0x7fae38149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3eaa2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae381528f0 0x7fae381496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3eaa2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae380b2390 con 0x7fae200039e0 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3eaa2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fae380b2210 con 0x7fae38152470 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3eaa2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fae380b2510 con 0x7fae381528f0 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 0x7fae3814d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 0x7fae3814d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46046/0 (socket says 172.21.15.28:46046) 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 -- 172.21.15.28:0/863539773 learned_addr learned my addr 172.21.15.28:0/863539773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae37fff640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae38152470 0x7fae38149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 -- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae381528f0 msgr2=0x7fae381496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae381528f0 0x7fae381496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 -- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae38152470 msgr2=0x7fae38149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae38152470 0x7fae38149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 -- 172.21.15.28:0/863539773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae38150d40 con 0x7fae200039e0 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae37fff640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae38152470 0x7fae38149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3c83e640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 0x7fae3814d650 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fae2802aa80 tx=0x7fae28006060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae357fa640 1 -- 172.21.15.28:0/863539773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae280037f0 con 0x7fae200039e0 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae357fa640 1 -- 172.21.15.28:0/863539773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae28042070 con 0x7fae200039e0 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae3814b1d0 con 0x7fae200039e0 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.008+0000 7fae357fa640 1 -- 172.21.15.28:0/863539773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28003c10 con 0x7fae200039e0 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.009+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae381509c0 con 0x7fae200039e0 2024-09-10T20:42:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.010+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae3814b6f0 con 0x7fae200039e0 2024-09-10T20:42:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.010+0000 7fae357fa640 1 -- 172.21.15.28:0/863539773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fae28047460 con 0x7fae200039e0 2024-09-10T20:42:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.011+0000 7fae357fa640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae24078090 0x7fae2407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.011+0000 7fae357fa640 1 -- 172.21.15.28:0/863539773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7fae280ba810 con 0x7fae200039e0 2024-09-10T20:42:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.011+0000 7fae37fff640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae24078090 0x7fae2407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.011+0000 7fae37fff640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae24078090 0x7fae2407a550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0015c0 tx=0x7fae2c001fd0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:18.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.015+0000 7fae357fa640 1 -- 172.21.15.28:0/863539773 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fae28082ae0 con 0x7fae200039e0 2024-09-10T20:42:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.203+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fae3815a2c0 con 0x7fae200039e0 2024-09-10T20:42:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.206+0000 7fae357fa640 1 -- 172.21.15.28:0/863539773 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7fae3815a2c0 con 0x7fae200039e0 2024-09-10T20:42:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae24078090 msgr2=0x7fae2407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae24078090 0x7fae2407a550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0015c0 tx=0x7fae2c001fd0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 msgr2=0x7fae3814d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 0x7fae3814d650 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fae2802aa80 tx=0x7fae28006060 comp rx=0 tx=0).stop 2024-09-10T20:42:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 shutdown_connections 2024-09-10T20:42:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae24078090 0x7fae2407a550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae381528f0 0x7fae381496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae38152470 0x7fae38149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 --2- 172.21.15.28:0/863539773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae200039e0 0x7fae3814d650 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 >> 172.21.15.28:0/863539773 conn(0x7fae3806d110 msgr2=0x7fae38159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 shutdown_connections 2024-09-10T20:42:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.209+0000 7fae3eaa2640 1 -- 172.21.15.28:0/863539773 wait complete. 2024-09-10T20:42:18.225 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:42:18.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:18.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:18.318 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:42:18.318 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:18.318 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:18.335 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:42:18.335 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:18.335 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:18.376 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:42:18.376 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:18.376 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:18.392 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:42:18.393 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:18.393 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:18.435 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:42:18.435 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:42:18.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 -- 172.21.15.28:0/233914080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d0bc0 msgr2=0x7f84380b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 --2- 172.21.15.28:0/233914080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d0bc0 0x7f84380b4130 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f843808e6e0 tx=0x7f843002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 -- 172.21.15.28:0/233914080 shutdown_connections 2024-09-10T20:42:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 --2- 172.21.15.28:0/233914080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d0bc0 0x7f84380b4130 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 --2- 172.21.15.28:0/233914080 >> v2:172.21.15.159:3300/0 conn(0x7f84380d4950 0x7f84380d2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 -- 172.21.15.28:0/233914080 >> 172.21.15.28:0/233914080 conn(0x7f843809d2d0 msgr2=0x7f84380d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 -- 172.21.15.28:0/233914080 shutdown_connections 2024-09-10T20:42:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.535+0000 7f844070e640 1 -- 172.21.15.28:0/233914080 wait complete. 2024-09-10T20:42:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.536+0000 7f844070e640 1 Processor -- start 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f844070e640 1 -- start start 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f844070e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d07e0 0x7f8438155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f844070e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84380d0bc0 0x7f84380b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f844070e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84380d4950 0x7f84380cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f844070e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84380cef90 con 0x7f84380d07e0 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f844070e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f84380cee10 con 0x7f84380d4950 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f844070e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f84380cf110 con 0x7f84380d0bc0 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f843e4aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d07e0 0x7f8438155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f843e4aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d07e0 0x7f8438155610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46066/0 (socket says 172.21.15.28:46066) 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.537+0000 7f843e4aa640 1 -- 172.21.15.28:0/2209222643 learned_addr learned my addr 172.21.15.28:0/2209222643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843ecab640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84380d4950 0x7f84380cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843dca9640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84380d0bc0 0x7f84380b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843e4aa640 1 -- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84380d0bc0 msgr2=0x7f84380b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843e4aa640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84380d0bc0 0x7f84380b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843e4aa640 1 -- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84380d4950 msgr2=0x7f84380cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843e4aa640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84380d4950 0x7f84380cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843e4aa640 1 -- 172.21.15.28:0/2209222643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843815fa40 con 0x7f84380d07e0 2024-09-10T20:42:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843ecab640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84380d4950 0x7f84380cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f843e4aa640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d07e0 0x7f8438155610 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f843400c480 tx=0x7f843400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f842effd640 1 -- 172.21.15.28:0/2209222643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8434016020 con 0x7f84380d07e0 2024-09-10T20:42:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f842effd640 1 -- 172.21.15.28:0/2209222643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8434012920 con 0x7f84380d07e0 2024-09-10T20:42:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84380cb1c0 con 0x7f84380d07e0 2024-09-10T20:42:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f842effd640 1 -- 172.21.15.28:0/2209222643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8434012d70 con 0x7f84380d07e0 2024-09-10T20:42:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.538+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84380b9340 con 0x7f84380d07e0 2024-09-10T20:42:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.539+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84380d2060 con 0x7f84380d07e0 2024-09-10T20:42:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.540+0000 7f842effd640 1 -- 172.21.15.28:0/2209222643 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f843401b400 con 0x7f84380d07e0 2024-09-10T20:42:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.540+0000 7f842effd640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f841c078090 0x7f841c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.540+0000 7f843dca9640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f841c078090 0x7f841c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.540+0000 7f842effd640 1 -- 172.21.15.28:0/2209222643 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7f843409a260 con 0x7f84380d07e0 2024-09-10T20:42:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.541+0000 7f843dca9640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f841c078090 0x7f841c07a550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8428005fd0 tx=0x7f8428005850 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.544+0000 7f842effd640 1 -- 172.21.15.28:0/2209222643 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8434020020 con 0x7f84380d07e0 2024-09-10T20:42:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.767+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84380d1a30 con 0x7f84380d07e0 2024-09-10T20:42:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.768+0000 7f842effd640 1 -- 172.21.15.28:0/2209222643 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+3911 (secure 0 0 0) 0x7f843815fa40 con 0x7f84380d07e0 2024-09-10T20:42:18.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:18.773 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"btime":"2024-09-10T20:42:12:366763+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1624510264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1624510264},{"type":"v1","addr":"172.21.15.28:6835","nonce":1624510264}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6836/102528641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":102528641},{"type":"v1","addr":"172.21.15.159:6836","nonce":102528641}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5704,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6837/4086133720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6835","nonce":4086133720},{"type":"v1","addr":"172.21.15.159:6837","nonce":4086133720}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6492,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/688933206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":688933206},{"type":"v1","addr":"172.21.15.28:6834","nonce":688933206}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[]} 2024-09-10T20:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-09-10T20:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f841c078090 msgr2=0x7f841c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f841c078090 0x7f841c07a550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8428005fd0 tx=0x7f8428005850 comp rx=0 tx=0).stop 2024-09-10T20:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d07e0 msgr2=0x7f8438155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d07e0 0x7f8438155610 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f843400c480 tx=0x7f843400c940 comp rx=0 tx=0).stop 2024-09-10T20:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 shutdown_connections 2024-09-10T20:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f841c078090 0x7f841c07a550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f84380d4950 0x7f84380cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f84380d0bc0 0x7f84380b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 --2- 172.21.15.28:0/2209222643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84380d07e0 0x7f8438155610 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 >> 172.21.15.28:0/2209222643 conn(0x7f843809d2d0 msgr2=0x7f8438158680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 shutdown_connections 2024-09-10T20:42:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.770+0000 7f844070e640 1 -- 172.21.15.28:0/2209222643 wait complete. 2024-09-10T20:42:18.786 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:42:18.786 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:42:18.786 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:42:18.787 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:18.787 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:42:18.787 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:42:18.787 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:42:18.787 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:18.787 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:42:18.787 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:42:18.788 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:42:18.788 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:18.788 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:42:18.788 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:42:18.788 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:42:18.788 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:18.796 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:42:18.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:42:18.797 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:42:18.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:42:18.800 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:42:18.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:42:18.801 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:42:18.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:42:18.839 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:42:18.839 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:18.840 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:42:18.840 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:18.843 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:42:18.844 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:18.844 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:42:18.844 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:18.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:42:18.917 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:42:18.918 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:42:18.928 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:42:18.929 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:42:18.929 INFO:tasks.ceph.mds.b.smithi159.stderr:2024-09-10T20:42:18.924+0000 7fdcbc9bd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-09-10T20:42:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 -- 172.21.15.28:0/2111682751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c00cf780 msgr2=0x7f55c00cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/2111682751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c00cf780 0x7f55c00cfbf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f55c008c020 tx=0x7f55b402a540 comp rx=0 tx=0).stop 2024-09-10T20:42:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 -- 172.21.15.28:0/2111682751 shutdown_connections 2024-09-10T20:42:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/2111682751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c00cf780 0x7f55c00cfbf0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/2111682751 >> v2:172.21.15.159:3300/0 conn(0x7f55c015de70 0x7f55c00d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 -- 172.21.15.28:0/2111682751 >> 172.21.15.28:0/2111682751 conn(0x7f55c00d3fb0 msgr2=0x7f55c00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 -- 172.21.15.28:0/2111682751 shutdown_connections 2024-09-10T20:42:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.950+0000 7f55c5f2c640 1 -- 172.21.15.28:0/2111682751 wait complete. 2024-09-10T20:42:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.951+0000 7f55c5f2c640 1 Processor -- start 2024-09-10T20:42:18.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55c5f2c640 1 -- start start 2024-09-10T20:42:18.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55c5f2c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55c00cf780 0x7f55c014f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55c5f2c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55c015de70 0x7f55c01f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55c5f2c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c0161b50 0x7f55c014caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55c5f2c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55c00b7690 con 0x7f55c0161b50 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55c5f2c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f55c00b7510 con 0x7f55c015de70 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55c5f2c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f55c00b7810 con 0x7f55c00cf780 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c0161b50 0x7f55c014caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c0161b50 0x7f55c014caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46114/0 (socket says 172.21.15.28:46114) 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55bffff640 1 -- 172.21.15.28:0/1180452574 learned_addr learned my addr 172.21.15.28:0/1180452574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55bf7fe640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55c00cf780 0x7f55c014f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.952+0000 7f55beffd640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55c015de70 0x7f55c01f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55bffff640 1 -- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55c00cf780 msgr2=0x7f55c014f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55bffff640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55c00cf780 0x7f55c014f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55bffff640 1 -- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55c015de70 msgr2=0x7f55c01f88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55bffff640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55c015de70 0x7f55c01f88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55bffff640 1 -- 172.21.15.28:0/1180452574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c014fbb0 con 0x7f55c0161b50 2024-09-10T20:42:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55bffff640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c0161b50 0x7f55c014caf0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f55b800cae0 tx=0x7f55b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55a7fff640 1 -- 172.21.15.28:0/1180452574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8016020 con 0x7f55c0161b50 2024-09-10T20:42:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c014fe90 con 0x7f55c0161b50 2024-09-10T20:42:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55a7fff640 1 -- 172.21.15.28:0/1180452574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55b800ad30 con 0x7f55c0161b50 2024-09-10T20:42:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55a7fff640 1 -- 172.21.15.28:0/1180452574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8012680 con 0x7f55c0161b50 2024-09-10T20:42:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.953+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c014d5c0 con 0x7f55c0161b50 2024-09-10T20:42:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.954+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55c00d38e0 con 0x7f55c0161b50 2024-09-10T20:42:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.955+0000 7f55a7fff640 1 -- 172.21.15.28:0/1180452574 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f55b801a3b0 con 0x7f55c0161b50 2024-09-10T20:42:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.955+0000 7f55a7fff640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55ac078090 0x7f55ac07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.955+0000 7f55a7fff640 1 -- 172.21.15.28:0/1180452574 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5502+0+0 (secure 0 0 0) 0x7f55b809a3d0 con 0x7f55c0161b50 2024-09-10T20:42:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.955+0000 7f55bf7fe640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55ac078090 0x7f55ac07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.956+0000 7f55bf7fe640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55ac078090 0x7f55ac07a550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f55c0161b20 tx=0x7f55b000b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:18.960+0000 7f55a7fff640 1 -- 172.21.15.28:0/1180452574 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f55c00d38e0 con 0x7f55c0161b50 2024-09-10T20:42:19.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:19.155+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f55c00cfbf0 con 0x7f55c0161b50 2024-09-10T20:42:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.219+0000 7f55a7fff640 1 -- 172.21.15.28:0/1180452574 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v77) ==== 97+0+0 (secure 0 0 0) 0x7f55b8020030 con 0x7f55c0161b50 2024-09-10T20:42:20.224 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:42:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55ac078090 msgr2=0x7f55ac07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55ac078090 0x7f55ac07a550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f55c0161b20 tx=0x7f55b000b390 comp rx=0 tx=0).stop 2024-09-10T20:42:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c0161b50 msgr2=0x7f55c014caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c0161b50 0x7f55c014caf0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f55b800cae0 tx=0x7f55b800a060 comp rx=0 tx=0).stop 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 shutdown_connections 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55ac078090 0x7f55ac07a550 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c0161b50 0x7f55c014caf0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55c015de70 0x7f55c01f88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 --2- 172.21.15.28:0/1180452574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55c00cf780 0x7f55c014f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 >> 172.21.15.28:0/1180452574 conn(0x7f55c00d3fb0 msgr2=0x7f55c00b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.222+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 shutdown_connections 2024-09-10T20:42:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.223+0000 7f55c5f2c640 1 -- 172.21.15.28:0/1180452574 wait complete. 2024-09-10T20:42:20.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.347+0000 7f930616b640 1 -- 172.21.15.28:0/292966989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93000d2b90 msgr2=0x7f9300161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.347+0000 7f930616b640 1 --2- 172.21.15.28:0/292966989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93000d2b90 0x7f9300161b50 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f92f4005fd0 tx=0x7f92f402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.347+0000 7f930616b640 1 -- 172.21.15.28:0/292966989 shutdown_connections 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.347+0000 7f930616b640 1 --2- 172.21.15.28:0/292966989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9300162090 0x7f93000b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.347+0000 7f930616b640 1 --2- 172.21.15.28:0/292966989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93000d2b90 0x7f9300161b50 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.347+0000 7f930616b640 1 -- 172.21.15.28:0/292966989 >> 172.21.15.28:0/292966989 conn(0x7f93000d3fb0 msgr2=0x7f93000d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.347+0000 7f930616b640 1 -- 172.21.15.28:0/292966989 shutdown_connections 2024-09-10T20:42:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.348+0000 7f930616b640 1 -- 172.21.15.28:0/292966989 wait complete. 2024-09-10T20:42:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.349+0000 7f930616b640 1 Processor -- start 2024-09-10T20:42:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.349+0000 7f930616b640 1 -- start start 2024-09-10T20:42:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f930616b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93000d2b90 0x7f9300160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f930616b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f930015de70 0x7f93000b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f930616b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9300162090 0x7f93000b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f930616b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f930015e3d0 con 0x7f9300162090 2024-09-10T20:42:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f930616b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f930015e250 con 0x7f93000d2b90 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f930616b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f93000b7340 con 0x7f930015de70 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9300162090 0x7f93000b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9300162090 0x7f93000b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41530/0 (socket says 172.21.15.28:41530) 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 -- 172.21.15.28:0/773627629 learned_addr learned my addr 172.21.15.28:0/773627629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92feffd640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f930015de70 0x7f93000b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92ff7fe640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93000d2b90 0x7f9300160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 -- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f930015de70 msgr2=0x7f93000b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f930015de70 0x7f93000b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 -- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93000d2b90 msgr2=0x7f9300160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93000d2b90 0x7f9300160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 -- 172.21.15.28:0/773627629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9300160d00 con 0x7f9300162090 2024-09-10T20:42:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92feffd640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f930015de70 0x7f93000b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92ff7fe640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93000d2b90 0x7f9300160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92fffff640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9300162090 0x7f93000b5db0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f92f800e960 tx=0x7f92f800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92e7fff640 1 -- 172.21.15.28:0/773627629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8004d20 con 0x7f9300162090 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f92e7fff640 1 -- 172.21.15.28:0/773627629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92f800cb20 con 0x7f9300162090 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.350+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9300160fe0 con 0x7f9300162090 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.351+0000 7f92e7fff640 1 -- 172.21.15.28:0/773627629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8005020 con 0x7f9300162090 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.351+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93001615a0 con 0x7f9300162090 2024-09-10T20:42:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.351+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93000cf4c0 con 0x7f9300162090 2024-09-10T20:42:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.352+0000 7f92e7fff640 1 -- 172.21.15.28:0/773627629 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f92f800f040 con 0x7f9300162090 2024-09-10T20:42:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.353+0000 7f92e7fff640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92ec078090 0x7f92ec07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.353+0000 7f92e7fff640 1 -- 172.21.15.28:0/773627629 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f92f809a1e0 con 0x7f9300162090 2024-09-10T20:42:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.353+0000 7f92ff7fe640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92ec078090 0x7f92ec07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.353+0000 7f92ff7fe640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92ec078090 0x7f92ec07a550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92f0007fd0 tx=0x7f92f0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.357+0000 7f92e7fff640 1 -- 172.21.15.28:0/773627629 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f92f8016030 con 0x7f9300162090 2024-09-10T20:42:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.611+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f93000b77a0 con 0x7f9300162090 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.612+0000 7f92e7fff640 1 -- 172.21.15.28:0/773627629 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v30) ==== 80+0+5304 (secure 0 0 0) 0x7f92f8062ea0 con 0x7f9300162090 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.617 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:20.618 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.619 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.620 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.622 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:42:20.624 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.626 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:42:20.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.628 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:42:20.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:20.630 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92ec078090 msgr2=0x7f92ec07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92ec078090 0x7f92ec07a550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92f0007fd0 tx=0x7f92f0009040 comp rx=0 tx=0).stop 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9300162090 msgr2=0x7f93000b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9300162090 0x7f93000b5db0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f92f800e960 tx=0x7f92f800ee20 comp rx=0 tx=0).stop 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 shutdown_connections 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92ec078090 0x7f92ec07a550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9300162090 0x7f93000b5db0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f930015de70 0x7f93000b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 --2- 172.21.15.28:0/773627629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93000d2b90 0x7f9300160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 >> 172.21.15.28:0/773627629 conn(0x7f93000d3fb0 msgr2=0x7f93000b7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 shutdown_connections 2024-09-10T20:42:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.614+0000 7f930616b640 1 -- 172.21.15.28:0/773627629 wait complete. 2024-09-10T20:42:20.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:42:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.734+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3372245352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba4004800 msgr2=0x7f7ba4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.734+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3372245352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba4004800 0x7f7ba4004c80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000baf0 tx=0x7f7ba4005830 comp rx=0 tx=0).stop 2024-09-10T20:42:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.734+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3372245352 shutdown_connections 2024-09-10T20:42:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.734+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3372245352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba4004800 0x7f7ba4004c80 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.734+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3372245352 >> 172.21.15.28:0/3372245352 conn(0x7f7bbc0d3fb0 msgr2=0x7f7bbc0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.734+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3372245352 shutdown_connections 2024-09-10T20:42:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.734+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3372245352 wait complete. 2024-09-10T20:42:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.735+0000 7f7bc11cd640 1 Processor -- start 2024-09-10T20:42:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.735+0000 7f7bc11cd640 1 -- start start 2024-09-10T20:42:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bc11cd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba4004800 0x7f7bbc0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bc11cd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bbc1540b0 0x7f7bbc1f7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bc11cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc1f8070 0x7f7bbc1f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bc11cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc161b50 con 0x7f7bbc1f8070 2024-09-10T20:42:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bc11cd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc15e460 con 0x7f7bbc1540b0 2024-09-10T20:42:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bc11cd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7bbc161cd0 con 0x7f7ba4004800 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bbb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc1f8070 0x7f7bbc1f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bbb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc1f8070 0x7f7bbc1f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41546/0 (socket says 172.21.15.28:41546) 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bbb59e640 1 -- 172.21.15.28:0/3546353148 learned_addr learned my addr 172.21.15.28:0/3546353148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bbad9d640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba4004800 0x7f7bbc0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bba59c640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bbc1540b0 0x7f7bbc1f7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bbb59e640 1 -- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba4004800 msgr2=0x7f7bbc0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.736+0000 7f7bbb59e640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba4004800 0x7f7bbc0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7bbb59e640 1 -- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bbc1540b0 msgr2=0x7f7bbc1f7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7bbb59e640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bbc1540b0 0x7f7bbc1f7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7bbb59e640 1 -- 172.21.15.28:0/3546353148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc156630 con 0x7f7bbc1f8070 2024-09-10T20:42:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7bbad9d640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba4004800 0x7f7bbc0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7bbb59e640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc1f8070 0x7f7bbc1f84c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4000f80 tx=0x7f7bb400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7ba37fe640 1 -- 172.21.15.28:0/3546353148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4015020 con 0x7f7bbc1f8070 2024-09-10T20:42:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7ba37fe640 1 -- 172.21.15.28:0/3546353148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7bb400cd30 con 0x7f7bbc1f8070 2024-09-10T20:42:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc156910 con 0x7f7bbc1f8070 2024-09-10T20:42:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7ba37fe640 1 -- 172.21.15.28:0/3546353148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40116c0 con 0x7f7bbc1f8070 2024-09-10T20:42:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.737+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc156ed0 con 0x7f7bbc1f8070 2024-09-10T20:42:20.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.738+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bbc0cf8f0 con 0x7f7bbc1f8070 2024-09-10T20:42:20.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.738+0000 7f7ba37fe640 1 -- 172.21.15.28:0/3546353148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f7bb4011860 con 0x7f7bbc1f8070 2024-09-10T20:42:20.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.739+0000 7f7ba37fe640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ba8078090 0x7f7ba807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:20.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.739+0000 7f7ba37fe640 1 -- 172.21.15.28:0/3546353148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f7bb409a400 con 0x7f7bbc1f8070 2024-09-10T20:42:20.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.739+0000 7f7bbad9d640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ba8078090 0x7f7ba807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:20.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.740+0000 7f7bbad9d640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ba8078090 0x7f7ba807a550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc15dbd0 tx=0x7f7bac006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:20.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.743+0000 7f7ba37fe640 1 -- 172.21.15.28:0/3546353148 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7bb401f030 con 0x7f7bbc1f8070 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.933+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7bbc0b9f30 con 0x7f7bbc1f8070 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.934+0000 7f7ba37fe640 1 -- 172.21.15.28:0/3546353148 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) ==== 84+0+65 (secure 0 0 0) 0x7f7bb40629f0 con 0x7f7bbc1f8070 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:20.939 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:42:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ba8078090 msgr2=0x7f7ba807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ba8078090 0x7f7ba807a550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc15dbd0 tx=0x7f7bac006b70 comp rx=0 tx=0).stop 2024-09-10T20:42:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc1f8070 msgr2=0x7f7bbc1f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc1f8070 0x7f7bbc1f84c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4000f80 tx=0x7f7bb400edb0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 shutdown_connections 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ba8078090 0x7f7ba807a550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc1f8070 0x7f7bbc1f84c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7bbc1540b0 0x7f7bbc1f7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 --2- 172.21.15.28:0/3546353148 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ba4004800 0x7f7bbc0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 >> 172.21.15.28:0/3546353148 conn(0x7f7bbc0d3fb0 msgr2=0x7f7bbc0b95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 shutdown_connections 2024-09-10T20:42:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:20.936+0000 7f7bc11cd640 1 -- 172.21.15.28:0/3546353148 wait complete. 2024-09-10T20:42:20.952 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:42:20.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 -- 172.21.15.28:0/2872816653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 msgr2=0x7fbdfc0741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 --2- 172.21.15.28:0/2872816653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc0741b0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc15ff60 tx=0x7fbdf402a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 -- 172.21.15.28:0/2872816653 shutdown_connections 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 --2- 172.21.15.28:0/2872816653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc0741b0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 --2- 172.21.15.28:0/2872816653 >> v2:172.21.15.159:3300/0 conn(0x7fbdfc06e390 0x7fbdfc15cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 -- 172.21.15.28:0/2872816653 >> 172.21.15.28:0/2872816653 conn(0x7fbdfc0cfc80 msgr2=0x7fbdfc0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 -- 172.21.15.28:0/2872816653 shutdown_connections 2024-09-10T20:42:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.057+0000 7fbe03208640 1 -- 172.21.15.28:0/2872816653 wait complete. 2024-09-10T20:42:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.058+0000 7fbe03208640 1 Processor -- start 2024-09-10T20:42:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.058+0000 7fbe03208640 1 -- start start 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe03208640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdfc06e390 0x7fbdfc1f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe03208640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc15a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe03208640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdfc0b5030 0x7fbdfc15adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe03208640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdfc0ca2d0 con 0x7fbdfc073d40 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe03208640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbdfc0ca450 con 0x7fbdfc0b5030 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe03208640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbdfc0ca5d0 con 0x7fbdfc06e390 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc15a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc15a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41554/0 (socket says 172.21.15.28:41554) 2024-09-10T20:42:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 -- 172.21.15.28:0/3023838912 learned_addr learned my addr 172.21.15.28:0/3023838912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe017a5640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdfc0b5030 0x7fbdfc15adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbe00fa4640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdfc06e390 0x7fbdfc1f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 -- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdfc06e390 msgr2=0x7fbdfc1f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdfc06e390 0x7fbdfc1f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 -- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdfc0b5030 msgr2=0x7fbdfc15adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdfc0b5030 0x7fbdfc15adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 -- 172.21.15.28:0/3023838912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdfc14dc80 con 0x7fbdfc073d40 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf3fff640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc15a880 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00d770 tx=0x7fbdec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.059+0000 7fbdf17fa640 1 -- 172.21.15.28:0/3023838912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec017070 con 0x7fbdfc073d40 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.060+0000 7fbdf17fa640 1 -- 172.21.15.28:0/3023838912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdec009d20 con 0x7fbdfc073d40 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.060+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdfc148ed0 con 0x7fbdfc073d40 2024-09-10T20:42:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.060+0000 7fbdf17fa640 1 -- 172.21.15.28:0/3023838912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec005280 con 0x7fbdfc073d40 2024-09-10T20:42:21.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.060+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdfc149490 con 0x7fbdfc073d40 2024-09-10T20:42:21.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.061+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdfc0caaf0 con 0x7fbdfc073d40 2024-09-10T20:42:21.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.061+0000 7fbdf17fa640 1 -- 172.21.15.28:0/3023838912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fbdec005420 con 0x7fbdfc073d40 2024-09-10T20:42:21.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.062+0000 7fbdf17fa640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbde8078090 0x7fbde807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:21.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.062+0000 7fbe00fa4640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbde8078090 0x7fbde807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.062+0000 7fbdf17fa640 1 -- 172.21.15.28:0/3023838912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7fbdec099db0 con 0x7fbdfc073d40 2024-09-10T20:42:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.062+0000 7fbe00fa4640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbde8078090 0x7fbde807a550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8007fd0 tx=0x7fbdf8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:21.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.066+0000 7fbdf17fa640 1 -- 172.21.15.28:0/3023838912 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbdfc14dc80 con 0x7fbdfc073d40 2024-09-10T20:42:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:21.250+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fbdfc15fd50 con 0x7fbdfc073d40 2024-09-10T20:42:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.223+0000 7fbdf17fa640 1 -- 172.21.15.28:0/3023838912 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) ==== 138+0+0 (secure 0 0 0) 0x7fbdec062a70 con 0x7fbdfc073d40 2024-09-10T20:42:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:42:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbde8078090 msgr2=0x7fbde807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbde8078090 0x7fbde807a550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8007fd0 tx=0x7fbdf8009040 comp rx=0 tx=0).stop 2024-09-10T20:42:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 msgr2=0x7fbdfc15a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc15a880 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00d770 tx=0x7fbdec00dc30 comp rx=0 tx=0).stop 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 shutdown_connections 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbde8078090 0x7fbde807a550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdfc0b5030 0x7fbdfc15adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdfc073d40 0x7fbdfc15a880 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 --2- 172.21.15.28:0/3023838912 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdfc06e390 0x7fbdfc1f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 >> 172.21.15.28:0/3023838912 conn(0x7fbdfc0cfc80 msgr2=0x7fbdfc0d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 shutdown_connections 2024-09-10T20:42:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.228+0000 7fbe03208640 1 -- 172.21.15.28:0/3023838912 wait complete. 2024-09-10T20:42:22.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:42:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.364+0000 7f39ddee2640 1 -- 172.21.15.28:0/3309860343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 msgr2=0x7f39d80d4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.364+0000 7f39ddee2640 1 --2- 172.21.15.28:0/3309860343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80d4b60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f39d80cecb0 tx=0x7f39c802a890 comp rx=0 tx=0).stop 2024-09-10T20:42:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.364+0000 7f39ddee2640 1 -- 172.21.15.28:0/3309860343 shutdown_connections 2024-09-10T20:42:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.364+0000 7f39ddee2640 1 --2- 172.21.15.28:0/3309860343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80d4b60 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.364+0000 7f39ddee2640 1 -- 172.21.15.28:0/3309860343 >> 172.21.15.28:0/3309860343 conn(0x7f39d80d4040 msgr2=0x7f39d80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.364+0000 7f39ddee2640 1 -- 172.21.15.28:0/3309860343 shutdown_connections 2024-09-10T20:42:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.365+0000 7f39ddee2640 1 -- 172.21.15.28:0/3309860343 wait complete. 2024-09-10T20:42:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.366+0000 7f39ddee2640 1 Processor -- start 2024-09-10T20:42:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.366+0000 7f39ddee2640 1 -- start start 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39ddee2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39d80b0c90 0x7f39d80b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39ddee2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39d80cfc50 0x7f39d81f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39ddee2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39ddee2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81624a0 con 0x7f39d815e130 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39ddee2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f39d8162320 con 0x7f39d80cfc50 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39ddee2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f39d8162620 con 0x7f39d80b0c90 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41570/0 (socket says 172.21.15.28:41570) 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 -- 172.21.15.28:0/4216801956 learned_addr learned my addr 172.21.15.28:0/4216801956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d77fe640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39d80b0c90 0x7f39d80b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d6ffd640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39d80cfc50 0x7f39d81f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 -- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39d80b0c90 msgr2=0x7f39d80b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39d80b0c90 0x7f39d80b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 -- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39d80cfc50 msgr2=0x7f39d81f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39d80cfc50 0x7f39d81f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 -- 172.21.15.28:0/4216801956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d81f5760 con 0x7f39d815e130 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.367+0000 7f39d7fff640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80cddf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c400 tx=0x7f39d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.368+0000 7f39bffff640 1 -- 172.21.15.28:0/4216801956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0017070 con 0x7f39d815e130 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.368+0000 7f39bffff640 1 -- 172.21.15.28:0/4216801956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39d00053c0 con 0x7f39d815e130 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.368+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d814db20 con 0x7f39d815e130 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.368+0000 7f39bffff640 1 -- 172.21.15.28:0/4216801956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0003850 con 0x7f39d815e130 2024-09-10T20:42:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.368+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d80b88f0 con 0x7f39d815e130 2024-09-10T20:42:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.369+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d80d4b60 con 0x7f39d815e130 2024-09-10T20:42:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.369+0000 7f39bffff640 1 -- 172.21.15.28:0/4216801956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f39d00039f0 con 0x7f39d815e130 2024-09-10T20:42:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.370+0000 7f39bffff640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39c4078090 0x7f39c407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.370+0000 7f39bffff640 1 -- 172.21.15.28:0/4216801956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5227+0+0 (secure 0 0 0) 0x7f39d00993c0 con 0x7f39d815e130 2024-09-10T20:42:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.370+0000 7f39d77fe640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39c4078090 0x7f39c407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.370+0000 7f39d77fe640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39c4078090 0x7f39c407a550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39c8034000 tx=0x7f39c8004ab0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.374+0000 7f39bffff640 1 -- 172.21.15.28:0/4216801956 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f39d81f5760 con 0x7f39d815e130 2024-09-10T20:42:22.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:22.560+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f39d80761d0 con 0x7f39d815e130 2024-09-10T20:42:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.230+0000 7f39bffff640 1 -- 172.21.15.28:0/4216801956 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) ==== 170+0+0 (secure 0 0 0) 0x7f39d00618c0 con 0x7f39d815e130 2024-09-10T20:42:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:42:23.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39c4078090 msgr2=0x7f39c407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:23.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39c4078090 0x7f39c407a550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39c8034000 tx=0x7f39c8004ab0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 msgr2=0x7f39d80cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:23.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80cddf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c400 tx=0x7f39d000c8c0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 shutdown_connections 2024-09-10T20:42:23.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39c4078090 0x7f39c407a550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39d815e130 0x7f39d80cddf0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39d80cfc50 0x7f39d81f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 --2- 172.21.15.28:0/4216801956 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39d80b0c90 0x7f39d80b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.233+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 >> 172.21.15.28:0/4216801956 conn(0x7f39d80d4040 msgr2=0x7f39d80d4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:23.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.234+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 shutdown_connections 2024-09-10T20:42:23.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.239+0000 7f39ddee2640 1 -- 172.21.15.28:0/4216801956 wait complete. 2024-09-10T20:42:23.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:42:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 -- 172.21.15.28:0/1372429033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c15e420 msgr2=0x7f2a3c06cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 --2- 172.21.15.28:0/1372429033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c15e420 0x7f2a3c06cae0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2a34005fd0 tx=0x7f2a3402a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 -- 172.21.15.28:0/1372429033 shutdown_connections 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 --2- 172.21.15.28:0/1372429033 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c15e420 0x7f2a3c06cae0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 --2- 172.21.15.28:0/1372429033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c0cbbf0 0x7f2a3c0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 -- 172.21.15.28:0/1372429033 >> 172.21.15.28:0/1372429033 conn(0x7f2a3c0d0d50 msgr2=0x7f2a3c0d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 -- 172.21.15.28:0/1372429033 shutdown_connections 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.378+0000 7f2a43f05640 1 -- 172.21.15.28:0/1372429033 wait complete. 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.379+0000 7f2a43f05640 1 Processor -- start 2024-09-10T20:42:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a43f05640 1 -- start start 2024-09-10T20:42:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a43f05640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c0cbbf0 0x7f2a3c0c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a43f05640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0cbfd0 0x7f2a3c0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a43f05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c15e420 0x7f2a3c0c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a43f05640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c073c20 con 0x7f2a3c15e420 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a43f05640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c073da0 con 0x7f2a3c0cbfd0 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a43f05640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2a3c073f20 con 0x7f2a3c0cbbf0 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a424a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c15e420 0x7f2a3c0c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a424a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c15e420 0x7f2a3c0c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41574/0 (socket says 172.21.15.28:41574) 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.380+0000 7f2a424a2640 1 -- 172.21.15.28:0/2343566911 learned_addr learned my addr 172.21.15.28:0/2343566911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a41ca1640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c0cbbf0 0x7f2a3c0c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a414a0640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0cbfd0 0x7f2a3c0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a424a2640 1 -- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c0cbbf0 msgr2=0x7f2a3c0c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a424a2640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c0cbbf0 0x7f2a3c0c3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a424a2640 1 -- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0cbfd0 msgr2=0x7f2a3c0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a424a2640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0cbfd0 0x7f2a3c0b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a424a2640 1 -- 172.21.15.28:0/2343566911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c0b5490 con 0x7f2a3c15e420 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a424a2640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c15e420 0x7f2a3c0c7e20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006010 tx=0x7f2a34005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a327fc640 1 -- 172.21.15.28:0/2343566911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a34031990 con 0x7f2a3c15e420 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a327fc640 1 -- 172.21.15.28:0/2343566911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a34031c20 con 0x7f2a3c15e420 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c0b5770 con 0x7f2a3c15e420 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a327fc640 1 -- 172.21.15.28:0/2343566911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a34033330 con 0x7f2a3c15e420 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.381+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c0b5d30 con 0x7f2a3c15e420 2024-09-10T20:42:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.382+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c074440 con 0x7f2a3c15e420 2024-09-10T20:42:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.383+0000 7f2a327fc640 1 -- 172.21.15.28:0/2343566911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f2a3404b030 con 0x7f2a3c15e420 2024-09-10T20:42:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.384+0000 7f2a327fc640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a200780c0 0x7f2a2007a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.384+0000 7f2a41ca1640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a200780c0 0x7f2a2007a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.384+0000 7f2a327fc640 1 -- 172.21.15.28:0/2343566911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5609+0+0 (secure 0 0 0) 0x7f2a340baec0 con 0x7f2a3c15e420 2024-09-10T20:42:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.384+0000 7f2a41ca1640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a200780c0 0x7f2a2007a580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2a380095f0 tx=0x7f2a38005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:23.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.389+0000 7f2a327fc640 1 -- 172.21.15.28:0/2343566911 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2a3403f030 con 0x7f2a3c15e420 2024-09-10T20:42:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:23.613+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2a3c0d1b50 con 0x7f2a3c15e420 2024-09-10T20:42:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.231+0000 7f2a327fc640 1 -- 172.21.15.28:0/2343566911 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v49) ==== 174+0+0 (secure 0 0 0) 0x7f2a340317b0 con 0x7f2a3c15e420 2024-09-10T20:42:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 12 and data pool 13 2024-09-10T20:42:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a200780c0 msgr2=0x7f2a2007a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a200780c0 0x7f2a2007a580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2a380095f0 tx=0x7f2a38005f90 comp rx=0 tx=0).stop 2024-09-10T20:42:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c15e420 msgr2=0x7f2a3c0c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c15e420 0x7f2a3c0c7e20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006010 tx=0x7f2a34005c50 comp rx=0 tx=0).stop 2024-09-10T20:42:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 shutdown_connections 2024-09-10T20:42:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a200780c0 0x7f2a2007a580 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c15e420 0x7f2a3c0c7e20 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a3c0cbfd0 0x7f2a3c0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 --2- 172.21.15.28:0/2343566911 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a3c0cbbf0 0x7f2a3c0c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.236+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 >> 172.21.15.28:0/2343566911 conn(0x7f2a3c0d0d50 msgr2=0x7f2a3c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.242+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 shutdown_connections 2024-09-10T20:42:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.242+0000 7f2a43f05640 1 -- 172.21.15.28:0/2343566911 wait complete. 2024-09-10T20:42:24.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 -- 172.21.15.28:0/1526998937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b98156130 msgr2=0x7f3b980caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/1526998937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b98156130 0x7f3b980caa70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3b90005fd0 tx=0x7f3b9002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 -- 172.21.15.28:0/1526998937 shutdown_connections 2024-09-10T20:42:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/1526998937 >> v2:172.21.15.159:3300/0 conn(0x7f3b9806d1f0 0x7f3b9806ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/1526998937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b98156130 0x7f3b980caa70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 -- 172.21.15.28:0/1526998937 >> 172.21.15.28:0/1526998937 conn(0x7f3b980d0420 msgr2=0x7f3b980d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 -- 172.21.15.28:0/1526998937 shutdown_connections 2024-09-10T20:42:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.380+0000 7f3ba00b0640 1 -- 172.21.15.28:0/1526998937 wait complete. 2024-09-10T20:42:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 Processor -- start 2024-09-10T20:42:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 -- start start 2024-09-10T20:42:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b9806d1f0 0x7f3b98154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b98156130 0x7f3b9815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b98156510 0x7f3b980c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b980733b0 con 0x7f3b98156510 2024-09-10T20:42:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3b98073230 con 0x7f3b98156130 2024-09-10T20:42:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3ba00b0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3b98073530 con 0x7f3b9806d1f0 2024-09-10T20:42:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.382+0000 7f3b9e64d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b98156510 0x7f3b980c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b98156510 0x7f3b980c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41586/0 (socket says 172.21.15.28:41586) 2024-09-10T20:42:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 -- 172.21.15.28:0/3160451043 learned_addr learned my addr 172.21.15.28:0/3160451043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9de4c640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b9806d1f0 0x7f3b98154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9d64b640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b98156130 0x7f3b9815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 -- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b9806d1f0 msgr2=0x7f3b98154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b9806d1f0 0x7f3b98154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 -- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b98156130 msgr2=0x7f3b9815bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b98156130 0x7f3b9815bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 -- 172.21.15.28:0/3160451043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b98155330 con 0x7f3b98156510 2024-09-10T20:42:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b9e64d640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b98156510 0x7f3b980c86e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800eb40 tx=0x7f3b8800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b967fc640 1 -- 172.21.15.28:0/3160451043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88004b20 con 0x7f3b98156510 2024-09-10T20:42:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b98156e90 con 0x7f3b98156510 2024-09-10T20:42:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b967fc640 1 -- 172.21.15.28:0/3160451043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b88004ce0 con 0x7f3b98156510 2024-09-10T20:42:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3b967fc640 1 -- 172.21.15.28:0/3160451043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88005280 con 0x7f3b98156510 2024-09-10T20:42:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.383+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b981575c0 con 0x7f3b98156510 2024-09-10T20:42:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.384+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b9806ef30 con 0x7f3b98156510 2024-09-10T20:42:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.385+0000 7f3b967fc640 1 -- 172.21.15.28:0/3160451043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f3b88005420 con 0x7f3b98156510 2024-09-10T20:42:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.385+0000 7f3b967fc640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b84078090 0x7f3b8407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.386+0000 7f3b967fc640 1 -- 172.21.15.28:0/3160451043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f3b8809abf0 con 0x7f3b98156510 2024-09-10T20:42:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.386+0000 7f3b9de4c640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b84078090 0x7f3b8407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.386+0000 7f3b9de4c640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b84078090 0x7f3b8407a550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3b90033000 tx=0x7f3b90004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:24.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.390+0000 7f3b967fc640 1 -- 172.21.15.28:0/3160451043 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3b88062e90 con 0x7f3b98156510 2024-09-10T20:42:24.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.575+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b980b1d00 con 0x7f3b98156510 2024-09-10T20:42:24.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.576+0000 7f3b967fc640 1 -- 172.21.15.28:0/3160451043 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16745 (secure 0 0 0) 0x7f3b880678c0 con 0x7f3b98156510 2024-09-10T20:42:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:24.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:23.614265+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:21.256150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:22.565350+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:24.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b84078090 msgr2=0x7f3b8407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b84078090 0x7f3b8407a550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3b90033000 tx=0x7f3b90004730 comp rx=0 tx=0).stop 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b98156510 msgr2=0x7f3b980c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b98156510 0x7f3b980c86e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800eb40 tx=0x7f3b8800c2d0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 shutdown_connections 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b84078090 0x7f3b8407a550 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b98156510 0x7f3b980c86e0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b98156130 0x7f3b9815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 --2- 172.21.15.28:0/3160451043 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b9806d1f0 0x7f3b98154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 >> 172.21.15.28:0/3160451043 conn(0x7f3b980d0420 msgr2=0x7f3b9815e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 shutdown_connections 2024-09-10T20:42:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.578+0000 7f3ba00b0640 1 -- 172.21.15.28:0/3160451043 wait complete. 2024-09-10T20:42:24.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 -- 172.21.15.28:0/918215185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 msgr2=0x7f8d9815e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 --2- 172.21.15.28:0/918215185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d9815e6e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c005fd0 tx=0x7f8d8c02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 -- 172.21.15.28:0/918215185 shutdown_connections 2024-09-10T20:42:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 --2- 172.21.15.28:0/918215185 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d9815ec20 0x7f8d98070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 --2- 172.21.15.28:0/918215185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d9815e6e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 -- 172.21.15.28:0/918215185 >> 172.21.15.28:0/918215185 conn(0x7f8d9806dbf0 msgr2=0x7f8d9806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 -- 172.21.15.28:0/918215185 shutdown_connections 2024-09-10T20:42:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.695+0000 7f8d9e260640 1 -- 172.21.15.28:0/918215185 wait complete. 2024-09-10T20:42:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.696+0000 7f8d9e260640 1 Processor -- start 2024-09-10T20:42:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.696+0000 7f8d9e260640 1 -- start start 2024-09-10T20:42:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d9e260640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980d0c80 0x7f8d980cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d9e260640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d98144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d9e260640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d9815ec20 0x7f8d980b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d9e260640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d98070940 con 0x7f8d9815bc40 2024-09-10T20:42:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d9e260640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8d980707c0 con 0x7f8d9815ec20 2024-09-10T20:42:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d9e260640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8d98070ac0 con 0x7f8d980d0c80 2024-09-10T20:42:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d98144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d98144720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41598/0 (socket says 172.21.15.28:41598) 2024-09-10T20:42:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d96ffd640 1 -- 172.21.15.28:0/2518279153 learned_addr learned my addr 172.21.15.28:0/2518279153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d97fff640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d9815ec20 0x7f8d980b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d977fe640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980d0c80 0x7f8d980cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d96ffd640 1 -- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980d0c80 msgr2=0x7f8d980cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.697+0000 7f8d96ffd640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980d0c80 0x7f8d980cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d96ffd640 1 -- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d9815ec20 msgr2=0x7f8d980b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d96ffd640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d9815ec20 0x7f8d980b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d96ffd640 1 -- 172.21.15.28:0/2518279153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d980b33d0 con 0x7f8d9815bc40 2024-09-10T20:42:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d96ffd640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d98144720 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009850 tx=0x7f8d8c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d7ffff640 1 -- 172.21.15.28:0/2518279153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c03f070 con 0x7f8d9815bc40 2024-09-10T20:42:24.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d7ffff640 1 -- 172.21.15.28:0/2518279153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d8c005600 con 0x7f8d9815bc40 2024-09-10T20:42:24.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d7ffff640 1 -- 172.21.15.28:0/2518279153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c035030 con 0x7f8d9815bc40 2024-09-10T20:42:24.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d980b36b0 con 0x7f8d9815bc40 2024-09-10T20:42:24.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.698+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d980cd330 con 0x7f8d9815bc40 2024-09-10T20:42:24.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.699+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d980b5620 con 0x7f8d9815bc40 2024-09-10T20:42:24.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.700+0000 7f8d7ffff640 1 -- 172.21.15.28:0/2518279153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f8d8c0049e0 con 0x7f8d9815bc40 2024-09-10T20:42:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.700+0000 7f8d7ffff640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d84078090 0x7f8d8407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.700+0000 7f8d7ffff640 1 -- 172.21.15.28:0/2518279153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f8d8c0badb0 con 0x7f8d9815bc40 2024-09-10T20:42:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.700+0000 7f8d977fe640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d84078090 0x7f8d8407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.701+0000 7f8d977fe640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d84078090 0x7f8d8407a550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d880017c0 tx=0x7f8d88009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.704+0000 7f8d7ffff640 1 -- 172.21.15.28:0/2518279153 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8d980b5620 con 0x7f8d9815bc40 2024-09-10T20:42:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.891+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d98155180 con 0x7f8d9815bc40 2024-09-10T20:42:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.892+0000 7f8d7ffff640 1 -- 172.21.15.28:0/2518279153 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16745 (secure 0 0 0) 0x7f8d98155180 con 0x7f8d9815bc40 2024-09-10T20:42:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:24.896 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:23.614265+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:21.256150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:22.565350+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d84078090 msgr2=0x7f8d8407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d84078090 0x7f8d8407a550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d880017c0 tx=0x7f8d88009040 comp rx=0 tx=0).stop 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 msgr2=0x7f8d98144720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d98144720 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009850 tx=0x7f8d8c0038a0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 shutdown_connections 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d84078090 0x7f8d8407a550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d9815ec20 0x7f8d980b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9815bc40 0x7f8d98144720 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 --2- 172.21.15.28:0/2518279153 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d980d0c80 0x7f8d980cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 >> 172.21.15.28:0/2518279153 conn(0x7f8d9806dbf0 msgr2=0x7f8d9815c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 shutdown_connections 2024-09-10T20:42:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:24.895+0000 7f8d9e260640 1 -- 172.21.15.28:0/2518279153 wait complete. 2024-09-10T20:42:24.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:42:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 -- 172.21.15.28:0/1853081577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 msgr2=0x7f965c0706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 --2- 172.21.15.28:0/1853081577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 0x7f965c0706e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f964c005fd0 tx=0x7f964c02a540 comp rx=0 tx=0).stop 2024-09-10T20:42:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 -- 172.21.15.28:0/1853081577 shutdown_connections 2024-09-10T20:42:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 --2- 172.21.15.28:0/1853081577 >> v2:172.21.15.159:3300/0 conn(0x7f965c15bd80 0x7f965c073790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 --2- 172.21.15.28:0/1853081577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 0x7f965c0706e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 -- 172.21.15.28:0/1853081577 >> 172.21.15.28:0/1853081577 conn(0x7f965c0d0d50 msgr2=0x7f965c0d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 -- 172.21.15.28:0/1853081577 shutdown_connections 2024-09-10T20:42:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.014+0000 7f96629ad640 1 -- 172.21.15.28:0/1853081577 wait complete. 2024-09-10T20:42:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.015+0000 7f96629ad640 1 Processor -- start 2024-09-10T20:42:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f96629ad640 1 -- start start 2024-09-10T20:42:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f96629ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c073cd0 0x7f965c1481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f96629ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 0x7f965c147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f96629ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c15bd80 0x7f965c1f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f96629ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f965c15f390 con 0x7f965c15bd80 2024-09-10T20:42:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f96629ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f965c15f210 con 0x7f965c073cd0 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f96629ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f965c15f510 con 0x7f965c151db0 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.016+0000 7f9660f4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c15bd80 0x7f965c1f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c15bd80 0x7f965c1f2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41600/0 (socket says 172.21.15.28:41600) 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 -- 172.21.15.28:0/3288988529 learned_addr learned my addr 172.21.15.28:0/3288988529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f965bfff640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c073cd0 0x7f965c1481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f965b7fe640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 0x7f965c147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 -- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 msgr2=0x7f965c147930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 0x7f965c147930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 -- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c073cd0 msgr2=0x7f965c1481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c073cd0 0x7f965c1481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 -- 172.21.15.28:0/3288988529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c154750 con 0x7f965c15bd80 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f965b7fe640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 0x7f965c147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9660f4a640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c15bd80 0x7f965c1f2ad0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f965400c7a0 tx=0x7f965400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9658ff9640 1 -- 172.21.15.28:0/3288988529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654003b70 con 0x7f965c15bd80 2024-09-10T20:42:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c154a30 con 0x7f965c15bd80 2024-09-10T20:42:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9658ff9640 1 -- 172.21.15.28:0/3288988529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9654003d30 con 0x7f965c15bd80 2024-09-10T20:42:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f9658ff9640 1 -- 172.21.15.28:0/3288988529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965400aab0 con 0x7f965c15bd80 2024-09-10T20:42:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.017+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c0b3e70 con 0x7f965c15bd80 2024-09-10T20:42:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.018+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965c0706e0 con 0x7f965c15bd80 2024-09-10T20:42:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.019+0000 7f9658ff9640 1 -- 172.21.15.28:0/3288988529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f965400ac50 con 0x7f965c15bd80 2024-09-10T20:42:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.020+0000 7f9658ff9640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9648078090 0x7f964807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.020+0000 7f965bfff640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9648078090 0x7f964807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.020+0000 7f9658ff9640 1 -- 172.21.15.28:0/3288988529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f9654099ed0 con 0x7f965c15bd80 2024-09-10T20:42:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.020+0000 7f965bfff640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9648078090 0x7f964807a550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f964c033000 tx=0x7f964c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.024+0000 7f9658ff9640 1 -- 172.21.15.28:0/3288988529 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9654062720 con 0x7f965c15bd80 2024-09-10T20:42:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:25.250+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f965c0749a0 con 0x7f965c15bd80 2024-09-10T20:42:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.249+0000 7f9658ff9640 1 -- 172.21.15.28:0/3288988529 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v52) ==== 118+0+0 (secure 0 0 0) 0x7f9654062020 con 0x7f965c15bd80 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9648078090 msgr2=0x7f964807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9648078090 0x7f964807a550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f964c033000 tx=0x7f964c004730 comp rx=0 tx=0).stop 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c15bd80 msgr2=0x7f965c1f2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c15bd80 0x7f965c1f2ad0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f965400c7a0 tx=0x7f965400cc60 comp rx=0 tx=0).stop 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 shutdown_connections 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9648078090 0x7f964807a550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c15bd80 0x7f965c1f2ad0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c151db0 0x7f965c147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 --2- 172.21.15.28:0/3288988529 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c073cd0 0x7f965c1481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 >> 172.21.15.28:0/3288988529 conn(0x7f965c0d0d50 msgr2=0x7f965c0cb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 shutdown_connections 2024-09-10T20:42:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.252+0000 7f96629ad640 1 -- 172.21.15.28:0/3288988529 wait complete. 2024-09-10T20:42:26.269 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:42:26.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:42:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.381+0000 7f3f39a40640 1 -- 172.21.15.28:0/486574201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 msgr2=0x7f3f3415b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.381+0000 7f3f39a40640 1 --2- 172.21.15.28:0/486574201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f3415b170 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3f24005fd0 tx=0x7f3f2402aaa0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.381+0000 7f3f39a40640 1 -- 172.21.15.28:0/486574201 shutdown_connections 2024-09-10T20:42:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.381+0000 7f3f39a40640 1 --2- 172.21.15.28:0/486574201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f3415b170 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.381+0000 7f3f39a40640 1 -- 172.21.15.28:0/486574201 >> 172.21.15.28:0/486574201 conn(0x7f3f3406d110 msgr2=0x7f3f340b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:26.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.381+0000 7f3f39a40640 1 -- 172.21.15.28:0/486574201 shutdown_connections 2024-09-10T20:42:26.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.381+0000 7f3f39a40640 1 -- 172.21.15.28:0/486574201 wait complete. 2024-09-10T20:42:26.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.382+0000 7f3f39a40640 1 Processor -- start 2024-09-10T20:42:26.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.382+0000 7f3f39a40640 1 -- start start 2024-09-10T20:42:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f39a40640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f3406fc20 0x7f3f341556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f39a40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f34148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f39a40640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3415e7d0 0x7f3f34072890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f39a40640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3415d670 con 0x7f3f341587a0 2024-09-10T20:42:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f39a40640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3415d4f0 con 0x7f3f3415e7d0 2024-09-10T20:42:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f39a40640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3415d7f0 con 0x7f3f3406fc20 2024-09-10T20:42:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f34148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f34148780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41624/0 (socket says 172.21.15.28:41624) 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.383+0000 7f3f327fc640 1 -- 172.21.15.28:0/874423590 learned_addr learned my addr 172.21.15.28:0/874423590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f337fe640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3415e7d0 0x7f3f34072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f32ffd640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f3406fc20 0x7f3f341556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f327fc640 1 -- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f3406fc20 msgr2=0x7f3f341556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f327fc640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f3406fc20 0x7f3f341556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f327fc640 1 -- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3415e7d0 msgr2=0x7f3f34072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f327fc640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3415e7d0 0x7f3f34072890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f327fc640 1 -- 172.21.15.28:0/874423590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f340af980 con 0x7f3f341587a0 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f337fe640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3415e7d0 0x7f3f34072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f32ffd640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f3406fc20 0x7f3f341556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f327fc640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f34148780 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00c400 tx=0x7f3f1c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f2bfff640 1 -- 172.21.15.28:0/874423590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c017070 con 0x7f3f341587a0 2024-09-10T20:42:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f2bfff640 1 -- 172.21.15.28:0/874423590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f1c0053c0 con 0x7f3f341587a0 2024-09-10T20:42:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f340afc60 con 0x7f3f341587a0 2024-09-10T20:42:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f2bfff640 1 -- 172.21.15.28:0/874423590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c003850 con 0x7f3f341587a0 2024-09-10T20:42:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.384+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f34072dd0 con 0x7f3f341587a0 2024-09-10T20:42:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.385+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3415dcc0 con 0x7f3f341587a0 2024-09-10T20:42:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.386+0000 7f3f2bfff640 1 -- 172.21.15.28:0/874423590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f3f1c0039f0 con 0x7f3f341587a0 2024-09-10T20:42:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.386+0000 7f3f2bfff640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f14078090 0x7f3f1407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.386+0000 7f3f2bfff640 1 -- 172.21.15.28:0/874423590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f3f1c099760 con 0x7f3f341587a0 2024-09-10T20:42:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.386+0000 7f3f32ffd640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f14078090 0x7f3f1407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.387+0000 7f3f32ffd640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f14078090 0x7f3f1407a550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f24035000 tx=0x7f3f24004ab0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.390+0000 7f3f2bfff640 1 -- 172.21.15.28:0/874423590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3f1c061fc0 con 0x7f3f341587a0 2024-09-10T20:42:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:26.616+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f3f34157e70 con 0x7f3f341587a0 2024-09-10T20:42:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.256+0000 7f3f2bfff640 1 -- 172.21.15.28:0/874423590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v53) ==== 122+0+0 (secure 0 0 0) 0x7f3f1c0618c0 con 0x7f3f341587a0 2024-09-10T20:42:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f14078090 msgr2=0x7f3f1407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f14078090 0x7f3f1407a550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f24035000 tx=0x7f3f24004ab0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 msgr2=0x7f3f34148780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f34148780 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00c400 tx=0x7f3f1c00c8c0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 shutdown_connections 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f14078090 0x7f3f1407a550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f3415e7d0 0x7f3f34072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f341587a0 0x7f3f34148780 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 --2- 172.21.15.28:0/874423590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f3406fc20 0x7f3f341556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 >> 172.21.15.28:0/874423590 conn(0x7f3f3406d110 msgr2=0x7f3f340adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.259+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 shutdown_connections 2024-09-10T20:42:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.260+0000 7f3f39a40640 1 -- 172.21.15.28:0/874423590 wait complete. 2024-09-10T20:42:27.278 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:42:27.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:27.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 -- 172.21.15.28:0/1278861252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 msgr2=0x7fd95c0b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 --2- 172.21.15.28:0/1278861252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0b2e80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd954009ca0 tx=0x7fd95402aa30 comp rx=0 tx=0).stop 2024-09-10T20:42:27.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 -- 172.21.15.28:0/1278861252 shutdown_connections 2024-09-10T20:42:27.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 --2- 172.21.15.28:0/1278861252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0b2e80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 --2- 172.21.15.28:0/1278861252 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd95c0cc570 0x7fd95c0d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 -- 172.21.15.28:0/1278861252 >> 172.21.15.28:0/1278861252 conn(0x7fd95c06cf00 msgr2=0x7fd95c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:27.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 -- 172.21.15.28:0/1278861252 shutdown_connections 2024-09-10T20:42:27.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.386+0000 7fd962c85640 1 -- 172.21.15.28:0/1278861252 wait complete. 2024-09-10T20:42:27.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.387+0000 7fd962c85640 1 Processor -- start 2024-09-10T20:42:27.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.387+0000 7fd962c85640 1 -- start start 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd962c85640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd95c0cc570 0x7fd95c157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd962c85640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd95c0cd3d0 0x7fd95c0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd962c85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd962c85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c0b33c0 con 0x7fd95c0d05d0 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd962c85640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c0b4b60 con 0x7fd95c0cc570 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd962c85640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd95c0b3540 con 0x7fd95c0cd3d0 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41636/0 (socket says 172.21.15.28:41636) 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 -- 172.21.15.28:0/2837837510 learned_addr learned my addr 172.21.15.28:0/2837837510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd95bfff640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd95c0cd3d0 0x7fd95c0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd960a21640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd95c0cc570 0x7fd95c157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 -- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd95c0cd3d0 msgr2=0x7fd95c0afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd95c0cd3d0 0x7fd95c0afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 -- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd95c0cc570 msgr2=0x7fd95c157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd95c0cc570 0x7fd95c157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd961222640 1 -- 172.21.15.28:0/2837837510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95c0b07d0 con 0x7fd95c0d05d0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.388+0000 7fd960a21640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd95c0cc570 0x7fd95c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.389+0000 7fd961222640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0cb110 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd954009dc0 tx=0x7fd954003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.389+0000 7fd9597fa640 1 -- 172.21.15.28:0/2837837510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954033050 con 0x7fd95c0d05d0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.389+0000 7fd9597fa640 1 -- 172.21.15.28:0/2837837510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd954004510 con 0x7fd95c0d05d0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.389+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd95c0b0ab0 con 0x7fd95c0d05d0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.389+0000 7fd9597fa640 1 -- 172.21.15.28:0/2837837510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954038450 con 0x7fd95c0d05d0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.389+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd95c1584e0 con 0x7fd95c0d05d0 2024-09-10T20:42:27.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.390+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd95c0b39d0 con 0x7fd95c0d05d0 2024-09-10T20:42:27.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.391+0000 7fd9597fa640 1 -- 172.21.15.28:0/2837837510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fd954040070 con 0x7fd95c0d05d0 2024-09-10T20:42:27.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.391+0000 7fd9597fa640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd940078090 0x7fd94007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.391+0000 7fd960a21640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd940078090 0x7fd94007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.391+0000 7fd9597fa640 1 -- 172.21.15.28:0/2837837510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fd9540bb1e0 con 0x7fd95c0d05d0 2024-09-10T20:42:27.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.392+0000 7fd960a21640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd940078090 0x7fd94007a550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd950003eb0 tx=0x7fd950009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:27.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.395+0000 7fd9597fa640 1 -- 172.21.15.28:0/2837837510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd954083b50 con 0x7fd95c0d05d0 2024-09-10T20:42:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.615+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd95c0b0c80 con 0x7fd95c0d05d0 2024-09-10T20:42:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.616+0000 7fd9597fa640 1 -- 172.21.15.28:0/2837837510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5231 (secure 0 0 0) 0x7fd954083450 con 0x7fd95c0d05d0 2024-09-10T20:42:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:27.620 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"btime":"2024-09-10T20:42:27:250005+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2983894865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2983894865},{"type":"v1","addr":"172.21.15.159:6801","nonce":2983894865}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2105531171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2105531171},{"type":"v1","addr":"172.21.15.159:6835","nonce":2105531171}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2239499017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2239499017},{"type":"v1","addr":"172.21.15.28:6834","nonce":2239499017}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:23.614228+0000","modified":"2024-09-10T20:42:26.617829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6570":{"gid":6570,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2216477518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2216477518},{"type":"v1","addr":"172.21.15.28:6835","nonce":2216477518}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6570,"qdb_cluster":[6570]},"id":6}]} 2024-09-10T20:42:27.620 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-09-10T20:42:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd940078090 msgr2=0x7fd94007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd940078090 0x7fd94007a550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd950003eb0 tx=0x7fd950009040 comp rx=0 tx=0).stop 2024-09-10T20:42:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 msgr2=0x7fd95c0cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0cb110 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd954009dc0 tx=0x7fd954003f90 comp rx=0 tx=0).stop 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 shutdown_connections 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd940078090 0x7fd94007a550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c0d05d0 0x7fd95c0cb110 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd95c0cd3d0 0x7fd95c0afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 --2- 172.21.15.28:0/2837837510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd95c0cc570 0x7fd95c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 >> 172.21.15.28:0/2837837510 conn(0x7fd95c06cf00 msgr2=0x7fd95c09aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 shutdown_connections 2024-09-10T20:42:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.618+0000 7fd962c85640 1 -- 172.21.15.28:0/2837837510 wait complete. 2024-09-10T20:42:27.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:27.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.735+0000 7ffadb594640 1 -- 172.21.15.28:0/2519925561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad40d05d0 msgr2=0x7ffad40d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.735+0000 7ffadb594640 1 --2- 172.21.15.28:0/2519925561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad40d05d0 0x7ffad40d2960 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009830 tx=0x7ffac402a510 comp rx=0 tx=0).stop 2024-09-10T20:42:27.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.735+0000 7ffadb594640 1 -- 172.21.15.28:0/2519925561 shutdown_connections 2024-09-10T20:42:27.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.735+0000 7ffadb594640 1 --2- 172.21.15.28:0/2519925561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad40d05d0 0x7ffad40d2960 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.735+0000 7ffadb594640 1 -- 172.21.15.28:0/2519925561 >> 172.21.15.28:0/2519925561 conn(0x7ffad40d1e40 msgr2=0x7ffad40d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:27.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.735+0000 7ffadb594640 1 -- 172.21.15.28:0/2519925561 shutdown_connections 2024-09-10T20:42:27.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.735+0000 7ffadb594640 1 -- 172.21.15.28:0/2519925561 wait complete. 2024-09-10T20:42:27.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.736+0000 7ffadb594640 1 Processor -- start 2024-09-10T20:42:27.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.736+0000 7ffadb594640 1 -- start start 2024-09-10T20:42:27.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffadb594640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad40b5610 0x7ffad415f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffadb594640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad40d05d0 0x7ffad41520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffadb594640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad415bf30 0x7ffad41f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffadb594640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffad40b6170 con 0x7ffad415bf30 2024-09-10T20:42:27.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffadb594640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffad40d0d50 con 0x7ffad40d05d0 2024-09-10T20:42:27.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffadb594640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffad40b62f0 con 0x7ffad40b5610 2024-09-10T20:42:27.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffad9b31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad415bf30 0x7ffad41f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffad9b31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad415bf30 0x7ffad41f5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41660/0 (socket says 172.21.15.28:41660) 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffad9b31640 1 -- 172.21.15.28:0/2491271922 learned_addr learned my addr 172.21.15.28:0/2491271922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffad9330640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad40b5610 0x7ffad415f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.737+0000 7ffad8b2f640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad40d05d0 0x7ffad41520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffad9b31640 1 -- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad40b5610 msgr2=0x7ffad415f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffad9b31640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad40b5610 0x7ffad415f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffad9b31640 1 -- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad40d05d0 msgr2=0x7ffad41520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffad9b31640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad40d05d0 0x7ffad41520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffad9b31640 1 -- 172.21.15.28:0/2491271922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad4152780 con 0x7ffad415bf30 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffad9b31640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad415bf30 0x7ffad41f5710 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00c770 tx=0x7ffacc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffac9ffb640 1 -- 172.21.15.28:0/2491271922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc017070 con 0x7ffad415bf30 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffac9ffb640 1 -- 172.21.15.28:0/2491271922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffacc005070 con 0x7ffad415bf30 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad4156dd0 con 0x7ffad415bf30 2024-09-10T20:42:27.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffac9ffb640 1 -- 172.21.15.28:0/2491271922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc0054c0 con 0x7ffad415bf30 2024-09-10T20:42:27.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.738+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad41578b0 con 0x7ffad415bf30 2024-09-10T20:42:27.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.739+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad40b6f60 con 0x7ffad415bf30 2024-09-10T20:42:27.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.740+0000 7ffac9ffb640 1 -- 172.21.15.28:0/2491271922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7ffacc005660 con 0x7ffad415bf30 2024-09-10T20:42:27.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.740+0000 7ffac9ffb640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffab8078090 0x7ffab807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:27.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.741+0000 7ffac9ffb640 1 -- 172.21.15.28:0/2491271922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7ffacc099990 con 0x7ffad415bf30 2024-09-10T20:42:27.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.741+0000 7ffad9330640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffab8078090 0x7ffab807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:27.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.741+0000 7ffad9330640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffab8078090 0x7ffab807a550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffad0006fd0 tx=0x7ffad0008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:27.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.745+0000 7ffac9ffb640 1 -- 172.21.15.28:0/2491271922 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffacc062260 con 0x7ffad415bf30 2024-09-10T20:42:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.930+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffad40cbf30 con 0x7ffad415bf30 2024-09-10T20:42:27.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.931+0000 7ffac9ffb640 1 -- 172.21.15.28:0/2491271922 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16732 (secure 0 0 0) 0x7ffacc061b60 con 0x7ffad415bf30 2024-09-10T20:42:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:25.239594+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:21.256150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:22.565350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffab8078090 msgr2=0x7ffab807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffab8078090 0x7ffab807a550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffad0006fd0 tx=0x7ffad0008040 comp rx=0 tx=0).stop 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad415bf30 msgr2=0x7ffad41f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad415bf30 0x7ffad41f5710 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00c770 tx=0x7ffacc00cc30 comp rx=0 tx=0).stop 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 shutdown_connections 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffab8078090 0x7ffab807a550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffad415bf30 0x7ffad41f5710 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffad40d05d0 0x7ffad41520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 --2- 172.21.15.28:0/2491271922 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffad40b5610 0x7ffad415f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 >> 172.21.15.28:0/2491271922 conn(0x7ffad40d1e40 msgr2=0x7ffad4071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 shutdown_connections 2024-09-10T20:42:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:27.934+0000 7ffadb594640 1 -- 172.21.15.28:0/2491271922 wait complete. 2024-09-10T20:42:27.949 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:42:27.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:42:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 -- 172.21.15.28:0/193720457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 msgr2=0x7fccd415e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 --2- 172.21.15.28:0/193720457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd415e7e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fccd4074360 tx=0x7fccc802a5f0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 -- 172.21.15.28:0/193720457 shutdown_connections 2024-09-10T20:42:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 --2- 172.21.15.28:0/193720457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd415e7e0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 --2- 172.21.15.28:0/193720457 >> v2:172.21.15.159:3300/0 conn(0x7fccd4151cc0 0x7fccd40cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 -- 172.21.15.28:0/193720457 >> 172.21.15.28:0/193720457 conn(0x7fccd40d0420 msgr2=0x7fccd40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 -- 172.21.15.28:0/193720457 shutdown_connections 2024-09-10T20:42:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.050+0000 7fccda087640 1 -- 172.21.15.28:0/193720457 wait complete. 2024-09-10T20:42:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.051+0000 7fccda087640 1 Processor -- start 2024-09-10T20:42:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccda087640 1 -- start start 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccda087640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd4147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccda087640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccd4073cf0 0x7fccd41f3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccda087640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccd4151cc0 0x7fccd4156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccda087640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fccd40cb920 con 0x7fccd40738b0 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccda087640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fccd415f640 con 0x7fccd4151cc0 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccda087640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fccd40cbaa0 con 0x7fccd4073cf0 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd4147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd4147840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41680/0 (socket says 172.21.15.28:41680) 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccd37fe640 1 -- 172.21.15.28:0/3851774243 learned_addr learned my addr 172.21.15.28:0/3851774243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.052+0000 7fccd2ffd640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccd4073cf0 0x7fccd41f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccd3fff640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccd4151cc0 0x7fccd4156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccd37fe640 1 -- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccd4073cf0 msgr2=0x7fccd41f3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccd37fe640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccd4073cf0 0x7fccd41f3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccd37fe640 1 -- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccd4151cc0 msgr2=0x7fccd4156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccd37fe640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccd4151cc0 0x7fccd4156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccd37fe640 1 -- 172.21.15.28:0/3851774243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd41f6600 con 0x7fccd40738b0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccd37fe640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd4147840 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fccc400d700 tx=0x7fccc400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccbbfff640 1 -- 172.21.15.28:0/3851774243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4017070 con 0x7fccd40738b0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccbbfff640 1 -- 172.21.15.28:0/3851774243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fccc4005310 con 0x7fccd40738b0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd41f68e0 con 0x7fccd40738b0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccbbfff640 1 -- 172.21.15.28:0/3851774243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4009820 con 0x7fccd40738b0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.053+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd41f6ea0 con 0x7fccd40738b0 2024-09-10T20:42:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.054+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd40ce190 con 0x7fccd40738b0 2024-09-10T20:42:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.055+0000 7fccbbfff640 1 -- 172.21.15.28:0/3851774243 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fccc40099c0 con 0x7fccd40738b0 2024-09-10T20:42:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.055+0000 7fccbbfff640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccc0078090 0x7fccc007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.056+0000 7fccbbfff640 1 -- 172.21.15.28:0/3851774243 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fccc409a0e0 con 0x7fccd40738b0 2024-09-10T20:42:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.056+0000 7fccd2ffd640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccc0078090 0x7fccc007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.056+0000 7fccd2ffd640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccc0078090 0x7fccc007a550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccc8033000 tx=0x7fccc8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.060+0000 7fccbbfff640 1 -- 172.21.15.28:0/3851774243 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fccd40ce190 con 0x7fccd40738b0 2024-09-10T20:42:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.240+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccd40cdf90 con 0x7fccc0078090 2024-09-10T20:42:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.244+0000 7fccbbfff640 1 -- 172.21.15.28:0/3851774243 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252076 (secure 0 0 0) 0x7fccd40cdf90 con 0x7fccc0078090 2024-09-10T20:42:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:42:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccc0078090 msgr2=0x7fccc007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccc0078090 0x7fccc007a550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccc8033000 tx=0x7fccc8004730 comp rx=0 tx=0).stop 2024-09-10T20:42:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 msgr2=0x7fccd4147840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd4147840 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fccc400d700 tx=0x7fccc400dbc0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 shutdown_connections 2024-09-10T20:42:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccc0078090 0x7fccc007a550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccd4151cc0 0x7fccd4156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccd4073cf0 0x7fccd41f3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 --2- 172.21.15.28:0/3851774243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd40738b0 0x7fccd4147840 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 >> 172.21.15.28:0/3851774243 conn(0x7fccd40d0420 msgr2=0x7fccd40cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 shutdown_connections 2024-09-10T20:42:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.247+0000 7fccda087640 1 -- 172.21.15.28:0/3851774243 wait complete. 2024-09-10T20:42:28.264 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-10T20:42:26.294009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254072,"kb_used_data":8708,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754720648,"statfs":{"total":773094113280,"available":772833943552,"internally_reserved":0,"allocated":8916992,"data_stored":4470113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.068902"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256527+0000","last_change":"2024-09-10T20:42:24.252046+0000","last_active":"2024-09-10T20:42:24.256527+0000","last_peered":"2024-09-10T20:42:24.256527+0000","last_clean":"2024-09-10T20:42:24.256527+0000","last_became_active":"2024-09-10T20:42:24.251464+0000","last_became_peered":"2024-09-10T20:42:24.251464+0000","last_unstale":"2024-09-10T20:42:24.256527+0000","last_undegraded":"2024-09-10T20:42:24.256527+0000","last_fullsized":"2024-09-10T20:42:24.256527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:49.716152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255934+0000","last_change":"2024-09-10T20:42:24.249553+0000","last_active":"2024-09-10T20:42:24.255934+0000","last_peered":"2024-09-10T20:42:24.255934+0000","last_clean":"2024-09-10T20:42:24.255934+0000","last_became_active":"2024-09-10T20:42:24.249206+0000","last_became_peered":"2024-09-10T20:42:24.249206+0000","last_unstale":"2024-09-10T20:42:24.255934+0000","last_undegraded":"2024-09-10T20:42:24.255934+0000","last_fullsized":"2024-09-10T20:42:24.255934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:56:52.562320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264425+0000","last_change":"2024-09-10T20:42:24.263659+0000","last_active":"2024-09-10T20:42:24.264425+0000","last_peered":"2024-09-10T20:42:24.264425+0000","last_clean":"2024-09-10T20:42:24.264425+0000","last_became_active":"2024-09-10T20:42:24.263351+0000","last_became_peered":"2024-09-10T20:42:24.263351+0000","last_unstale":"2024-09-10T20:42:24.264425+0000","last_undegraded":"2024-09-10T20:42:24.264425+0000","last_fullsized":"2024-09-10T20:42:24.264425+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:28.917202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255149+0000","last_change":"2024-09-10T20:42:24.248059+0000","last_active":"2024-09-10T20:42:24.255149+0000","last_peered":"2024-09-10T20:42:24.255149+0000","last_clean":"2024-09-10T20:42:24.255149+0000","last_became_active":"2024-09-10T20:42:24.247750+0000","last_became_peered":"2024-09-10T20:42:24.247750+0000","last_unstale":"2024-09-10T20:42:24.255149+0000","last_undegraded":"2024-09-10T20:42:24.255149+0000","last_fullsized":"2024-09-10T20:42:24.255149+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:34:20.849386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260866+0000","last_change":"2024-09-10T20:42:24.250941+0000","last_active":"2024-09-10T20:42:24.260866+0000","last_peered":"2024-09-10T20:42:24.260866+0000","last_clean":"2024-09-10T20:42:24.260866+0000","last_became_active":"2024-09-10T20:42:24.250646+0000","last_became_peered":"2024-09-10T20:42:24.250646+0000","last_unstale":"2024-09-10T20:42:24.260866+0000","last_undegraded":"2024-09-10T20:42:24.260866+0000","last_fullsized":"2024-09-10T20:42:24.260866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:04.772094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256294+0000","last_change":"2024-09-10T20:42:24.250491+0000","last_active":"2024-09-10T20:42:24.256294+0000","last_peered":"2024-09-10T20:42:24.256294+0000","last_clean":"2024-09-10T20:42:24.256294+0000","last_became_active":"2024-09-10T20:42:24.250197+0000","last_became_peered":"2024-09-10T20:42:24.250197+0000","last_unstale":"2024-09-10T20:42:24.256294+0000","last_undegraded":"2024-09-10T20:42:24.256294+0000","last_fullsized":"2024-09-10T20:42:24.256294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:31:05.117623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.252338+0000","last_change":"2024-09-10T20:42:24.246602+0000","last_active":"2024-09-10T20:42:24.252338+0000","last_peered":"2024-09-10T20:42:24.252338+0000","last_clean":"2024-09-10T20:42:24.252338+0000","last_became_active":"2024-09-10T20:42:24.246346+0000","last_became_peered":"2024-09-10T20:42:24.246346+0000","last_unstale":"2024-09-10T20:42:24.252338+0000","last_undegraded":"2024-09-10T20:42:24.252338+0000","last_fullsized":"2024-09-10T20:42:24.252338+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:16:50.457371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264517+0000","last_change":"2024-09-10T20:42:24.261493+0000","last_active":"2024-09-10T20:42:24.264517+0000","last_peered":"2024-09-10T20:42:24.264517+0000","last_clean":"2024-09-10T20:42:24.264517+0000","last_became_active":"2024-09-10T20:42:24.261160+0000","last_became_peered":"2024-09-10T20:42:24.261160+0000","last_unstale":"2024-09-10T20:42:24.264517+0000","last_undegraded":"2024-09-10T20:42:24.264517+0000","last_fullsized":"2024-09-10T20:42:24.264517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:41.626813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.246813+0000","last_change":"2024-09-10T20:42:24.243110+0000","last_active":"2024-09-10T20:42:24.246813+0000","last_peered":"2024-09-10T20:42:24.246813+0000","last_clean":"2024-09-10T20:42:24.246813+0000","last_became_active":"2024-09-10T20:42:24.242832+0000","last_became_peered":"2024-09-10T20:42:24.242832+0000","last_unstale":"2024-09-10T20:42:24.246813+0000","last_undegraded":"2024-09-10T20:42:24.246813+0000","last_fullsized":"2024-09-10T20:42:24.246813+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:13.194381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.257084+0000","last_change":"2024-09-10T20:42:24.253465+0000","last_active":"2024-09-10T20:42:24.257084+0000","last_peered":"2024-09-10T20:42:24.257084+0000","last_clean":"2024-09-10T20:42:24.257084+0000","last_became_active":"2024-09-10T20:42:24.253175+0000","last_became_peered":"2024-09-10T20:42:24.253175+0000","last_unstale":"2024-09-10T20:42:24.257084+0000","last_undegraded":"2024-09-10T20:42:24.257084+0000","last_fullsized":"2024-09-10T20:42:24.257084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:41.626813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260983+0000","last_change":"2024-09-10T20:42:24.255079+0000","last_active":"2024-09-10T20:42:24.260983+0000","last_peered":"2024-09-10T20:42:24.260983+0000","last_clean":"2024-09-10T20:42:24.260983+0000","last_became_active":"2024-09-10T20:42:24.254591+0000","last_became_peered":"2024-09-10T20:42:24.254591+0000","last_unstale":"2024-09-10T20:42:24.260983+0000","last_undegraded":"2024-09-10T20:42:24.260983+0000","last_fullsized":"2024-09-10T20:42:24.260983+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:21:08.468715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262382+0000","last_change":"2024-09-10T20:42:24.255464+0000","last_active":"2024-09-10T20:42:24.262382+0000","last_peered":"2024-09-10T20:42:24.262382+0000","last_clean":"2024-09-10T20:42:24.262382+0000","last_became_active":"2024-09-10T20:42:24.255056+0000","last_became_peered":"2024-09-10T20:42:24.255056+0000","last_unstale":"2024-09-10T20:42:24.262382+0000","last_undegraded":"2024-09-10T20:42:24.262382+0000","last_fullsized":"2024-09-10T20:42:24.262382+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:56.600475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266217+0000","last_change":"2024-09-10T20:42:24.260343+0000","last_active":"2024-09-10T20:42:24.266217+0000","last_peered":"2024-09-10T20:42:24.266217+0000","last_clean":"2024-09-10T20:42:24.266217+0000","last_became_active":"2024-09-10T20:42:24.259812+0000","last_became_peered":"2024-09-10T20:42:24.259812+0000","last_unstale":"2024-09-10T20:42:24.266217+0000","last_undegraded":"2024-09-10T20:42:24.266217+0000","last_fullsized":"2024-09-10T20:42:24.266217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:15.203081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.249456+0000","last_change":"2024-09-10T20:42:24.245659+0000","last_active":"2024-09-10T20:42:24.249456+0000","last_peered":"2024-09-10T20:42:24.249456+0000","last_clean":"2024-09-10T20:42:24.249456+0000","last_became_active":"2024-09-10T20:42:24.245357+0000","last_became_peered":"2024-09-10T20:42:24.245357+0000","last_unstale":"2024-09-10T20:42:24.249456+0000","last_undegraded":"2024-09-10T20:42:24.249456+0000","last_fullsized":"2024-09-10T20:42:24.249456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:49.739293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.249939+0000","last_change":"2024-09-10T20:42:24.241653+0000","last_active":"2024-09-10T20:42:24.249939+0000","last_peered":"2024-09-10T20:42:24.249939+0000","last_clean":"2024-09-10T20:42:24.249939+0000","last_became_active":"2024-09-10T20:42:24.241359+0000","last_became_peered":"2024-09-10T20:42:24.241359+0000","last_unstale":"2024-09-10T20:42:24.249939+0000","last_undegraded":"2024-09-10T20:42:24.249939+0000","last_fullsized":"2024-09-10T20:42:24.249939+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:17:23.346363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256360+0000","last_change":"2024-09-10T20:42:24.251074+0000","last_active":"2024-09-10T20:42:24.256360+0000","last_peered":"2024-09-10T20:42:24.256360+0000","last_clean":"2024-09-10T20:42:24.256360+0000","last_became_active":"2024-09-10T20:42:24.250784+0000","last_became_peered":"2024-09-10T20:42:24.250784+0000","last_unstale":"2024-09-10T20:42:24.256360+0000","last_undegraded":"2024-09-10T20:42:24.256360+0000","last_fullsized":"2024-09-10T20:42:24.256360+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:39.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.259069+0000","last_change":"2024-09-10T20:42:24.254012+0000","last_active":"2024-09-10T20:42:24.259069+0000","last_peered":"2024-09-10T20:42:24.259069+0000","last_clean":"2024-09-10T20:42:24.259069+0000","last_became_active":"2024-09-10T20:42:24.253691+0000","last_became_peered":"2024-09-10T20:42:24.253691+0000","last_unstale":"2024-09-10T20:42:24.259069+0000","last_undegraded":"2024-09-10T20:42:24.259069+0000","last_fullsized":"2024-09-10T20:42:24.259069+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:51.778063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260855+0000","last_change":"2024-09-10T20:42:24.253511+0000","last_active":"2024-09-10T20:42:24.260855+0000","last_peered":"2024-09-10T20:42:24.260855+0000","last_clean":"2024-09-10T20:42:24.260855+0000","last_became_active":"2024-09-10T20:42:24.253082+0000","last_became_peered":"2024-09-10T20:42:24.253082+0000","last_unstale":"2024-09-10T20:42:24.260855+0000","last_undegraded":"2024-09-10T20:42:24.260855+0000","last_fullsized":"2024-09-10T20:42:24.260855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:39.725236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.252400+0000","last_change":"2024-09-10T20:42:24.248623+0000","last_active":"2024-09-10T20:42:24.252400+0000","last_peered":"2024-09-10T20:42:24.252400+0000","last_clean":"2024-09-10T20:42:24.252400+0000","last_became_active":"2024-09-10T20:42:24.248380+0000","last_became_peered":"2024-09-10T20:42:24.248380+0000","last_unstale":"2024-09-10T20:42:24.252400+0000","last_undegraded":"2024-09-10T20:42:24.252400+0000","last_fullsized":"2024-09-10T20:42:24.252400+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:34.794709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.254730+0000","last_change":"2024-09-10T20:42:24.249339+0000","last_active":"2024-09-10T20:42:24.254730+0000","last_peered":"2024-09-10T20:42:24.254730+0000","last_clean":"2024-09-10T20:42:24.254730+0000","last_became_active":"2024-09-10T20:42:24.249080+0000","last_became_peered":"2024-09-10T20:42:24.249080+0000","last_unstale":"2024-09-10T20:42:24.254730+0000","last_undegraded":"2024-09-10T20:42:24.254730+0000","last_fullsized":"2024-09-10T20:42:24.254730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:20:14.247696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.244998+0000","last_change":"2024-09-10T20:42:24.240390+0000","last_active":"2024-09-10T20:42:24.244998+0000","last_peered":"2024-09-10T20:42:24.244998+0000","last_clean":"2024-09-10T20:42:24.244998+0000","last_became_active":"2024-09-10T20:42:24.239950+0000","last_became_peered":"2024-09-10T20:42:24.239950+0000","last_unstale":"2024-09-10T20:42:24.244998+0000","last_undegraded":"2024-09-10T20:42:24.244998+0000","last_fullsized":"2024-09-10T20:42:24.244998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:31:16.945421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255104+0000","last_change":"2024-09-10T20:42:24.250755+0000","last_active":"2024-09-10T20:42:24.255104+0000","last_peered":"2024-09-10T20:42:24.255104+0000","last_clean":"2024-09-10T20:42:24.255104+0000","last_became_active":"2024-09-10T20:42:24.250468+0000","last_became_peered":"2024-09-10T20:42:24.250468+0000","last_unstale":"2024-09-10T20:42:24.255104+0000","last_undegraded":"2024-09-10T20:42:24.255104+0000","last_fullsized":"2024-09-10T20:42:24.255104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:57:37.616646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262510+0000","last_change":"2024-09-10T20:42:24.259183+0000","last_active":"2024-09-10T20:42:24.262510+0000","last_peered":"2024-09-10T20:42:24.262510+0000","last_clean":"2024-09-10T20:42:24.262510+0000","last_became_active":"2024-09-10T20:42:24.258885+0000","last_became_peered":"2024-09-10T20:42:24.258885+0000","last_unstale":"2024-09-10T20:42:24.262510+0000","last_undegraded":"2024-09-10T20:42:24.262510+0000","last_fullsized":"2024-09-10T20:42:24.262510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:39.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255964+0000","last_change":"2024-09-10T20:42:24.250947+0000","last_active":"2024-09-10T20:42:24.255964+0000","last_peered":"2024-09-10T20:42:24.255964+0000","last_clean":"2024-09-10T20:42:24.255964+0000","last_became_active":"2024-09-10T20:42:24.250691+0000","last_became_peered":"2024-09-10T20:42:24.250691+0000","last_unstale":"2024-09-10T20:42:24.255964+0000","last_undegraded":"2024-09-10T20:42:24.255964+0000","last_fullsized":"2024-09-10T20:42:24.255964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:42:22.244222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.269135+0000","last_change":"2024-09-10T20:42:24.267779+0000","last_active":"2024-09-10T20:42:24.269135+0000","last_peered":"2024-09-10T20:42:24.269135+0000","last_clean":"2024-09-10T20:42:24.269135+0000","last_became_active":"2024-09-10T20:42:24.267405+0000","last_became_peered":"2024-09-10T20:42:24.267405+0000","last_unstale":"2024-09-10T20:42:24.269135+0000","last_undegraded":"2024-09-10T20:42:24.269135+0000","last_fullsized":"2024-09-10T20:42:24.269135+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:15.449103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266230+0000","last_change":"2024-09-10T20:42:24.261575+0000","last_active":"2024-09-10T20:42:24.266230+0000","last_peered":"2024-09-10T20:42:24.266230+0000","last_clean":"2024-09-10T20:42:24.266230+0000","last_became_active":"2024-09-10T20:42:24.261238+0000","last_became_peered":"2024-09-10T20:42:24.261238+0000","last_unstale":"2024-09-10T20:42:24.266230+0000","last_undegraded":"2024-09-10T20:42:24.266230+0000","last_fullsized":"2024-09-10T20:42:24.266230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:20.623882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260240+0000","last_change":"2024-09-10T20:42:24.251130+0000","last_active":"2024-09-10T20:42:24.260240+0000","last_peered":"2024-09-10T20:42:24.260240+0000","last_clean":"2024-09-10T20:42:24.260240+0000","last_became_active":"2024-09-10T20:42:24.250824+0000","last_became_peered":"2024-09-10T20:42:24.250824+0000","last_unstale":"2024-09-10T20:42:24.260240+0000","last_undegraded":"2024-09-10T20:42:24.260240+0000","last_fullsized":"2024-09-10T20:42:24.260240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:44:49.605060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266421+0000","last_change":"2024-09-10T20:42:24.265003+0000","last_active":"2024-09-10T20:42:24.266421+0000","last_peered":"2024-09-10T20:42:24.266421+0000","last_clean":"2024-09-10T20:42:24.266421+0000","last_became_active":"2024-09-10T20:42:24.264690+0000","last_became_peered":"2024-09-10T20:42:24.264690+0000","last_unstale":"2024-09-10T20:42:24.266421+0000","last_undegraded":"2024-09-10T20:42:24.266421+0000","last_fullsized":"2024-09-10T20:42:24.266421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:30:08.734512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.247929+0000","last_change":"2024-09-10T20:42:24.242217+0000","last_active":"2024-09-10T20:42:24.247929+0000","last_peered":"2024-09-10T20:42:24.247929+0000","last_clean":"2024-09-10T20:42:24.247929+0000","last_became_active":"2024-09-10T20:42:24.241929+0000","last_became_peered":"2024-09-10T20:42:24.241929+0000","last_unstale":"2024-09-10T20:42:24.247929+0000","last_undegraded":"2024-09-10T20:42:24.247929+0000","last_fullsized":"2024-09-10T20:42:24.247929+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:58:10.673710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255056+0000","last_change":"2024-09-10T20:42:24.249606+0000","last_active":"2024-09-10T20:42:24.255056+0000","last_peered":"2024-09-10T20:42:24.255056+0000","last_clean":"2024-09-10T20:42:24.255056+0000","last_became_active":"2024-09-10T20:42:24.249288+0000","last_became_peered":"2024-09-10T20:42:24.249288+0000","last_unstale":"2024-09-10T20:42:24.255056+0000","last_undegraded":"2024-09-10T20:42:24.255056+0000","last_fullsized":"2024-09-10T20:42:24.255056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:18:13.328287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255988+0000","last_change":"2024-09-10T20:42:24.249671+0000","last_active":"2024-09-10T20:42:24.255988+0000","last_peered":"2024-09-10T20:42:24.255988+0000","last_clean":"2024-09-10T20:42:24.255988+0000","last_became_active":"2024-09-10T20:42:24.249396+0000","last_became_peered":"2024-09-10T20:42:24.249396+0000","last_unstale":"2024-09-10T20:42:24.255988+0000","last_undegraded":"2024-09-10T20:42:24.255988+0000","last_fullsized":"2024-09-10T20:42:24.255988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:15.234387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256336+0000","last_change":"2024-09-10T20:42:24.249638+0000","last_active":"2024-09-10T20:42:24.256336+0000","last_peered":"2024-09-10T20:42:24.256336+0000","last_clean":"2024-09-10T20:42:24.256336+0000","last_became_active":"2024-09-10T20:42:24.249341+0000","last_became_peered":"2024-09-10T20:42:24.249341+0000","last_unstale":"2024-09-10T20:42:24.256336+0000","last_undegraded":"2024-09-10T20:42:24.256336+0000","last_fullsized":"2024-09-10T20:42:24.256336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:56.600475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.268443+0000","last_change":"2024-09-10T20:42:24.267438+0000","last_active":"2024-09-10T20:42:24.268443+0000","last_peered":"2024-09-10T20:42:24.268443+0000","last_clean":"2024-09-10T20:42:24.268443+0000","last_became_active":"2024-09-10T20:42:24.267159+0000","last_became_peered":"2024-09-10T20:42:24.267159+0000","last_unstale":"2024-09-10T20:42:24.268443+0000","last_undegraded":"2024-09-10T20:42:24.268443+0000","last_fullsized":"2024-09-10T20:42:24.268443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:28.917202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.269313+0000","last_change":"2024-09-10T20:42:23.253219+0000","last_active":"2024-09-10T20:42:24.269313+0000","last_peered":"2024-09-10T20:42:24.269313+0000","last_clean":"2024-09-10T20:42:24.269313+0000","last_became_active":"2024-09-10T20:42:23.252705+0000","last_became_peered":"2024-09-10T20:42:23.252705+0000","last_unstale":"2024-09-10T20:42:24.269313+0000","last_undegraded":"2024-09-10T20:42:24.269313+0000","last_fullsized":"2024-09-10T20:42:24.269313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:04:21.695369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256497+0000","last_change":"2024-09-10T20:42:24.251751+0000","last_active":"2024-09-10T20:42:24.256497+0000","last_peered":"2024-09-10T20:42:24.256497+0000","last_clean":"2024-09-10T20:42:24.256497+0000","last_became_active":"2024-09-10T20:42:24.251088+0000","last_became_peered":"2024-09-10T20:42:24.251088+0000","last_unstale":"2024-09-10T20:42:24.256497+0000","last_undegraded":"2024-09-10T20:42:24.256497+0000","last_fullsized":"2024-09-10T20:42:24.256497+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:59.741077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267082+0000","last_change":"2024-09-10T20:42:23.250981+0000","last_active":"2024-09-10T20:42:24.267082+0000","last_peered":"2024-09-10T20:42:24.267082+0000","last_clean":"2024-09-10T20:42:24.267082+0000","last_became_active":"2024-09-10T20:42:23.250669+0000","last_became_peered":"2024-09-10T20:42:23.250669+0000","last_unstale":"2024-09-10T20:42:24.267082+0000","last_undegraded":"2024-09-10T20:42:24.267082+0000","last_fullsized":"2024-09-10T20:42:24.267082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:03:04.402951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262423+0000","last_change":"2024-09-10T20:42:24.258631+0000","last_active":"2024-09-10T20:42:24.262423+0000","last_peered":"2024-09-10T20:42:24.262423+0000","last_clean":"2024-09-10T20:42:24.262423+0000","last_became_active":"2024-09-10T20:42:24.258293+0000","last_became_peered":"2024-09-10T20:42:24.258293+0000","last_unstale":"2024-09-10T20:42:24.262423+0000","last_undegraded":"2024-09-10T20:42:24.262423+0000","last_fullsized":"2024-09-10T20:42:24.262423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:15.234387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264222+0000","last_change":"2024-09-10T20:42:23.243429+0000","last_active":"2024-09-10T20:42:24.264222+0000","last_peered":"2024-09-10T20:42:24.264222+0000","last_clean":"2024-09-10T20:42:24.264222+0000","last_became_active":"2024-09-10T20:42:23.242486+0000","last_became_peered":"2024-09-10T20:42:23.242486+0000","last_unstale":"2024-09-10T20:42:24.264222+0000","last_undegraded":"2024-09-10T20:42:24.264222+0000","last_fullsized":"2024-09-10T20:42:24.264222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:58.738335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.259278+0000","last_change":"2024-09-10T20:42:24.257400+0000","last_active":"2024-09-10T20:42:24.259278+0000","last_peered":"2024-09-10T20:42:24.259278+0000","last_clean":"2024-09-10T20:42:24.259278+0000","last_became_active":"2024-09-10T20:42:24.256006+0000","last_became_peered":"2024-09-10T20:42:24.256006+0000","last_unstale":"2024-09-10T20:42:24.259278+0000","last_undegraded":"2024-09-10T20:42:24.259278+0000","last_fullsized":"2024-09-10T20:42:24.259278+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:03:04.919447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266443+0000","last_change":"2024-09-10T20:42:23.247283+0000","last_active":"2024-09-10T20:42:24.266443+0000","last_peered":"2024-09-10T20:42:24.266443+0000","last_clean":"2024-09-10T20:42:24.266443+0000","last_became_active":"2024-09-10T20:42:23.246969+0000","last_became_peered":"2024-09-10T20:42:23.246969+0000","last_unstale":"2024-09-10T20:42:24.266443+0000","last_undegraded":"2024-09-10T20:42:24.266443+0000","last_fullsized":"2024-09-10T20:42:24.266443+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:29:13.545500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256001+0000","last_change":"2024-09-10T20:42:24.251220+0000","last_active":"2024-09-10T20:42:24.256001+0000","last_peered":"2024-09-10T20:42:24.256001+0000","last_clean":"2024-09-10T20:42:24.256001+0000","last_became_active":"2024-09-10T20:42:24.250688+0000","last_became_peered":"2024-09-10T20:42:24.250688+0000","last_unstale":"2024-09-10T20:42:24.256001+0000","last_undegraded":"2024-09-10T20:42:24.256001+0000","last_fullsized":"2024-09-10T20:42:24.256001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:37:40.196769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262867+0000","last_change":"2024-09-10T20:42:23.249368+0000","last_active":"2024-09-10T20:42:24.262867+0000","last_peered":"2024-09-10T20:42:24.262867+0000","last_clean":"2024-09-10T20:42:24.262867+0000","last_became_active":"2024-09-10T20:42:23.249075+0000","last_became_peered":"2024-09-10T20:42:23.249075+0000","last_unstale":"2024-09-10T20:42:24.262867+0000","last_undegraded":"2024-09-10T20:42:24.262867+0000","last_fullsized":"2024-09-10T20:42:24.262867+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:50.459124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.253510+0000","last_change":"2024-09-10T20:42:24.248311+0000","last_active":"2024-09-10T20:42:24.253510+0000","last_peered":"2024-09-10T20:42:24.253510+0000","last_clean":"2024-09-10T20:42:24.253510+0000","last_became_active":"2024-09-10T20:42:24.247053+0000","last_became_peered":"2024-09-10T20:42:24.247053+0000","last_unstale":"2024-09-10T20:42:24.253510+0000","last_undegraded":"2024-09-10T20:42:24.253510+0000","last_fullsized":"2024-09-10T20:42:24.253510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:51.778063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267026+0000","last_change":"2024-09-10T20:42:23.245633+0000","last_active":"2024-09-10T20:42:24.267026+0000","last_peered":"2024-09-10T20:42:24.267026+0000","last_clean":"2024-09-10T20:42:24.267026+0000","last_became_active":"2024-09-10T20:42:23.245376+0000","last_became_peered":"2024-09-10T20:42:23.245376+0000","last_unstale":"2024-09-10T20:42:24.267026+0000","last_undegraded":"2024-09-10T20:42:24.267026+0000","last_fullsized":"2024-09-10T20:42:24.267026+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:06.736064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.257394+0000","last_change":"2024-09-10T20:42:24.256670+0000","last_active":"2024-09-10T20:42:24.257394+0000","last_peered":"2024-09-10T20:42:24.257394+0000","last_clean":"2024-09-10T20:42:24.257394+0000","last_became_active":"2024-09-10T20:42:24.256293+0000","last_became_peered":"2024-09-10T20:42:24.256293+0000","last_unstale":"2024-09-10T20:42:24.257394+0000","last_undegraded":"2024-09-10T20:42:24.257394+0000","last_fullsized":"2024-09-10T20:42:24.257394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:42:22.244222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.239377+0000","last_change":"2024-09-10T20:42:23.249360+0000","last_active":"2024-09-10T20:42:24.239377+0000","last_peered":"2024-09-10T20:42:24.239377+0000","last_clean":"2024-09-10T20:42:24.239377+0000","last_became_active":"2024-09-10T20:42:23.249041+0000","last_became_peered":"2024-09-10T20:42:23.249041+0000","last_unstale":"2024-09-10T20:42:24.239377+0000","last_undegraded":"2024-09-10T20:42:24.239377+0000","last_fullsized":"2024-09-10T20:42:24.239377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:54.690307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.265477+0000","last_change":"2024-09-10T20:42:24.263533+0000","last_active":"2024-09-10T20:42:24.265477+0000","last_peered":"2024-09-10T20:42:24.265477+0000","last_clean":"2024-09-10T20:42:24.265477+0000","last_became_active":"2024-09-10T20:42:24.263259+0000","last_became_peered":"2024-09-10T20:42:24.263259+0000","last_unstale":"2024-09-10T20:42:24.265477+0000","last_undegraded":"2024-09-10T20:42:24.265477+0000","last_fullsized":"2024-09-10T20:42:24.265477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:20:24.107503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.239130+0000","last_change":"2024-09-10T20:42:23.253926+0000","last_active":"2024-09-10T20:42:24.239130+0000","last_peered":"2024-09-10T20:42:24.239130+0000","last_clean":"2024-09-10T20:42:24.239130+0000","last_became_active":"2024-09-10T20:42:23.253668+0000","last_became_peered":"2024-09-10T20:42:23.253668+0000","last_unstale":"2024-09-10T20:42:24.239130+0000","last_undegraded":"2024-09-10T20:42:24.239130+0000","last_fullsized":"2024-09-10T20:42:24.239130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:50.409359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255565+0000","last_change":"2024-09-10T20:42:24.253189+0000","last_active":"2024-09-10T20:42:24.255565+0000","last_peered":"2024-09-10T20:42:24.255565+0000","last_clean":"2024-09-10T20:42:24.255565+0000","last_became_active":"2024-09-10T20:42:24.252850+0000","last_became_peered":"2024-09-10T20:42:24.252850+0000","last_unstale":"2024-09-10T20:42:24.255565+0000","last_undegraded":"2024-09-10T20:42:24.255565+0000","last_fullsized":"2024-09-10T20:42:24.255565+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:29:41.776801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.237102+0000","last_change":"2024-09-10T20:42:23.245177+0000","last_active":"2024-09-10T20:42:24.237102+0000","last_peered":"2024-09-10T20:42:24.237102+0000","last_clean":"2024-09-10T20:42:24.237102+0000","last_became_active":"2024-09-10T20:42:23.244736+0000","last_became_peered":"2024-09-10T20:42:23.244736+0000","last_unstale":"2024-09-10T20:42:24.237102+0000","last_undegraded":"2024-09-10T20:42:24.237102+0000","last_fullsized":"2024-09-10T20:42:24.237102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:38:17.077537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256552+0000","last_change":"2024-09-10T20:42:24.253781+0000","last_active":"2024-09-10T20:42:24.256552+0000","last_peered":"2024-09-10T20:42:24.256552+0000","last_clean":"2024-09-10T20:42:24.256552+0000","last_became_active":"2024-09-10T20:42:24.253524+0000","last_became_peered":"2024-09-10T20:42:24.253524+0000","last_unstale":"2024-09-10T20:42:24.256552+0000","last_undegraded":"2024-09-10T20:42:24.256552+0000","last_fullsized":"2024-09-10T20:42:24.256552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:25.940389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.243006+0000","last_change":"2024-09-10T20:42:23.247121+0000","last_active":"2024-09-10T20:42:24.243006+0000","last_peered":"2024-09-10T20:42:24.243006+0000","last_clean":"2024-09-10T20:42:24.243006+0000","last_became_active":"2024-09-10T20:42:23.246816+0000","last_became_peered":"2024-09-10T20:42:23.246816+0000","last_unstale":"2024-09-10T20:42:24.243006+0000","last_undegraded":"2024-09-10T20:42:24.243006+0000","last_fullsized":"2024-09-10T20:42:24.243006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:15.265502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.268806+0000","last_change":"2024-09-10T20:42:24.262104+0000","last_active":"2024-09-10T20:42:24.268806+0000","last_peered":"2024-09-10T20:42:24.268806+0000","last_clean":"2024-09-10T20:42:24.268806+0000","last_became_active":"2024-09-10T20:42:24.261758+0000","last_became_peered":"2024-09-10T20:42:24.261758+0000","last_unstale":"2024-09-10T20:42:24.268806+0000","last_undegraded":"2024-09-10T20:42:24.268806+0000","last_fullsized":"2024-09-10T20:42:24.268806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:07.738806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.263688+0000","last_change":"2024-09-10T20:42:23.251025+0000","last_active":"2024-09-10T20:42:24.263688+0000","last_peered":"2024-09-10T20:42:24.263688+0000","last_clean":"2024-09-10T20:42:24.263688+0000","last_became_active":"2024-09-10T20:42:23.250757+0000","last_became_peered":"2024-09-10T20:42:23.250757+0000","last_unstale":"2024-09-10T20:42:24.263688+0000","last_undegraded":"2024-09-10T20:42:24.263688+0000","last_fullsized":"2024-09-10T20:42:24.263688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:27:59.341727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264579+0000","last_change":"2024-09-10T20:42:24.260041+0000","last_active":"2024-09-10T20:42:24.264579+0000","last_peered":"2024-09-10T20:42:24.264579+0000","last_clean":"2024-09-10T20:42:24.264579+0000","last_became_active":"2024-09-10T20:42:24.259715+0000","last_became_peered":"2024-09-10T20:42:24.259715+0000","last_unstale":"2024-09-10T20:42:24.264579+0000","last_undegraded":"2024-09-10T20:42:24.264579+0000","last_fullsized":"2024-09-10T20:42:24.264579+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:55.967204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266500+0000","last_change":"2024-09-10T20:42:23.241836+0000","last_active":"2024-09-10T20:42:24.266500+0000","last_peered":"2024-09-10T20:42:24.266500+0000","last_clean":"2024-09-10T20:42:24.266500+0000","last_became_active":"2024-09-10T20:42:23.241522+0000","last_became_peered":"2024-09-10T20:42:23.241522+0000","last_unstale":"2024-09-10T20:42:24.266500+0000","last_undegraded":"2024-09-10T20:42:24.266500+0000","last_fullsized":"2024-09-10T20:42:24.266500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:15.265502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262241+0000","last_change":"2024-09-10T20:42:24.260342+0000","last_active":"2024-09-10T20:42:24.262241+0000","last_peered":"2024-09-10T20:42:24.262241+0000","last_clean":"2024-09-10T20:42:24.262241+0000","last_became_active":"2024-09-10T20:42:24.260044+0000","last_became_peered":"2024-09-10T20:42:24.260044+0000","last_unstale":"2024-09-10T20:42:24.262241+0000","last_undegraded":"2024-09-10T20:42:24.262241+0000","last_fullsized":"2024-09-10T20:42:24.262241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:21:34.514066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.265222+0000","last_change":"2024-09-10T20:42:23.244794+0000","last_active":"2024-09-10T20:42:24.265222+0000","last_peered":"2024-09-10T20:42:24.265222+0000","last_clean":"2024-09-10T20:42:24.265222+0000","last_became_active":"2024-09-10T20:42:23.244466+0000","last_became_peered":"2024-09-10T20:42:23.244466+0000","last_unstale":"2024-09-10T20:42:24.265222+0000","last_undegraded":"2024-09-10T20:42:24.265222+0000","last_fullsized":"2024-09-10T20:42:24.265222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:13:02.552327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.259243+0000","last_change":"2024-09-10T20:42:24.257623+0000","last_active":"2024-09-10T20:42:24.259243+0000","last_peered":"2024-09-10T20:42:24.259243+0000","last_clean":"2024-09-10T20:42:24.259243+0000","last_became_active":"2024-09-10T20:42:24.257055+0000","last_became_peered":"2024-09-10T20:42:24.257055+0000","last_unstale":"2024-09-10T20:42:24.259243+0000","last_undegraded":"2024-09-10T20:42:24.259243+0000","last_fullsized":"2024-09-10T20:42:24.259243+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:39.725236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.236403+0000","last_change":"2024-09-10T20:42:23.246728+0000","last_active":"2024-09-10T20:42:24.236403+0000","last_peered":"2024-09-10T20:42:24.236403+0000","last_clean":"2024-09-10T20:42:24.236403+0000","last_became_active":"2024-09-10T20:42:23.245892+0000","last_became_peered":"2024-09-10T20:42:23.245892+0000","last_unstale":"2024-09-10T20:42:24.236403+0000","last_undegraded":"2024-09-10T20:42:24.236403+0000","last_fullsized":"2024-09-10T20:42:24.236403+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:47.440987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255620+0000","last_change":"2024-09-10T20:42:24.253960+0000","last_active":"2024-09-10T20:42:24.255620+0000","last_peered":"2024-09-10T20:42:24.255620+0000","last_clean":"2024-09-10T20:42:24.255620+0000","last_became_active":"2024-09-10T20:42:24.253697+0000","last_became_peered":"2024-09-10T20:42:24.253697+0000","last_unstale":"2024-09-10T20:42:24.255620+0000","last_undegraded":"2024-09-10T20:42:24.255620+0000","last_fullsized":"2024-09-10T20:42:24.255620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:05.073601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.241984+0000","last_change":"2024-09-10T20:42:23.249368+0000","last_active":"2024-09-10T20:42:24.241984+0000","last_peered":"2024-09-10T20:42:24.241984+0000","last_clean":"2024-09-10T20:42:24.241984+0000","last_became_active":"2024-09-10T20:42:23.249095+0000","last_became_peered":"2024-09-10T20:42:23.249095+0000","last_unstale":"2024-09-10T20:42:24.241984+0000","last_undegraded":"2024-09-10T20:42:24.241984+0000","last_fullsized":"2024-09-10T20:42:24.241984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:50.459124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260891+0000","last_change":"2024-09-10T20:42:24.253368+0000","last_active":"2024-09-10T20:42:24.260891+0000","last_peered":"2024-09-10T20:42:24.260891+0000","last_clean":"2024-09-10T20:42:24.260891+0000","last_became_active":"2024-09-10T20:42:24.253019+0000","last_became_peered":"2024-09-10T20:42:24.253019+0000","last_unstale":"2024-09-10T20:42:24.260891+0000","last_undegraded":"2024-09-10T20:42:24.260891+0000","last_fullsized":"2024-09-10T20:42:24.260891+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:03:04.919447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267904+0000","last_change":"2024-09-10T20:42:23.248363+0000","last_active":"2024-09-10T20:42:24.267904+0000","last_peered":"2024-09-10T20:42:24.267904+0000","last_clean":"2024-09-10T20:42:24.267904+0000","last_became_active":"2024-09-10T20:42:23.247975+0000","last_became_peered":"2024-09-10T20:42:23.247975+0000","last_unstale":"2024-09-10T20:42:24.267904+0000","last_undegraded":"2024-09-10T20:42:24.267904+0000","last_fullsized":"2024-09-10T20:42:24.267904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:24.937647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.258804+0000","last_change":"2024-09-10T20:42:24.254541+0000","last_active":"2024-09-10T20:42:24.258804+0000","last_peered":"2024-09-10T20:42:24.258804+0000","last_clean":"2024-09-10T20:42:24.258804+0000","last_became_active":"2024-09-10T20:42:24.254251+0000","last_became_peered":"2024-09-10T20:42:24.254251+0000","last_unstale":"2024-09-10T20:42:24.258804+0000","last_undegraded":"2024-09-10T20:42:24.258804+0000","last_fullsized":"2024-09-10T20:42:24.258804+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:12:41.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266762+0000","last_change":"2024-09-10T20:42:23.252893+0000","last_active":"2024-09-10T20:42:24.266762+0000","last_peered":"2024-09-10T20:42:24.266762+0000","last_clean":"2024-09-10T20:42:24.266762+0000","last_became_active":"2024-09-10T20:42:23.252544+0000","last_became_peered":"2024-09-10T20:42:23.252544+0000","last_unstale":"2024-09-10T20:42:24.266762+0000","last_undegraded":"2024-09-10T20:42:24.266762+0000","last_fullsized":"2024-09-10T20:42:24.266762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:14:55.067011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.253451+0000","last_change":"2024-09-10T20:42:24.248801+0000","last_active":"2024-09-10T20:42:24.253451+0000","last_peered":"2024-09-10T20:42:24.253451+0000","last_clean":"2024-09-10T20:42:24.253451+0000","last_became_active":"2024-09-10T20:42:24.248477+0000","last_became_peered":"2024-09-10T20:42:24.248477+0000","last_unstale":"2024-09-10T20:42:24.253451+0000","last_undegraded":"2024-09-10T20:42:24.253451+0000","last_fullsized":"2024-09-10T20:42:24.253451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:12:41.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.263349+0000","last_change":"2024-09-10T20:42:23.252895+0000","last_active":"2024-09-10T20:42:24.263349+0000","last_peered":"2024-09-10T20:42:24.263349+0000","last_clean":"2024-09-10T20:42:24.263349+0000","last_became_active":"2024-09-10T20:42:23.252592+0000","last_became_peered":"2024-09-10T20:42:23.252592+0000","last_unstale":"2024-09-10T20:42:24.263349+0000","last_undegraded":"2024-09-10T20:42:24.263349+0000","last_fullsized":"2024-09-10T20:42:24.263349+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:12:44.327544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.258704+0000","last_change":"2024-09-10T20:42:24.253114+0000","last_active":"2024-09-10T20:42:24.258704+0000","last_peered":"2024-09-10T20:42:24.258704+0000","last_clean":"2024-09-10T20:42:24.258704+0000","last_became_active":"2024-09-10T20:42:24.252817+0000","last_became_peered":"2024-09-10T20:42:24.252817+0000","last_unstale":"2024-09-10T20:42:24.258704+0000","last_undegraded":"2024-09-10T20:42:24.258704+0000","last_fullsized":"2024-09-10T20:42:24.258704+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:53:10.278617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.240721+0000","last_change":"2024-09-10T20:42:23.250304+0000","last_active":"2024-09-10T20:42:24.240721+0000","last_peered":"2024-09-10T20:42:24.240721+0000","last_clean":"2024-09-10T20:42:24.240721+0000","last_became_active":"2024-09-10T20:42:23.250025+0000","last_became_peered":"2024-09-10T20:42:23.250025+0000","last_unstale":"2024-09-10T20:42:24.240721+0000","last_undegraded":"2024-09-10T20:42:24.240721+0000","last_fullsized":"2024-09-10T20:42:24.240721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:02.302834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255492+0000","last_change":"2024-09-10T20:42:24.251276+0000","last_active":"2024-09-10T20:42:24.255492+0000","last_peered":"2024-09-10T20:42:24.255492+0000","last_clean":"2024-09-10T20:42:24.255492+0000","last_became_active":"2024-09-10T20:42:24.251007+0000","last_became_peered":"2024-09-10T20:42:24.251007+0000","last_unstale":"2024-09-10T20:42:24.255492+0000","last_undegraded":"2024-09-10T20:42:24.255492+0000","last_fullsized":"2024-09-10T20:42:24.255492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:25:51.538368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262294+0000","last_change":"2024-09-10T20:42:23.249350+0000","last_active":"2024-09-10T20:42:24.262294+0000","last_peered":"2024-09-10T20:42:24.262294+0000","last_clean":"2024-09-10T20:42:24.262294+0000","last_became_active":"2024-09-10T20:42:23.249058+0000","last_became_peered":"2024-09-10T20:42:23.249058+0000","last_unstale":"2024-09-10T20:42:24.262294+0000","last_undegraded":"2024-09-10T20:42:24.262294+0000","last_fullsized":"2024-09-10T20:42:24.262294+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:16:10.406185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260892+0000","last_change":"2024-09-10T20:42:24.254958+0000","last_active":"2024-09-10T20:42:24.260892+0000","last_peered":"2024-09-10T20:42:24.260892+0000","last_clean":"2024-09-10T20:42:24.260892+0000","last_became_active":"2024-09-10T20:42:24.254668+0000","last_became_peered":"2024-09-10T20:42:24.254668+0000","last_unstale":"2024-09-10T20:42:24.260892+0000","last_undegraded":"2024-09-10T20:42:24.260892+0000","last_fullsized":"2024-09-10T20:42:24.260892+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:47:33.211289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.236779+0000","last_change":"2024-09-10T20:42:23.246818+0000","last_active":"2024-09-10T20:42:24.236779+0000","last_peered":"2024-09-10T20:42:24.236779+0000","last_clean":"2024-09-10T20:42:24.236779+0000","last_became_active":"2024-09-10T20:42:23.246510+0000","last_became_peered":"2024-09-10T20:42:23.246510+0000","last_unstale":"2024-09-10T20:42:24.236779+0000","last_undegraded":"2024-09-10T20:42:24.236779+0000","last_fullsized":"2024-09-10T20:42:24.236779+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:32.249681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256201+0000","last_change":"2024-09-10T20:42:24.254492+0000","last_active":"2024-09-10T20:42:24.256201+0000","last_peered":"2024-09-10T20:42:24.256201+0000","last_clean":"2024-09-10T20:42:24.256201+0000","last_became_active":"2024-09-10T20:42:24.254157+0000","last_became_peered":"2024-09-10T20:42:24.254157+0000","last_unstale":"2024-09-10T20:42:24.256201+0000","last_undegraded":"2024-09-10T20:42:24.256201+0000","last_fullsized":"2024-09-10T20:42:24.256201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:40.202138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.246102+0000","last_change":"2024-09-10T20:42:23.244187+0000","last_active":"2024-09-10T20:42:24.246102+0000","last_peered":"2024-09-10T20:42:24.246102+0000","last_clean":"2024-09-10T20:42:24.246102+0000","last_became_active":"2024-09-10T20:42:23.243905+0000","last_became_peered":"2024-09-10T20:42:23.243905+0000","last_unstale":"2024-09-10T20:42:24.246102+0000","last_undegraded":"2024-09-10T20:42:24.246102+0000","last_fullsized":"2024-09-10T20:42:24.246102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:48.713410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262481+0000","last_change":"2024-09-10T20:42:24.259044+0000","last_active":"2024-09-10T20:42:24.262481+0000","last_peered":"2024-09-10T20:42:24.262481+0000","last_clean":"2024-09-10T20:42:24.262481+0000","last_became_active":"2024-09-10T20:42:24.257959+0000","last_became_peered":"2024-09-10T20:42:24.257959+0000","last_unstale":"2024-09-10T20:42:24.262481+0000","last_undegraded":"2024-09-10T20:42:24.262481+0000","last_fullsized":"2024-09-10T20:42:24.262481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:31:05.117623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.240101+0000","last_change":"2024-09-10T20:42:23.250015+0000","last_active":"2024-09-10T20:42:24.240101+0000","last_peered":"2024-09-10T20:42:24.240101+0000","last_clean":"2024-09-10T20:42:24.240101+0000","last_became_active":"2024-09-10T20:42:23.249203+0000","last_became_peered":"2024-09-10T20:42:23.249203+0000","last_unstale":"2024-09-10T20:42:24.240101+0000","last_undegraded":"2024-09-10T20:42:24.240101+0000","last_fullsized":"2024-09-10T20:42:24.240101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:42.109627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266201+0000","last_change":"2024-09-10T20:42:24.259245+0000","last_active":"2024-09-10T20:42:24.266201+0000","last_peered":"2024-09-10T20:42:24.266201+0000","last_clean":"2024-09-10T20:42:24.266201+0000","last_became_active":"2024-09-10T20:42:24.258793+0000","last_became_peered":"2024-09-10T20:42:24.258793+0000","last_unstale":"2024-09-10T20:42:24.266201+0000","last_undegraded":"2024-09-10T20:42:24.266201+0000","last_fullsized":"2024-09-10T20:42:24.266201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:41.961270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267931+0000","last_change":"2024-09-10T20:42:23.252693+0000","last_active":"2024-09-10T20:42:24.267931+0000","last_peered":"2024-09-10T20:42:24.267931+0000","last_clean":"2024-09-10T20:42:24.267931+0000","last_became_active":"2024-09-10T20:42:23.252461+0000","last_became_peered":"2024-09-10T20:42:23.252461+0000","last_unstale":"2024-09-10T20:42:24.267931+0000","last_undegraded":"2024-09-10T20:42:24.267931+0000","last_fullsized":"2024-09-10T20:42:24.267931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:27:21.791560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266257+0000","last_change":"2024-09-10T20:42:24.262908+0000","last_active":"2024-09-10T20:42:24.266257+0000","last_peered":"2024-09-10T20:42:24.266257+0000","last_clean":"2024-09-10T20:42:24.266257+0000","last_became_active":"2024-09-10T20:42:24.262614+0000","last_became_peered":"2024-09-10T20:42:24.262614+0000","last_unstale":"2024-09-10T20:42:24.266257+0000","last_undegraded":"2024-09-10T20:42:24.266257+0000","last_fullsized":"2024-09-10T20:42:24.266257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:46.383244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.244301+0000","last_change":"2024-09-10T20:42:23.247648+0000","last_active":"2024-09-10T20:42:24.244301+0000","last_peered":"2024-09-10T20:42:24.244301+0000","last_clean":"2024-09-10T20:42:24.244301+0000","last_became_active":"2024-09-10T20:42:23.247345+0000","last_became_peered":"2024-09-10T20:42:23.247345+0000","last_unstale":"2024-09-10T20:42:24.244301+0000","last_undegraded":"2024-09-10T20:42:24.244301+0000","last_fullsized":"2024-09-10T20:42:24.244301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:49:11.666986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260129+0000","last_change":"2024-09-10T20:42:24.255585+0000","last_active":"2024-09-10T20:42:24.260129+0000","last_peered":"2024-09-10T20:42:24.260129+0000","last_clean":"2024-09-10T20:42:24.260129+0000","last_became_active":"2024-09-10T20:42:24.255340+0000","last_became_peered":"2024-09-10T20:42:24.255340+0000","last_unstale":"2024-09-10T20:42:24.260129+0000","last_undegraded":"2024-09-10T20:42:24.260129+0000","last_fullsized":"2024-09-10T20:42:24.260129+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:55.045900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.240007+0000","last_change":"2024-09-10T20:42:23.249288+0000","last_active":"2024-09-10T20:42:24.240007+0000","last_peered":"2024-09-10T20:42:24.240007+0000","last_clean":"2024-09-10T20:42:24.240007+0000","last_became_active":"2024-09-10T20:42:23.248991+0000","last_became_peered":"2024-09-10T20:42:23.248991+0000","last_unstale":"2024-09-10T20:42:24.240007+0000","last_undegraded":"2024-09-10T20:42:24.240007+0000","last_fullsized":"2024-09-10T20:42:24.240007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:41.414236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.248573+0000","last_change":"2024-09-10T20:42:24.245873+0000","last_active":"2024-09-10T20:42:24.248573+0000","last_peered":"2024-09-10T20:42:24.248573+0000","last_clean":"2024-09-10T20:42:24.248573+0000","last_became_active":"2024-09-10T20:42:24.245555+0000","last_became_peered":"2024-09-10T20:42:24.245555+0000","last_unstale":"2024-09-10T20:42:24.248573+0000","last_undegraded":"2024-09-10T20:42:24.248573+0000","last_fullsized":"2024-09-10T20:42:24.248573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:53:10.278617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.242169+0000","last_change":"2024-09-10T20:42:23.250318+0000","last_active":"2024-09-10T20:42:24.242169+0000","last_peered":"2024-09-10T20:42:24.242169+0000","last_clean":"2024-09-10T20:42:24.242169+0000","last_became_active":"2024-09-10T20:42:23.249984+0000","last_became_peered":"2024-09-10T20:42:23.249984+0000","last_unstale":"2024-09-10T20:42:24.242169+0000","last_undegraded":"2024-09-10T20:42:24.242169+0000","last_fullsized":"2024-09-10T20:42:24.242169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:41.414236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256345+0000","last_change":"2024-09-10T20:42:24.250834+0000","last_active":"2024-09-10T20:42:24.256345+0000","last_peered":"2024-09-10T20:42:24.256345+0000","last_clean":"2024-09-10T20:42:24.256345+0000","last_became_active":"2024-09-10T20:42:24.250522+0000","last_became_peered":"2024-09-10T20:42:24.250522+0000","last_unstale":"2024-09-10T20:42:24.256345+0000","last_undegraded":"2024-09-10T20:42:24.256345+0000","last_fullsized":"2024-09-10T20:42:24.256345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:49:12.669728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.235738+0000","last_change":"2024-09-10T20:42:23.245079+0000","last_active":"2024-09-10T20:42:24.235738+0000","last_peered":"2024-09-10T20:42:24.235738+0000","last_clean":"2024-09-10T20:42:24.235738+0000","last_became_active":"2024-09-10T20:42:23.244765+0000","last_became_peered":"2024-09-10T20:42:23.244765+0000","last_unstale":"2024-09-10T20:42:24.235738+0000","last_undegraded":"2024-09-10T20:42:24.235738+0000","last_fullsized":"2024-09-10T20:42:24.235738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:16:05.597476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260963+0000","last_change":"2024-09-10T20:42:24.254607+0000","last_active":"2024-09-10T20:42:24.260963+0000","last_peered":"2024-09-10T20:42:24.260963+0000","last_clean":"2024-09-10T20:42:24.260963+0000","last_became_active":"2024-09-10T20:42:24.254287+0000","last_became_peered":"2024-09-10T20:42:24.254287+0000","last_unstale":"2024-09-10T20:42:24.260963+0000","last_undegraded":"2024-09-10T20:42:24.260963+0000","last_fullsized":"2024-09-10T20:42:24.260963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:30:08.734512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.241754+0000","last_change":"2024-09-10T20:42:23.243965+0000","last_active":"2024-09-10T20:42:24.241754+0000","last_peered":"2024-09-10T20:42:24.241754+0000","last_clean":"2024-09-10T20:42:24.241754+0000","last_became_active":"2024-09-10T20:42:23.243684+0000","last_became_peered":"2024-09-10T20:42:23.243684+0000","last_unstale":"2024-09-10T20:42:24.241754+0000","last_undegraded":"2024-09-10T20:42:24.241754+0000","last_fullsized":"2024-09-10T20:42:24.241754+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:14.446361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256582+0000","last_change":"2024-09-10T20:42:24.247470+0000","last_active":"2024-09-10T20:42:24.256582+0000","last_peered":"2024-09-10T20:42:24.256582+0000","last_clean":"2024-09-10T20:42:24.256582+0000","last_became_active":"2024-09-10T20:42:24.247218+0000","last_became_peered":"2024-09-10T20:42:24.247218+0000","last_unstale":"2024-09-10T20:42:24.256582+0000","last_undegraded":"2024-09-10T20:42:24.256582+0000","last_fullsized":"2024-09-10T20:42:24.256582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:22.324477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.235299+0000","last_change":"2024-09-10T20:42:23.250191+0000","last_active":"2024-09-10T20:42:24.235299+0000","last_peered":"2024-09-10T20:42:24.235299+0000","last_clean":"2024-09-10T20:42:24.235299+0000","last_became_active":"2024-09-10T20:42:23.249930+0000","last_became_peered":"2024-09-10T20:42:23.249930+0000","last_unstale":"2024-09-10T20:42:24.235299+0000","last_undegraded":"2024-09-10T20:42:24.235299+0000","last_fullsized":"2024-09-10T20:42:24.235299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:15.534284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.243492+0000","last_change":"2024-09-10T20:42:24.242253+0000","last_active":"2024-09-10T20:42:24.243492+0000","last_peered":"2024-09-10T20:42:24.243492+0000","last_clean":"2024-09-10T20:42:24.243492+0000","last_became_active":"2024-09-10T20:42:24.241966+0000","last_became_peered":"2024-09-10T20:42:24.241966+0000","last_unstale":"2024-09-10T20:42:24.243492+0000","last_undegraded":"2024-09-10T20:42:24.243492+0000","last_fullsized":"2024-09-10T20:42:24.243492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:58:37.573858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.241114+0000","last_change":"2024-09-10T20:42:23.248404+0000","last_active":"2024-09-10T20:42:24.241114+0000","last_peered":"2024-09-10T20:42:24.241114+0000","last_clean":"2024-09-10T20:42:24.241114+0000","last_became_active":"2024-09-10T20:42:23.247973+0000","last_became_peered":"2024-09-10T20:42:23.247973+0000","last_unstale":"2024-09-10T20:42:24.241114+0000","last_undegraded":"2024-09-10T20:42:24.241114+0000","last_fullsized":"2024-09-10T20:42:24.241114+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:03:04.402951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266099+0000","last_change":"2024-09-10T20:42:24.261854+0000","last_active":"2024-09-10T20:42:24.266099+0000","last_peered":"2024-09-10T20:42:24.266099+0000","last_clean":"2024-09-10T20:42:24.266099+0000","last_became_active":"2024-09-10T20:42:24.261431+0000","last_became_peered":"2024-09-10T20:42:24.261431+0000","last_unstale":"2024-09-10T20:42:24.266099+0000","last_undegraded":"2024-09-10T20:42:24.266099+0000","last_fullsized":"2024-09-10T20:42:24.266099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:02.685852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262812+0000","last_change":"2024-09-10T20:42:23.252862+0000","last_active":"2024-09-10T20:42:24.262812+0000","last_peered":"2024-09-10T20:42:24.262812+0000","last_clean":"2024-09-10T20:42:24.262812+0000","last_became_active":"2024-09-10T20:42:23.252563+0000","last_became_peered":"2024-09-10T20:42:23.252563+0000","last_unstale":"2024-09-10T20:42:24.262812+0000","last_undegraded":"2024-09-10T20:42:24.262812+0000","last_fullsized":"2024-09-10T20:42:24.262812+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:42.952894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":247,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.235567+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:42:24.235567+0000","last_peered":"2024-09-10T20:42:24.235567+0000","last_clean":"2024-09-10T20:42:24.235567+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:42:24.235567+0000","last_undegraded":"2024-09-10T20:42:24.235567+0000","last_fullsized":"2024-09-10T20:42:24.235567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542199,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32712,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1556480,"data_stored":1000489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542199,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509497,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509496,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509496,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32712,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1556480,"data_stored":1000489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509496,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":991232,"data_stored":418530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:42:28.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.406+0000 7f74abd2b640 1 -- 172.21.15.28:0/1613145980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7490003fd0 msgr2=0x7f7490004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.406+0000 7f74abd2b640 1 --2- 172.21.15.28:0/1613145980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7490003fd0 0x7f7490004800 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7494009b20 tx=0x7f7490005830 comp rx=0 tx=0).stop 2024-09-10T20:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.406+0000 7f74aa2c8640 1 -- 172.21.15.28:0/1613145980 reap_dead start 2024-09-10T20:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.406+0000 7f74abd2b640 1 -- 172.21.15.28:0/1613145980 shutdown_connections 2024-09-10T20:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.406+0000 7f74abd2b640 1 -- 172.21.15.28:0/1613145980 >> 172.21.15.28:0/1613145980 conn(0x7f74a406dbf0 msgr2=0x7f74a406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.407+0000 7f74abd2b640 1 -- 172.21.15.28:0/1613145980 shutdown_connections 2024-09-10T20:42:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.407+0000 7f74abd2b640 1 -- 172.21.15.28:0/1613145980 wait complete. 2024-09-10T20:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 Processor -- start 2024-09-10T20:42:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 -- start start 2024-09-10T20:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7490003fd0 0x7f74a415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a40aff40 0x7f74a40b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74a40b0900 0x7f74a41f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74a40d1a70 con 0x7f74a40aff40 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f74a40d18f0 con 0x7f74a40b0900 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.408+0000 7f74abd2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f74a406f550 con 0x7f7490003fd0 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a40aff40 0x7f74a40b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a40aff40 0x7f74a40b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41708/0 (socket says 172.21.15.28:41708) 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 -- 172.21.15.28:0/3599795430 learned_addr learned my addr 172.21.15.28:0/3599795430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74aa2c8640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74a40b0900 0x7f74a41f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a9ac7640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7490003fd0 0x7f74a415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 -- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7490003fd0 msgr2=0x7f74a415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7490003fd0 0x7f74a415da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 -- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74a40b0900 msgr2=0x7f74a41f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74a40b0900 0x7f74a41f7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 -- 172.21.15.28:0/3599795430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a40b0d60 con 0x7f74a40aff40 2024-09-10T20:42:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74a92c6640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a40aff40 0x7f74a40b03c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7494009c40 tx=0x7f7494033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f749a7fc640 1 -- 172.21.15.28:0/3599795430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494038040 con 0x7f74a40aff40 2024-09-10T20:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f749a7fc640 1 -- 172.21.15.28:0/3599795430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7494041070 con 0x7f74a40aff40 2024-09-10T20:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a40b36c0 con 0x7f74a40aff40 2024-09-10T20:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f749a7fc640 1 -- 172.21.15.28:0/3599795430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494033d00 con 0x7f74a40aff40 2024-09-10T20:42:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.409+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a40c5f10 con 0x7f74a40aff40 2024-09-10T20:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.410+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a4161950 con 0x7f74a40aff40 2024-09-10T20:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.411+0000 7f749a7fc640 1 -- 172.21.15.28:0/3599795430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f749403b070 con 0x7f74a40aff40 2024-09-10T20:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.412+0000 7f749a7fc640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7488078090 0x7f748807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.412+0000 7f749a7fc640 1 -- 172.21.15.28:0/3599795430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f74940ba640 con 0x7f74a40aff40 2024-09-10T20:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.412+0000 7f74a9ac7640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7488078090 0x7f748807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.412+0000 7f74a9ac7640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7488078090 0x7f748807a550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74a00099c0 tx=0x7f74a0005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.416+0000 7f749a7fc640 1 -- 172.21.15.28:0/3599795430 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7494082e80 con 0x7f74a40aff40 2024-09-10T20:42:28.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.598+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f74a4069a80 con 0x7f7488078090 2024-09-10T20:42:28.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.602+0000 7f749a7fc640 1 -- 172.21.15.28:0/3599795430 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252076 (secure 0 0 0) 0x7f74a4069a80 con 0x7f7488078090 2024-09-10T20:42:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7488078090 msgr2=0x7f748807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7488078090 0x7f748807a550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74a00099c0 tx=0x7f74a0005f90 comp rx=0 tx=0).stop 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a40aff40 msgr2=0x7f74a40b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a40aff40 0x7f74a40b03c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7494009c40 tx=0x7f7494033670 comp rx=0 tx=0).stop 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 shutdown_connections 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7488078090 0x7f748807a550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74a40b0900 0x7f74a41f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a40aff40 0x7f74a40b03c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 --2- 172.21.15.28:0/3599795430 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7490003fd0 0x7f74a415da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 >> 172.21.15.28:0/3599795430 conn(0x7f74a406dbf0 msgr2=0x7f74a4144f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 shutdown_connections 2024-09-10T20:42:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.605+0000 7f74abd2b640 1 -- 172.21.15.28:0/3599795430 wait complete. 2024-09-10T20:42:28.621 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-10T20:42:28.294518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254072,"kb_used_data":8708,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754720648,"statfs":{"total":773094113280,"available":772833943552,"internally_reserved":0,"allocated":8916992,"data_stored":4470113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001226"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256527+0000","last_change":"2024-09-10T20:42:24.252046+0000","last_active":"2024-09-10T20:42:24.256527+0000","last_peered":"2024-09-10T20:42:24.256527+0000","last_clean":"2024-09-10T20:42:24.256527+0000","last_became_active":"2024-09-10T20:42:24.251464+0000","last_became_peered":"2024-09-10T20:42:24.251464+0000","last_unstale":"2024-09-10T20:42:24.256527+0000","last_undegraded":"2024-09-10T20:42:24.256527+0000","last_fullsized":"2024-09-10T20:42:24.256527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:49.716152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255934+0000","last_change":"2024-09-10T20:42:24.249553+0000","last_active":"2024-09-10T20:42:24.255934+0000","last_peered":"2024-09-10T20:42:24.255934+0000","last_clean":"2024-09-10T20:42:24.255934+0000","last_became_active":"2024-09-10T20:42:24.249206+0000","last_became_peered":"2024-09-10T20:42:24.249206+0000","last_unstale":"2024-09-10T20:42:24.255934+0000","last_undegraded":"2024-09-10T20:42:24.255934+0000","last_fullsized":"2024-09-10T20:42:24.255934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:56:52.562320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264425+0000","last_change":"2024-09-10T20:42:24.263659+0000","last_active":"2024-09-10T20:42:24.264425+0000","last_peered":"2024-09-10T20:42:24.264425+0000","last_clean":"2024-09-10T20:42:24.264425+0000","last_became_active":"2024-09-10T20:42:24.263351+0000","last_became_peered":"2024-09-10T20:42:24.263351+0000","last_unstale":"2024-09-10T20:42:24.264425+0000","last_undegraded":"2024-09-10T20:42:24.264425+0000","last_fullsized":"2024-09-10T20:42:24.264425+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:28.917202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255149+0000","last_change":"2024-09-10T20:42:24.248059+0000","last_active":"2024-09-10T20:42:24.255149+0000","last_peered":"2024-09-10T20:42:24.255149+0000","last_clean":"2024-09-10T20:42:24.255149+0000","last_became_active":"2024-09-10T20:42:24.247750+0000","last_became_peered":"2024-09-10T20:42:24.247750+0000","last_unstale":"2024-09-10T20:42:24.255149+0000","last_undegraded":"2024-09-10T20:42:24.255149+0000","last_fullsized":"2024-09-10T20:42:24.255149+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:34:20.849386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260866+0000","last_change":"2024-09-10T20:42:24.250941+0000","last_active":"2024-09-10T20:42:24.260866+0000","last_peered":"2024-09-10T20:42:24.260866+0000","last_clean":"2024-09-10T20:42:24.260866+0000","last_became_active":"2024-09-10T20:42:24.250646+0000","last_became_peered":"2024-09-10T20:42:24.250646+0000","last_unstale":"2024-09-10T20:42:24.260866+0000","last_undegraded":"2024-09-10T20:42:24.260866+0000","last_fullsized":"2024-09-10T20:42:24.260866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:04.772094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256294+0000","last_change":"2024-09-10T20:42:24.250491+0000","last_active":"2024-09-10T20:42:24.256294+0000","last_peered":"2024-09-10T20:42:24.256294+0000","last_clean":"2024-09-10T20:42:24.256294+0000","last_became_active":"2024-09-10T20:42:24.250197+0000","last_became_peered":"2024-09-10T20:42:24.250197+0000","last_unstale":"2024-09-10T20:42:24.256294+0000","last_undegraded":"2024-09-10T20:42:24.256294+0000","last_fullsized":"2024-09-10T20:42:24.256294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:31:05.117623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.252338+0000","last_change":"2024-09-10T20:42:24.246602+0000","last_active":"2024-09-10T20:42:24.252338+0000","last_peered":"2024-09-10T20:42:24.252338+0000","last_clean":"2024-09-10T20:42:24.252338+0000","last_became_active":"2024-09-10T20:42:24.246346+0000","last_became_peered":"2024-09-10T20:42:24.246346+0000","last_unstale":"2024-09-10T20:42:24.252338+0000","last_undegraded":"2024-09-10T20:42:24.252338+0000","last_fullsized":"2024-09-10T20:42:24.252338+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:16:50.457371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264517+0000","last_change":"2024-09-10T20:42:24.261493+0000","last_active":"2024-09-10T20:42:24.264517+0000","last_peered":"2024-09-10T20:42:24.264517+0000","last_clean":"2024-09-10T20:42:24.264517+0000","last_became_active":"2024-09-10T20:42:24.261160+0000","last_became_peered":"2024-09-10T20:42:24.261160+0000","last_unstale":"2024-09-10T20:42:24.264517+0000","last_undegraded":"2024-09-10T20:42:24.264517+0000","last_fullsized":"2024-09-10T20:42:24.264517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:41.626813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.246813+0000","last_change":"2024-09-10T20:42:24.243110+0000","last_active":"2024-09-10T20:42:24.246813+0000","last_peered":"2024-09-10T20:42:24.246813+0000","last_clean":"2024-09-10T20:42:24.246813+0000","last_became_active":"2024-09-10T20:42:24.242832+0000","last_became_peered":"2024-09-10T20:42:24.242832+0000","last_unstale":"2024-09-10T20:42:24.246813+0000","last_undegraded":"2024-09-10T20:42:24.246813+0000","last_fullsized":"2024-09-10T20:42:24.246813+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:13.194381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.257084+0000","last_change":"2024-09-10T20:42:24.253465+0000","last_active":"2024-09-10T20:42:24.257084+0000","last_peered":"2024-09-10T20:42:24.257084+0000","last_clean":"2024-09-10T20:42:24.257084+0000","last_became_active":"2024-09-10T20:42:24.253175+0000","last_became_peered":"2024-09-10T20:42:24.253175+0000","last_unstale":"2024-09-10T20:42:24.257084+0000","last_undegraded":"2024-09-10T20:42:24.257084+0000","last_fullsized":"2024-09-10T20:42:24.257084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:26:41.626813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260983+0000","last_change":"2024-09-10T20:42:24.255079+0000","last_active":"2024-09-10T20:42:24.260983+0000","last_peered":"2024-09-10T20:42:24.260983+0000","last_clean":"2024-09-10T20:42:24.260983+0000","last_became_active":"2024-09-10T20:42:24.254591+0000","last_became_peered":"2024-09-10T20:42:24.254591+0000","last_unstale":"2024-09-10T20:42:24.260983+0000","last_undegraded":"2024-09-10T20:42:24.260983+0000","last_fullsized":"2024-09-10T20:42:24.260983+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:21:08.468715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262382+0000","last_change":"2024-09-10T20:42:24.255464+0000","last_active":"2024-09-10T20:42:24.262382+0000","last_peered":"2024-09-10T20:42:24.262382+0000","last_clean":"2024-09-10T20:42:24.262382+0000","last_became_active":"2024-09-10T20:42:24.255056+0000","last_became_peered":"2024-09-10T20:42:24.255056+0000","last_unstale":"2024-09-10T20:42:24.262382+0000","last_undegraded":"2024-09-10T20:42:24.262382+0000","last_fullsized":"2024-09-10T20:42:24.262382+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:56.600475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266217+0000","last_change":"2024-09-10T20:42:24.260343+0000","last_active":"2024-09-10T20:42:24.266217+0000","last_peered":"2024-09-10T20:42:24.266217+0000","last_clean":"2024-09-10T20:42:24.266217+0000","last_became_active":"2024-09-10T20:42:24.259812+0000","last_became_peered":"2024-09-10T20:42:24.259812+0000","last_unstale":"2024-09-10T20:42:24.266217+0000","last_undegraded":"2024-09-10T20:42:24.266217+0000","last_fullsized":"2024-09-10T20:42:24.266217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:15.203081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.249456+0000","last_change":"2024-09-10T20:42:24.245659+0000","last_active":"2024-09-10T20:42:24.249456+0000","last_peered":"2024-09-10T20:42:24.249456+0000","last_clean":"2024-09-10T20:42:24.249456+0000","last_became_active":"2024-09-10T20:42:24.245357+0000","last_became_peered":"2024-09-10T20:42:24.245357+0000","last_unstale":"2024-09-10T20:42:24.249456+0000","last_undegraded":"2024-09-10T20:42:24.249456+0000","last_fullsized":"2024-09-10T20:42:24.249456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:49.739293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.249939+0000","last_change":"2024-09-10T20:42:24.241653+0000","last_active":"2024-09-10T20:42:24.249939+0000","last_peered":"2024-09-10T20:42:24.249939+0000","last_clean":"2024-09-10T20:42:24.249939+0000","last_became_active":"2024-09-10T20:42:24.241359+0000","last_became_peered":"2024-09-10T20:42:24.241359+0000","last_unstale":"2024-09-10T20:42:24.249939+0000","last_undegraded":"2024-09-10T20:42:24.249939+0000","last_fullsized":"2024-09-10T20:42:24.249939+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:17:23.346363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256360+0000","last_change":"2024-09-10T20:42:24.251074+0000","last_active":"2024-09-10T20:42:24.256360+0000","last_peered":"2024-09-10T20:42:24.256360+0000","last_clean":"2024-09-10T20:42:24.256360+0000","last_became_active":"2024-09-10T20:42:24.250784+0000","last_became_peered":"2024-09-10T20:42:24.250784+0000","last_unstale":"2024-09-10T20:42:24.256360+0000","last_undegraded":"2024-09-10T20:42:24.256360+0000","last_fullsized":"2024-09-10T20:42:24.256360+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:39.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.259069+0000","last_change":"2024-09-10T20:42:24.254012+0000","last_active":"2024-09-10T20:42:24.259069+0000","last_peered":"2024-09-10T20:42:24.259069+0000","last_clean":"2024-09-10T20:42:24.259069+0000","last_became_active":"2024-09-10T20:42:24.253691+0000","last_became_peered":"2024-09-10T20:42:24.253691+0000","last_unstale":"2024-09-10T20:42:24.259069+0000","last_undegraded":"2024-09-10T20:42:24.259069+0000","last_fullsized":"2024-09-10T20:42:24.259069+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:51.778063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260855+0000","last_change":"2024-09-10T20:42:24.253511+0000","last_active":"2024-09-10T20:42:24.260855+0000","last_peered":"2024-09-10T20:42:24.260855+0000","last_clean":"2024-09-10T20:42:24.260855+0000","last_became_active":"2024-09-10T20:42:24.253082+0000","last_became_peered":"2024-09-10T20:42:24.253082+0000","last_unstale":"2024-09-10T20:42:24.260855+0000","last_undegraded":"2024-09-10T20:42:24.260855+0000","last_fullsized":"2024-09-10T20:42:24.260855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:39.725236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.252400+0000","last_change":"2024-09-10T20:42:24.248623+0000","last_active":"2024-09-10T20:42:24.252400+0000","last_peered":"2024-09-10T20:42:24.252400+0000","last_clean":"2024-09-10T20:42:24.252400+0000","last_became_active":"2024-09-10T20:42:24.248380+0000","last_became_peered":"2024-09-10T20:42:24.248380+0000","last_unstale":"2024-09-10T20:42:24.252400+0000","last_undegraded":"2024-09-10T20:42:24.252400+0000","last_fullsized":"2024-09-10T20:42:24.252400+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:34.794709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.254730+0000","last_change":"2024-09-10T20:42:24.249339+0000","last_active":"2024-09-10T20:42:24.254730+0000","last_peered":"2024-09-10T20:42:24.254730+0000","last_clean":"2024-09-10T20:42:24.254730+0000","last_became_active":"2024-09-10T20:42:24.249080+0000","last_became_peered":"2024-09-10T20:42:24.249080+0000","last_unstale":"2024-09-10T20:42:24.254730+0000","last_undegraded":"2024-09-10T20:42:24.254730+0000","last_fullsized":"2024-09-10T20:42:24.254730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:20:14.247696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.244998+0000","last_change":"2024-09-10T20:42:24.240390+0000","last_active":"2024-09-10T20:42:24.244998+0000","last_peered":"2024-09-10T20:42:24.244998+0000","last_clean":"2024-09-10T20:42:24.244998+0000","last_became_active":"2024-09-10T20:42:24.239950+0000","last_became_peered":"2024-09-10T20:42:24.239950+0000","last_unstale":"2024-09-10T20:42:24.244998+0000","last_undegraded":"2024-09-10T20:42:24.244998+0000","last_fullsized":"2024-09-10T20:42:24.244998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:31:16.945421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255104+0000","last_change":"2024-09-10T20:42:24.250755+0000","last_active":"2024-09-10T20:42:24.255104+0000","last_peered":"2024-09-10T20:42:24.255104+0000","last_clean":"2024-09-10T20:42:24.255104+0000","last_became_active":"2024-09-10T20:42:24.250468+0000","last_became_peered":"2024-09-10T20:42:24.250468+0000","last_unstale":"2024-09-10T20:42:24.255104+0000","last_undegraded":"2024-09-10T20:42:24.255104+0000","last_fullsized":"2024-09-10T20:42:24.255104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:57:37.616646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262510+0000","last_change":"2024-09-10T20:42:24.259183+0000","last_active":"2024-09-10T20:42:24.262510+0000","last_peered":"2024-09-10T20:42:24.262510+0000","last_clean":"2024-09-10T20:42:24.262510+0000","last_became_active":"2024-09-10T20:42:24.258885+0000","last_became_peered":"2024-09-10T20:42:24.258885+0000","last_unstale":"2024-09-10T20:42:24.262510+0000","last_undegraded":"2024-09-10T20:42:24.262510+0000","last_fullsized":"2024-09-10T20:42:24.262510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:39.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255964+0000","last_change":"2024-09-10T20:42:24.250947+0000","last_active":"2024-09-10T20:42:24.255964+0000","last_peered":"2024-09-10T20:42:24.255964+0000","last_clean":"2024-09-10T20:42:24.255964+0000","last_became_active":"2024-09-10T20:42:24.250691+0000","last_became_peered":"2024-09-10T20:42:24.250691+0000","last_unstale":"2024-09-10T20:42:24.255964+0000","last_undegraded":"2024-09-10T20:42:24.255964+0000","last_fullsized":"2024-09-10T20:42:24.255964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:42:22.244222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.269135+0000","last_change":"2024-09-10T20:42:24.267779+0000","last_active":"2024-09-10T20:42:24.269135+0000","last_peered":"2024-09-10T20:42:24.269135+0000","last_clean":"2024-09-10T20:42:24.269135+0000","last_became_active":"2024-09-10T20:42:24.267405+0000","last_became_peered":"2024-09-10T20:42:24.267405+0000","last_unstale":"2024-09-10T20:42:24.269135+0000","last_undegraded":"2024-09-10T20:42:24.269135+0000","last_fullsized":"2024-09-10T20:42:24.269135+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:15.449103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266230+0000","last_change":"2024-09-10T20:42:24.261575+0000","last_active":"2024-09-10T20:42:24.266230+0000","last_peered":"2024-09-10T20:42:24.266230+0000","last_clean":"2024-09-10T20:42:24.266230+0000","last_became_active":"2024-09-10T20:42:24.261238+0000","last_became_peered":"2024-09-10T20:42:24.261238+0000","last_unstale":"2024-09-10T20:42:24.266230+0000","last_undegraded":"2024-09-10T20:42:24.266230+0000","last_fullsized":"2024-09-10T20:42:24.266230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:20.623882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260240+0000","last_change":"2024-09-10T20:42:24.251130+0000","last_active":"2024-09-10T20:42:24.260240+0000","last_peered":"2024-09-10T20:42:24.260240+0000","last_clean":"2024-09-10T20:42:24.260240+0000","last_became_active":"2024-09-10T20:42:24.250824+0000","last_became_peered":"2024-09-10T20:42:24.250824+0000","last_unstale":"2024-09-10T20:42:24.260240+0000","last_undegraded":"2024-09-10T20:42:24.260240+0000","last_fullsized":"2024-09-10T20:42:24.260240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:44:49.605060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266421+0000","last_change":"2024-09-10T20:42:24.265003+0000","last_active":"2024-09-10T20:42:24.266421+0000","last_peered":"2024-09-10T20:42:24.266421+0000","last_clean":"2024-09-10T20:42:24.266421+0000","last_became_active":"2024-09-10T20:42:24.264690+0000","last_became_peered":"2024-09-10T20:42:24.264690+0000","last_unstale":"2024-09-10T20:42:24.266421+0000","last_undegraded":"2024-09-10T20:42:24.266421+0000","last_fullsized":"2024-09-10T20:42:24.266421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:30:08.734512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.247929+0000","last_change":"2024-09-10T20:42:24.242217+0000","last_active":"2024-09-10T20:42:24.247929+0000","last_peered":"2024-09-10T20:42:24.247929+0000","last_clean":"2024-09-10T20:42:24.247929+0000","last_became_active":"2024-09-10T20:42:24.241929+0000","last_became_peered":"2024-09-10T20:42:24.241929+0000","last_unstale":"2024-09-10T20:42:24.247929+0000","last_undegraded":"2024-09-10T20:42:24.247929+0000","last_fullsized":"2024-09-10T20:42:24.247929+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:58:10.673710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255056+0000","last_change":"2024-09-10T20:42:24.249606+0000","last_active":"2024-09-10T20:42:24.255056+0000","last_peered":"2024-09-10T20:42:24.255056+0000","last_clean":"2024-09-10T20:42:24.255056+0000","last_became_active":"2024-09-10T20:42:24.249288+0000","last_became_peered":"2024-09-10T20:42:24.249288+0000","last_unstale":"2024-09-10T20:42:24.255056+0000","last_undegraded":"2024-09-10T20:42:24.255056+0000","last_fullsized":"2024-09-10T20:42:24.255056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:18:13.328287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255988+0000","last_change":"2024-09-10T20:42:24.249671+0000","last_active":"2024-09-10T20:42:24.255988+0000","last_peered":"2024-09-10T20:42:24.255988+0000","last_clean":"2024-09-10T20:42:24.255988+0000","last_became_active":"2024-09-10T20:42:24.249396+0000","last_became_peered":"2024-09-10T20:42:24.249396+0000","last_unstale":"2024-09-10T20:42:24.255988+0000","last_undegraded":"2024-09-10T20:42:24.255988+0000","last_fullsized":"2024-09-10T20:42:24.255988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:15.234387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256336+0000","last_change":"2024-09-10T20:42:24.249638+0000","last_active":"2024-09-10T20:42:24.256336+0000","last_peered":"2024-09-10T20:42:24.256336+0000","last_clean":"2024-09-10T20:42:24.256336+0000","last_became_active":"2024-09-10T20:42:24.249341+0000","last_became_peered":"2024-09-10T20:42:24.249341+0000","last_unstale":"2024-09-10T20:42:24.256336+0000","last_undegraded":"2024-09-10T20:42:24.256336+0000","last_fullsized":"2024-09-10T20:42:24.256336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:56.600475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.268443+0000","last_change":"2024-09-10T20:42:24.267438+0000","last_active":"2024-09-10T20:42:24.268443+0000","last_peered":"2024-09-10T20:42:24.268443+0000","last_clean":"2024-09-10T20:42:24.268443+0000","last_became_active":"2024-09-10T20:42:24.267159+0000","last_became_peered":"2024-09-10T20:42:24.267159+0000","last_unstale":"2024-09-10T20:42:24.268443+0000","last_undegraded":"2024-09-10T20:42:24.268443+0000","last_fullsized":"2024-09-10T20:42:24.268443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:28.917202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.269313+0000","last_change":"2024-09-10T20:42:23.253219+0000","last_active":"2024-09-10T20:42:24.269313+0000","last_peered":"2024-09-10T20:42:24.269313+0000","last_clean":"2024-09-10T20:42:24.269313+0000","last_became_active":"2024-09-10T20:42:23.252705+0000","last_became_peered":"2024-09-10T20:42:23.252705+0000","last_unstale":"2024-09-10T20:42:24.269313+0000","last_undegraded":"2024-09-10T20:42:24.269313+0000","last_fullsized":"2024-09-10T20:42:24.269313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:04:21.695369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256497+0000","last_change":"2024-09-10T20:42:24.251751+0000","last_active":"2024-09-10T20:42:24.256497+0000","last_peered":"2024-09-10T20:42:24.256497+0000","last_clean":"2024-09-10T20:42:24.256497+0000","last_became_active":"2024-09-10T20:42:24.251088+0000","last_became_peered":"2024-09-10T20:42:24.251088+0000","last_unstale":"2024-09-10T20:42:24.256497+0000","last_undegraded":"2024-09-10T20:42:24.256497+0000","last_fullsized":"2024-09-10T20:42:24.256497+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:59.741077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267082+0000","last_change":"2024-09-10T20:42:23.250981+0000","last_active":"2024-09-10T20:42:24.267082+0000","last_peered":"2024-09-10T20:42:24.267082+0000","last_clean":"2024-09-10T20:42:24.267082+0000","last_became_active":"2024-09-10T20:42:23.250669+0000","last_became_peered":"2024-09-10T20:42:23.250669+0000","last_unstale":"2024-09-10T20:42:24.267082+0000","last_undegraded":"2024-09-10T20:42:24.267082+0000","last_fullsized":"2024-09-10T20:42:24.267082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:03:04.402951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262423+0000","last_change":"2024-09-10T20:42:24.258631+0000","last_active":"2024-09-10T20:42:24.262423+0000","last_peered":"2024-09-10T20:42:24.262423+0000","last_clean":"2024-09-10T20:42:24.262423+0000","last_became_active":"2024-09-10T20:42:24.258293+0000","last_became_peered":"2024-09-10T20:42:24.258293+0000","last_unstale":"2024-09-10T20:42:24.262423+0000","last_undegraded":"2024-09-10T20:42:24.262423+0000","last_fullsized":"2024-09-10T20:42:24.262423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:15.234387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264222+0000","last_change":"2024-09-10T20:42:23.243429+0000","last_active":"2024-09-10T20:42:24.264222+0000","last_peered":"2024-09-10T20:42:24.264222+0000","last_clean":"2024-09-10T20:42:24.264222+0000","last_became_active":"2024-09-10T20:42:23.242486+0000","last_became_peered":"2024-09-10T20:42:23.242486+0000","last_unstale":"2024-09-10T20:42:24.264222+0000","last_undegraded":"2024-09-10T20:42:24.264222+0000","last_fullsized":"2024-09-10T20:42:24.264222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:58.738335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.259278+0000","last_change":"2024-09-10T20:42:24.257400+0000","last_active":"2024-09-10T20:42:24.259278+0000","last_peered":"2024-09-10T20:42:24.259278+0000","last_clean":"2024-09-10T20:42:24.259278+0000","last_became_active":"2024-09-10T20:42:24.256006+0000","last_became_peered":"2024-09-10T20:42:24.256006+0000","last_unstale":"2024-09-10T20:42:24.259278+0000","last_undegraded":"2024-09-10T20:42:24.259278+0000","last_fullsized":"2024-09-10T20:42:24.259278+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:03:04.919447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266443+0000","last_change":"2024-09-10T20:42:23.247283+0000","last_active":"2024-09-10T20:42:24.266443+0000","last_peered":"2024-09-10T20:42:24.266443+0000","last_clean":"2024-09-10T20:42:24.266443+0000","last_became_active":"2024-09-10T20:42:23.246969+0000","last_became_peered":"2024-09-10T20:42:23.246969+0000","last_unstale":"2024-09-10T20:42:24.266443+0000","last_undegraded":"2024-09-10T20:42:24.266443+0000","last_fullsized":"2024-09-10T20:42:24.266443+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:29:13.545500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256001+0000","last_change":"2024-09-10T20:42:24.251220+0000","last_active":"2024-09-10T20:42:24.256001+0000","last_peered":"2024-09-10T20:42:24.256001+0000","last_clean":"2024-09-10T20:42:24.256001+0000","last_became_active":"2024-09-10T20:42:24.250688+0000","last_became_peered":"2024-09-10T20:42:24.250688+0000","last_unstale":"2024-09-10T20:42:24.256001+0000","last_undegraded":"2024-09-10T20:42:24.256001+0000","last_fullsized":"2024-09-10T20:42:24.256001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:37:40.196769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262867+0000","last_change":"2024-09-10T20:42:23.249368+0000","last_active":"2024-09-10T20:42:24.262867+0000","last_peered":"2024-09-10T20:42:24.262867+0000","last_clean":"2024-09-10T20:42:24.262867+0000","last_became_active":"2024-09-10T20:42:23.249075+0000","last_became_peered":"2024-09-10T20:42:23.249075+0000","last_unstale":"2024-09-10T20:42:24.262867+0000","last_undegraded":"2024-09-10T20:42:24.262867+0000","last_fullsized":"2024-09-10T20:42:24.262867+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:50.459124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.253510+0000","last_change":"2024-09-10T20:42:24.248311+0000","last_active":"2024-09-10T20:42:24.253510+0000","last_peered":"2024-09-10T20:42:24.253510+0000","last_clean":"2024-09-10T20:42:24.253510+0000","last_became_active":"2024-09-10T20:42:24.247053+0000","last_became_peered":"2024-09-10T20:42:24.247053+0000","last_unstale":"2024-09-10T20:42:24.253510+0000","last_undegraded":"2024-09-10T20:42:24.253510+0000","last_fullsized":"2024-09-10T20:42:24.253510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:51.778063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267026+0000","last_change":"2024-09-10T20:42:23.245633+0000","last_active":"2024-09-10T20:42:24.267026+0000","last_peered":"2024-09-10T20:42:24.267026+0000","last_clean":"2024-09-10T20:42:24.267026+0000","last_became_active":"2024-09-10T20:42:23.245376+0000","last_became_peered":"2024-09-10T20:42:23.245376+0000","last_unstale":"2024-09-10T20:42:24.267026+0000","last_undegraded":"2024-09-10T20:42:24.267026+0000","last_fullsized":"2024-09-10T20:42:24.267026+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:06.736064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.257394+0000","last_change":"2024-09-10T20:42:24.256670+0000","last_active":"2024-09-10T20:42:24.257394+0000","last_peered":"2024-09-10T20:42:24.257394+0000","last_clean":"2024-09-10T20:42:24.257394+0000","last_became_active":"2024-09-10T20:42:24.256293+0000","last_became_peered":"2024-09-10T20:42:24.256293+0000","last_unstale":"2024-09-10T20:42:24.257394+0000","last_undegraded":"2024-09-10T20:42:24.257394+0000","last_fullsized":"2024-09-10T20:42:24.257394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:42:22.244222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.239377+0000","last_change":"2024-09-10T20:42:23.249360+0000","last_active":"2024-09-10T20:42:24.239377+0000","last_peered":"2024-09-10T20:42:24.239377+0000","last_clean":"2024-09-10T20:42:24.239377+0000","last_became_active":"2024-09-10T20:42:23.249041+0000","last_became_peered":"2024-09-10T20:42:23.249041+0000","last_unstale":"2024-09-10T20:42:24.239377+0000","last_undegraded":"2024-09-10T20:42:24.239377+0000","last_fullsized":"2024-09-10T20:42:24.239377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:54.690307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.265477+0000","last_change":"2024-09-10T20:42:24.263533+0000","last_active":"2024-09-10T20:42:24.265477+0000","last_peered":"2024-09-10T20:42:24.265477+0000","last_clean":"2024-09-10T20:42:24.265477+0000","last_became_active":"2024-09-10T20:42:24.263259+0000","last_became_peered":"2024-09-10T20:42:24.263259+0000","last_unstale":"2024-09-10T20:42:24.265477+0000","last_undegraded":"2024-09-10T20:42:24.265477+0000","last_fullsized":"2024-09-10T20:42:24.265477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:20:24.107503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.239130+0000","last_change":"2024-09-10T20:42:23.253926+0000","last_active":"2024-09-10T20:42:24.239130+0000","last_peered":"2024-09-10T20:42:24.239130+0000","last_clean":"2024-09-10T20:42:24.239130+0000","last_became_active":"2024-09-10T20:42:23.253668+0000","last_became_peered":"2024-09-10T20:42:23.253668+0000","last_unstale":"2024-09-10T20:42:24.239130+0000","last_undegraded":"2024-09-10T20:42:24.239130+0000","last_fullsized":"2024-09-10T20:42:24.239130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:50.409359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255565+0000","last_change":"2024-09-10T20:42:24.253189+0000","last_active":"2024-09-10T20:42:24.255565+0000","last_peered":"2024-09-10T20:42:24.255565+0000","last_clean":"2024-09-10T20:42:24.255565+0000","last_became_active":"2024-09-10T20:42:24.252850+0000","last_became_peered":"2024-09-10T20:42:24.252850+0000","last_unstale":"2024-09-10T20:42:24.255565+0000","last_undegraded":"2024-09-10T20:42:24.255565+0000","last_fullsized":"2024-09-10T20:42:24.255565+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:29:41.776801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.237102+0000","last_change":"2024-09-10T20:42:23.245177+0000","last_active":"2024-09-10T20:42:24.237102+0000","last_peered":"2024-09-10T20:42:24.237102+0000","last_clean":"2024-09-10T20:42:24.237102+0000","last_became_active":"2024-09-10T20:42:23.244736+0000","last_became_peered":"2024-09-10T20:42:23.244736+0000","last_unstale":"2024-09-10T20:42:24.237102+0000","last_undegraded":"2024-09-10T20:42:24.237102+0000","last_fullsized":"2024-09-10T20:42:24.237102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:38:17.077537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256552+0000","last_change":"2024-09-10T20:42:24.253781+0000","last_active":"2024-09-10T20:42:24.256552+0000","last_peered":"2024-09-10T20:42:24.256552+0000","last_clean":"2024-09-10T20:42:24.256552+0000","last_became_active":"2024-09-10T20:42:24.253524+0000","last_became_peered":"2024-09-10T20:42:24.253524+0000","last_unstale":"2024-09-10T20:42:24.256552+0000","last_undegraded":"2024-09-10T20:42:24.256552+0000","last_fullsized":"2024-09-10T20:42:24.256552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:25.940389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.243006+0000","last_change":"2024-09-10T20:42:23.247121+0000","last_active":"2024-09-10T20:42:24.243006+0000","last_peered":"2024-09-10T20:42:24.243006+0000","last_clean":"2024-09-10T20:42:24.243006+0000","last_became_active":"2024-09-10T20:42:23.246816+0000","last_became_peered":"2024-09-10T20:42:23.246816+0000","last_unstale":"2024-09-10T20:42:24.243006+0000","last_undegraded":"2024-09-10T20:42:24.243006+0000","last_fullsized":"2024-09-10T20:42:24.243006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:15.265502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.268806+0000","last_change":"2024-09-10T20:42:24.262104+0000","last_active":"2024-09-10T20:42:24.268806+0000","last_peered":"2024-09-10T20:42:24.268806+0000","last_clean":"2024-09-10T20:42:24.268806+0000","last_became_active":"2024-09-10T20:42:24.261758+0000","last_became_peered":"2024-09-10T20:42:24.261758+0000","last_unstale":"2024-09-10T20:42:24.268806+0000","last_undegraded":"2024-09-10T20:42:24.268806+0000","last_fullsized":"2024-09-10T20:42:24.268806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:07.738806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.263688+0000","last_change":"2024-09-10T20:42:23.251025+0000","last_active":"2024-09-10T20:42:24.263688+0000","last_peered":"2024-09-10T20:42:24.263688+0000","last_clean":"2024-09-10T20:42:24.263688+0000","last_became_active":"2024-09-10T20:42:23.250757+0000","last_became_peered":"2024-09-10T20:42:23.250757+0000","last_unstale":"2024-09-10T20:42:24.263688+0000","last_undegraded":"2024-09-10T20:42:24.263688+0000","last_fullsized":"2024-09-10T20:42:24.263688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:27:59.341727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.264579+0000","last_change":"2024-09-10T20:42:24.260041+0000","last_active":"2024-09-10T20:42:24.264579+0000","last_peered":"2024-09-10T20:42:24.264579+0000","last_clean":"2024-09-10T20:42:24.264579+0000","last_became_active":"2024-09-10T20:42:24.259715+0000","last_became_peered":"2024-09-10T20:42:24.259715+0000","last_unstale":"2024-09-10T20:42:24.264579+0000","last_undegraded":"2024-09-10T20:42:24.264579+0000","last_fullsized":"2024-09-10T20:42:24.264579+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:55.967204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266500+0000","last_change":"2024-09-10T20:42:23.241836+0000","last_active":"2024-09-10T20:42:24.266500+0000","last_peered":"2024-09-10T20:42:24.266500+0000","last_clean":"2024-09-10T20:42:24.266500+0000","last_became_active":"2024-09-10T20:42:23.241522+0000","last_became_peered":"2024-09-10T20:42:23.241522+0000","last_unstale":"2024-09-10T20:42:24.266500+0000","last_undegraded":"2024-09-10T20:42:24.266500+0000","last_fullsized":"2024-09-10T20:42:24.266500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:15.265502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262241+0000","last_change":"2024-09-10T20:42:24.260342+0000","last_active":"2024-09-10T20:42:24.262241+0000","last_peered":"2024-09-10T20:42:24.262241+0000","last_clean":"2024-09-10T20:42:24.262241+0000","last_became_active":"2024-09-10T20:42:24.260044+0000","last_became_peered":"2024-09-10T20:42:24.260044+0000","last_unstale":"2024-09-10T20:42:24.262241+0000","last_undegraded":"2024-09-10T20:42:24.262241+0000","last_fullsized":"2024-09-10T20:42:24.262241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:21:34.514066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.265222+0000","last_change":"2024-09-10T20:42:23.244794+0000","last_active":"2024-09-10T20:42:24.265222+0000","last_peered":"2024-09-10T20:42:24.265222+0000","last_clean":"2024-09-10T20:42:24.265222+0000","last_became_active":"2024-09-10T20:42:23.244466+0000","last_became_peered":"2024-09-10T20:42:23.244466+0000","last_unstale":"2024-09-10T20:42:24.265222+0000","last_undegraded":"2024-09-10T20:42:24.265222+0000","last_fullsized":"2024-09-10T20:42:24.265222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:13:02.552327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.259243+0000","last_change":"2024-09-10T20:42:24.257623+0000","last_active":"2024-09-10T20:42:24.259243+0000","last_peered":"2024-09-10T20:42:24.259243+0000","last_clean":"2024-09-10T20:42:24.259243+0000","last_became_active":"2024-09-10T20:42:24.257055+0000","last_became_peered":"2024-09-10T20:42:24.257055+0000","last_unstale":"2024-09-10T20:42:24.259243+0000","last_undegraded":"2024-09-10T20:42:24.259243+0000","last_fullsized":"2024-09-10T20:42:24.259243+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:31:39.725236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.236403+0000","last_change":"2024-09-10T20:42:23.246728+0000","last_active":"2024-09-10T20:42:24.236403+0000","last_peered":"2024-09-10T20:42:24.236403+0000","last_clean":"2024-09-10T20:42:24.236403+0000","last_became_active":"2024-09-10T20:42:23.245892+0000","last_became_peered":"2024-09-10T20:42:23.245892+0000","last_unstale":"2024-09-10T20:42:24.236403+0000","last_undegraded":"2024-09-10T20:42:24.236403+0000","last_fullsized":"2024-09-10T20:42:24.236403+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:47.440987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255620+0000","last_change":"2024-09-10T20:42:24.253960+0000","last_active":"2024-09-10T20:42:24.255620+0000","last_peered":"2024-09-10T20:42:24.255620+0000","last_clean":"2024-09-10T20:42:24.255620+0000","last_became_active":"2024-09-10T20:42:24.253697+0000","last_became_peered":"2024-09-10T20:42:24.253697+0000","last_unstale":"2024-09-10T20:42:24.255620+0000","last_undegraded":"2024-09-10T20:42:24.255620+0000","last_fullsized":"2024-09-10T20:42:24.255620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:05.073601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.241984+0000","last_change":"2024-09-10T20:42:23.249368+0000","last_active":"2024-09-10T20:42:24.241984+0000","last_peered":"2024-09-10T20:42:24.241984+0000","last_clean":"2024-09-10T20:42:24.241984+0000","last_became_active":"2024-09-10T20:42:23.249095+0000","last_became_peered":"2024-09-10T20:42:23.249095+0000","last_unstale":"2024-09-10T20:42:24.241984+0000","last_undegraded":"2024-09-10T20:42:24.241984+0000","last_fullsized":"2024-09-10T20:42:24.241984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:50.459124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260891+0000","last_change":"2024-09-10T20:42:24.253368+0000","last_active":"2024-09-10T20:42:24.260891+0000","last_peered":"2024-09-10T20:42:24.260891+0000","last_clean":"2024-09-10T20:42:24.260891+0000","last_became_active":"2024-09-10T20:42:24.253019+0000","last_became_peered":"2024-09-10T20:42:24.253019+0000","last_unstale":"2024-09-10T20:42:24.260891+0000","last_undegraded":"2024-09-10T20:42:24.260891+0000","last_fullsized":"2024-09-10T20:42:24.260891+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:03:04.919447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267904+0000","last_change":"2024-09-10T20:42:23.248363+0000","last_active":"2024-09-10T20:42:24.267904+0000","last_peered":"2024-09-10T20:42:24.267904+0000","last_clean":"2024-09-10T20:42:24.267904+0000","last_became_active":"2024-09-10T20:42:23.247975+0000","last_became_peered":"2024-09-10T20:42:23.247975+0000","last_unstale":"2024-09-10T20:42:24.267904+0000","last_undegraded":"2024-09-10T20:42:24.267904+0000","last_fullsized":"2024-09-10T20:42:24.267904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:36:24.937647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.258804+0000","last_change":"2024-09-10T20:42:24.254541+0000","last_active":"2024-09-10T20:42:24.258804+0000","last_peered":"2024-09-10T20:42:24.258804+0000","last_clean":"2024-09-10T20:42:24.258804+0000","last_became_active":"2024-09-10T20:42:24.254251+0000","last_became_peered":"2024-09-10T20:42:24.254251+0000","last_unstale":"2024-09-10T20:42:24.258804+0000","last_undegraded":"2024-09-10T20:42:24.258804+0000","last_fullsized":"2024-09-10T20:42:24.258804+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:12:41.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266762+0000","last_change":"2024-09-10T20:42:23.252893+0000","last_active":"2024-09-10T20:42:24.266762+0000","last_peered":"2024-09-10T20:42:24.266762+0000","last_clean":"2024-09-10T20:42:24.266762+0000","last_became_active":"2024-09-10T20:42:23.252544+0000","last_became_peered":"2024-09-10T20:42:23.252544+0000","last_unstale":"2024-09-10T20:42:24.266762+0000","last_undegraded":"2024-09-10T20:42:24.266762+0000","last_fullsized":"2024-09-10T20:42:24.266762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:14:55.067011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.253451+0000","last_change":"2024-09-10T20:42:24.248801+0000","last_active":"2024-09-10T20:42:24.253451+0000","last_peered":"2024-09-10T20:42:24.253451+0000","last_clean":"2024-09-10T20:42:24.253451+0000","last_became_active":"2024-09-10T20:42:24.248477+0000","last_became_peered":"2024-09-10T20:42:24.248477+0000","last_unstale":"2024-09-10T20:42:24.253451+0000","last_undegraded":"2024-09-10T20:42:24.253451+0000","last_fullsized":"2024-09-10T20:42:24.253451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:12:41.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.263349+0000","last_change":"2024-09-10T20:42:23.252895+0000","last_active":"2024-09-10T20:42:24.263349+0000","last_peered":"2024-09-10T20:42:24.263349+0000","last_clean":"2024-09-10T20:42:24.263349+0000","last_became_active":"2024-09-10T20:42:23.252592+0000","last_became_peered":"2024-09-10T20:42:23.252592+0000","last_unstale":"2024-09-10T20:42:24.263349+0000","last_undegraded":"2024-09-10T20:42:24.263349+0000","last_fullsized":"2024-09-10T20:42:24.263349+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:12:44.327544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.258704+0000","last_change":"2024-09-10T20:42:24.253114+0000","last_active":"2024-09-10T20:42:24.258704+0000","last_peered":"2024-09-10T20:42:24.258704+0000","last_clean":"2024-09-10T20:42:24.258704+0000","last_became_active":"2024-09-10T20:42:24.252817+0000","last_became_peered":"2024-09-10T20:42:24.252817+0000","last_unstale":"2024-09-10T20:42:24.258704+0000","last_undegraded":"2024-09-10T20:42:24.258704+0000","last_fullsized":"2024-09-10T20:42:24.258704+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:53:10.278617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.240721+0000","last_change":"2024-09-10T20:42:23.250304+0000","last_active":"2024-09-10T20:42:24.240721+0000","last_peered":"2024-09-10T20:42:24.240721+0000","last_clean":"2024-09-10T20:42:24.240721+0000","last_became_active":"2024-09-10T20:42:23.250025+0000","last_became_peered":"2024-09-10T20:42:23.250025+0000","last_unstale":"2024-09-10T20:42:24.240721+0000","last_undegraded":"2024-09-10T20:42:24.240721+0000","last_fullsized":"2024-09-10T20:42:24.240721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:02.302834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.255492+0000","last_change":"2024-09-10T20:42:24.251276+0000","last_active":"2024-09-10T20:42:24.255492+0000","last_peered":"2024-09-10T20:42:24.255492+0000","last_clean":"2024-09-10T20:42:24.255492+0000","last_became_active":"2024-09-10T20:42:24.251007+0000","last_became_peered":"2024-09-10T20:42:24.251007+0000","last_unstale":"2024-09-10T20:42:24.255492+0000","last_undegraded":"2024-09-10T20:42:24.255492+0000","last_fullsized":"2024-09-10T20:42:24.255492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:25:51.538368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262294+0000","last_change":"2024-09-10T20:42:23.249350+0000","last_active":"2024-09-10T20:42:24.262294+0000","last_peered":"2024-09-10T20:42:24.262294+0000","last_clean":"2024-09-10T20:42:24.262294+0000","last_became_active":"2024-09-10T20:42:23.249058+0000","last_became_peered":"2024-09-10T20:42:23.249058+0000","last_unstale":"2024-09-10T20:42:24.262294+0000","last_undegraded":"2024-09-10T20:42:24.262294+0000","last_fullsized":"2024-09-10T20:42:24.262294+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:16:10.406185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260892+0000","last_change":"2024-09-10T20:42:24.254958+0000","last_active":"2024-09-10T20:42:24.260892+0000","last_peered":"2024-09-10T20:42:24.260892+0000","last_clean":"2024-09-10T20:42:24.260892+0000","last_became_active":"2024-09-10T20:42:24.254668+0000","last_became_peered":"2024-09-10T20:42:24.254668+0000","last_unstale":"2024-09-10T20:42:24.260892+0000","last_undegraded":"2024-09-10T20:42:24.260892+0000","last_fullsized":"2024-09-10T20:42:24.260892+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:47:33.211289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.236779+0000","last_change":"2024-09-10T20:42:23.246818+0000","last_active":"2024-09-10T20:42:24.236779+0000","last_peered":"2024-09-10T20:42:24.236779+0000","last_clean":"2024-09-10T20:42:24.236779+0000","last_became_active":"2024-09-10T20:42:23.246510+0000","last_became_peered":"2024-09-10T20:42:23.246510+0000","last_unstale":"2024-09-10T20:42:24.236779+0000","last_undegraded":"2024-09-10T20:42:24.236779+0000","last_fullsized":"2024-09-10T20:42:24.236779+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:32.249681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256201+0000","last_change":"2024-09-10T20:42:24.254492+0000","last_active":"2024-09-10T20:42:24.256201+0000","last_peered":"2024-09-10T20:42:24.256201+0000","last_clean":"2024-09-10T20:42:24.256201+0000","last_became_active":"2024-09-10T20:42:24.254157+0000","last_became_peered":"2024-09-10T20:42:24.254157+0000","last_unstale":"2024-09-10T20:42:24.256201+0000","last_undegraded":"2024-09-10T20:42:24.256201+0000","last_fullsized":"2024-09-10T20:42:24.256201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:40.202138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.246102+0000","last_change":"2024-09-10T20:42:23.244187+0000","last_active":"2024-09-10T20:42:24.246102+0000","last_peered":"2024-09-10T20:42:24.246102+0000","last_clean":"2024-09-10T20:42:24.246102+0000","last_became_active":"2024-09-10T20:42:23.243905+0000","last_became_peered":"2024-09-10T20:42:23.243905+0000","last_unstale":"2024-09-10T20:42:24.246102+0000","last_undegraded":"2024-09-10T20:42:24.246102+0000","last_fullsized":"2024-09-10T20:42:24.246102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:48.713410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262481+0000","last_change":"2024-09-10T20:42:24.259044+0000","last_active":"2024-09-10T20:42:24.262481+0000","last_peered":"2024-09-10T20:42:24.262481+0000","last_clean":"2024-09-10T20:42:24.262481+0000","last_became_active":"2024-09-10T20:42:24.257959+0000","last_became_peered":"2024-09-10T20:42:24.257959+0000","last_unstale":"2024-09-10T20:42:24.262481+0000","last_undegraded":"2024-09-10T20:42:24.262481+0000","last_fullsized":"2024-09-10T20:42:24.262481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:31:05.117623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.240101+0000","last_change":"2024-09-10T20:42:23.250015+0000","last_active":"2024-09-10T20:42:24.240101+0000","last_peered":"2024-09-10T20:42:24.240101+0000","last_clean":"2024-09-10T20:42:24.240101+0000","last_became_active":"2024-09-10T20:42:23.249203+0000","last_became_peered":"2024-09-10T20:42:23.249203+0000","last_unstale":"2024-09-10T20:42:24.240101+0000","last_undegraded":"2024-09-10T20:42:24.240101+0000","last_fullsized":"2024-09-10T20:42:24.240101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:04:42.109627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266201+0000","last_change":"2024-09-10T20:42:24.259245+0000","last_active":"2024-09-10T20:42:24.266201+0000","last_peered":"2024-09-10T20:42:24.266201+0000","last_clean":"2024-09-10T20:42:24.266201+0000","last_became_active":"2024-09-10T20:42:24.258793+0000","last_became_peered":"2024-09-10T20:42:24.258793+0000","last_unstale":"2024-09-10T20:42:24.266201+0000","last_undegraded":"2024-09-10T20:42:24.266201+0000","last_fullsized":"2024-09-10T20:42:24.266201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:11:41.961270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.267931+0000","last_change":"2024-09-10T20:42:23.252693+0000","last_active":"2024-09-10T20:42:24.267931+0000","last_peered":"2024-09-10T20:42:24.267931+0000","last_clean":"2024-09-10T20:42:24.267931+0000","last_became_active":"2024-09-10T20:42:23.252461+0000","last_became_peered":"2024-09-10T20:42:23.252461+0000","last_unstale":"2024-09-10T20:42:24.267931+0000","last_undegraded":"2024-09-10T20:42:24.267931+0000","last_fullsized":"2024-09-10T20:42:24.267931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:27:21.791560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266257+0000","last_change":"2024-09-10T20:42:24.262908+0000","last_active":"2024-09-10T20:42:24.266257+0000","last_peered":"2024-09-10T20:42:24.266257+0000","last_clean":"2024-09-10T20:42:24.266257+0000","last_became_active":"2024-09-10T20:42:24.262614+0000","last_became_peered":"2024-09-10T20:42:24.262614+0000","last_unstale":"2024-09-10T20:42:24.266257+0000","last_undegraded":"2024-09-10T20:42:24.266257+0000","last_fullsized":"2024-09-10T20:42:24.266257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:46.383244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.244301+0000","last_change":"2024-09-10T20:42:23.247648+0000","last_active":"2024-09-10T20:42:24.244301+0000","last_peered":"2024-09-10T20:42:24.244301+0000","last_clean":"2024-09-10T20:42:24.244301+0000","last_became_active":"2024-09-10T20:42:23.247345+0000","last_became_peered":"2024-09-10T20:42:23.247345+0000","last_unstale":"2024-09-10T20:42:24.244301+0000","last_undegraded":"2024-09-10T20:42:24.244301+0000","last_fullsized":"2024-09-10T20:42:24.244301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:49:11.666986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260129+0000","last_change":"2024-09-10T20:42:24.255585+0000","last_active":"2024-09-10T20:42:24.260129+0000","last_peered":"2024-09-10T20:42:24.260129+0000","last_clean":"2024-09-10T20:42:24.260129+0000","last_became_active":"2024-09-10T20:42:24.255340+0000","last_became_peered":"2024-09-10T20:42:24.255340+0000","last_unstale":"2024-09-10T20:42:24.260129+0000","last_undegraded":"2024-09-10T20:42:24.260129+0000","last_fullsized":"2024-09-10T20:42:24.260129+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:55.045900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.240007+0000","last_change":"2024-09-10T20:42:23.249288+0000","last_active":"2024-09-10T20:42:24.240007+0000","last_peered":"2024-09-10T20:42:24.240007+0000","last_clean":"2024-09-10T20:42:24.240007+0000","last_became_active":"2024-09-10T20:42:23.248991+0000","last_became_peered":"2024-09-10T20:42:23.248991+0000","last_unstale":"2024-09-10T20:42:24.240007+0000","last_undegraded":"2024-09-10T20:42:24.240007+0000","last_fullsized":"2024-09-10T20:42:24.240007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:41.414236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.248573+0000","last_change":"2024-09-10T20:42:24.245873+0000","last_active":"2024-09-10T20:42:24.248573+0000","last_peered":"2024-09-10T20:42:24.248573+0000","last_clean":"2024-09-10T20:42:24.248573+0000","last_became_active":"2024-09-10T20:42:24.245555+0000","last_became_peered":"2024-09-10T20:42:24.245555+0000","last_unstale":"2024-09-10T20:42:24.248573+0000","last_undegraded":"2024-09-10T20:42:24.248573+0000","last_fullsized":"2024-09-10T20:42:24.248573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:53:10.278617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.242169+0000","last_change":"2024-09-10T20:42:23.250318+0000","last_active":"2024-09-10T20:42:24.242169+0000","last_peered":"2024-09-10T20:42:24.242169+0000","last_clean":"2024-09-10T20:42:24.242169+0000","last_became_active":"2024-09-10T20:42:23.249984+0000","last_became_peered":"2024-09-10T20:42:23.249984+0000","last_unstale":"2024-09-10T20:42:24.242169+0000","last_undegraded":"2024-09-10T20:42:24.242169+0000","last_fullsized":"2024-09-10T20:42:24.242169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:41.414236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256345+0000","last_change":"2024-09-10T20:42:24.250834+0000","last_active":"2024-09-10T20:42:24.256345+0000","last_peered":"2024-09-10T20:42:24.256345+0000","last_clean":"2024-09-10T20:42:24.256345+0000","last_became_active":"2024-09-10T20:42:24.250522+0000","last_became_peered":"2024-09-10T20:42:24.250522+0000","last_unstale":"2024-09-10T20:42:24.256345+0000","last_undegraded":"2024-09-10T20:42:24.256345+0000","last_fullsized":"2024-09-10T20:42:24.256345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:49:12.669728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.235738+0000","last_change":"2024-09-10T20:42:23.245079+0000","last_active":"2024-09-10T20:42:24.235738+0000","last_peered":"2024-09-10T20:42:24.235738+0000","last_clean":"2024-09-10T20:42:24.235738+0000","last_became_active":"2024-09-10T20:42:23.244765+0000","last_became_peered":"2024-09-10T20:42:23.244765+0000","last_unstale":"2024-09-10T20:42:24.235738+0000","last_undegraded":"2024-09-10T20:42:24.235738+0000","last_fullsized":"2024-09-10T20:42:24.235738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:16:05.597476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.260963+0000","last_change":"2024-09-10T20:42:24.254607+0000","last_active":"2024-09-10T20:42:24.260963+0000","last_peered":"2024-09-10T20:42:24.260963+0000","last_clean":"2024-09-10T20:42:24.260963+0000","last_became_active":"2024-09-10T20:42:24.254287+0000","last_became_peered":"2024-09-10T20:42:24.254287+0000","last_unstale":"2024-09-10T20:42:24.260963+0000","last_undegraded":"2024-09-10T20:42:24.260963+0000","last_fullsized":"2024-09-10T20:42:24.260963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:30:08.734512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.241754+0000","last_change":"2024-09-10T20:42:23.243965+0000","last_active":"2024-09-10T20:42:24.241754+0000","last_peered":"2024-09-10T20:42:24.241754+0000","last_clean":"2024-09-10T20:42:24.241754+0000","last_became_active":"2024-09-10T20:42:23.243684+0000","last_became_peered":"2024-09-10T20:42:23.243684+0000","last_unstale":"2024-09-10T20:42:24.241754+0000","last_undegraded":"2024-09-10T20:42:24.241754+0000","last_fullsized":"2024-09-10T20:42:24.241754+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:14.446361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.256582+0000","last_change":"2024-09-10T20:42:24.247470+0000","last_active":"2024-09-10T20:42:24.256582+0000","last_peered":"2024-09-10T20:42:24.256582+0000","last_clean":"2024-09-10T20:42:24.256582+0000","last_became_active":"2024-09-10T20:42:24.247218+0000","last_became_peered":"2024-09-10T20:42:24.247218+0000","last_unstale":"2024-09-10T20:42:24.256582+0000","last_undegraded":"2024-09-10T20:42:24.256582+0000","last_fullsized":"2024-09-10T20:42:24.256582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:22.324477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.235299+0000","last_change":"2024-09-10T20:42:23.250191+0000","last_active":"2024-09-10T20:42:24.235299+0000","last_peered":"2024-09-10T20:42:24.235299+0000","last_clean":"2024-09-10T20:42:24.235299+0000","last_became_active":"2024-09-10T20:42:23.249930+0000","last_became_peered":"2024-09-10T20:42:23.249930+0000","last_unstale":"2024-09-10T20:42:24.235299+0000","last_undegraded":"2024-09-10T20:42:24.235299+0000","last_fullsized":"2024-09-10T20:42:24.235299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:15.534284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.243492+0000","last_change":"2024-09-10T20:42:24.242253+0000","last_active":"2024-09-10T20:42:24.243492+0000","last_peered":"2024-09-10T20:42:24.243492+0000","last_clean":"2024-09-10T20:42:24.243492+0000","last_became_active":"2024-09-10T20:42:24.241966+0000","last_became_peered":"2024-09-10T20:42:24.241966+0000","last_unstale":"2024-09-10T20:42:24.243492+0000","last_undegraded":"2024-09-10T20:42:24.243492+0000","last_fullsized":"2024-09-10T20:42:24.243492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:58:37.573858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.241114+0000","last_change":"2024-09-10T20:42:23.248404+0000","last_active":"2024-09-10T20:42:24.241114+0000","last_peered":"2024-09-10T20:42:24.241114+0000","last_clean":"2024-09-10T20:42:24.241114+0000","last_became_active":"2024-09-10T20:42:23.247973+0000","last_became_peered":"2024-09-10T20:42:23.247973+0000","last_unstale":"2024-09-10T20:42:24.241114+0000","last_undegraded":"2024-09-10T20:42:24.241114+0000","last_fullsized":"2024-09-10T20:42:24.241114+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:03:04.402951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.266099+0000","last_change":"2024-09-10T20:42:24.261854+0000","last_active":"2024-09-10T20:42:24.266099+0000","last_peered":"2024-09-10T20:42:24.266099+0000","last_clean":"2024-09-10T20:42:24.266099+0000","last_became_active":"2024-09-10T20:42:24.261431+0000","last_became_peered":"2024-09-10T20:42:24.261431+0000","last_unstale":"2024-09-10T20:42:24.266099+0000","last_undegraded":"2024-09-10T20:42:24.266099+0000","last_fullsized":"2024-09-10T20:42:24.266099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:23.224946+0000","last_clean_scrub_stamp":"2024-09-10T20:42:23.224946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:50:02.685852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.262812+0000","last_change":"2024-09-10T20:42:23.252862+0000","last_active":"2024-09-10T20:42:24.262812+0000","last_peered":"2024-09-10T20:42:24.262812+0000","last_clean":"2024-09-10T20:42:24.262812+0000","last_became_active":"2024-09-10T20:42:23.252563+0000","last_became_peered":"2024-09-10T20:42:23.252563+0000","last_unstale":"2024-09-10T20:42:24.262812+0000","last_undegraded":"2024-09-10T20:42:24.262812+0000","last_fullsized":"2024-09-10T20:42:24.262812+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:22.222204+0000","last_clean_scrub_stamp":"2024-09-10T20:42:22.222204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:42.952894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":247,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T20:42:24.235567+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:42:24.235567+0000","last_peered":"2024-09-10T20:42:24.235567+0000","last_clean":"2024-09-10T20:42:24.235567+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:42:24.235567+0000","last_undegraded":"2024-09-10T20:42:24.235567+0000","last_fullsized":"2024-09-10T20:42:24.235567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542199,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32712,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1556480,"data_stored":1000489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542199,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509497,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509496,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":962560,"data_stored":410121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509496,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32712,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1556480,"data_stored":1000489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509496,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":991232,"data_stored":418530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:42:28.626 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:42:28.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:42:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 -- 172.21.15.28:0/1881343662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 msgr2=0x7f9eb4161c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 --2- 172.21.15.28:0/1881343662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4161c10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9eb408e230 tx=0x7f9eac02a820 comp rx=0 tx=0).stop 2024-09-10T20:42:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 -- 172.21.15.28:0/1881343662 shutdown_connections 2024-09-10T20:42:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 --2- 172.21.15.28:0/1881343662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4161c10 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 --2- 172.21.15.28:0/1881343662 >> v2:172.21.15.159:3300/0 conn(0x7f9eb40d4b60 0x7f9eb40cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 -- 172.21.15.28:0/1881343662 >> 172.21.15.28:0/1881343662 conn(0x7f9eb40d4040 msgr2=0x7f9eb40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 -- 172.21.15.28:0/1881343662 shutdown_connections 2024-09-10T20:42:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.770+0000 7f9eba852640 1 -- 172.21.15.28:0/1881343662 wait complete. 2024-09-10T20:42:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.771+0000 7f9eba852640 1 Processor -- start 2024-09-10T20:42:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.772+0000 7f9eba852640 1 -- start start 2024-09-10T20:42:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.772+0000 7f9eba852640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.772+0000 7f9eba852640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9eb40d4b60 0x7f9eb40cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.772+0000 7f9eba852640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9eb415e130 0x7f9eb40cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.772+0000 7f9eba852640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41622d0 con 0x7f9eb40cfa80 2024-09-10T20:42:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.772+0000 7f9eba852640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4162150 con 0x7f9eb415e130 2024-09-10T20:42:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.772+0000 7f9eba852640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb4162450 con 0x7f9eb40d4b60 2024-09-10T20:42:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4154900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41732/0 (socket says 172.21.15.28:41732) 2024-09-10T20:42:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 -- 172.21.15.28:0/2854092682 learned_addr learned my addr 172.21.15.28:0/2854092682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb8def640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9eb415e130 0x7f9eb40cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb37fe640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9eb40d4b60 0x7f9eb40cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 -- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9eb40d4b60 msgr2=0x7f9eb40cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9eb40d4b60 0x7f9eb40cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 -- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9eb415e130 msgr2=0x7f9eb40cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9eb415e130 0x7f9eb40cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 -- 172.21.15.28:0/2854092682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb41f8110 con 0x7f9eb40cfa80 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb37fe640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9eb40d4b60 0x7f9eb40cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb3fff640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4154900 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c430 tx=0x7f9ea400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb0ff9640 1 -- 172.21.15.28:0/2854092682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4011790 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb0ff9640 1 -- 172.21.15.28:0/2854092682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ea4017070 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eb0ff9640 1 -- 172.21.15.28:0/2854092682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4011ce0 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb41f83f0 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.773+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb41f89b0 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.774+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb4162980 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.775+0000 7f9eb0ff9640 1 -- 172.21.15.28:0/2854092682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f9ea401c460 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.776+0000 7f9eb0ff9640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ea0078090 0x7f9ea007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.776+0000 7f9eb37fe640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ea0078090 0x7f9ea007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.776+0000 7f9eb0ff9640 1 -- 172.21.15.28:0/2854092682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f9ea4099d40 con 0x7f9eb40cfa80 2024-09-10T20:42:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.776+0000 7f9eb37fe640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ea0078090 0x7f9ea007a550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40b11e0 tx=0x7f9ea8005ec0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:28.780+0000 7f9eb0ff9640 1 -- 172.21.15.28:0/2854092682 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ea4062580 con 0x7f9eb40cfa80 2024-09-10T20:42:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.041+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f9eb40b58c0 con 0x7f9eb40cfa80 2024-09-10T20:42:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.043+0000 7f9eb0ff9640 1 -- 172.21.15.28:0/2854092682 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v31) ==== 190+0+147 (secure 0 0 0) 0x7f9ea4061e80 con 0x7f9eb40cfa80 2024-09-10T20:42:29.047 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:42:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:42:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:42:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:42:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:42:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:42:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ea0078090 msgr2=0x7f9ea007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ea0078090 0x7f9ea007a550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40b11e0 tx=0x7f9ea8005ec0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 msgr2=0x7f9eb4154900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4154900 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c430 tx=0x7f9ea400c8f0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 shutdown_connections 2024-09-10T20:42:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ea0078090 0x7f9ea007a550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9eb415e130 0x7f9eb40cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9eb40d4b60 0x7f9eb40cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 --2- 172.21.15.28:0/2854092682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb40cfa80 0x7f9eb4154900 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.045+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 >> 172.21.15.28:0/2854092682 conn(0x7f9eb40d4040 msgr2=0x7f9eb406ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.046+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 shutdown_connections 2024-09-10T20:42:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.046+0000 7f9eba852640 1 -- 172.21.15.28:0/2854092682 wait complete. 2024-09-10T20:42:29.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 -- 172.21.15.28:0/4048236685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f0162090 msgr2=0x7fe4f00b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/4048236685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f0162090 0x7fe4f00b6e00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009830 tx=0x7fe4e002a510 comp rx=0 tx=0).stop 2024-09-10T20:42:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 -- 172.21.15.28:0/4048236685 shutdown_connections 2024-09-10T20:42:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/4048236685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f0162090 0x7fe4f00b6e00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/4048236685 >> v2:172.21.15.159:3300/0 conn(0x7fe4f00d2b90 0x7fe4f0161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 -- 172.21.15.28:0/4048236685 >> 172.21.15.28:0/4048236685 conn(0x7fe4f00d3fb0 msgr2=0x7fe4f00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 -- 172.21.15.28:0/4048236685 shutdown_connections 2024-09-10T20:42:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.163+0000 7fe4f4a33640 1 -- 172.21.15.28:0/4048236685 wait complete. 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.164+0000 7fe4f4a33640 1 Processor -- start 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.164+0000 7fe4f4a33640 1 -- start start 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4f4a33640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f00d2b90 0x7fe4f014d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4f4a33640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4f015de70 0x7fe4f0160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4f4a33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4f0162090 0x7fe4f01502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4f4a33640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f015e3d0 con 0x7fe4f0162090 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4f4a33640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f015e250 con 0x7fe4f015de70 2024-09-10T20:42:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4f4a33640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f00b7340 con 0x7fe4f00d2b90 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4f0162090 0x7fe4f01502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4f0162090 0x7fe4f01502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41746/0 (socket says 172.21.15.28:41746) 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 -- 172.21.15.28:0/1436144615 learned_addr learned my addr 172.21.15.28:0/1436144615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4edd9b640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4f015de70 0x7fe4f0160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4ee59c640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f00d2b90 0x7fe4f014d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 -- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f00d2b90 msgr2=0x7fe4f014d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f00d2b90 0x7fe4f014d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 -- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4f015de70 msgr2=0x7fe4f0160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4f015de70 0x7fe4f0160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.165+0000 7fe4eed9d640 1 -- 172.21.15.28:0/1436144615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f01507e0 con 0x7fe4f0162090 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.166+0000 7fe4eed9d640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4f0162090 0x7fe4f01502a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe4e002ab00 tx=0x7fe4e0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.166+0000 7fe4e6ffd640 1 -- 172.21.15.28:0/1436144615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0005c90 con 0x7fe4f0162090 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.166+0000 7fe4e6ffd640 1 -- 172.21.15.28:0/1436144615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4e0031e50 con 0x7fe4f0162090 2024-09-10T20:42:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.166+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f015fec0 con 0x7fe4f0162090 2024-09-10T20:42:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.166+0000 7fe4e6ffd640 1 -- 172.21.15.28:0/1436144615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0033330 con 0x7fe4f0162090 2024-09-10T20:42:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.166+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f0160480 con 0x7fe4f0162090 2024-09-10T20:42:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.167+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4f00b65d0 con 0x7fe4f0162090 2024-09-10T20:42:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.168+0000 7fe4e6ffd640 1 -- 172.21.15.28:0/1436144615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fe4e00334d0 con 0x7fe4f0162090 2024-09-10T20:42:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.168+0000 7fe4e6ffd640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4d00780c0 0x7fe4d007a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.168+0000 7fe4e6ffd640 1 -- 172.21.15.28:0/1436144615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fe4e00bb0f0 con 0x7fe4f0162090 2024-09-10T20:42:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.168+0000 7fe4ee59c640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4d00780c0 0x7fe4d007a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.169+0000 7fe4ee59c640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4d00780c0 0x7fe4d007a580 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8001ec0 tx=0x7fe4e8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.172+0000 7fe4e6ffd640 1 -- 172.21.15.28:0/1436144615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe4f00b65d0 con 0x7fe4f0162090 2024-09-10T20:42:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.394+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4f00b77a0 con 0x7fe4f0162090 2024-09-10T20:42:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.395+0000 7fe4e6ffd640 1 -- 172.21.15.28:0/1436144615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5231 (secure 0 0 0) 0x7fe4e003f030 con 0x7fe4f0162090 2024-09-10T20:42:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:29.399 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"btime":"2024-09-10T20:42:27:250005+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2983894865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2983894865},{"type":"v1","addr":"172.21.15.159:6801","nonce":2983894865}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2105531171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2105531171},{"type":"v1","addr":"172.21.15.159:6835","nonce":2105531171}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2239499017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2239499017},{"type":"v1","addr":"172.21.15.28:6834","nonce":2239499017}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:23.614228+0000","modified":"2024-09-10T20:42:26.617829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6570":{"gid":6570,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2216477518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2216477518},{"type":"v1","addr":"172.21.15.28:6835","nonce":2216477518}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6570,"qdb_cluster":[6570]},"id":6}]} 2024-09-10T20:42:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-09-10T20:42:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4d00780c0 msgr2=0x7fe4d007a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4d00780c0 0x7fe4d007a580 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8001ec0 tx=0x7fe4e8009040 comp rx=0 tx=0).stop 2024-09-10T20:42:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4f0162090 msgr2=0x7fe4f01502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4f0162090 0x7fe4f01502a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe4e002ab00 tx=0x7fe4e0033c20 comp rx=0 tx=0).stop 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 shutdown_connections 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4d00780c0 0x7fe4d007a580 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4f0162090 0x7fe4f01502a0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4f015de70 0x7fe4f0160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 --2- 172.21.15.28:0/1436144615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4f00d2b90 0x7fe4f014d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 >> 172.21.15.28:0/1436144615 conn(0x7fe4f00d3fb0 msgr2=0x7fe4f00b7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.397+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 shutdown_connections 2024-09-10T20:42:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.398+0000 7fe4f4a33640 1 -- 172.21.15.28:0/1436144615 wait complete. 2024-09-10T20:42:29.573 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-10T20:42:29.570+0000 7fbeb250c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:42:29.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:42:23.614228+0000', 'modified': '2024-09-10T20:42:26.617829+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6570': {'gid': 6570, 'name': 'a', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2216477518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 2216477518}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2216477518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6570, 'qdb_cluster': [6570]} 2024-09-10T20:42:29.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:42:29.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-10T20:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.922+0000 7f8266a2b640 1 -- 172.21.15.28:0/2154782928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600747a0 msgr2=0x7f826015c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.922+0000 7f8266a2b640 1 --2- 172.21.15.28:0/2154782928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600747a0 0x7f826015c160 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8250009830 tx=0x7f825002a510 comp rx=0 tx=0).stop 2024-09-10T20:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.922+0000 7f8266a2b640 1 -- 172.21.15.28:0/2154782928 shutdown_connections 2024-09-10T20:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.922+0000 7f8266a2b640 1 --2- 172.21.15.28:0/2154782928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600747a0 0x7f826015c160 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.922+0000 7f8266a2b640 1 -- 172.21.15.28:0/2154782928 >> 172.21.15.28:0/2154782928 conn(0x7f826006dbf0 msgr2=0x7f826006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.922+0000 7f8266a2b640 1 -- 172.21.15.28:0/2154782928 shutdown_connections 2024-09-10T20:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.922+0000 7f8266a2b640 1 -- 172.21.15.28:0/2154782928 wait complete. 2024-09-10T20:42:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.924+0000 7f8266a2b640 1 Processor -- start 2024-09-10T20:42:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.924+0000 7f8266a2b640 1 -- start start 2024-09-10T20:42:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8266a2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826006cae0 0x7f826014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8266a2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82600747a0 0x7f82600b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8266a2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600b5a50 0x7f82600b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8266a2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f826006cf30 con 0x7f82600b5a50 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8266a2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f82600b05d0 con 0x7f82600747a0 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8266a2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f826006d0b0 con 0x7f826006cae0 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600b5a50 0x7f82600b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600b5a50 0x7f82600b23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41764/0 (socket says 172.21.15.28:41764) 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 -- 172.21.15.28:0/4280627634 learned_addr learned my addr 172.21.15.28:0/4280627634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f825ffff640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826006cae0 0x7f826014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f825f7fe640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82600747a0 0x7f82600b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 -- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826006cae0 msgr2=0x7f826014eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826006cae0 0x7f826014eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 -- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82600747a0 msgr2=0x7f82600b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82600747a0 0x7f82600b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 -- 172.21.15.28:0/4280627634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826009ead0 con 0x7f82600b5a50 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f8264fc8640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600b5a50 0x7f82600b23b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f825800c780 tx=0x7f825800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f825cff9640 1 -- 172.21.15.28:0/4280627634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258004b60 con 0x7f82600b5a50 2024-09-10T20:42:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.925+0000 7f825cff9640 1 -- 172.21.15.28:0/4280627634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8258004df0 con 0x7f82600b5a50 2024-09-10T20:42:29.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.926+0000 7f825cff9640 1 -- 172.21.15.28:0/4280627634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825800aa10 con 0x7f82600b5a50 2024-09-10T20:42:29.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.926+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826009e510 con 0x7f82600b5a50 2024-09-10T20:42:29.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.926+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826009aad0 con 0x7f82600b5a50 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.927+0000 7f825cff9640 1 -- 172.21.15.28:0/4280627634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f825800abb0 con 0x7f82600b5a50 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.928+0000 7f825cff9640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f824c078090 0x7f824c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.928+0000 7f825cff9640 1 -- 172.21.15.28:0/4280627634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f825809a0a0 con 0x7f82600b5a50 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.928+0000 7f825ffff640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f824c078090 0x7f824c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.928+0000 7f825ffff640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f824c078090 0x7f824c07a550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8250034000 tx=0x7f8250004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 Processor -- start 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 -- start start 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf746b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf78f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf7445f0 0x55f5cf744a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf6bb820 con 0x55f5cf78edf0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf6aac50 con 0x55f5cf7445f0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.929+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55f5cf6aea60 con 0x55f5cf76f1b0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf78f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf78f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41768/0 (socket says 172.21.15.28:41768) 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 -- 172.21.15.28:0/1391970849 learned_addr learned my addr 172.21.15.28:0/1391970849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825ffff640 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf746b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f8264fc8640 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf7445f0 0x55f5cf744a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 -- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 msgr2=0x55f5cf746b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf746b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 -- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf7445f0 msgr2=0x55f5cf744a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf7445f0 0x55f5cf744a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 -- 172.21.15.28:0/1391970849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5cf40c850 con 0x55f5cf78edf0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f825f7fe640 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf78f1f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8254007f20 tx=0x7f8254001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b8bb9cbb6afc0c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f82467fc640 1 -- 172.21.15.28:0/1391970849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254018020 con 0x55f5cf78edf0 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f82467fc640 1 -- 172.21.15.28:0/1391970849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f825400fb70 con 0x55f5cf78edf0 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.930+0000 7f82467fc640 1 -- 172.21.15.28:0/1391970849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254014470 con 0x55f5cf78edf0 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 -- 172.21.15.28:0/1391970849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 msgr2=0x55f5cf78f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf78f1f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8254007f20 tx=0x7f8254001cb0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 -- 172.21.15.28:0/1391970849 shutdown_connections 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf7445f0 0x55f5cf744a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf78f1f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 --2- 172.21.15.28:0/1391970849 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf746b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 -- 172.21.15.28:0/1391970849 >> 172.21.15.28:0/1391970849 conn(0x55f5cf7456a0 msgr2=0x55f5cf7490c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 -- 172.21.15.28:0/1391970849 shutdown_connections 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 -- 172.21.15.28:0/1391970849 wait complete. 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.931+0000 7f82697f4740 1 -- start start 2024-09-10T20:42:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 1 Processor -- start 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf759ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 0x55f5cf76df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf7693e0 0x55f5cf76e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf41b620 con 0x55f5cf7693e0 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf6505f0 con 0x55f5cf78edf0 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55f5cf743b30 con 0x55f5cf76f1b0 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.932+0000 7f82697f4740 10 client.0 _ll_register_callbacks cb 0x55f5cf6540a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf7693e0 0x55f5cf76e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f825ffff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf759ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f825ffff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf759ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:43100/0 (socket says 172.21.15.28:43100) 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf7693e0 0x55f5cf76e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41778/0 (socket says 172.21.15.28:41778) 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f825ffff640 1 -- 172.21.15.28:0/3388563800 learned_addr learned my addr 172.21.15.28:0/3388563800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f825f7fe640 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 0x55f5cf76df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 -- 172.21.15.28:0/3388563800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 msgr2=0x55f5cf759ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf759ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 -- 172.21.15.28:0/3388563800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 msgr2=0x55f5cf76df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 0x55f5cf76df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 -- 172.21.15.28:0/3388563800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5cf4836b0 con 0x55f5cf7693e0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f8264fc8640 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf7693e0 0x55f5cf76e480 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f8258098340 tx=0x7f825805e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ceb2e8cb4f0166 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f823effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f823effd640 1 -- 172.21.15.28:0/3388563800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258061e40 con 0x55f5cf7693e0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f823effd640 1 -- 172.21.15.28:0/3388563800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f825806b2f0 con 0x55f5cf7693e0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825400fb70 con 0x55f5cf7693e0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f823effd640 1 -- 172.21.15.28:0/3388563800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f8258062330 con 0x55f5cf7693e0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f82697f4740 10 client.6702 fetch_fsmap learned FSMap version 53 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.933+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55f5cf758590 con 0x55f5cf7693e0 2024-09-10T20:42:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f823effd640 1 -- 172.21.15.28:0/3388563800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f825400fb70 con 0x55f5cf7693e0 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f823effd640 1 -- 172.21.15.28:0/3388563800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 53) ==== 3502+0+0 (secure 0 0 0) 0x7f8258006b50 con 0x55f5cf7693e0 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f82697f4740 10 client.6702 fetch_fsmap finished waiting for FSMap version 53 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f82697f4740 10 client.6702 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f82697f4740 20 client.6702 populate_metadata read hostname 'smithi028' 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f82697f4740 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf7a1620 0x55f5cf7a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f82697f4740 4 client.6702 mds_command: new command op to 6570 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 --> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5cf6bd770 con 0x55f5cf7a1620 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f825ffff640 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf7a1620 0x55f5cf7a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f825ffff640 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf7a1620 0x55f5cf7a1a00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8754cdc683d2653d server_cookie=cc9178b612d568ef in_seq=0 out_seq=0 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.934+0000 7f823effd640 10 client.6702 ms_handle_connect on v2:172.21.15.28:6833/2216477518 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f823effd640 1 -- 172.21.15.28:0/3388563800 <== mds.0 v2:172.21.15.28:6833/2216477518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f5cf6bd770 con 0x55f5cf7a1620 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f823effd640 10 client.6702 handle_command_reply: tid=0 2024-09-10T20:42:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 client.6702 shutdown 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf7693e0 msgr2=0x55f5cf76e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf7693e0 0x55f5cf76e480 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f8258098340 tx=0x7f825805e820 comp rx=0 tx=0).stop 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 shutdown_connections 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf7693e0 0x55f5cf76e480 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 0x55f5cf76df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf7a1620 0x55f5cf7a1a00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 --2- 172.21.15.28:0/3388563800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf76f1b0 0x55f5cf759ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 >> 172.21.15.28:0/3388563800 conn(0x55f5cf7445f0 msgr2=0x55f5cf748ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f823effd640 0 client.6702 ms_handle_reset on v2:172.21.15.28:6833/2216477518 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 shutdown_connections 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 1 -- 172.21.15.28:0/3388563800 wait complete. 2024-09-10T20:42:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.935+0000 7f82697f4740 20 client.6702 trim_cache size 0 max 16384 2024-09-10T20:42:29.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 Processor -- start 2024-09-10T20:42:29.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 -- start start 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf758bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf76f1b0 0x55f5cf7b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 0x55f5cf7b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82601f8d70 con 0x55f5cf76f1b0 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f826009b870 con 0x55f5cf78edf0 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f82601f86e0 con 0x55f5cf7693e0 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f825f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf76f1b0 0x55f5cf7b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f825f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf76f1b0 0x55f5cf7b2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41782/0 (socket says 172.21.15.28:41782) 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.954+0000 7f825f7fe640 1 -- 172.21.15.28:0/3935443226 learned_addr learned my addr 172.21.15.28:0/3935443226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f825f7fe640 1 -- 172.21.15.28:0/3935443226 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 msgr2=0x55f5cf758bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f825f7fe640 1 --2- 172.21.15.28:0/3935443226 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf758bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f825f7fe640 1 -- 172.21.15.28:0/3935443226 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 msgr2=0x55f5cf7b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f825f7fe640 1 --2- 172.21.15.28:0/3935443226 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 0x55f5cf7b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f825f7fe640 1 -- 172.21.15.28:0/3935443226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5cf769800 con 0x55f5cf76f1b0 2024-09-10T20:42:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f825f7fe640 1 --2- 172.21.15.28:0/3935443226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf76f1b0 0x55f5cf7b2b90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f825400f5a0 tx=0x7f8254004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5835b028f8ab47ee server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f8245ffb640 1 -- 172.21.15.28:0/3935443226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254018040 con 0x55f5cf76f1b0 2024-09-10T20:42:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f8245ffb640 1 -- 172.21.15.28:0/3935443226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8254005020 con 0x55f5cf76f1b0 2024-09-10T20:42:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f8245ffb640 1 -- 172.21.15.28:0/3935443226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254005470 con 0x55f5cf76f1b0 2024-09-10T20:42:29.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.955+0000 7f82697f4740 1 -- 172.21.15.28:0/3935443226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf76f1b0 msgr2=0x55f5cf7b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 --2- 172.21.15.28:0/3935443226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf76f1b0 0x55f5cf7b2b90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f825400f5a0 tx=0x7f8254004620 comp rx=0 tx=0).stop 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 -- 172.21.15.28:0/3935443226 shutdown_connections 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 --2- 172.21.15.28:0/3935443226 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf78edf0 0x55f5cf7b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 --2- 172.21.15.28:0/3935443226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf76f1b0 0x55f5cf7b2b90 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 --2- 172.21.15.28:0/3935443226 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf758bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 -- 172.21.15.28:0/3935443226 >> 172.21.15.28:0/3935443226 conn(0x55f5cf7445f0 msgr2=0x55f5cf787d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 -- 172.21.15.28:0/3935443226 shutdown_connections 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 -- 172.21.15.28:0/3935443226 wait complete. 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 -- start start 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.956+0000 7f82697f4740 1 Processor -- start 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf787a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f82697f4740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf76f1b0 0x55f5cf787f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f82697f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf785e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f82697f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf79da90 con 0x55f5cf78edf0 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf41b620 con 0x55f5cf76f1b0 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f82697f4740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55f5cf787230 con 0x55f5cf7693e0 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f82697f4740 10 client.0 _ll_register_callbacks cb 0x7f8258003a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf785e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf785e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41796/0 (socket says 172.21.15.28:41796) 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 -- 172.21.15.28:0/3687548092 learned_addr learned my addr 172.21.15.28:0/3687548092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f825ffff640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf787a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f825f7fe640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf76f1b0 0x55f5cf787f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 -- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 msgr2=0x55f5cf787a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf787a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 -- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf76f1b0 msgr2=0x55f5cf787f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf76f1b0 0x55f5cf787f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f8264fc8640 1 -- 172.21.15.28:0/3687548092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5cf759110 con 0x55f5cf78edf0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f825ffff640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf787a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.957+0000 7f825f7fe640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf76f1b0 0x55f5cf787f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f8264fc8640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf785e40 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f825805eaf0 tx=0x7f8258066d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcceb0ec586b8a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f823effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f823effd640 1 -- 172.21.15.28:0/3687548092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258061e40 con 0x55f5cf78edf0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f823effd640 1 -- 172.21.15.28:0/3687548092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f8258061590 con 0x55f5cf78edf0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8254005020 con 0x55f5cf78edf0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f823effd640 1 -- 172.21.15.28:0/3687548092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f8258061790 con 0x55f5cf78edf0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 10 client.6708 fetch_fsmap learned FSMap version 53 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55f5cf747050 con 0x55f5cf78edf0 2024-09-10T20:42:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f823effd640 1 -- 172.21.15.28:0/3687548092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8254005020 con 0x55f5cf78edf0 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f823effd640 1 -- 172.21.15.28:0/3687548092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 53) ==== 3502+0+0 (secure 0 0 0) 0x7f825805dc80 con 0x55f5cf78edf0 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 10 client.6708 fetch_fsmap finished waiting for FSMap version 53 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 10 client.6708 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 20 client.6708 populate_metadata read hostname 'smithi028' 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf786520 0x55f5cf78d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 4 client.6708 mds_command: new command op to 6570 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 --> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] -- command(tid 0: {"prefix": "status"}) -- 0x55f5cf6b9220 con 0x55f5cf786520 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.958+0000 7f825ffff640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf786520 0x55f5cf78d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f823effd640 10 client.6708 ms_handle_connect on v2:172.21.15.28:6833/2216477518 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f825ffff640 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf786520 0x55f5cf78d680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72cdf48c93ae541e server_cookie=2caca148406f27ef in_seq=0 out_seq=0 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f823effd640 1 -- 172.21.15.28:0/3687548092 <== mds.0 v2:172.21.15.28:6833/2216477518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55f5cf6b9220 con 0x55f5cf786520 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f823effd640 10 client.6708 handle_command_reply: tid=0 2024-09-10T20:42:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 client.6708 shutdown 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 msgr2=0x55f5cf785e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf785e40 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f825805eaf0 tx=0x7f8258066d60 comp rx=0 tx=0).stop 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 shutdown_connections 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5cf78edf0 0x55f5cf785e40 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55f5cf76f1b0 0x55f5cf787f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.28:6833/2216477518,v1:172.21.15.28:6835/2216477518] conn(0x55f5cf786520 0x55f5cf78d680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 --2- 172.21.15.28:0/3687548092 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55f5cf7693e0 0x55f5cf787a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f823effd640 0 client.6708 ms_handle_reset on v2:172.21.15.28:6833/2216477518 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 >> 172.21.15.28:0/3687548092 conn(0x55f5cf7445f0 msgr2=0x55f5cf7449d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 shutdown_connections 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 1 -- 172.21.15.28:0/3687548092 wait complete. 2024-09-10T20:42:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.959+0000 7f82697f4740 20 client.6708 trim_cache size 0 max 16384 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6570, 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7269068790000004, 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 53, 2024-09-10T20:42:29.970 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 81, 2024-09-10T20:42:29.971 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 81, 2024-09-10T20:42:29.971 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.043225128 2024-09-10T20:42:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f824c078090 msgr2=0x7f824c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f824c078090 0x7f824c07a550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8250034000 tx=0x7f8250004730 comp rx=0 tx=0).stop 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600b5a50 msgr2=0x7f82600b23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600b5a50 0x7f82600b23b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f825800c780 tx=0x7f825800cc40 comp rx=0 tx=0).stop 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 shutdown_connections 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f824c078090 0x7f824c07a550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82600b5a50 0x7f82600b23b0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82600747a0 0x7f82600b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 --2- 172.21.15.28:0/4280627634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f826006cae0 0x7f826014eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 >> 172.21.15.28:0/4280627634 conn(0x7f826006dbf0 msgr2=0x7f8260070540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 shutdown_connections 2024-09-10T20:42:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:29.960+0000 7f8266a2b640 1 -- 172.21.15.28:0/4280627634 wait complete. 2024-09-10T20:42:29.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:42:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.076+0000 7ff2baeb0640 1 -- 172.21.15.28:0/1257006330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c003c60 msgr2=0x7ff29c004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.076+0000 7ff2baeb0640 1 --2- 172.21.15.28:0/1257006330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c003c60 0x7ff29c004060 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4001830 tx=0x7ff29c005830 comp rx=0 tx=0).stop 2024-09-10T20:42:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.076+0000 7ff2b944d640 1 -- 172.21.15.28:0/1257006330 reap_dead start 2024-09-10T20:42:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.076+0000 7ff2baeb0640 1 -- 172.21.15.28:0/1257006330 shutdown_connections 2024-09-10T20:42:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.076+0000 7ff2baeb0640 1 -- 172.21.15.28:0/1257006330 >> 172.21.15.28:0/1257006330 conn(0x7ff2b40d3fb0 msgr2=0x7ff2b40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.076+0000 7ff2baeb0640 1 -- 172.21.15.28:0/1257006330 shutdown_connections 2024-09-10T20:42:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.076+0000 7ff2baeb0640 1 -- 172.21.15.28:0/1257006330 wait complete. 2024-09-10T20:42:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.077+0000 7ff2baeb0640 1 Processor -- start 2024-09-10T20:42:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.077+0000 7ff2baeb0640 1 -- start start 2024-09-10T20:42:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2baeb0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff29c003c60 0x7ff2b41f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2baeb0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b40a17e0 0x7ff2b41f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2baeb0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2b41f8640 0x7ff2b40d0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2baeb0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b40b6e00 con 0x7ff2b41f8640 2024-09-10T20:42:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2baeb0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b415e460 con 0x7ff2b40a17e0 2024-09-10T20:42:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2baeb0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b40b6f80 con 0x7ff29c003c60 2024-09-10T20:42:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2b41f8640 0x7ff2b40d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2b41f8640 0x7ff2b40d0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41810/0 (socket says 172.21.15.28:41810) 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 -- 172.21.15.28:0/2674492924 learned_addr learned my addr 172.21.15.28:0/2674492924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b8c4c640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff29c003c60 0x7ff2b41f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b3fff640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b40a17e0 0x7ff2b41f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 -- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff29c003c60 msgr2=0x7ff2b41f5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff29c003c60 0x7ff2b41f5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 -- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b40a17e0 msgr2=0x7ff2b41f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b40a17e0 0x7ff2b41f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b944d640 1 -- 172.21.15.28:0/2674492924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b41f8b80 con 0x7ff2b41f8640 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b8c4c640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff29c003c60 0x7ff2b41f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.078+0000 7ff2b3fff640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b40a17e0 0x7ff2b41f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.079+0000 7ff2b944d640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2b41f8640 0x7ff2b40d0780 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac005fd0 tx=0x7ff2ac00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.079+0000 7ff2b17fa640 1 -- 172.21.15.28:0/2674492924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac004b20 con 0x7ff2b41f8640 2024-09-10T20:42:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.079+0000 7ff2b17fa640 1 -- 172.21.15.28:0/2674492924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2ac004cc0 con 0x7ff2b41f8640 2024-09-10T20:42:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.079+0000 7ff2b17fa640 1 -- 172.21.15.28:0/2674492924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac003bd0 con 0x7ff2b41f8640 2024-09-10T20:42:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.079+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b41f8e60 con 0x7ff2b41f8640 2024-09-10T20:42:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.079+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b41617a0 con 0x7ff2b41f8640 2024-09-10T20:42:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.080+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b4161b50 con 0x7ff2b41f8640 2024-09-10T20:42:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.081+0000 7ff2b17fa640 1 -- 172.21.15.28:0/2674492924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7ff2ac021060 con 0x7ff2b41f8640 2024-09-10T20:42:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.081+0000 7ff2b17fa640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2a0078090 0x7ff2a007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.081+0000 7ff2b17fa640 1 -- 172.21.15.28:0/2674492924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7ff2ac09a1e0 con 0x7ff2b41f8640 2024-09-10T20:42:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.081+0000 7ff2b8c4c640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2a0078090 0x7ff2a007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.082+0000 7ff2b8c4c640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2a0078090 0x7ff2a007a550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4032000 tx=0x7ff2a4001360 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.085+0000 7ff2b17fa640 1 -- 172.21.15.28:0/2674492924 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff2ac062ac0 con 0x7ff2b41f8640 2024-09-10T20:42:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.308+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff2b40b5440 con 0x7ff2b41f8640 2024-09-10T20:42:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.308+0000 7ff2b17fa640 1 -- 172.21.15.28:0/2674492924 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) ==== 53+0+73 (secure 0 0 0) 0x7ff2ac0623c0 con 0x7ff2b41f8640 2024-09-10T20:42:30.312 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:42:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.310+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2a0078090 msgr2=0x7ff2a007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.310+0000 7ff2baeb0640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2a0078090 0x7ff2a007a550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4032000 tx=0x7ff2a4001360 comp rx=0 tx=0).stop 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.310+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2b41f8640 msgr2=0x7ff2b40d0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.310+0000 7ff2baeb0640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2b41f8640 0x7ff2b40d0780 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac005fd0 tx=0x7ff2ac00f3c0 comp rx=0 tx=0).stop 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 shutdown_connections 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2a0078090 0x7ff2a007a550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2b41f8640 0x7ff2b40d0780 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b40a17e0 0x7ff2b41f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 --2- 172.21.15.28:0/2674492924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff29c003c60 0x7ff2b41f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 >> 172.21.15.28:0/2674492924 conn(0x7ff2b40d3fb0 msgr2=0x7ff2b40b9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 shutdown_connections 2024-09-10T20:42:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30.311+0000 7ff2baeb0640 1 -- 172.21.15.28:0/2674492924 wait complete. 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:42:30.326 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:42:30.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:30.327 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:42:30.343 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:42:30.343 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:42:30.343 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:42:30.343 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:30.343 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:42:30.343 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:30.343 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42480sec preferred_lft 42480sec 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:30.344 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:30.345 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:42:30.346 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:42:30.346 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:30.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:30.346 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:42:30.409 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:42:30.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:30.410 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:42:30.467 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:42:30.467 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:42:30.467 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:42:30.467 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:42:30.467 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:42:30.468 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:42:30.468 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:42:30.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:42:30.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:42:30.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:42:30.468 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:42:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:42:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:30.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:30.596 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:42:30.612 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:42:30.612 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:42:30.612 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:42:30.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:30.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:42:30.679 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:30.679 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:30.735 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:42:30.735 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:30.735 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:30.791 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:42:30.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:30.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:30.876 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:30.877 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:42:30.877 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:42:30.885 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:30.886 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:42:30.886 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:42:30.886 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:30.886 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:42:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:42:30.892 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:42:30.892 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:42:30.892 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:42:30.892 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:42:30.892 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:42:30.893 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:42:30.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:30.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:30.920 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:42:31.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:42:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:31.021 INFO:teuthology.orchestra.run.smithi028.stdout:client6717 2024-09-10T20:42:31.022 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:31.022 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:31.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:42:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 -- 172.21.15.28:0/3848315333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc15e9e0 msgr2=0x7fc0bc06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 --2- 172.21.15.28:0/3848315333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc15e9e0 0x7fc0bc06faa0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4005fd0 tx=0x7fc0b402aa10 comp rx=0 tx=0).stop 2024-09-10T20:42:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 -- 172.21.15.28:0/3848315333 shutdown_connections 2024-09-10T20:42:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 --2- 172.21.15.28:0/3848315333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc15e9e0 0x7fc0bc06faa0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 --2- 172.21.15.28:0/3848315333 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0bc072b50 0x7fc0bc0af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 -- 172.21.15.28:0/3848315333 >> 172.21.15.28:0/3848315333 conn(0x7fc0bc0b1c40 msgr2=0x7fc0bc0b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 -- 172.21.15.28:0/3848315333 shutdown_connections 2024-09-10T20:42:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.181+0000 7fc0c3285640 1 -- 172.21.15.28:0/3848315333 wait complete. 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.182+0000 7fc0c3285640 1 Processor -- start 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.182+0000 7fc0c3285640 1 -- start start 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c3285640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc072b50 0x7fc0bc148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c3285640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0bc15e600 0x7fc0bc0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c3285640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0bc15e9e0 0x7fc0bc1f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c3285640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc15f200 con 0x7fc0bc072b50 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c3285640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc15f080 con 0x7fc0bc15e9e0 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c3285640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc0bc06ffe0 con 0x7fc0bc15e600 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc072b50 0x7fc0bc148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc072b50 0x7fc0bc148ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51376/0 (socket says 172.21.15.28:51376) 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 -- 172.21.15.28:0/1643838083 learned_addr learned my addr 172.21.15.28:0/1643838083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c0820640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0bc15e600 0x7fc0bc0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1822640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0bc15e9e0 0x7fc0bc1f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 -- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0bc15e600 msgr2=0x7fc0bc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0bc15e600 0x7fc0bc0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 -- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0bc15e9e0 msgr2=0x7fc0bc1f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0bc15e9e0 0x7fc0bc1f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 -- 172.21.15.28:0/1643838083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc0b5010 con 0x7fc0bc072b50 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c1021640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc072b50 0x7fc0bc148ee0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800c480 tx=0x7fc0b800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0b1ffb640 1 -- 172.21.15.28:0/1643838083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8017070 con 0x7fc0bc072b50 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0b1ffb640 1 -- 172.21.15.28:0/1643838083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0b80035f0 con 0x7fc0bc072b50 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0b1ffb640 1 -- 172.21.15.28:0/1643838083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8003a80 con 0x7fc0bc072b50 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.183+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0bc1f59e0 con 0x7fc0bc072b50 2024-09-10T20:42:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.184+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0bc1f5670 con 0x7fc0bc072b50 2024-09-10T20:42:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.185+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0bc073710 con 0x7fc0bc072b50 2024-09-10T20:42:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.185+0000 7fc0b1ffb640 1 -- 172.21.15.28:0/1643838083 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fc0b8003c20 con 0x7fc0bc072b50 2024-09-10T20:42:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.186+0000 7fc0b1ffb640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0a0078090 0x7fc0a007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.186+0000 7fc0c0820640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0a0078090 0x7fc0a007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:31.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.186+0000 7fc0b1ffb640 1 -- 172.21.15.28:0/1643838083 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc0b8099730 con 0x7fc0bc072b50 2024-09-10T20:42:31.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.186+0000 7fc0c0820640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0a0078090 0x7fc0a007a550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0cfc10 tx=0x7fc0ac00a4c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:31.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.190+0000 7fc0b1ffb640 1 -- 172.21.15.28:0/1643838083 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc0bc073710 con 0x7fc0bc072b50 2024-09-10T20:42:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.415+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc0bc0af530 con 0x7fc0bc072b50 2024-09-10T20:42:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.415+0000 7fc0b1ffb640 1 -- 172.21.15.28:0/1643838083 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) ==== 53+0+73 (secure 0 0 0) 0x7fc0b8062000 con 0x7fc0bc072b50 2024-09-10T20:42:31.419 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0a0078090 msgr2=0x7fc0a007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0a0078090 0x7fc0a007a550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0cfc10 tx=0x7fc0ac00a4c0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc072b50 msgr2=0x7fc0bc148ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc072b50 0x7fc0bc148ee0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800c480 tx=0x7fc0b800c940 comp rx=0 tx=0).stop 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 shutdown_connections 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0a0078090 0x7fc0a007a550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0bc15e9e0 0x7fc0bc1f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0bc15e600 0x7fc0bc0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 --2- 172.21.15.28:0/1643838083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0bc072b50 0x7fc0bc148ee0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 >> 172.21.15.28:0/1643838083 conn(0x7fc0bc0b1c40 msgr2=0x7fc0bc0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 shutdown_connections 2024-09-10T20:42:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:31.418+0000 7fc0c3285640 1 -- 172.21.15.28:0/1643838083 wait complete. 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:42:31.433 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:42:31.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:31.434 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:42:31.454 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:42:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:42:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:42:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:42:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:31.455 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:42:31.455 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42498sec preferred_lft 42498sec 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:42:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:42:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:42:31.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:31.458 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:42:31.522 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:42:31.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:31.524 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:42:31.582 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:42:31.583 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:42:31.583 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:42:31.583 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:42:31.583 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:42:31.583 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:42:31.583 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:42:31.583 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:42:31.584 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:42:31.584 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:42:31.584 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:42:31.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:42:31.675 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:42:31.677 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:42:31.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:31.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:31.715 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:42:31.771 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:42:31.771 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:42:31.771 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:42:31.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:31.772 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:42:31.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:31.836 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:31.911 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:42:31.911 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:42:31.911 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:31.966 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:42:31.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:31.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:32.049 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:32.050 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:42:32.050 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:42:32.058 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:32.058 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:42:32.058 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:42:32.058 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:32.059 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:32.063 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:42:32.063 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:42:32.063 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:42:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:42:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:42:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:42:32.066 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:42:32.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:32.066 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:32.131 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:42:32.219 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:42:32.221 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:42:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:42:32.226 INFO:teuthology.orchestra.run.smithi159.stdout:client5825 2024-09-10T20:42:32.227 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:32.227 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:32.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:42:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 -- 172.21.15.28:0/1729695591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 msgr2=0x7f5c280ce7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/1729695591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 0x7f5c280ce7f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009830 tx=0x7f5c1802a5a0 comp rx=0 tx=0).stop 2024-09-10T20:42:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 -- 172.21.15.28:0/1729695591 shutdown_connections 2024-09-10T20:42:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/1729695591 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c280ced30 0x7f5c280afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/1729695591 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 0x7f5c280ce7f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 -- 172.21.15.28:0/1729695591 >> 172.21.15.28:0/1729695591 conn(0x7f5c280d0410 msgr2=0x7f5c280d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 -- 172.21.15.28:0/1729695591 shutdown_connections 2024-09-10T20:42:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.355+0000 7f5c2cd52640 1 -- 172.21.15.28:0/1729695591 wait complete. 2024-09-10T20:42:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.356+0000 7f5c2cd52640 1 Processor -- start 2024-09-10T20:42:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.356+0000 7f5c2cd52640 1 -- start start 2024-09-10T20:42:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c2cd52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c280b02b0 0x7f5c280b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c2cd52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c280ced30 0x7f5c280c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c2cd52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 0x7f5c280993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c2cd52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2806d850 con 0x7f5c280ced30 2024-09-10T20:42:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c2cd52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5c280739e0 con 0x7f5c2815e4a0 2024-09-10T20:42:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c2cd52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5c2806d9d0 con 0x7f5c280b02b0 2024-09-10T20:42:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c280ced30 0x7f5c280c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c280ced30 0x7f5c280c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51382/0 (socket says 172.21.15.28:51382) 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 -- 172.21.15.28:0/3920602423 learned_addr learned my addr 172.21.15.28:0/3920602423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c2659c640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c280b02b0 0x7f5c280b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c26d9d640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 0x7f5c280993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 -- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c280b02b0 msgr2=0x7f5c280b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c280b02b0 0x7f5c280b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 -- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 msgr2=0x7f5c280993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 0x7f5c280993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c1ffff640 1 -- 172.21.15.28:0/3920602423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c280b5b80 con 0x7f5c280ced30 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.357+0000 7f5c26d9d640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 0x7f5c280993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.358+0000 7f5c1ffff640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c280ced30 0x7f5c280c74e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000e9c0 tx=0x7f5c1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.358+0000 7f5c1effd640 1 -- 172.21.15.28:0/3920602423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1000cd80 con 0x7f5c280ced30 2024-09-10T20:42:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.358+0000 7f5c1effd640 1 -- 172.21.15.28:0/3920602423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c10004020 con 0x7f5c280ced30 2024-09-10T20:42:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.358+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c280ca140 con 0x7f5c280ced30 2024-09-10T20:42:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.358+0000 7f5c1effd640 1 -- 172.21.15.28:0/3920602423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c10004450 con 0x7f5c280ced30 2024-09-10T20:42:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.358+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c280b4e40 con 0x7f5c280ced30 2024-09-10T20:42:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.359+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c2815e9f0 con 0x7f5c280ced30 2024-09-10T20:42:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.360+0000 7f5c1effd640 1 -- 172.21.15.28:0/3920602423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f5c100035f0 con 0x7f5c280ced30 2024-09-10T20:42:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.360+0000 7f5c1effd640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5c08078090 0x7f5c0807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.360+0000 7f5c1effd640 1 -- 172.21.15.28:0/3920602423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f5c1009aad0 con 0x7f5c280ced30 2024-09-10T20:42:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.360+0000 7f5c2659c640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5c08078090 0x7f5c0807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.361+0000 7f5c2659c640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5c08078090 0x7f5c0807a550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c18034000 tx=0x7f5c18004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:32.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.364+0000 7f5c1effd640 1 -- 172.21.15.28:0/3920602423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5c10063330 con 0x7f5c280ced30 2024-09-10T20:42:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:32.588+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f5c28076660 con 0x7f5c280ced30 2024-09-10T20:42:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.275+0000 7f5c1effd640 1 -- 172.21.15.28:0/3920602423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v54) ==== 136+0+0 (secure 0 0 0) 0x7f5c10062c30 con 0x7f5c280ced30 2024-09-10T20:42:33.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5c08078090 msgr2=0x7f5c0807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5c08078090 0x7f5c0807a550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c18034000 tx=0x7f5c18004730 comp rx=0 tx=0).stop 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c280ced30 msgr2=0x7f5c280c74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c280ced30 0x7f5c280c74e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000e9c0 tx=0x7f5c1000ee80 comp rx=0 tx=0).stop 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 shutdown_connections 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5c08078090 0x7f5c0807a550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5c2815e4a0 0x7f5c280993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c280ced30 0x7f5c280c74e0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 --2- 172.21.15.28:0/3920602423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5c280b02b0 0x7f5c280b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 >> 172.21.15.28:0/3920602423 conn(0x7f5c280d0410 msgr2=0x7f5c2815d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 shutdown_connections 2024-09-10T20:42:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.278+0000 7f5c2cd52640 1 -- 172.21.15.28:0/3920602423 wait complete. 2024-09-10T20:42:33.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:42:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.406+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2258851802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d0162260 msgr2=0x7fb2d00cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.407+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2258851802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d0162260 0x7fb2d00cfbc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8005fd0 tx=0x7fb2c802aa90 comp rx=0 tx=0).stop 2024-09-10T20:42:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.407+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2258851802 shutdown_connections 2024-09-10T20:42:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.407+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2258851802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d0162260 0x7fb2d00cfbc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.407+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2258851802 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2d015de70 0x7fb2d00d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.407+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2258851802 >> 172.21.15.28:0/2258851802 conn(0x7fb2d00d3fb0 msgr2=0x7fb2d00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.407+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2258851802 shutdown_connections 2024-09-10T20:42:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.407+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2258851802 wait complete. 2024-09-10T20:42:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.408+0000 7fb2d79fb640 1 Processor -- start 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.408+0000 7fb2d79fb640 1 -- start start 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d79fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d00cf780 0x7fb2d00cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d79fb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2d015de70 0x7fb2d014caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d79fb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2d0162260 0x7fb2d00b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d79fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d00b7190 con 0x7fb2d00cf780 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d79fb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d00b7010 con 0x7fb2d015de70 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d79fb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d00b7310 con 0x7fb2d0162260 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d00cf780 0x7fb2d00cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d00cf780 0x7fb2d00cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51410/0 (socket says 172.21.15.28:51410) 2024-09-10T20:42:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 -- 172.21.15.28:0/2951126655 learned_addr learned my addr 172.21.15.28:0/2951126655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d4f96640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2d015de70 0x7fb2d014caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5f98640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2d0162260 0x7fb2d00b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 -- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2d0162260 msgr2=0x7fb2d00b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2d0162260 0x7fb2d00b34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 -- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2d015de70 msgr2=0x7fb2d014caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2d015de70 0x7fb2d014caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 -- 172.21.15.28:0/2951126655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d00cc690 con 0x7fb2d00cf780 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d4f96640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2d015de70 0x7fb2d014caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.409+0000 7fb2d5797640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d00cf780 0x7fb2d00cd730 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00d950 tx=0x7fb2cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.410+0000 7fb2c5ffb640 1 -- 172.21.15.28:0/2951126655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc017070 con 0x7fb2d00cf780 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.410+0000 7fb2c5ffb640 1 -- 172.21.15.28:0/2951126655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2cc009d20 con 0x7fb2d00cf780 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.410+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d00cc970 con 0x7fb2d00cf780 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.410+0000 7fb2c5ffb640 1 -- 172.21.15.28:0/2951126655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc0106c0 con 0x7fb2d00cf780 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.410+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d00ccf30 con 0x7fb2d00cf780 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.411+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2d00b7830 con 0x7fb2d00cf780 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.411+0000 7fb2c5ffb640 1 -- 172.21.15.28:0/2951126655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fb2cc010860 con 0x7fb2d00cf780 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.412+0000 7fb2c5ffb640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2b4078090 0x7fb2b407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.412+0000 7fb2c5ffb640 1 -- 172.21.15.28:0/2951126655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb2cc09a480 con 0x7fb2d00cf780 2024-09-10T20:42:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.412+0000 7fb2d4f96640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2b4078090 0x7fb2b407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.413+0000 7fb2d4f96640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2b4078090 0x7fb2b407a550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00d0a00 tx=0x7fb2c0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.416+0000 7fb2c5ffb640 1 -- 172.21.15.28:0/2951126655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb2cc062d50 con 0x7fb2d00cf780 2024-09-10T20:42:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.624+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fb2d00b9f30 con 0x7fb2b4078090 2024-09-10T20:42:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.625+0000 7fb2c5ffb640 1 -- 172.21.15.28:0/2951126655 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fb2d00b9f30 con 0x7fb2b4078090 2024-09-10T20:42:33.629 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:42:33.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:33.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:42:33.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:33.629 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:42:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2b4078090 msgr2=0x7fb2b407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2b4078090 0x7fb2b407a550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00d0a00 tx=0x7fb2c0009040 comp rx=0 tx=0).stop 2024-09-10T20:42:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d00cf780 msgr2=0x7fb2d00cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d00cf780 0x7fb2d00cd730 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00d950 tx=0x7fb2cc00de10 comp rx=0 tx=0).stop 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 shutdown_connections 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2b4078090 0x7fb2b407a550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2d0162260 0x7fb2d00b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2d015de70 0x7fb2d014caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 --2- 172.21.15.28:0/2951126655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d00cf780 0x7fb2d00cd730 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 >> 172.21.15.28:0/2951126655 conn(0x7fb2d00d3fb0 msgr2=0x7fb2d00d1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 shutdown_connections 2024-09-10T20:42:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.627+0000 7fb2d79fb640 1 -- 172.21.15.28:0/2951126655 wait complete. 2024-09-10T20:42:33.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:42:33.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.748+0000 7fc31315d640 1 -- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f8004800 msgr2=0x7fc2f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.748+0000 7fc31315d640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f8004800 0x7fc2f8004c80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc30c15b1f0 tx=0x7fc2f8005830 comp rx=0 tx=0).stop 2024-09-10T20:42:33.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.748+0000 7fc3116fa640 1 -- 172.21.15.28:0/145559217 reap_dead start 2024-09-10T20:42:33.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.748+0000 7fc31315d640 1 -- 172.21.15.28:0/145559217 shutdown_connections 2024-09-10T20:42:33.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.748+0000 7fc31315d640 1 -- 172.21.15.28:0/145559217 >> 172.21.15.28:0/145559217 conn(0x7fc30c06cf00 msgr2=0x7fc30c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:33.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.748+0000 7fc31315d640 1 -- 172.21.15.28:0/145559217 shutdown_connections 2024-09-10T20:42:33.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.748+0000 7fc31315d640 1 -- 172.21.15.28:0/145559217 wait complete. 2024-09-10T20:42:33.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.749+0000 7fc31315d640 1 Processor -- start 2024-09-10T20:42:33.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.749+0000 7fc31315d640 1 -- start start 2024-09-10T20:42:33.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc31315d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2f8004800 0x7fc30c152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc31315d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c157780 0x7fc30c150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc31315d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c155d30 0x7fc30c0cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc31315d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c06fde0 con 0x7fc30c157780 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc31315d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0d1110 con 0x7fc2f8004800 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc31315d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c06ff60 con 0x7fc30c155d30 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c157780 0x7fc30c150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c157780 0x7fc30c150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51422/0 (socket says 172.21.15.28:51422) 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 -- 172.21.15.28:0/1081662908 learned_addr learned my addr 172.21.15.28:0/1081662908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc310ef9640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2f8004800 0x7fc30c152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc3116fa640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c155d30 0x7fc30c0cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 -- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c155d30 msgr2=0x7fc30c0cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c155d30 0x7fc30c0cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 -- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2f8004800 msgr2=0x7fc30c152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2f8004800 0x7fc30c152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc30bfff640 1 -- 172.21.15.28:0/1081662908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c157f10 con 0x7fc30c157780 2024-09-10T20:42:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.750+0000 7fc3116fa640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c155d30 0x7fc30c0cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.751+0000 7fc30bfff640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c157780 0x7fc30c150f00 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc2f402b410 tx=0x7fc2f4004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.751+0000 7fc3097fa640 1 -- 172.21.15.28:0/1081662908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f402b760 con 0x7fc30c157780 2024-09-10T20:42:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.751+0000 7fc3097fa640 1 -- 172.21.15.28:0/1081662908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f402b900 con 0x7fc30c157780 2024-09-10T20:42:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.751+0000 7fc3097fa640 1 -- 172.21.15.28:0/1081662908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f402bd30 con 0x7fc30c157780 2024-09-10T20:42:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.751+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c151750 con 0x7fc30c157780 2024-09-10T20:42:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.751+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c151d10 con 0x7fc30c157780 2024-09-10T20:42:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.752+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c0cb230 con 0x7fc30c157780 2024-09-10T20:42:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.753+0000 7fc3097fa640 1 -- 172.21.15.28:0/1081662908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fc2f4006070 con 0x7fc30c157780 2024-09-10T20:42:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.753+0000 7fc3097fa640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078090 0x7fc2ec07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.753+0000 7fc310ef9640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078090 0x7fc2ec07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.753+0000 7fc3097fa640 1 -- 172.21.15.28:0/1081662908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc2f40ba6b0 con 0x7fc30c157780 2024-09-10T20:42:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.754+0000 7fc310ef9640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078090 0x7fc2ec07a550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009b90 tx=0x7fc2fc005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.758+0000 7fc3097fa640 1 -- 172.21.15.28:0/1081662908 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc2f4082e60 con 0x7fc30c157780 2024-09-10T20:42:33.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.942+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc30c0af9a0 con 0x7fc30c157780 2024-09-10T20:42:33.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.942+0000 7fc3097fa640 1 -- 172.21.15.28:0/1081662908 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7fc2f4082760 con 0x7fc30c157780 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078090 msgr2=0x7fc2ec07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078090 0x7fc2ec07a550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009b90 tx=0x7fc2fc005f90 comp rx=0 tx=0).stop 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c157780 msgr2=0x7fc30c150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c157780 0x7fc30c150f00 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc2f402b410 tx=0x7fc2f4004bb0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 shutdown_connections 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078090 0x7fc2ec07a550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c155d30 0x7fc30c0cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c157780 0x7fc30c150f00 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 --2- 172.21.15.28:0/1081662908 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2f8004800 0x7fc30c152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:33.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 >> 172.21.15.28:0/1081662908 conn(0x7fc30c06cf00 msgr2=0x7fc30c0cf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:33.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 shutdown_connections 2024-09-10T20:42:33.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:33.944+0000 7fc31315d640 1 -- 172.21.15.28:0/1081662908 wait complete. 2024-09-10T20:42:33.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create 'abcd@#' 2024-09-10T20:42:34.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 -- 172.21.15.28:0/3266631366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca680b4440 msgr2=0x7fca6806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:34.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 --2- 172.21.15.28:0/3266631366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca680b4440 0x7fca6806f1b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fca68076790 tx=0x7fca5802a5f0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 -- 172.21.15.28:0/3266631366 shutdown_connections 2024-09-10T20:42:34.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 --2- 172.21.15.28:0/3266631366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca680b4440 0x7fca6806f1b0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 --2- 172.21.15.28:0/3266631366 >> v2:172.21.15.159:3300/0 conn(0x7fca680b3ff0 0x7fca6806ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 -- 172.21.15.28:0/3266631366 >> 172.21.15.28:0/3266631366 conn(0x7fca680b1800 msgr2=0x7fca680b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 -- 172.21.15.28:0/3266631366 shutdown_connections 2024-09-10T20:42:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.062+0000 7fca6d910640 1 -- 172.21.15.28:0/3266631366 wait complete. 2024-09-10T20:42:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.063+0000 7fca6d910640 1 Processor -- start 2024-09-10T20:42:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.063+0000 7fca6d910640 1 -- start start 2024-09-10T20:42:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca6d910640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca680b3ff0 0x7fca6814fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca6d910640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca680b4440 0x7fca68144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca6d910640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca6815a4d0 0x7fca681f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca6d910640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca6806e560 con 0x7fca680b4440 2024-09-10T20:42:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca6d910640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fca6806e3e0 con 0x7fca6815a4d0 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca6d910640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fca6806e6e0 con 0x7fca680b3ff0 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca680b4440 0x7fca68144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca680b4440 0x7fca68144850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51438/0 (socket says 172.21.15.28:51438) 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca667fc640 1 -- 172.21.15.28:0/2668846056 learned_addr learned my addr 172.21.15.28:0/2668846056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca677fe640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca6815a4d0 0x7fca681f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.064+0000 7fca66ffd640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca680b3ff0 0x7fca6814fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca667fc640 1 -- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca680b3ff0 msgr2=0x7fca6814fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca667fc640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca680b3ff0 0x7fca6814fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca667fc640 1 -- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca6815a4d0 msgr2=0x7fca681f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca667fc640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca6815a4d0 0x7fca681f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca667fc640 1 -- 172.21.15.28:0/2668846056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca681f34f0 con 0x7fca680b4440 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca677fe640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fca6815a4d0 0x7fca681f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca66ffd640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fca680b3ff0 0x7fca6814fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca667fc640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca680b4440 0x7fca68144850 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fca5000ea20 tx=0x7fca5000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca5ffff640 1 -- 172.21.15.28:0/2668846056 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5000cd80 con 0x7fca680b4440 2024-09-10T20:42:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca5ffff640 1 -- 172.21.15.28:0/2668846056 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca50010430 con 0x7fca680b4440 2024-09-10T20:42:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca6d910640 1 -- 172.21.15.28:0/2668846056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca681510c0 con 0x7fca680b4440 2024-09-10T20:42:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca5ffff640 1 -- 172.21.15.28:0/2668846056 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca50010840 con 0x7fca680b4440 2024-09-10T20:42:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.065+0000 7fca6d910640 1 -- 172.21.15.28:0/2668846056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca6814f020 con 0x7fca680b4440 2024-09-10T20:42:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.066+0000 7fca6d910640 1 -- 172.21.15.28:0/2668846056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca6806fee0 con 0x7fca680b4440 2024-09-10T20:42:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.067+0000 7fca5ffff640 1 -- 172.21.15.28:0/2668846056 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fca500109e0 con 0x7fca680b4440 2024-09-10T20:42:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.067+0000 7fca5ffff640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca48078090 0x7fca4807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.067+0000 7fca5ffff640 1 -- 172.21.15.28:0/2668846056 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fca5009ab20 con 0x7fca680b4440 2024-09-10T20:42:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.067+0000 7fca66ffd640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca48078090 0x7fca4807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.068+0000 7fca66ffd640 1 --2- 172.21.15.28:0/2668846056 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca48078090 0x7fca4807a550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fca60001d10 tx=0x7fca60009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.072+0000 7fca5ffff640 1 -- 172.21.15.28:0/2668846056 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fca681f34f0 con 0x7fca680b4440 2024-09-10T20:42:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:Invalid command: invalid chars #@ in abcd@# 2024-09-10T20:42:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:fs volume create [] : Create a CephFS volume 2024-09-10T20:42:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-09-10T20:42:34.288 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:42:34.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create testvol 2024-09-10T20:42:34.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.392+0000 7f76c4440640 1 -- 172.21.15.28:0/910157635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0d09a0 msgr2=0x7f76bc0d2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:34.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.392+0000 7f76c4440640 1 --2- 172.21.15.28:0/910157635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0d09a0 0x7f76bc0d2970 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f76ac005fd0 tx=0x7f76ac02a4b0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.392+0000 7f76c4440640 1 -- 172.21.15.28:0/910157635 shutdown_connections 2024-09-10T20:42:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.392+0000 7f76c4440640 1 --2- 172.21.15.28:0/910157635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0d09a0 0x7f76bc0d2970 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.392+0000 7f76c4440640 1 -- 172.21.15.28:0/910157635 >> 172.21.15.28:0/910157635 conn(0x7f76bc0d1e50 msgr2=0x7f76bc0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.392+0000 7f76c4440640 1 -- 172.21.15.28:0/910157635 shutdown_connections 2024-09-10T20:42:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.392+0000 7f76c4440640 1 -- 172.21.15.28:0/910157635 wait complete. 2024-09-10T20:42:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.393+0000 7f76c4440640 1 Processor -- start 2024-09-10T20:42:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.393+0000 7f76c4440640 1 -- start start 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c4440640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76bc0cfa20 0x7f76bc06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c4440640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76bc0d09a0 0x7f76bc14e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c4440640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc15bf40 0x7f76bc0b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c4440640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc0cf020 con 0x7f76bc15bf40 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c4440640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc0ceea0 con 0x7f76bc0d09a0 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c4440640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc0cf1a0 con 0x7f76bc0cfa20 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c29dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc15bf40 0x7f76bc0b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c29dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc15bf40 0x7f76bc0b6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51444/0 (socket says 172.21.15.28:51444) 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c29dd640 1 -- 172.21.15.28:0/996367869 learned_addr learned my addr 172.21.15.28:0/996367869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c19db640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76bc0d09a0 0x7f76bc14e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.394+0000 7f76c21dc640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76bc0cfa20 0x7f76bc06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c29dd640 1 -- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76bc0cfa20 msgr2=0x7f76bc06eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c29dd640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76bc0cfa20 0x7f76bc06eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c29dd640 1 -- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76bc0d09a0 msgr2=0x7f76bc14e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c29dd640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76bc0d09a0 0x7f76bc14e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c29dd640 1 -- 172.21.15.28:0/996367869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc06de40 con 0x7f76bc15bf40 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c29dd640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc15bf40 0x7f76bc0b6a60 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f76b400c780 tx=0x7f76b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76b2ffd640 1 -- 172.21.15.28:0/996367869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4017070 con 0x7f76bc15bf40 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76b2ffd640 1 -- 172.21.15.28:0/996367869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76b400ce30 con 0x7f76bc15bf40 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc06e120 con 0x7f76bc15bf40 2024-09-10T20:42:34.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76b2ffd640 1 -- 172.21.15.28:0/996367869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4005290 con 0x7f76bc15bf40 2024-09-10T20:42:34.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.395+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76bc06e6e0 con 0x7f76bc15bf40 2024-09-10T20:42:34.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.396+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76bc0d2970 con 0x7f76bc15bf40 2024-09-10T20:42:34.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.397+0000 7f76b2ffd640 1 -- 172.21.15.28:0/996367869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f76b4005430 con 0x7f76bc15bf40 2024-09-10T20:42:34.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.397+0000 7f76b2ffd640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76a0078090 0x7f76a007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:34.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.397+0000 7f76c21dc640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76a0078090 0x7f76a007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:34.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.397+0000 7f76b2ffd640 1 -- 172.21.15.28:0/996367869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f76b40995a0 con 0x7f76bc15bf40 2024-09-10T20:42:34.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.398+0000 7f76c21dc640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76a0078090 0x7f76a007a550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f76b8000f60 tx=0x7f76b8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:34.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.402+0000 7f76b2ffd640 1 -- 172.21.15.28:0/996367869 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f76b40617a0 con 0x7f76bc15bf40 2024-09-10T20:42:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:34.611+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "testvol", "target": ["mon-mgr", ""]}) -- 0x7f76bc0d05f0 con 0x7f76a0078090 2024-09-10T20:42:37.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.303+0000 7f76b2ffd640 1 -- 172.21.15.28:0/996367869 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 Volume created successfully (no MDS daemons created)) ==== 60+0+0 (secure 0 0 0) 0x7f76bc0d05f0 con 0x7f76a0078090 2024-09-10T20:42:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:Volume created successfully (no MDS daemons created) 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76a0078090 msgr2=0x7f76a007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76a0078090 0x7f76a007a550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f76b8000f60 tx=0x7f76b8009040 comp rx=0 tx=0).stop 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc15bf40 msgr2=0x7f76bc0b6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc15bf40 0x7f76bc0b6a60 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f76b400c780 tx=0x7f76b400cc40 comp rx=0 tx=0).stop 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 shutdown_connections 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76a0078090 0x7f76a007a550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc15bf40 0x7f76bc0b6a60 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76bc0d09a0 0x7f76bc14e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 --2- 172.21.15.28:0/996367869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76bc0cfa20 0x7f76bc06eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.310+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 >> 172.21.15.28:0/996367869 conn(0x7f76bc0d1e50 msgr2=0x7f76bc0aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.312+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 shutdown_connections 2024-09-10T20:42:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.312+0000 7f76c4440640 1 -- 172.21.15.28:0/996367869 wait complete. 2024-09-10T20:42:37.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create testvol 'abcd@#' 2024-09-10T20:42:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.443+0000 7f7b43271640 1 -- 172.21.15.28:0/1425936962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b24004800 msgr2=0x7f7b24004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.443+0000 7f7b43271640 1 --2- 172.21.15.28:0/1425936962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b24004800 0x7f7b24004c80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0d0d40 tx=0x7f7b24005830 comp rx=0 tx=0).stop 2024-09-10T20:42:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.443+0000 7f7b4180e640 1 -- 172.21.15.28:0/1425936962 reap_dead start 2024-09-10T20:42:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.443+0000 7f7b43271640 1 -- 172.21.15.28:0/1425936962 shutdown_connections 2024-09-10T20:42:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.443+0000 7f7b43271640 1 -- 172.21.15.28:0/1425936962 >> 172.21.15.28:0/1425936962 conn(0x7f7b3c06ccc0 msgr2=0x7f7b3c06d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.443+0000 7f7b43271640 1 -- 172.21.15.28:0/1425936962 shutdown_connections 2024-09-10T20:42:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.443+0000 7f7b43271640 1 -- 172.21.15.28:0/1425936962 wait complete. 2024-09-10T20:42:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.444+0000 7f7b43271640 1 Processor -- start 2024-09-10T20:42:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.444+0000 7f7b43271640 1 -- start start 2024-09-10T20:42:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b43271640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b24004800 0x7f7b3c150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b43271640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b3c14c4b0 0x7f7b3c14e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b43271640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b3c150ec0 0x7f7b3c14eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b43271640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c161980 con 0x7f7b3c150ec0 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b43271640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c161800 con 0x7f7b3c14c4b0 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b43271640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c161b00 con 0x7f7b24004800 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b4180e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b3c150ec0 0x7f7b3c14eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b4180e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b3c150ec0 0x7f7b3c14eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51464/0 (socket says 172.21.15.28:51464) 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b4180e640 1 -- 172.21.15.28:0/2722444526 learned_addr learned my addr 172.21.15.28:0/2722444526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b4100d640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b24004800 0x7f7b3c150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.445+0000 7f7b4080c640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b3c14c4b0 0x7f7b3c14e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b4180e640 1 -- 172.21.15.28:0/2722444526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b24004800 msgr2=0x7f7b3c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b4180e640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b24004800 0x7f7b3c150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b4180e640 1 -- 172.21.15.28:0/2722444526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b3c14c4b0 msgr2=0x7f7b3c14e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b4180e640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b3c14c4b0 0x7f7b3c14e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b4180e640 1 -- 172.21.15.28:0/2722444526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c0af210 con 0x7f7b3c150ec0 2024-09-10T20:42:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b4180e640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b3c150ec0 0x7f7b3c14eeb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0cc970 tx=0x7f7b34001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b39ffb640 1 -- 172.21.15.28:0/2722444526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34011840 con 0x7f7b3c150ec0 2024-09-10T20:42:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b39ffb640 1 -- 172.21.15.28:0/2722444526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b340119e0 con 0x7f7b3c150ec0 2024-09-10T20:42:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b43271640 1 -- 172.21.15.28:0/2722444526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3c0ae690 con 0x7f7b3c150ec0 2024-09-10T20:42:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b39ffb640 1 -- 172.21.15.28:0/2722444526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34011df0 con 0x7f7b3c150ec0 2024-09-10T20:42:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.446+0000 7f7b43271640 1 -- 172.21.15.28:0/2722444526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3c0aec50 con 0x7f7b3c150ec0 2024-09-10T20:42:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.447+0000 7f7b43271640 1 -- 172.21.15.28:0/2722444526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3c15b7b0 con 0x7f7b3c150ec0 2024-09-10T20:42:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.448+0000 7f7b39ffb640 1 -- 172.21.15.28:0/2722444526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f7b3401d470 con 0x7f7b3c150ec0 2024-09-10T20:42:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.449+0000 7f7b39ffb640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b280780c0 0x7f7b2807a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.449+0000 7f7b39ffb640 1 -- 172.21.15.28:0/2722444526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 6543+0+0 (secure 0 0 0) 0x7f7b3409a910 con 0x7f7b3c150ec0 2024-09-10T20:42:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.449+0000 7f7b4100d640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b280780c0 0x7f7b2807a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.449+0000 7f7b4100d640 1 --2- 172.21.15.28:0/2722444526 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b280780c0 0x7f7b2807a580 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0015a0 tx=0x7f7b2c001490 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:37.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.453+0000 7f7b39ffb640 1 -- 172.21.15.28:0/2722444526 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7b34001d00 con 0x7f7b3c150ec0 2024-09-10T20:42:37.665 INFO:teuthology.orchestra.run.smithi028.stderr:Invalid command: invalid chars #@ in abcd@# 2024-09-10T20:42:37.665 INFO:teuthology.orchestra.run.smithi028.stderr:fs subvolumegroup create [] [] [] [] [] : Create a CephFS subvolume group in a volume, and optionally, with a specific data pool layout, and a specific numeric mode 2024-09-10T20:42:37.665 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-09-10T20:42:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:42:37.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create testvol 'abcd@#' 2024-09-10T20:42:37.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 -- 172.21.15.28:0/575310627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab000d4b90 msgr2=0x7fab000d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 --2- 172.21.15.28:0/575310627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab000d4b90 0x7fab000d2620 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7faaf8005fd0 tx=0x7faaf802aa10 comp rx=0 tx=0).stop 2024-09-10T20:42:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 -- 172.21.15.28:0/575310627 shutdown_connections 2024-09-10T20:42:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 --2- 172.21.15.28:0/575310627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab000d4b90 0x7fab000d2620 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 --2- 172.21.15.28:0/575310627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab00161770 0x7fab000d20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 -- 172.21.15.28:0/575310627 >> 172.21.15.28:0/575310627 conn(0x7fab000d4070 msgr2=0x7fab000d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 -- 172.21.15.28:0/575310627 shutdown_connections 2024-09-10T20:42:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.773+0000 7fab05f2b640 1 -- 172.21.15.28:0/575310627 wait complete. 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.774+0000 7fab05f2b640 1 Processor -- start 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.774+0000 7fab05f2b640 1 -- start start 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7fab05f2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab000d4b90 0x7fab000cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7fab05f2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab00161770 0x7fab001e95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7fab05f2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab00161b50 0x7fab0015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7fab05f2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab000b2f50 con 0x7fab000d4b90 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7fab05f2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fab000b2dd0 con 0x7fab00161770 2024-09-10T20:42:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7fab05f2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fab000b30d0 con 0x7fab00161b50 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7faaff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab000d4b90 0x7fab000cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7faaff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab000d4b90 0x7fab000cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51480/0 (socket says 172.21.15.28:51480) 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.775+0000 7faaff7fe640 1 -- 172.21.15.28:0/306939087 learned_addr learned my addr 172.21.15.28:0/306939087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faafffff640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab00161b50 0x7fab0015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faafeffd640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab00161770 0x7fab001e95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faaff7fe640 1 -- 172.21.15.28:0/306939087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab00161b50 msgr2=0x7fab0015f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faaff7fe640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab00161b50 0x7fab0015f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faaff7fe640 1 -- 172.21.15.28:0/306939087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab00161770 msgr2=0x7fab001e95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faaff7fe640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab00161770 0x7fab001e95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faaff7fe640 1 -- 172.21.15.28:0/306939087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab000cda20 con 0x7fab000d4b90 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faafeffd640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab00161770 0x7fab001e95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faaff7fe640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab000d4b90 0x7fab000cd4e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7faaf000c890 tx=0x7faaf000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faae7fff640 1 -- 172.21.15.28:0/306939087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0004220 con 0x7fab000d4b90 2024-09-10T20:42:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faae7fff640 1 -- 172.21.15.28:0/306939087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faaf00079b0 con 0x7fab000d4b90 2024-09-10T20:42:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7fab05f2b640 1 -- 172.21.15.28:0/306939087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab000cb1a0 con 0x7fab000d4b90 2024-09-10T20:42:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7faae7fff640 1 -- 172.21.15.28:0/306939087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0007de0 con 0x7fab000d4b90 2024-09-10T20:42:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.776+0000 7fab05f2b640 1 -- 172.21.15.28:0/306939087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab000cb9e0 con 0x7fab000d4b90 2024-09-10T20:42:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.777+0000 7fab05f2b640 1 -- 172.21.15.28:0/306939087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab000b3870 con 0x7fab000d4b90 2024-09-10T20:42:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.778+0000 7faae7fff640 1 -- 172.21.15.28:0/306939087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7faaf0005020 con 0x7fab000d4b90 2024-09-10T20:42:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.778+0000 7faae7fff640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faaec078090 0x7faaec07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.778+0000 7faafeffd640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faaec078090 0x7faaec07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.778+0000 7faae7fff640 1 -- 172.21.15.28:0/306939087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 6543+0+0 (secure 0 0 0) 0x7faaf009a860 con 0x7fab000d4b90 2024-09-10T20:42:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.779+0000 7faafeffd640 1 --2- 172.21.15.28:0/306939087 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faaec078090 0x7faaec07a550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab000cf0b0 tx=0x7faaf4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:37.783+0000 7faae7fff640 1 -- 172.21.15.28:0/306939087 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fab000cb1a0 con 0x7fab000d4b90 2024-09-10T20:42:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:Invalid command: invalid chars #@ in abcd@# 2024-09-10T20:42:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:fs subvolume create [] [] [] [] [] [] [--namespace-isolated] : Create a CephFS subvolume in a volume, and optionally, with a specific size (in bytes), a specific data pool layout, a specific mode, in a specific subvolume group and in separate RADOS namespace 2024-09-10T20:42:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-09-10T20:42:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T20:42:38.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm testvol --yes-i-really-mean-it 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/2522927105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf015f580 msgr2=0x7f0cf00ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/2522927105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf015f580 0x7f0cf00ce750 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0073dd0 tx=0x7f0ce002a560 comp rx=0 tx=0).stop 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/2522927105 shutdown_connections 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/2522927105 >> v2:172.21.15.159:3300/0 conn(0x7f0cf00cdb30 0x7f0cf00cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/2522927105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf015f580 0x7f0cf00ce750 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/2522927105 >> 172.21.15.28:0/2522927105 conn(0x7f0cf00d1f30 msgr2=0x7f0cf00d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/2522927105 shutdown_connections 2024-09-10T20:42:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.104+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/2522927105 wait complete. 2024-09-10T20:42:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.106+0000 7f0cf5b3d640 1 Processor -- start 2024-09-10T20:42:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.106+0000 7f0cf5b3d640 1 -- start start 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.106+0000 7f0cf5b3d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf00b3110 0x7f0cf0153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cf5b3d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf00cdb30 0x7f0cf00c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cf5b3d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf015f580 0x7f0cf00c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cf5b3d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf00d1030 con 0x7f0cf00cdb30 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cf5b3d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf00d0eb0 con 0x7f0cf00b3110 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cf5b3d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0cf00b5470 con 0x7f0cf015f580 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf00cdb30 0x7f0cf00c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf00cdb30 0x7f0cf00c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51492/0 (socket says 172.21.15.28:51492) 2024-09-10T20:42:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 -- 172.21.15.28:0/1629661282 learned_addr learned my addr 172.21.15.28:0/1629661282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cef7fe640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf00b3110 0x7f0cf0153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceffff640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf015f580 0x7f0cf00c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 -- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf015f580 msgr2=0x7f0cf00c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf015f580 0x7f0cf00c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 -- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf00b3110 msgr2=0x7f0cf0153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf00b3110 0x7f0cf0153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 -- 172.21.15.28:0/1629661282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf014b170 con 0x7f0cf00cdb30 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceffff640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf015f580 0x7f0cf00c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0ceeffd640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf00cdb30 0x7f0cf00c8190 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400c400 tx=0x7f0ce400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cd7fff640 1 -- 172.21.15.28:0/1629661282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4017070 con 0x7f0cf00cdb30 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cd7fff640 1 -- 172.21.15.28:0/1629661282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ce40035f0 con 0x7f0cf00cdb30 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf0154400 con 0x7f0cf00cdb30 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cd7fff640 1 -- 172.21.15.28:0/1629661282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4003a80 con 0x7f0cf00cdb30 2024-09-10T20:42:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.107+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf0156a90 con 0x7f0cf00cdb30 2024-09-10T20:42:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.108+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf00b5a00 con 0x7f0cf00cdb30 2024-09-10T20:42:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.109+0000 7f0cd7fff640 1 -- 172.21.15.28:0/1629661282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f0ce4003c20 con 0x7f0cf00cdb30 2024-09-10T20:42:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.110+0000 7f0cd7fff640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cdc078090 0x7f0cdc07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.110+0000 7f0cd7fff640 1 -- 172.21.15.28:0/1629661282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 6543+0+0 (secure 0 0 0) 0x7f0ce4099890 con 0x7f0cf00cdb30 2024-09-10T20:42:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.110+0000 7f0cef7fe640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cdc078090 0x7f0cdc07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.110+0000 7f0cef7fe640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cdc078090 0x7f0cdc07a550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0033000 tx=0x7f0ce0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:38.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.114+0000 7f0cd7fff640 1 -- 172.21.15.28:0/1629661282 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0cf014b170 con 0x7f0cf00cdb30 2024-09-10T20:42:38.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:38.332+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "testvol", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) -- 0x7f0cf00ce290 con 0x7f0cdc078090 2024-09-10T20:42:38.339 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:42:38.336+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:42:38.369 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:42:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.360+0000 7f0cd7fff640 1 -- 172.21.15.28:0/1629661282 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+78 (secure 0 0 0) 0x7f0cf00ce290 con 0x7f0cdc078090 2024-09-10T20:42:41.364 INFO:teuthology.orchestra.run.smithi028.stdout:metadata pool: cephfs.testvol.meta data pool: ['cephfs.testvol.data'] removed 2024-09-10T20:42:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cdc078090 msgr2=0x7f0cdc07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cdc078090 0x7f0cdc07a550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0033000 tx=0x7f0ce0004730 comp rx=0 tx=0).stop 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf00cdb30 msgr2=0x7f0cf00c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf00cdb30 0x7f0cf00c8190 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400c400 tx=0x7f0ce400c8c0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 shutdown_connections 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cdc078090 0x7f0cdc07a550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cf015f580 0x7f0cf00c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf00cdb30 0x7f0cf00c8190 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 --2- 172.21.15.28:0/1629661282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cf00b3110 0x7f0cf0153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.363+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 >> 172.21.15.28:0/1629661282 conn(0x7f0cf00d1f30 msgr2=0x7f0cf00b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.364+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 shutdown_connections 2024-09-10T20:42:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.364+0000 7f0cf5b3d640 1 -- 172.21.15.28:0/1629661282 wait complete. 2024-09-10T20:42:41.382 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:42:41.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:41.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:41.492 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:42:41.492 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:41.492 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:41.492 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:41.510 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:41.510 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:41.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:42:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.713+0000 7f902759e640 1 -- 172.21.15.28:0/4024581869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90280cfb30 msgr2=0x7f9028162410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.713+0000 7f902759e640 1 --2- 172.21.15.28:0/4024581869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90280cfb30 0x7f9028162410 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9010009900 tx=0x7f901002a560 comp rx=0 tx=0).stop 2024-09-10T20:42:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.713+0000 7f902759e640 1 -- 172.21.15.28:0/4024581869 shutdown_connections 2024-09-10T20:42:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.713+0000 7f902759e640 1 --2- 172.21.15.28:0/4024581869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90280cfb30 0x7f9028162410 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.713+0000 7f902759e640 1 --2- 172.21.15.28:0/4024581869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f902815df70 0x7f90280d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.713+0000 7f902759e640 1 -- 172.21.15.28:0/4024581869 >> 172.21.15.28:0/4024581869 conn(0x7f902809d070 msgr2=0x7f90280d3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.713+0000 7f902759e640 1 -- 172.21.15.28:0/4024581869 shutdown_connections 2024-09-10T20:42:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.714+0000 7f902759e640 1 -- 172.21.15.28:0/4024581869 wait complete. 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.715+0000 7f902759e640 1 Processor -- start 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.715+0000 7f902759e640 1 -- start start 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902759e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90280cfb30 0x7f90280b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902759e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90280d0890 0x7f90280b8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f902815df70 0x7f90281f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902759e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90280d2610 con 0x7f902815df70 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902759e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f90280d10f0 con 0x7f90280cfb30 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902759e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f90280d2790 con 0x7f90280d0890 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f902815df70 0x7f90281f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f902815df70 0x7f90281f4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57290/0 (socket says 172.21.15.28:57290) 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 -- 172.21.15.28:0/3195311815 learned_addr learned my addr 172.21.15.28:0/3195311815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902659c640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90280cfb30 0x7f90280b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9025d9b640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90280d0890 0x7f90280b8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 -- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90280d0890 msgr2=0x7f90280b8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90280d0890 0x7f90280b8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 -- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90280cfb30 msgr2=0x7f90280b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90280cfb30 0x7f90280b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 -- 172.21.15.28:0/3195311815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90281f5370 con 0x7f902815df70 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f9026d9d640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f902815df70 0x7f90281f4600 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f901800c430 tx=0x7f901800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f901effd640 1 -- 172.21.15.28:0/3195311815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018017070 con 0x7f902815df70 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f901effd640 1 -- 172.21.15.28:0/3195311815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90180053c0 con 0x7f902815df70 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90280caff0 con 0x7f902815df70 2024-09-10T20:42:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.716+0000 7f901effd640 1 -- 172.21.15.28:0/3195311815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018003850 con 0x7f902815df70 2024-09-10T20:42:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.717+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90280cb5b0 con 0x7f902815df70 2024-09-10T20:42:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.717+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90280d2c40 con 0x7f902815df70 2024-09-10T20:42:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.718+0000 7f901effd640 1 -- 172.21.15.28:0/3195311815 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f90180039f0 con 0x7f902815df70 2024-09-10T20:42:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.719+0000 7f901effd640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9008078090 0x7f900807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.719+0000 7f902659c640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9008078090 0x7f900807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.719+0000 7f901effd640 1 -- 172.21.15.28:0/3195311815 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7f90180995c0 con 0x7f902815df70 2024-09-10T20:42:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.719+0000 7f902659c640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9008078090 0x7f900807a550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9020005fd0 tx=0x7f9020009500 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.723+0000 7f901effd640 1 -- 172.21.15.28:0/3195311815 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9018061880 con 0x7f902815df70 2024-09-10T20:42:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.918+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f902809e2d0 con 0x7f902815df70 2024-09-10T20:42:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.918+0000 7f901effd640 1 -- 172.21.15.28:0/3195311815 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v89) ==== 80+0+122 (secure 0 0 0) 0x7f90180662b0 con 0x7f902815df70 2024-09-10T20:42:41.922 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6834/2239499017 2024-09-11T20:42:38.336838+0000 2024-09-10T20:42:41.922 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6832/2239499017 2024-09-11T20:42:38.336838+0000 2024-09-10T20:42:41.923 INFO:teuthology.orchestra.run.smithi028.stderr:listed 2 entries 2024-09-10T20:42:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9008078090 msgr2=0x7f900807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9008078090 0x7f900807a550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9020005fd0 tx=0x7f9020009500 comp rx=0 tx=0).stop 2024-09-10T20:42:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f902815df70 msgr2=0x7f90281f4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f902815df70 0x7f90281f4600 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f901800c430 tx=0x7f901800c8f0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 shutdown_connections 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9008078090 0x7f900807a550 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f902815df70 0x7f90281f4600 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f90280d0890 0x7f90280b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 --2- 172.21.15.28:0/3195311815 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f90280cfb30 0x7f90280b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 >> 172.21.15.28:0/3195311815 conn(0x7f902809d070 msgr2=0x7f9028161a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 shutdown_connections 2024-09-10T20:42:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:41.920+0000 7f902759e640 1 -- 172.21.15.28:0/3195311815 wait complete. 2024-09-10T20:42:41.935 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:42:41.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:41.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:42:41.965 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:42:41.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:41.966 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:42:42.021 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:42.022 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:42.022 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:42.041 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:42.041 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:42.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 -- 172.21.15.28:0/3620003879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 msgr2=0x7fadb806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/3620003879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb806fdf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fadb0005fd0 tx=0x7fadb002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 -- 172.21.15.28:0/3620003879 shutdown_connections 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/3620003879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb806fdf0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/3620003879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadb815e4c0 0x7fadb806ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 -- 172.21.15.28:0/3620003879 >> 172.21.15.28:0/3620003879 conn(0x7fadb80d0420 msgr2=0x7fadb80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 -- 172.21.15.28:0/3620003879 shutdown_connections 2024-09-10T20:42:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.203+0000 7fadbf2dd640 1 -- 172.21.15.28:0/3620003879 wait complete. 2024-09-10T20:42:42.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.204+0000 7fadbf2dd640 1 Processor -- start 2024-09-10T20:42:42.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.204+0000 7fadbf2dd640 1 -- start start 2024-09-10T20:42:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbf2dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb8156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbf2dd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadb815e4c0 0x7fadb8159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbf2dd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadb8156680 0x7fadb80b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbf2dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadb80c9e60 con 0x7fadb806f940 2024-09-10T20:42:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbf2dd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fadb8068bd0 con 0x7fadb8156680 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbf2dd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fadb80c9fe0 con 0x7fadb815e4c0 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb8156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb8156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57310/0 (socket says 172.21.15.28:57310) 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 -- 172.21.15.28:0/2243385847 learned_addr learned my addr 172.21.15.28:0/2243385847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbc878640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadb815e4c0 0x7fadb8159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 -- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadb815e4c0 msgr2=0x7fadb8159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd87a640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadb8156680 0x7fadb80b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadb815e4c0 0x7fadb8159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 -- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadb8156680 msgr2=0x7fadb80b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadb8156680 0x7fadb80b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 -- 172.21.15.28:0/2243385847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb81f4c30 con 0x7fadb806f940 2024-09-10T20:42:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbc878640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadb815e4c0 0x7fadb8159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbd079640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb8156050 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fadb400c450 tx=0x7fadb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadadffb640 1 -- 172.21.15.28:0/2243385847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4016020 con 0x7fadb806f940 2024-09-10T20:42:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadadffb640 1 -- 172.21.15.28:0/2243385847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadb4012920 con 0x7fadb806f940 2024-09-10T20:42:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb81f8770 con 0x7fadb806f940 2024-09-10T20:42:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.205+0000 7fadadffb640 1 -- 172.21.15.28:0/2243385847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4012d70 con 0x7fadb806f940 2024-09-10T20:42:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.206+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb81f8d30 con 0x7fadb806f940 2024-09-10T20:42:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.207+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb80ca500 con 0x7fadb806f940 2024-09-10T20:42:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.207+0000 7fadadffb640 1 -- 172.21.15.28:0/2243385847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fadb401b400 con 0x7fadb806f940 2024-09-10T20:42:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.208+0000 7fadadffb640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fad9c078090 0x7fad9c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.208+0000 7fadadffb640 1 -- 172.21.15.28:0/2243385847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7fadb4099fc0 con 0x7fadb806f940 2024-09-10T20:42:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.208+0000 7fadbc878640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fad9c078090 0x7fad9c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.208+0000 7fadbc878640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fad9c078090 0x7fad9c07a550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fadb806f470 tx=0x7fada8008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:42.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.212+0000 7fadadffb640 1 -- 172.21.15.28:0/2243385847 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fadb4020030 con 0x7fadb806f940 2024-09-10T20:42:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.411+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fadb815bde0 con 0x7fadb806f940 2024-09-10T20:42:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.412+0000 7fadadffb640 1 -- 172.21.15.28:0/2243385847 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v89) ==== 80+0+122 (secure 0 0 0) 0x7fadb40628d0 con 0x7fadb806f940 2024-09-10T20:42:42.416 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6834/2239499017 2024-09-11T20:42:38.336838+0000 2024-09-10T20:42:42.416 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6832/2239499017 2024-09-11T20:42:38.336838+0000 2024-09-10T20:42:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:listed 2 entries 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fad9c078090 msgr2=0x7fad9c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fad9c078090 0x7fad9c07a550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fadb806f470 tx=0x7fada8008040 comp rx=0 tx=0).stop 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 msgr2=0x7fadb8156050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb8156050 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fadb400c450 tx=0x7fadb400c910 comp rx=0 tx=0).stop 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 shutdown_connections 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fad9c078090 0x7fad9c07a550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadb8156680 0x7fadb80b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadb815e4c0 0x7fadb8159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 --2- 172.21.15.28:0/2243385847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb806f940 0x7fadb8156050 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 >> 172.21.15.28:0/2243385847 conn(0x7fadb80d0420 msgr2=0x7fadb8160700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 shutdown_connections 2024-09-10T20:42:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.414+0000 7fadbf2dd640 1 -- 172.21.15.28:0/2243385847 wait complete. 2024-09-10T20:42:42.429 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:42:42.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:42.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:42:42.460 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:42:42.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:42:42.461 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:42:42.518 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:42.518 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:42.518 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:42.535 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:42:42.535 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:42:42.535 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:42.535 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:42.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:42:42.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.677+0000 7fd68bb3a640 1 -- 172.21.15.28:0/4220338360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 msgr2=0x7fd6840b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.677+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/4220338360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd6840b2e80 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009be0 tx=0x7fd67c02a960 comp rx=0 tx=0).stop 2024-09-10T20:42:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.677+0000 7fd68bb3a640 1 -- 172.21.15.28:0/4220338360 shutdown_connections 2024-09-10T20:42:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.677+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/4220338360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd6840b2e80 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.678+0000 7fd68bb3a640 1 -- 172.21.15.28:0/4220338360 >> 172.21.15.28:0/4220338360 conn(0x7fd68406cf00 msgr2=0x7fd68406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.678+0000 7fd68bb3a640 1 -- 172.21.15.28:0/4220338360 shutdown_connections 2024-09-10T20:42:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.678+0000 7fd68bb3a640 1 -- 172.21.15.28:0/4220338360 wait complete. 2024-09-10T20:42:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.679+0000 7fd68bb3a640 1 Processor -- start 2024-09-10T20:42:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.679+0000 7fd68bb3a640 1 -- start start 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.679+0000 7fd68bb3a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6840cc570 0x7fd6840b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68bb3a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6840cd3d0 0x7fd68409ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68bb3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd68414e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68bb3a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6840b3540 con 0x7fd6840d07a0 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68bb3a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd6840b33c0 con 0x7fd6840cc570 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68bb3a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd6840b36c0 con 0x7fd6840cd3d0 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd68414e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd68414e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57334/0 (socket says 172.21.15.28:57334) 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 -- 172.21.15.28:0/28007836 learned_addr learned my addr 172.21.15.28:0/28007836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd6898d6640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6840cc570 0x7fd6840b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd6890d5640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6840cd3d0 0x7fd68409ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 -- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6840cd3d0 msgr2=0x7fd68409ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6840cd3d0 0x7fd68409ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 -- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6840cc570 msgr2=0x7fd6840b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6840cc570 0x7fd6840b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 -- 172.21.15.28:0/28007836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6841f22f0 con 0x7fd6840d07a0 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68a0d7640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd68414e790 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009be0 tx=0x7fd67c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd67a7fc640 1 -- 172.21.15.28:0/28007836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c030c50 con 0x7fd6840d07a0 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd67a7fc640 1 -- 172.21.15.28:0/28007836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd67c03e070 con 0x7fd6840d07a0 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd684159f50 con 0x7fd6840d07a0 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.680+0000 7fd67a7fc640 1 -- 172.21.15.28:0/28007836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c032330 con 0x7fd6840d07a0 2024-09-10T20:42:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.681+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd684158240 con 0x7fd6840d07a0 2024-09-10T20:42:42.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.682+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6840b3b40 con 0x7fd6840d07a0 2024-09-10T20:42:42.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.682+0000 7fd67a7fc640 1 -- 172.21.15.28:0/28007836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fd67c0324d0 con 0x7fd6840d07a0 2024-09-10T20:42:42.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.683+0000 7fd67a7fc640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd668078090 0x7fd66807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:42.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.683+0000 7fd6898d6640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd668078090 0x7fd66807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:42.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.683+0000 7fd67a7fc640 1 -- 172.21.15.28:0/28007836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7fd67c0bb740 con 0x7fd6840d07a0 2024-09-10T20:42:42.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.683+0000 7fd6898d6640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd668078090 0x7fd66807a550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd680000f60 tx=0x7fd680009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:42.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.687+0000 7fd67a7fc640 1 -- 172.21.15.28:0/28007836 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd67c02d070 con 0x7fd6840d07a0 2024-09-10T20:42:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.911+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6840b0730 con 0x7fd6840d07a0 2024-09-10T20:42:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.912+0000 7fd67a7fc640 1 -- 172.21.15.28:0/28007836 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5231 (secure 0 0 0) 0x7fd67c083970 con 0x7fd6840d07a0 2024-09-10T20:42:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:42.916 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":58,"btime":"2024-09-10T20:42:39:338547+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2983894865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2983894865},{"type":"v1","addr":"172.21.15.159:6801","nonce":2983894865}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2105531171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2105531171},{"type":"v1","addr":"172.21.15.159:6835","nonce":2105531171}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1370248803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1370248803},{"type":"v1","addr":"172.21.15.28:6834","nonce":1370248803}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:23.614228+0000","modified":"2024-09-10T20:42:26.617829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6570":{"gid":6570,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2216477518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2216477518},{"type":"v1","addr":"172.21.15.28:6835","nonce":2216477518}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6570,"qdb_cluster":[6570]},"id":6}]} 2024-09-10T20:42:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 58 2024-09-10T20:42:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd668078090 msgr2=0x7fd66807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd668078090 0x7fd66807a550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd680000f60 tx=0x7fd680009040 comp rx=0 tx=0).stop 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 msgr2=0x7fd68414e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd68414e790 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009be0 tx=0x7fd67c003ba0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 shutdown_connections 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd668078090 0x7fd66807a550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6840d07a0 0x7fd68414e790 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6840cd3d0 0x7fd68409ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 --2- 172.21.15.28:0/28007836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6840cc570 0x7fd6840b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.914+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 >> 172.21.15.28:0/28007836 conn(0x7fd68406cf00 msgr2=0x7fd6840b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.915+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 shutdown_connections 2024-09-10T20:42:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:42.915+0000 7fd68bb3a640 1 -- 172.21.15.28:0/28007836 wait complete. 2024-09-10T20:42:42.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 -- 172.21.15.28:0/1876102881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540b6620 msgr2=0x7f6b540d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 --2- 172.21.15.28:0/1876102881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540b6620 0x7f6b540d0330 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6b54073cb0 tx=0x7f6b4402a560 comp rx=0 tx=0).stop 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 -- 172.21.15.28:0/1876102881 shutdown_connections 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 --2- 172.21.15.28:0/1876102881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540b6620 0x7f6b540d0330 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 --2- 172.21.15.28:0/1876102881 >> v2:172.21.15.159:3300/0 conn(0x7f6b5415bf30 0x7f6b540d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 -- 172.21.15.28:0/1876102881 >> 172.21.15.28:0/1876102881 conn(0x7f6b540d1e40 msgr2=0x7f6b540d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 -- 172.21.15.28:0/1876102881 shutdown_connections 2024-09-10T20:42:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.030+0000 7f6b59590640 1 -- 172.21.15.28:0/1876102881 wait complete. 2024-09-10T20:42:43.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 Processor -- start 2024-09-10T20:42:43.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 -- start start 2024-09-10T20:42:43.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b540b6620 0x7f6b541f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540cfec0 0x7f6b541f8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5415bf30 0x7f6b54147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b540d0cc0 con 0x7f6b540cfec0 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6b540d0b40 con 0x7f6b540b6620 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.032+0000 7f6b59590640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6b540ced00 con 0x7f6b5415bf30 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540cfec0 0x7f6b541f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540cfec0 0x7f6b541f8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57348/0 (socket says 172.21.15.28:57348) 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 -- 172.21.15.28:0/183411379 learned_addr learned my addr 172.21.15.28:0/183411379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b52ffd640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b540b6620 0x7f6b541f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b537fe640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5415bf30 0x7f6b54147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 -- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5415bf30 msgr2=0x7f6b54147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5415bf30 0x7f6b54147bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 -- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b540b6620 msgr2=0x7f6b541f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b540b6620 0x7f6b541f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 -- 172.21.15.28:0/183411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b541eadb0 con 0x7f6b540cfec0 2024-09-10T20:42:43.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b527fc640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540cfec0 0x7f6b541f8970 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00c430 tx=0x7f6b3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:43.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b4b7fe640 1 -- 172.21.15.28:0/183411379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c011790 con 0x7f6b540cfec0 2024-09-10T20:42:43.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b4b7fe640 1 -- 172.21.15.28:0/183411379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b3c017070 con 0x7f6b540cfec0 2024-09-10T20:42:43.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b4b7fe640 1 -- 172.21.15.28:0/183411379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c011d00 con 0x7f6b540cfec0 2024-09-10T20:42:43.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b540ca950 con 0x7f6b540cfec0 2024-09-10T20:42:43.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.033+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b540ca640 con 0x7f6b540cfec0 2024-09-10T20:42:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.035+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b540cf120 con 0x7f6b540cfec0 2024-09-10T20:42:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.035+0000 7f6b4b7fe640 1 -- 172.21.15.28:0/183411379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f6b3c01c460 con 0x7f6b540cfec0 2024-09-10T20:42:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.036+0000 7f6b4b7fe640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6b34078090 0x7f6b3407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.036+0000 7f6b52ffd640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6b34078090 0x7f6b3407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.036+0000 7f6b4b7fe640 1 -- 172.21.15.28:0/183411379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7f6b3c099ce0 con 0x7f6b540cfec0 2024-09-10T20:42:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.036+0000 7f6b52ffd640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6b34078090 0x7f6b3407a550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c005fd0 tx=0x7f6b4c0095e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.040+0000 7f6b4b7fe640 1 -- 172.21.15.28:0/183411379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6b3c061fa0 con 0x7f6b540cfec0 2024-09-10T20:42:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.266+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b540cab20 con 0x7f6b540cfec0 2024-09-10T20:42:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.266+0000 7f6b4b7fe640 1 -- 172.21.15.28:0/183411379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5231 (secure 0 0 0) 0x7f6b3c0669d0 con 0x7f6b540cfec0 2024-09-10T20:42:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":58,"btime":"2024-09-10T20:42:39:338547+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2983894865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2983894865},{"type":"v1","addr":"172.21.15.159:6801","nonce":2983894865}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2105531171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2105531171},{"type":"v1","addr":"172.21.15.159:6835","nonce":2105531171}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1370248803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1370248803},{"type":"v1","addr":"172.21.15.28:6834","nonce":1370248803}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:23.614228+0000","modified":"2024-09-10T20:42:26.617829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6570":{"gid":6570,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2216477518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2216477518},{"type":"v1","addr":"172.21.15.28:6835","nonce":2216477518}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6570,"qdb_cluster":[6570]},"id":6}]} 2024-09-10T20:42:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 58 2024-09-10T20:42:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6b34078090 msgr2=0x7f6b3407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6b34078090 0x7f6b3407a550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c005fd0 tx=0x7f6b4c0095e0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540cfec0 msgr2=0x7f6b541f8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540cfec0 0x7f6b541f8970 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00c430 tx=0x7f6b3c00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 shutdown_connections 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6b34078090 0x7f6b3407a550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b5415bf30 0x7f6b54147bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b540cfec0 0x7f6b541f8970 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 --2- 172.21.15.28:0/183411379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b540b6620 0x7f6b541f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.268+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 >> 172.21.15.28:0/183411379 conn(0x7f6b540d1e40 msgr2=0x7f6b5415b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.269+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 shutdown_connections 2024-09-10T20:42:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.269+0000 7f6b59590640 1 -- 172.21.15.28:0/183411379 wait complete. 2024-09-10T20:42:43.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.385+0000 7f3bdc416640 1 -- 172.21.15.28:0/2696538902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415eef0 msgr2=0x7f3bd40cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.385+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2696538902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415eef0 0x7f3bd40cb210 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc005fd0 tx=0x7f3bcc02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.385+0000 7f3bdc416640 1 -- 172.21.15.28:0/2696538902 shutdown_connections 2024-09-10T20:42:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.385+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2696538902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415eef0 0x7f3bd40cb210 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.385+0000 7f3bdc416640 1 -- 172.21.15.28:0/2696538902 >> 172.21.15.28:0/2696538902 conn(0x7f3bd40d0420 msgr2=0x7f3bd40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.385+0000 7f3bdc416640 1 -- 172.21.15.28:0/2696538902 shutdown_connections 2024-09-10T20:42:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.385+0000 7f3bdc416640 1 -- 172.21.15.28:0/2696538902 wait complete. 2024-09-10T20:42:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.386+0000 7f3bdc416640 1 Processor -- start 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.386+0000 7f3bdc416640 1 -- start start 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bdc416640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40cf0f0 0x7f3bd4142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bdc416640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3bd4151cc0 0x7f3bd4154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bdc416640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3bd415eef0 0x7f3bd415d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bdc416640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd415f5b0 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bdc416640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd415f430 con 0x7f3bd4151cc0 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bdc416640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd40cb750 con 0x7f3bd415eef0 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40cf0f0 0x7f3bd4142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40cf0f0 0x7f3bd4142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57366/0 (socket says 172.21.15.28:57366) 2024-09-10T20:42:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 -- 172.21.15.28:0/2890385193 learned_addr learned my addr 172.21.15.28:0/2890385193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bd99b1640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3bd4151cc0 0x7f3bd4154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 -- 172.21.15.28:0/2890385193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3bd415eef0 msgr2=0x7f3bd415d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3bd415eef0 0x7f3bd415d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 -- 172.21.15.28:0/2890385193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3bd4151cc0 msgr2=0x7f3bd4154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3bd4151cc0 0x7f3bd4154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 -- 172.21.15.28:0/2890385193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd41533a0 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.387+0000 7f3bda1b2640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40cf0f0 0x7f3bd4142f90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000c480 tx=0x7f3bd000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.388+0000 7f3bcaffd640 1 -- 172.21.15.28:0/2890385193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0016020 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.388+0000 7f3bcaffd640 1 -- 172.21.15.28:0/2890385193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bd0012920 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.388+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd4153680 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.388+0000 7f3bcaffd640 1 -- 172.21.15.28:0/2890385193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0012d70 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.388+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd4153c40 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.389+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd40cbc00 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.389+0000 7f3bcaffd640 1 -- 172.21.15.28:0/2890385193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f3bd001b400 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.390+0000 7f3bcaffd640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3bb8078090 0x7f3bb807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.390+0000 7f3bd99b1640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3bb8078090 0x7f3bb807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.390+0000 7f3bcaffd640 1 -- 172.21.15.28:0/2890385193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7f3bd009a3e0 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.391+0000 7f3bd99b1640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3bb8078090 0x7f3bb807a550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406d5b0 tx=0x7f3bc4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.394+0000 7f3bcaffd640 1 -- 172.21.15.28:0/2890385193 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3bd0020020 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.580+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bd4068d10 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.581+0000 7f3bcaffd640 1 -- 172.21.15.28:0/2890385193 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16861 (secure 0 0 0) 0x7f3bd0062cf0 con 0x7f3bd40cf0f0 2024-09-10T20:42:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:43.585 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:41.354414+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:21.256150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:22.565350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/2239499017":"2024-09-11T20:42:38.336838+0000","172.21.15.28:6832/2239499017":"2024-09-11T20:42:38.336838+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3bb8078090 msgr2=0x7f3bb807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3bb8078090 0x7f3bb807a550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406d5b0 tx=0x7f3bc4009040 comp rx=0 tx=0).stop 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40cf0f0 msgr2=0x7f3bd4142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40cf0f0 0x7f3bd4142f90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000c480 tx=0x7f3bd000c940 comp rx=0 tx=0).stop 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 shutdown_connections 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3bb8078090 0x7f3bb807a550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3bd415eef0 0x7f3bd415d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3bd4151cc0 0x7f3bd4154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 --2- 172.21.15.28:0/2890385193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40cf0f0 0x7f3bd4142f90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 >> 172.21.15.28:0/2890385193 conn(0x7f3bd40d0420 msgr2=0x7f3bd40ca0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 shutdown_connections 2024-09-10T20:42:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.584+0000 7f3bdc416640 1 -- 172.21.15.28:0/2890385193 wait complete. 2024-09-10T20:42:43.599 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:42:43.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.701+0000 7f16caf3d640 1 -- 172.21.15.28:0/541907108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b4003c10 msgr2=0x7f16b4004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.701+0000 7f16caf3d640 1 --2- 172.21.15.28:0/541907108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b4003c10 0x7f16b4004010 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f16b8006fb0 tx=0x7f16b4005830 comp rx=0 tx=0).stop 2024-09-10T20:42:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.701+0000 7f16c94da640 1 -- 172.21.15.28:0/541907108 reap_dead start 2024-09-10T20:42:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.701+0000 7f16caf3d640 1 -- 172.21.15.28:0/541907108 shutdown_connections 2024-09-10T20:42:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.701+0000 7f16caf3d640 1 -- 172.21.15.28:0/541907108 >> 172.21.15.28:0/541907108 conn(0x7f16c40d3fb0 msgr2=0x7f16c40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.701+0000 7f16caf3d640 1 -- 172.21.15.28:0/541907108 shutdown_connections 2024-09-10T20:42:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.701+0000 7f16caf3d640 1 -- 172.21.15.28:0/541907108 wait complete. 2024-09-10T20:42:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.702+0000 7f16caf3d640 1 Processor -- start 2024-09-10T20:42:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.702+0000 7f16caf3d640 1 -- start start 2024-09-10T20:42:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16caf3d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b4003c10 0x7f16c414d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16caf3d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c414caf0 0x7f16c40a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16caf3d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c414cf70 0x7f16c40a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16caf3d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16c415e3d0 con 0x7f16c414cf70 2024-09-10T20:42:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16caf3d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f16c415e250 con 0x7f16b4003c10 2024-09-10T20:42:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16caf3d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f16c4161b50 con 0x7f16c414caf0 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16c94da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c414cf70 0x7f16c40a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16c94da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c414cf70 0x7f16c40a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57390/0 (socket says 172.21.15.28:57390) 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16c94da640 1 -- 172.21.15.28:0/824774272 learned_addr learned my addr 172.21.15.28:0/824774272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16c8cd9640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b4003c10 0x7f16c414d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.703+0000 7f16c3fff640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c414caf0 0x7f16c40a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c94da640 1 -- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c414caf0 msgr2=0x7f16c40a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c94da640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c414caf0 0x7f16c40a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c94da640 1 -- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b4003c10 msgr2=0x7f16c414d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c94da640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b4003c10 0x7f16c414d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c94da640 1 -- 172.21.15.28:0/824774272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c4150510 con 0x7f16c414cf70 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c3fff640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c414caf0 0x7f16c40a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c94da640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c414cf70 0x7f16c40a0d10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f16c40d04a0 tx=0x7f16bc001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c17fa640 1 -- 172.21.15.28:0/824774272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc010890 con 0x7f16c414cf70 2024-09-10T20:42:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c17fa640 1 -- 172.21.15.28:0/824774272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16bc017070 con 0x7f16c414cf70 2024-09-10T20:42:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c41f2390 con 0x7f16c414cf70 2024-09-10T20:42:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16c17fa640 1 -- 172.21.15.28:0/824774272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc010e00 con 0x7f16c414cf70 2024-09-10T20:42:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.704+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c40cc4c0 con 0x7f16c414cf70 2024-09-10T20:42:43.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.705+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c40cf780 con 0x7f16c414cf70 2024-09-10T20:42:43.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.706+0000 7f16c17fa640 1 -- 172.21.15.28:0/824774272 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f16bc01c460 con 0x7f16c414cf70 2024-09-10T20:42:43.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.706+0000 7f16c17fa640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f16a8078090 0x7f16a807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:43.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.706+0000 7f16c17fa640 1 -- 172.21.15.28:0/824774272 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7f16bc09b360 con 0x7f16c414cf70 2024-09-10T20:42:43.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.706+0000 7f16c8cd9640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f16a8078090 0x7f16a807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:43.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.707+0000 7f16c8cd9640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f16a8078090 0x7f16a807a550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f16b8006fb0 tx=0x7f16b803b040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.710+0000 7f16c17fa640 1 -- 172.21.15.28:0/824774272 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f16bc063cc0 con 0x7f16c414cf70 2024-09-10T20:42:43.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.931+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16c40d16e0 con 0x7f16c414cf70 2024-09-10T20:42:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.932+0000 7f16c17fa640 1 -- 172.21.15.28:0/824774272 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5231 (secure 0 0 0) 0x7f16bc06c3e0 con 0x7f16c414cf70 2024-09-10T20:42:43.936 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:43.936 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":58,"btime":"2024-09-10T20:42:39:338547+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2983894865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2983894865},{"type":"v1","addr":"172.21.15.159:6801","nonce":2983894865}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2105531171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2105531171},{"type":"v1","addr":"172.21.15.159:6835","nonce":2105531171}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1370248803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1370248803},{"type":"v1","addr":"172.21.15.28:6834","nonce":1370248803}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:23.614228+0000","modified":"2024-09-10T20:42:26.617829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6570":{"gid":6570,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2216477518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2216477518},{"type":"v1","addr":"172.21.15.28:6835","nonce":2216477518}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6570,"qdb_cluster":[6570]},"id":6}]} 2024-09-10T20:42:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 58 2024-09-10T20:42:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f16a8078090 msgr2=0x7f16a807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f16a8078090 0x7f16a807a550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f16b8006fb0 tx=0x7f16b803b040 comp rx=0 tx=0).stop 2024-09-10T20:42:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c414cf70 msgr2=0x7f16c40a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c414cf70 0x7f16c40a0d10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f16c40d04a0 tx=0x7f16bc001b80 comp rx=0 tx=0).stop 2024-09-10T20:42:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 shutdown_connections 2024-09-10T20:42:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f16a8078090 0x7f16a807a550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c414cf70 0x7f16c40a0d10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16c414caf0 0x7f16c40a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 --2- 172.21.15.28:0/824774272 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b4003c10 0x7f16c414d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 >> 172.21.15.28:0/824774272 conn(0x7f16c40d3fb0 msgr2=0x7f16c40b86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 shutdown_connections 2024-09-10T20:42:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:43.934+0000 7f16caf3d640 1 -- 172.21.15.28:0/824774272 wait complete. 2024-09-10T20:42:43.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.051+0000 7f15bfea3640 1 -- 172.21.15.28:0/4020874926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 msgr2=0x7f15b80cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.051+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4020874926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 0x7f15b80cb210 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f15b0005fd0 tx=0x7f15b002aa90 comp rx=0 tx=0).stop 2024-09-10T20:42:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.051+0000 7f15bfea3640 1 -- 172.21.15.28:0/4020874926 shutdown_connections 2024-09-10T20:42:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.051+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4020874926 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 0x7f15b80cb210 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.051+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4020874926 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8151cc0 0x7f15b80cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.051+0000 7f15bfea3640 1 -- 172.21.15.28:0/4020874926 >> 172.21.15.28:0/4020874926 conn(0x7f15b80d0420 msgr2=0x7f15b80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.052+0000 7f15bfea3640 1 -- 172.21.15.28:0/4020874926 shutdown_connections 2024-09-10T20:42:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.052+0000 7f15bfea3640 1 -- 172.21.15.28:0/4020874926 wait complete. 2024-09-10T20:42:44.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.053+0000 7f15bfea3640 1 Processor -- start 2024-09-10T20:42:44.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.053+0000 7f15bfea3640 1 -- start start 2024-09-10T20:42:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bfea3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b80738b0 0x7f15b814ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bfea3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8151cc0 0x7f15b814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bfea3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 0x7f15b81504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bfea3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15b8073e70 con 0x7f15b80738b0 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bfea3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f15b8073cf0 con 0x7f15b815eef0 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bfea3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f15b8073ff0 con 0x7f15b8151cc0 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b80738b0 0x7f15b814ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b80738b0 0x7f15b814ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57394/0 (socket says 172.21.15.28:57394) 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 -- 172.21.15.28:0/4125998582 learned_addr learned my addr 172.21.15.28:0/4125998582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bd43e640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8151cc0 0x7f15b814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15be440640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 0x7f15b81504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 -- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8151cc0 msgr2=0x7f15b814ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8151cc0 0x7f15b814ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 -- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 msgr2=0x7f15b81504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 0x7f15b81504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 -- 172.21.15.28:0/4125998582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b80b1000 con 0x7f15b80738b0 2024-09-10T20:42:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bdc3f640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b80738b0 0x7f15b814ee00 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f15b400e9f0 tx=0x7f15b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15ae7fc640 1 -- 172.21.15.28:0/4125998582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b400cd80 con 0x7f15b80738b0 2024-09-10T20:42:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15ae7fc640 1 -- 172.21.15.28:0/4125998582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15b4010430 con 0x7f15b80738b0 2024-09-10T20:42:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b80b0240 con 0x7f15b80738b0 2024-09-10T20:42:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.054+0000 7f15ae7fc640 1 -- 172.21.15.28:0/4125998582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4010840 con 0x7f15b80738b0 2024-09-10T20:42:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.055+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b80b0800 con 0x7f15b80738b0 2024-09-10T20:42:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.056+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b80cb9f0 con 0x7f15b80738b0 2024-09-10T20:42:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.056+0000 7f15ae7fc640 1 -- 172.21.15.28:0/4125998582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f15b40109e0 con 0x7f15b80738b0 2024-09-10T20:42:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.057+0000 7f15ae7fc640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f159c078090 0x7f159c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.057+0000 7f15ae7fc640 1 -- 172.21.15.28:0/4125998582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7f15b409abf0 con 0x7f15b80738b0 2024-09-10T20:42:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.057+0000 7f15bd43e640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f159c078090 0x7f159c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.057+0000 7f15bd43e640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f159c078090 0x7f159c07a550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15b81f3680 tx=0x7f15a8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.061+0000 7f15ae7fc640 1 -- 172.21.15.28:0/4125998582 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f15b4062e40 con 0x7f15b80738b0 2024-09-10T20:42:44.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.281+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15b8068d10 con 0x7f15b80738b0 2024-09-10T20:42:44.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.281+0000 7f15ae7fc640 1 -- 172.21.15.28:0/4125998582 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5231 (secure 0 0 0) 0x7f15b4067870 con 0x7f15b80738b0 2024-09-10T20:42:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:44.286 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":58,"btime":"2024-09-10T20:42:39:338547+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2983894865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2983894865},{"type":"v1","addr":"172.21.15.159:6801","nonce":2983894865}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2105531171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2105531171},{"type":"v1","addr":"172.21.15.159:6835","nonce":2105531171}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1370248803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1370248803},{"type":"v1","addr":"172.21.15.28:6834","nonce":1370248803}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:23.614228+0000","modified":"2024-09-10T20:42:26.617829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6570":{"gid":6570,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2216477518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2216477518},{"type":"v1","addr":"172.21.15.28:6835","nonce":2216477518}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6570,"qdb_cluster":[6570]},"id":6}]} 2024-09-10T20:42:44.286 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 58 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f159c078090 msgr2=0x7f159c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f159c078090 0x7f159c07a550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15b81f3680 tx=0x7f15a8009040 comp rx=0 tx=0).stop 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b80738b0 msgr2=0x7f15b814ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b80738b0 0x7f15b814ee00 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f15b400e9f0 tx=0x7f15b400eeb0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 shutdown_connections 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f159c078090 0x7f159c07a550 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15b815eef0 0x7f15b81504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15b8151cc0 0x7f15b814ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 --2- 172.21.15.28:0/4125998582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b80738b0 0x7f15b814ee00 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 >> 172.21.15.28:0/4125998582 conn(0x7f15b80d0420 msgr2=0x7f15b80cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 shutdown_connections 2024-09-10T20:42:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.284+0000 7f15bfea3640 1 -- 172.21.15.28:0/4125998582 wait complete. 2024-09-10T20:42:44.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 -- 172.21.15.28:0/2264802504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 msgr2=0x7ffa200cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 --2- 172.21.15.28:0/2264802504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 0x7ffa200cfa10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009830 tx=0x7ffa1c02a510 comp rx=0 tx=0).stop 2024-09-10T20:42:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 -- 172.21.15.28:0/2264802504 shutdown_connections 2024-09-10T20:42:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 --2- 172.21.15.28:0/2264802504 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa200d2960 0x7ffa200ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 --2- 172.21.15.28:0/2264802504 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 0x7ffa200cfa10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 -- 172.21.15.28:0/2264802504 >> 172.21.15.28:0/2264802504 conn(0x7ffa200d1e40 msgr2=0x7ffa200d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 -- 172.21.15.28:0/2264802504 shutdown_connections 2024-09-10T20:42:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.401+0000 7ffa277b5640 1 -- 172.21.15.28:0/2264802504 wait complete. 2024-09-10T20:42:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.402+0000 7ffa277b5640 1 Processor -- start 2024-09-10T20:42:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.402+0000 7ffa277b5640 1 -- start start 2024-09-10T20:42:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa277b5640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa200cd9b0 0x7ffa201594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa277b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa200d2960 0x7ffa200c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa277b5640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 0x7ffa200c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa277b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200b1af0 con 0x7ffa200d2960 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa277b5640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200cf110 con 0x7ffa2015c050 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa277b5640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffa200b1c70 con 0x7ffa200cd9b0 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa24d50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa200d2960 0x7ffa200c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa24d50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa200d2960 0x7ffa200c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57410/0 (socket says 172.21.15.28:57410) 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.403+0000 7ffa24d50640 1 -- 172.21.15.28:0/497845528 learned_addr learned my addr 172.21.15.28:0/497845528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa25551640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa200cd9b0 0x7ffa201594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa25d52640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 0x7ffa200c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa24d50640 1 -- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa200cd9b0 msgr2=0x7ffa201594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa24d50640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa200cd9b0 0x7ffa201594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa24d50640 1 -- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 msgr2=0x7ffa200c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa24d50640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 0x7ffa200c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa24d50640 1 -- 172.21.15.28:0/497845528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2006e600 con 0x7ffa200d2960 2024-09-10T20:42:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa24d50640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa200d2960 0x7ffa200c5580 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000e9a0 tx=0x7ffa1000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa15ffb640 1 -- 172.21.15.28:0/497845528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1000cd20 con 0x7ffa200d2960 2024-09-10T20:42:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa15ffb640 1 -- 172.21.15.28:0/497845528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa10010430 con 0x7ffa200d2960 2024-09-10T20:42:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2006e940 con 0x7ffa200d2960 2024-09-10T20:42:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa15ffb640 1 -- 172.21.15.28:0/497845528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10010880 con 0x7ffa200d2960 2024-09-10T20:42:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.404+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2006f180 con 0x7ffa200d2960 2024-09-10T20:42:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.405+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa200724d0 con 0x7ffa200d2960 2024-09-10T20:42:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.406+0000 7ffa15ffb640 1 -- 172.21.15.28:0/497845528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7ffa10010a20 con 0x7ffa200d2960 2024-09-10T20:42:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.406+0000 7ffa15ffb640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa04078090 0x7ffa0407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.407+0000 7ffa15ffb640 1 -- 172.21.15.28:0/497845528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7ffa1009abc0 con 0x7ffa200d2960 2024-09-10T20:42:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.407+0000 7ffa25551640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa04078090 0x7ffa0407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.407+0000 7ffa25551640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa04078090 0x7ffa0407a550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c034000 tx=0x7ffa1c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.411+0000 7ffa15ffb640 1 -- 172.21.15.28:0/497845528 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffa200724d0 con 0x7ffa200d2960 2024-09-10T20:42:44.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.598+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa20076360 con 0x7ffa200d2960 2024-09-10T20:42:44.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.599+0000 7ffa15ffb640 1 -- 172.21.15.28:0/497845528 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16861 (secure 0 0 0) 0x7ffa2006e940 con 0x7ffa200d2960 2024-09-10T20:42:44.603 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:44.603 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:41.354414+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:21.256150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:22.565350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/2239499017":"2024-09-11T20:42:38.336838+0000","172.21.15.28:6832/2239499017":"2024-09-11T20:42:38.336838+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa04078090 msgr2=0x7ffa0407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa04078090 0x7ffa0407a550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c034000 tx=0x7ffa1c004730 comp rx=0 tx=0).stop 2024-09-10T20:42:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa200d2960 msgr2=0x7ffa200c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa200d2960 0x7ffa200c5580 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000e9a0 tx=0x7ffa1000ee60 comp rx=0 tx=0).stop 2024-09-10T20:42:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 shutdown_connections 2024-09-10T20:42:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa04078090 0x7ffa0407a550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa2015c050 0x7ffa200c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa200d2960 0x7ffa200c5580 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 --2- 172.21.15.28:0/497845528 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa200cd9b0 0x7ffa201594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 >> 172.21.15.28:0/497845528 conn(0x7ffa200d1e40 msgr2=0x7ffa20071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 shutdown_connections 2024-09-10T20:42:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.601+0000 7ffa277b5640 1 -- 172.21.15.28:0/497845528 wait complete. 2024-09-10T20:42:44.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 -- 172.21.15.28:0/3332495365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40d05d0 msgr2=0x7faec40b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 --2- 172.21.15.28:0/3332495365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40d05d0 0x7faec40b2e80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7faebc009be0 tx=0x7faebc02a8d0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 -- 172.21.15.28:0/3332495365 shutdown_connections 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 --2- 172.21.15.28:0/3332495365 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec40cc570 0x7faec40d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 --2- 172.21.15.28:0/3332495365 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40d05d0 0x7faec40b2e80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 -- 172.21.15.28:0/3332495365 >> 172.21.15.28:0/3332495365 conn(0x7faec406cf00 msgr2=0x7faec406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 -- 172.21.15.28:0/3332495365 shutdown_connections 2024-09-10T20:42:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.719+0000 7faeca56b640 1 -- 172.21.15.28:0/3332495365 wait complete. 2024-09-10T20:42:44.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.721+0000 7faeca56b640 1 Processor -- start 2024-09-10T20:42:44.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.721+0000 7faeca56b640 1 -- start start 2024-09-10T20:42:44.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.721+0000 7faeca56b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec40cc570 0x7faec4147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.721+0000 7faeca56b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40cd3d0 0x7faec40b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.721+0000 7faeca56b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec40d05d0 0x7faec41465f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faeca56b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faec40b33c0 con 0x7faec40d05d0 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faeca56b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faec40b4b60 con 0x7faec40cd3d0 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faeca56b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faec40b3540 con 0x7faec40cc570 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec40d05d0 0x7faec41465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec40d05d0 0x7faec41465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57420/0 (socket says 172.21.15.28:57420) 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 -- 172.21.15.28:0/3917770891 learned_addr learned my addr 172.21.15.28:0/3917770891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec3fff640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec40cc570 0x7faec4147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec37fe640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40cd3d0 0x7faec40b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 -- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec40cc570 msgr2=0x7faec4147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec40cc570 0x7faec4147280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 -- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40cd3d0 msgr2=0x7faec40b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40cd3d0 0x7faec40b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 -- 172.21.15.28:0/3917770891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec4157850 con 0x7faec40d05d0 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec8b08640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec40d05d0 0x7faec41465f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7faebc006010 tx=0x7faebc033f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec0ff9640 1 -- 172.21.15.28:0/3917770891 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc031d80 con 0x7faec40d05d0 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec0ff9640 1 -- 172.21.15.28:0/3917770891 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faebc003a60 con 0x7faec40d05d0 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec4157b30 con 0x7faec40d05d0 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.722+0000 7faec0ff9640 1 -- 172.21.15.28:0/3917770891 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc0330d0 con 0x7faec40d05d0 2024-09-10T20:42:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.723+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec41580f0 con 0x7faec40d05d0 2024-09-10T20:42:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.724+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec40b39d0 con 0x7faec40d05d0 2024-09-10T20:42:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.724+0000 7faec0ff9640 1 -- 172.21.15.28:0/3917770891 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7faebc04b030 con 0x7faec40d05d0 2024-09-10T20:42:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.725+0000 7faec0ff9640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faeb0078090 0x7faeb007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.725+0000 7faec3fff640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faeb0078090 0x7faeb007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.725+0000 7faec0ff9640 1 -- 172.21.15.28:0/3917770891 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5795+0+0 (secure 0 0 0) 0x7faebc0bae30 con 0x7faec40d05d0 2024-09-10T20:42:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.725+0000 7faec3fff640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faeb0078090 0x7faeb007a550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faeb40017c0 tx=0x7faeb4007e90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:44.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.729+0000 7faec0ff9640 1 -- 172.21.15.28:0/3917770891 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faebc03f030 con 0x7faec40d05d0 2024-09-10T20:42:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.949+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faec406ad20 con 0x7faec40d05d0 2024-09-10T20:42:44.954 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:42:44.950+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:42:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.952+0000 7faec0ff9640 1 -- 172.21.15.28:0/3917770891 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v59) ==== 161+0+0 (secure 0 0 0) 0x7faebc031790 con 0x7faec40d05d0 2024-09-10T20:42:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:42:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faeb0078090 msgr2=0x7faeb007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faeb0078090 0x7faeb007a550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faeb40017c0 tx=0x7faeb4007e90 comp rx=0 tx=0).stop 2024-09-10T20:42:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec40d05d0 msgr2=0x7faec41465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec40d05d0 0x7faec41465f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7faebc006010 tx=0x7faebc033f30 comp rx=0 tx=0).stop 2024-09-10T20:42:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 shutdown_connections 2024-09-10T20:42:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faeb0078090 0x7faeb007a550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec40d05d0 0x7faec41465f0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faec40cd3d0 0x7faec40b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 --2- 172.21.15.28:0/3917770891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faec40cc570 0x7faec4147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 >> 172.21.15.28:0/3917770891 conn(0x7faec406cf00 msgr2=0x7faec4070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 shutdown_connections 2024-09-10T20:42:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:44.955+0000 7faeca56b640 1 -- 172.21.15.28:0/3917770891 wait complete. 2024-09-10T20:42:44.960 INFO:tasks.ceph.mds.a.smithi028.stderr: -2567> 2024-09-10T20:42:29.570+0000 7fbeb250c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:42:44.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:42:44.989 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:42:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.077+0000 7f2ad9a23640 1 -- 172.21.15.28:0/2055199960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 msgr2=0x7f2ad40d0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.077+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/2055199960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40d0ba0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f2ad40d0080 tx=0x7f2abc02a5c0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.077+0000 7f2ad9a23640 1 -- 172.21.15.28:0/2055199960 shutdown_connections 2024-09-10T20:42:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.077+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/2055199960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40d0ba0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.077+0000 7f2ad9a23640 1 -- 172.21.15.28:0/2055199960 >> 172.21.15.28:0/2055199960 conn(0x7f2ad40d00b0 msgr2=0x7f2ad40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.077+0000 7f2ad9a23640 1 -- 172.21.15.28:0/2055199960 shutdown_connections 2024-09-10T20:42:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.077+0000 7f2ad9a23640 1 -- 172.21.15.28:0/2055199960 wait complete. 2024-09-10T20:42:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.078+0000 7f2ad9a23640 1 Processor -- start 2024-09-10T20:42:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.078+0000 7f2ad9a23640 1 -- start start 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad9a23640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad40706a0 0x7f2ad415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad9a23640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad40ca970 0x7f2ad40ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad9a23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad9a23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4166150 con 0x7f2ad40cad50 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad9a23640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4073db0 con 0x7f2ad40ca970 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad9a23640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad41662d0 con 0x7f2ad40706a0 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40c5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57464/0 (socket says 172.21.15.28:57464) 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad37fe640 1 -- 172.21.15.28:0/1484186749 learned_addr learned my addr 172.21.15.28:0/1484186749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad27fc640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad40ca970 0x7f2ad40ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.079+0000 7f2ad2ffd640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad40706a0 0x7f2ad415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad37fe640 1 -- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad40706a0 msgr2=0x7f2ad415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad37fe640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad40706a0 0x7f2ad415ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad37fe640 1 -- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad40ca970 msgr2=0x7f2ad40ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad37fe640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad40ca970 0x7f2ad40ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad37fe640 1 -- 172.21.15.28:0/1484186749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad40b5090 con 0x7f2ad40cad50 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad37fe640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40c5fa0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c400 tx=0x7f2acc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2acbfff640 1 -- 172.21.15.28:0/1484186749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc011790 con 0x7f2ad40cad50 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2acbfff640 1 -- 172.21.15.28:0/1484186749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2acc017070 con 0x7f2ad40cad50 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad40b5370 con 0x7f2ad40cad50 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2acbfff640 1 -- 172.21.15.28:0/1484186749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc011ce0 con 0x7f2ad40cad50 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.080+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad4152590 con 0x7f2ad40cad50 2024-09-10T20:42:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.081+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad40d0ba0 con 0x7f2ad40cad50 2024-09-10T20:42:45.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.082+0000 7f2acbfff640 1 -- 172.21.15.28:0/1484186749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f2acc01c460 con 0x7f2ad40cad50 2024-09-10T20:42:45.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.082+0000 7f2acbfff640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2ab4078090 0x7f2ab407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.082+0000 7f2acbfff640 1 -- 172.21.15.28:0/1484186749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5881+0+0 (secure 0 0 0) 0x7f2acc099d90 con 0x7f2ad40cad50 2024-09-10T20:42:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.082+0000 7f2ad2ffd640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2ab4078090 0x7f2ab407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.083+0000 7f2ad2ffd640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2ab4078090 0x7f2ab407a550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4001ec0 tx=0x7f2ac4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:45.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.087+0000 7f2acbfff640 1 -- 172.21.15.28:0/1484186749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2acc061ed0 con 0x7f2ad40cad50 2024-09-10T20:42:45.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.306+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2ad415b3d0 con 0x7f2ad40cad50 2024-09-10T20:42:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.953+0000 7f2acbfff640 1 -- 172.21.15.28:0/1484186749 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v60) ==== 104+0+0 (secure 0 0 0) 0x7f2acc066900 con 0x7f2ad40cad50 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.956+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2ab4078090 msgr2=0x7f2ab407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.956+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2ab4078090 0x7f2ab407a550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4001ec0 tx=0x7f2ac4009040 comp rx=0 tx=0).stop 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 msgr2=0x7f2ad40c5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40c5fa0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c400 tx=0x7f2acc00c8c0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 shutdown_connections 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2ab4078090 0x7f2ab407a550 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ad40cad50 0x7f2ad40c5fa0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2ad40ca970 0x7f2ad40ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 --2- 172.21.15.28:0/1484186749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2ad40706a0 0x7f2ad415ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 >> 172.21.15.28:0/1484186749 conn(0x7f2ad40d00b0 msgr2=0x7f2ad40cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 shutdown_connections 2024-09-10T20:42:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:45.957+0000 7f2ad9a23640 1 -- 172.21.15.28:0/1484186749 wait complete. 2024-09-10T20:42:45.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:42:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.131+0000 7f337447f640 1 -- 172.21.15.28:0/659990899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3358004800 msgr2=0x7f3358004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.131+0000 7f337447f640 1 --2- 172.21.15.28:0/659990899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3358004800 0x7f3358004c80 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f335c006fc0 tx=0x7f3358005830 comp rx=0 tx=0).stop 2024-09-10T20:42:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.131+0000 7f337447f640 1 -- 172.21.15.28:0/659990899 shutdown_connections 2024-09-10T20:42:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.131+0000 7f337447f640 1 --2- 172.21.15.28:0/659990899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3358004800 0x7f3358004c80 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.131+0000 7f337447f640 1 -- 172.21.15.28:0/659990899 >> 172.21.15.28:0/659990899 conn(0x7f336c0b2920 msgr2=0x7f336c0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.131+0000 7f337447f640 1 -- 172.21.15.28:0/659990899 shutdown_connections 2024-09-10T20:42:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.131+0000 7f337447f640 1 -- 172.21.15.28:0/659990899 wait complete. 2024-09-10T20:42:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.132+0000 7f337447f640 1 Processor -- start 2024-09-10T20:42:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.132+0000 7f337447f640 1 -- start start 2024-09-10T20:42:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f337447f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3358004800 0x7f336c143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f337447f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f336c157300 0x7f336c1f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f337447f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336c157780 0x7f336c1f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f337447f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f336c15aee0 con 0x7f336c157780 2024-09-10T20:42:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f337447f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f336c071aa0 con 0x7f336c157300 2024-09-10T20:42:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f337447f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f336c15b060 con 0x7f3358004800 2024-09-10T20:42:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336c157780 0x7f336c1f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336c157780 0x7f336c1f6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57476/0 (socket says 172.21.15.28:57476) 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 -- 172.21.15.28:0/277195714 learned_addr learned my addr 172.21.15.28:0/277195714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f337221b640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3358004800 0x7f336c143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3371a1a640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f336c157300 0x7f336c1f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 -- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3358004800 msgr2=0x7f336c143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3358004800 0x7f336c143e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 -- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f336c157300 msgr2=0x7f336c1f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f336c157300 0x7f336c1f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.133+0000 7f3372a1c640 1 -- 172.21.15.28:0/277195714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336c157cc0 con 0x7f336c157780 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.134+0000 7f3372a1c640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336c157780 0x7f336c1f6080 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3364005fd0 tx=0x7f3364011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.134+0000 7f3362ffd640 1 -- 172.21.15.28:0/277195714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364004b20 con 0x7f336c157780 2024-09-10T20:42:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.134+0000 7f3362ffd640 1 -- 172.21.15.28:0/277195714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3364004cc0 con 0x7f336c157780 2024-09-10T20:42:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.134+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336c157fa0 con 0x7f336c157780 2024-09-10T20:42:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.134+0000 7f3362ffd640 1 -- 172.21.15.28:0/277195714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364003bd0 con 0x7f336c157780 2024-09-10T20:42:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.134+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f336c158560 con 0x7f336c157780 2024-09-10T20:42:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.135+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f336c158d40 con 0x7f336c157780 2024-09-10T20:42:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.136+0000 7f3362ffd640 1 -- 172.21.15.28:0/277195714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f336402a080 con 0x7f336c157780 2024-09-10T20:42:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.136+0000 7f3362ffd640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3350078090 0x7f335007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.136+0000 7f3362ffd640 1 -- 172.21.15.28:0/277195714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5881+0+0 (secure 0 0 0) 0x7f336409b250 con 0x7f336c157780 2024-09-10T20:42:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.136+0000 7f337221b640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3350078090 0x7f335007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.137+0000 7f337221b640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3350078090 0x7f335007a550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3368006680 tx=0x7f3368006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:46.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.140+0000 7f3362ffd640 1 -- 172.21.15.28:0/277195714 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f336c158d40 con 0x7f336c157780 2024-09-10T20:42:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.330+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f336c0739f0 con 0x7f336c157780 2024-09-10T20:42:46.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.957+0000 7f3362ffd640 1 -- 172.21.15.28:0/277195714 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) ==== 181+0+0 (secure 0 0 0) 0x7f3364063410 con 0x7f336c157780 2024-09-10T20:42:46.963 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:42:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3350078090 msgr2=0x7f335007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3350078090 0x7f335007a550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3368006680 tx=0x7f3368006b70 comp rx=0 tx=0).stop 2024-09-10T20:42:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336c157780 msgr2=0x7f336c1f6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336c157780 0x7f336c1f6080 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3364005fd0 tx=0x7f3364011f80 comp rx=0 tx=0).stop 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 shutdown_connections 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3350078090 0x7f335007a550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336c157780 0x7f336c1f6080 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f336c157300 0x7f336c1f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 --2- 172.21.15.28:0/277195714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3358004800 0x7f336c143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 >> 172.21.15.28:0/277195714 conn(0x7f336c0b2920 msgr2=0x7f336c15b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.961+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 shutdown_connections 2024-09-10T20:42:46.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:46.962+0000 7f337447f640 1 -- 172.21.15.28:0/277195714 wait complete. 2024-09-10T20:42:46.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:42:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.091+0000 7fedf21d3640 1 -- 172.21.15.28:0/4192219501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 msgr2=0x7fedd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.091+0000 7fedf21d3640 1 --2- 172.21.15.28:0/4192219501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 0x7fedd4004c80 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fede000bb20 tx=0x7fedd4005830 comp rx=0 tx=0).stop 2024-09-10T20:42:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.092+0000 7fedebfff640 1 -- 172.21.15.28:0/4192219501 reap_dead start 2024-09-10T20:42:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.092+0000 7fedf21d3640 1 -- 172.21.15.28:0/4192219501 shutdown_connections 2024-09-10T20:42:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.092+0000 7fedf21d3640 1 -- 172.21.15.28:0/4192219501 >> 172.21.15.28:0/4192219501 conn(0x7fedec0cfc80 msgr2=0x7fedec0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.092+0000 7fedf21d3640 1 -- 172.21.15.28:0/4192219501 shutdown_connections 2024-09-10T20:42:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.092+0000 7fedf21d3640 1 -- 172.21.15.28:0/4192219501 wait complete. 2024-09-10T20:42:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.093+0000 7fedf21d3640 1 Processor -- start 2024-09-10T20:42:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.093+0000 7fedf21d3640 1 -- start start 2024-09-10T20:42:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedf21d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 0x7fedec0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedf21d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedec14ecc0 0x7fedec0b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedf21d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedec0b0920 0x7fedec0b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedf21d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedec0ce130 con 0x7fedd4004800 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedf21d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fedec0cdfb0 con 0x7fedec0b0920 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedf21d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fedec0ce2b0 con 0x7fedec14ecc0 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 0x7fedec0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 0x7fedec0b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57490/0 (socket says 172.21.15.28:57490) 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 -- 172.21.15.28:0/1022565508 learned_addr learned my addr 172.21.15.28:0/1022565508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeaffd640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedec14ecc0 0x7fedec0b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedebfff640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedec0b0920 0x7fedec0b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 -- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedec14ecc0 msgr2=0x7fedec0b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedec14ecc0 0x7fedec0b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 -- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedec0b0920 msgr2=0x7fedec0b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedec0b0920 0x7fedec0b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 -- 172.21.15.28:0/1022565508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedec0b3b60 con 0x7fedd4004800 2024-09-10T20:42:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedeb7fe640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 0x7fedec0b3580 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7feddc00e600 tx=0x7feddc00eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.094+0000 7fedd3fff640 1 -- 172.21.15.28:0/1022565508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc004220 con 0x7fedd4004800 2024-09-10T20:42:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.095+0000 7fedd3fff640 1 -- 172.21.15.28:0/1022565508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feddc015070 con 0x7fedd4004800 2024-09-10T20:42:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.095+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedec0b3e40 con 0x7fedd4004800 2024-09-10T20:42:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.095+0000 7fedd3fff640 1 -- 172.21.15.28:0/1022565508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc011b20 con 0x7fedd4004800 2024-09-10T20:42:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.095+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedec14f530 con 0x7fedd4004800 2024-09-10T20:42:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.095+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedec074860 con 0x7fedd4004800 2024-09-10T20:42:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.096+0000 7fedd3fff640 1 -- 172.21.15.28:0/1022565508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7feddc021030 con 0x7fedd4004800 2024-09-10T20:42:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.097+0000 7fedd3fff640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fedd8078090 0x7fedd807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.097+0000 7fedd3fff640 1 -- 172.21.15.28:0/1022565508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5443+0+0 (secure 0 0 0) 0x7feddc09bad0 con 0x7fedd4004800 2024-09-10T20:42:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.097+0000 7fedeaffd640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fedd8078090 0x7fedd807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.097+0000 7fedeaffd640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fedd8078090 0x7fedd807a550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fede0036000 tx=0x7fede003c040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.101+0000 7fedd3fff640 1 -- 172.21.15.28:0/1022565508 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feddc064540 con 0x7fedd4004800 2024-09-10T20:42:47.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.288+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fedec0ce7d0 con 0x7fedd4004800 2024-09-10T20:42:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.959+0000 7fedd3fff640 1 -- 172.21.15.28:0/1022565508 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) ==== 169+0+0 (secure 0 0 0) 0x7feddc063e40 con 0x7fedd4004800 2024-09-10T20:42:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:42:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fedd8078090 msgr2=0x7fedd807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fedd8078090 0x7fedd807a550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fede0036000 tx=0x7fede003c040 comp rx=0 tx=0).stop 2024-09-10T20:42:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 msgr2=0x7fedec0b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 0x7fedec0b3580 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7feddc00e600 tx=0x7feddc00eac0 comp rx=0 tx=0).stop 2024-09-10T20:42:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 shutdown_connections 2024-09-10T20:42:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fedd8078090 0x7fedd807a550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fedec0b0920 0x7fedec0b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fedec14ecc0 0x7fedec0b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 --2- 172.21.15.28:0/1022565508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedd4004800 0x7fedec0b3580 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 >> 172.21.15.28:0/1022565508 conn(0x7fedec0cfc80 msgr2=0x7fedec0725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 shutdown_connections 2024-09-10T20:42:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:47.963+0000 7fedf21d3640 1 -- 172.21.15.28:0/1022565508 wait complete. 2024-09-10T20:42:47.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:42:48.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 -- 172.21.15.28:0/3754872580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc30156130 msgr2=0x7fdc300caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3754872580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc30156130 0x7fdc300caa70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fdc300b64e0 tx=0x7fdc2802aa30 comp rx=0 tx=0).stop 2024-09-10T20:42:48.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 -- 172.21.15.28:0/3754872580 shutdown_connections 2024-09-10T20:42:48.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3754872580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc300af700 0x7fdc3006cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3754872580 >> v2:172.21.15.159:3300/0 conn(0x7fdc30156510 0x7fdc3006ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3754872580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc30156130 0x7fdc300caa70 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 -- 172.21.15.28:0/3754872580 >> 172.21.15.28:0/3754872580 conn(0x7fdc300d0420 msgr2=0x7fdc300d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 -- 172.21.15.28:0/3754872580 shutdown_connections 2024-09-10T20:42:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.091+0000 7fdc353c6640 1 -- 172.21.15.28:0/3754872580 wait complete. 2024-09-10T20:42:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.092+0000 7fdc353c6640 1 Processor -- start 2024-09-10T20:42:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.092+0000 7fdc353c6640 1 -- start start 2024-09-10T20:42:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc353c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc300af700 0x7fdc301f88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc353c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc30156130 0x7fdc30147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc353c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc30156510 0x7fdc30143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc353c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc300731f0 con 0x7fdc300af700 2024-09-10T20:42:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc353c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3006d3f0 con 0x7fdc30156130 2024-09-10T20:42:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc353c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdc30073370 con 0x7fdc30156510 2024-09-10T20:42:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc2effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc300af700 0x7fdc301f88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc2effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc300af700 0x7fdc301f88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57518/0 (socket says 172.21.15.28:57518) 2024-09-10T20:42:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc2effd640 1 -- 172.21.15.28:0/3332752258 learned_addr learned my addr 172.21.15.28:0/3332752258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc2f7fe640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc30156510 0x7fdc30143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.093+0000 7fdc2e7fc640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc30156130 0x7fdc30147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc2effd640 1 -- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc30156510 msgr2=0x7fdc30143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc2effd640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc30156510 0x7fdc30143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc2effd640 1 -- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc30156130 msgr2=0x7fdc30147810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc2effd640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc30156130 0x7fdc30147810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc2effd640 1 -- 172.21.15.28:0/3332752258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc301f8de0 con 0x7fdc300af700 2024-09-10T20:42:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc2e7fc640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc30156130 0x7fdc30147810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc2effd640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc300af700 0x7fdc301f88a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009d60 tx=0x7fdc28003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc277fe640 1 -- 172.21.15.28:0/3332752258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28046070 con 0x7fdc300af700 2024-09-10T20:42:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc277fe640 1 -- 172.21.15.28:0/3332752258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc28004250 con 0x7fdc300af700 2024-09-10T20:42:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc30144530 con 0x7fdc300af700 2024-09-10T20:42:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc277fe640 1 -- 172.21.15.28:0/3332752258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28033050 con 0x7fdc300af700 2024-09-10T20:42:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.094+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc301f80a0 con 0x7fdc300af700 2024-09-10T20:42:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.095+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc300caa70 con 0x7fdc300af700 2024-09-10T20:42:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.096+0000 7fdc277fe640 1 -- 172.21.15.28:0/3332752258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fdc28004660 con 0x7fdc300af700 2024-09-10T20:42:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.096+0000 7fdc277fe640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdc10078090 0x7fdc1007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.096+0000 7fdc277fe640 1 -- 172.21.15.28:0/3332752258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7fdc280bb060 con 0x7fdc300af700 2024-09-10T20:42:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.096+0000 7fdc2e7fc640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdc10078090 0x7fdc1007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.097+0000 7fdc2e7fc640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdc10078090 0x7fdc1007a550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc300cf580 tx=0x7fdc18008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.100+0000 7fdc277fe640 1 -- 172.21.15.28:0/3332752258 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdc28083510 con 0x7fdc300af700 2024-09-10T20:42:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.286+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fdc300d11c0 con 0x7fdc300af700 2024-09-10T20:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.290+0000 7fdc277fe640 1 -- 172.21.15.28:0/3332752258 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v37) ==== 71+0+0 (secure 0 0 0) 0x7fdc28087f40 con 0x7fdc300af700 2024-09-10T20:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdc10078090 msgr2=0x7fdc1007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdc10078090 0x7fdc1007a550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc300cf580 tx=0x7fdc18008040 comp rx=0 tx=0).stop 2024-09-10T20:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc300af700 msgr2=0x7fdc301f88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc300af700 0x7fdc301f88a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009d60 tx=0x7fdc28003f10 comp rx=0 tx=0).stop 2024-09-10T20:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 shutdown_connections 2024-09-10T20:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdc10078090 0x7fdc1007a550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdc30156510 0x7fdc30143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdc30156130 0x7fdc30147810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 --2- 172.21.15.28:0/3332752258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc300af700 0x7fdc301f88a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 >> 172.21.15.28:0/3332752258 conn(0x7fdc300d0420 msgr2=0x7fdc3015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 shutdown_connections 2024-09-10T20:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.292+0000 7fdc353c6640 1 -- 172.21.15.28:0/3332752258 wait complete. 2024-09-10T20:42:48.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-09-10T20:42:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 -- 172.21.15.28:0/2410641318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 msgr2=0x7f8a80070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 --2- 172.21.15.28:0/2410641318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a80070280 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8a800b4370 tx=0x7f8a7802aa30 comp rx=0 tx=0).stop 2024-09-10T20:42:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 -- 172.21.15.28:0/2410641318 shutdown_connections 2024-09-10T20:42:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 --2- 172.21.15.28:0/2410641318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a80070280 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 --2- 172.21.15.28:0/2410641318 >> v2:172.21.15.159:3300/0 conn(0x7f8a8015bc40 0x7f8a8015e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 -- 172.21.15.28:0/2410641318 >> 172.21.15.28:0/2410641318 conn(0x7f8a8006dbf0 msgr2=0x7f8a8006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 -- 172.21.15.28:0/2410641318 shutdown_connections 2024-09-10T20:42:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.409+0000 7f8a862e2640 1 -- 172.21.15.28:0/2410641318 wait complete. 2024-09-10T20:42:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.410+0000 7f8a862e2640 1 Processor -- start 2024-09-10T20:42:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.410+0000 7f8a862e2640 1 -- start start 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a862e2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a800d0c80 0x7f8a8014c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a862e2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8015bc40 0x7f8a8014c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a862e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a8009e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a862e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a80070940 con 0x7f8a8015ec20 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a862e2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8a800707c0 con 0x7f8a8015bc40 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a862e2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8a80070ac0 con 0x7f8a800d0c80 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a8487f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a8009e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a8487f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a8009e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57536/0 (socket says 172.21.15.28:57536) 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a8487f640 1 -- 172.21.15.28:0/3734918551 learned_addr learned my addr 172.21.15.28:0/3734918551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a7ffff640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a800d0c80 0x7f8a8014c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.411+0000 7f8a7f7fe640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8015bc40 0x7f8a8014c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a8487f640 1 -- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a800d0c80 msgr2=0x7f8a8014c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a8487f640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a800d0c80 0x7f8a8014c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a8487f640 1 -- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8015bc40 msgr2=0x7f8a8014c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a8487f640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8015bc40 0x7f8a8014c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a8487f640 1 -- 172.21.15.28:0/3734918551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a801f5e60 con 0x7f8a8015ec20 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a8487f640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a8009e5a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009d60 tx=0x7f8a78003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a7cff9640 1 -- 172.21.15.28:0/3734918551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78033050 con 0x7f8a8015ec20 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a7cff9640 1 -- 172.21.15.28:0/3734918551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a78004450 con 0x7f8a8015ec20 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a801f6140 con 0x7f8a8015ec20 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a7cff9640 1 -- 172.21.15.28:0/3734918551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78038450 con 0x7f8a8015ec20 2024-09-10T20:42:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.412+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a801f6700 con 0x7f8a8015ec20 2024-09-10T20:42:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.413+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a8015e6e0 con 0x7f8a8015ec20 2024-09-10T20:42:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.414+0000 7f8a7cff9640 1 -- 172.21.15.28:0/3734918551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f8a78040070 con 0x7f8a8015ec20 2024-09-10T20:42:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.414+0000 7f8a7cff9640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8a6c078090 0x7f8a6c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.414+0000 7f8a7cff9640 1 -- 172.21.15.28:0/3734918551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7f8a780bafc0 con 0x7f8a8015ec20 2024-09-10T20:42:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.414+0000 7f8a7ffff640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8a6c078090 0x7f8a6c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.415+0000 7f8a7ffff640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8a6c078090 0x7f8a6c07a550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a700017c0 tx=0x7f8a70009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.418+0000 7f8a7cff9640 1 -- 172.21.15.28:0/3734918551 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8a78083500 con 0x7f8a8015ec20 2024-09-10T20:42:48.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.678+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7f8a800d0a60 con 0x7f8a8015ec20 2024-09-10T20:42:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.960+0000 7f8a7cff9640 1 -- 172.21.15.28:0/3734918551 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v257) ==== 146+0+0 (secure 0 0 0) 0x7f8a78087f30 con 0x7f8a8015ec20 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8a6c078090 msgr2=0x7f8a6c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8a6c078090 0x7f8a6c07a550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a700017c0 tx=0x7f8a70009040 comp rx=0 tx=0).stop 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 msgr2=0x7f8a8009e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a8009e5a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009d60 tx=0x7f8a78003f10 comp rx=0 tx=0).stop 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 shutdown_connections 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8a6c078090 0x7f8a6c07a550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a8015ec20 0x7f8a8009e5a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a8015bc40 0x7f8a8014c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 --2- 172.21.15.28:0/3734918551 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a800d0c80 0x7f8a8014c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 >> 172.21.15.28:0/3734918551 conn(0x7f8a8006dbf0 msgr2=0x7f8a80070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 shutdown_connections 2024-09-10T20:42:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:48.963+0000 7f8a862e2640 1 -- 172.21.15.28:0/3734918551 wait complete. 2024-09-10T20:42:48.980 INFO:tasks.cephfs_test_runner:Test the creating vols, subvols subvolgroups fails when their names uses ... ok 2024-09-10T20:42:48.980 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-10T20:42:48.980 INFO:tasks.cephfs_test_runner:test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:42:48.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.086+0000 7f6340da1640 1 -- 172.21.15.28:0/1670750180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328003fd0 msgr2=0x7f6328004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.086+0000 7f6340da1640 1 --2- 172.21.15.28:0/1670750180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328003fd0 0x7f6328004800 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f632400ac90 tx=0x7f6328005830 comp rx=0 tx=0).stop 2024-09-10T20:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.086+0000 7f633bfff640 1 -- 172.21.15.28:0/1670750180 reap_dead start 2024-09-10T20:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.086+0000 7f6340da1640 1 -- 172.21.15.28:0/1670750180 shutdown_connections 2024-09-10T20:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.086+0000 7f6340da1640 1 -- 172.21.15.28:0/1670750180 >> 172.21.15.28:0/1670750180 conn(0x7f633c0cf990 msgr2=0x7f633c0cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.086+0000 7f6340da1640 1 -- 172.21.15.28:0/1670750180 shutdown_connections 2024-09-10T20:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.086+0000 7f6340da1640 1 -- 172.21.15.28:0/1670750180 wait complete. 2024-09-10T20:42:49.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.087+0000 7f6340da1640 1 Processor -- start 2024-09-10T20:42:49.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.087+0000 7f6340da1640 1 -- start start 2024-09-10T20:42:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f6340da1640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6328003fd0 0x7f633c150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f6340da1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f633c152c80 0x7f633c0c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f6340da1640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f633c0cc620 0x7f633c0c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f6340da1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f633c15ae70 con 0x7f633c152c80 2024-09-10T20:42:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f6340da1640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f633c15acf0 con 0x7f6328003fd0 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f6340da1640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f633c15aff0 con 0x7f633c0cc620 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f633c152c80 0x7f633c0c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f633c152c80 0x7f633c0c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57560/0 (socket says 172.21.15.28:57560) 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 -- 172.21.15.28:0/1620601158 learned_addr learned my addr 172.21.15.28:0/1620601158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633bfff640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f633c0cc620 0x7f633c0c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633b7fe640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6328003fd0 0x7f633c150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 -- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f633c0cc620 msgr2=0x7f633c0c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f633c0cc620 0x7f633c0c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 -- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6328003fd0 msgr2=0x7f633c150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6328003fd0 0x7f633c150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 -- 172.21.15.28:0/1620601158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633c153180 con 0x7f633c152c80 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.088+0000 7f633affd640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f633c152c80 0x7f633c0c8ca0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f632400ab80 tx=0x7f6324034e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.089+0000 7f6333fff640 1 -- 172.21.15.28:0/1620601158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324039040 con 0x7f633c152c80 2024-09-10T20:42:49.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.089+0000 7f6333fff640 1 -- 172.21.15.28:0/1620601158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f632402c9e0 con 0x7f633c152c80 2024-09-10T20:42:49.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.089+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f633c153460 con 0x7f633c152c80 2024-09-10T20:42:49.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.089+0000 7f6333fff640 1 -- 172.21.15.28:0/1620601158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632403f6b0 con 0x7f633c152c80 2024-09-10T20:42:49.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.089+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f633c153a20 con 0x7f633c152c80 2024-09-10T20:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.090+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f633c1e5d80 con 0x7f633c152c80 2024-09-10T20:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.090+0000 7f6333fff640 1 -- 172.21.15.28:0/1620601158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f632400b040 con 0x7f633c152c80 2024-09-10T20:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.091+0000 7f6333fff640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f631c078090 0x7f631c07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.091+0000 7f633b7fe640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f631c078090 0x7f631c07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.091+0000 7f6333fff640 1 -- 172.21.15.28:0/1620601158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7f63240ba730 con 0x7f633c152c80 2024-09-10T20:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.091+0000 7f633b7fe640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f631c078090 0x7f631c07a550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f632c004b40 tx=0x7f632c006040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:49.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.095+0000 7f6333fff640 1 -- 172.21.15.28:0/1620601158 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f633c1e5d80 con 0x7f633c152c80 2024-09-10T20:42:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.280+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f633c0743c0 con 0x7f633c152c80 2024-09-10T20:42:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.281+0000 7f6333fff640 1 -- 172.21.15.28:0/1620601158 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+13163 (secure 0 0 0) 0x7f632404d060 con 0x7f633c152c80 2024-09-10T20:42:49.286 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:49.286 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:47.959443+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/2216477518":"2024-09-11T20:42:44.950966+0000","172.21.15.28:6833/2216477518":"2024-09-11T20:42:44.950966+0000","172.21.15.28:6834/2239499017":"2024-09-11T20:42:38.336838+0000","172.21.15.28:6832/2239499017":"2024-09-11T20:42:38.336838+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:49.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f631c078090 msgr2=0x7f631c07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f631c078090 0x7f631c07a550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f632c004b40 tx=0x7f632c006040 comp rx=0 tx=0).stop 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f633c152c80 msgr2=0x7f633c0c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f633c152c80 0x7f633c0c8ca0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f632400ab80 tx=0x7f6324034e40 comp rx=0 tx=0).stop 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 shutdown_connections 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f631c078090 0x7f631c07a550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f633c0cc620 0x7f633c0c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f633c152c80 0x7f633c0c8ca0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 --2- 172.21.15.28:0/1620601158 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6328003fd0 0x7f633c150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 >> 172.21.15.28:0/1620601158 conn(0x7f633c0cf990 msgr2=0x7f633c0b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 shutdown_connections 2024-09-10T20:42:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.283+0000 7f6340da1640 1 -- 172.21.15.28:0/1620601158 wait complete. 2024-09-10T20:42:49.300 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:42:49.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:42:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.404+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1298816554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c015e220 msgr2=0x7fc0c00d1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.404+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1298816554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c015e220 0x7fc0c00d1f00 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8009830 tx=0x7fc0b802a5a0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.404+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1298816554 shutdown_connections 2024-09-10T20:42:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.404+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1298816554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c015e220 0x7fc0c00d1f00 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.404+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1298816554 >> 172.21.15.28:0/1298816554 conn(0x7fc0c00d4130 msgr2=0x7fc0c00d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.404+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1298816554 shutdown_connections 2024-09-10T20:42:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.404+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1298816554 wait complete. 2024-09-10T20:42:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.405+0000 7fc0c79c2640 1 Processor -- start 2024-09-10T20:42:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.405+0000 7fc0c79c2640 1 -- start start 2024-09-10T20:42:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c79c2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c00cf620 0x7fc0c01f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c79c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c00d2d70 0x7fc0c0155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c79c2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c015e220 0x7fc0c01f8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c79c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00b2310 con 0x7fc0c00d2d70 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c79c2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00b2190 con 0x7fc0c00cf620 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c79c2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c00b2490 con 0x7fc0c015e220 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c4f5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c00d2d70 0x7fc0c0155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c4f5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c00d2d70 0x7fc0c0155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57570/0 (socket says 172.21.15.28:57570) 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c4f5d640 1 -- 172.21.15.28:0/1396729483 learned_addr learned my addr 172.21.15.28:0/1396729483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.406+0000 7fc0c575e640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c00cf620 0x7fc0c01f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c5f5f640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c015e220 0x7fc0c01f8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c4f5d640 1 -- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c015e220 msgr2=0x7fc0c01f8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c4f5d640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c015e220 0x7fc0c01f8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c4f5d640 1 -- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c00cf620 msgr2=0x7fc0c01f54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c4f5d640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c00cf620 0x7fc0c01f54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c4f5d640 1 -- 172.21.15.28:0/1396729483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c00b5450 con 0x7fc0c00d2d70 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c575e640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c00cf620 0x7fc0c01f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c4f5d640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c00d2d70 0x7fc0c0155220 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c430 tx=0x7fc0b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0bdffb640 1 -- 172.21.15.28:0/1396729483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4011790 con 0x7fc0c00d2d70 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0bdffb640 1 -- 172.21.15.28:0/1396729483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0b4017070 con 0x7fc0c00d2d70 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c0155760 con 0x7fc0c00d2d70 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0bdffb640 1 -- 172.21.15.28:0/1396729483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4011d00 con 0x7fc0c00d2d70 2024-09-10T20:42:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.407+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c01f95e0 con 0x7fc0c00d2d70 2024-09-10T20:42:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.408+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c00b2980 con 0x7fc0c00d2d70 2024-09-10T20:42:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.409+0000 7fc0bdffb640 1 -- 172.21.15.28:0/1396729483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fc0b401c460 con 0x7fc0c00d2d70 2024-09-10T20:42:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.410+0000 7fc0bdffb640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0ac078090 0x7fc0ac07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.410+0000 7fc0c575e640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0ac078090 0x7fc0ac07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.410+0000 7fc0bdffb640 1 -- 172.21.15.28:0/1396729483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7fc0b4099c50 con 0x7fc0c00d2d70 2024-09-10T20:42:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.410+0000 7fc0c575e640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0ac078090 0x7fc0ac07a550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8034000 tx=0x7fc0b8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:49.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.414+0000 7fc0bdffb640 1 -- 172.21.15.28:0/1396729483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc0b40627c0 con 0x7fc0c00d2d70 2024-09-10T20:42:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.598+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0c0069580 con 0x7fc0c00d2d70 2024-09-10T20:42:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.599+0000 7fc0bdffb640 1 -- 172.21.15.28:0/1396729483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fc0b40620c0 con 0x7fc0c00d2d70 2024-09-10T20:42:49.604 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:42:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0ac078090 msgr2=0x7fc0ac07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0ac078090 0x7fc0ac07a550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8034000 tx=0x7fc0b8004730 comp rx=0 tx=0).stop 2024-09-10T20:42:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c00d2d70 msgr2=0x7fc0c0155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c00d2d70 0x7fc0c0155220 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c430 tx=0x7fc0b400c8f0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 shutdown_connections 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0ac078090 0x7fc0ac07a550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c015e220 0x7fc0c01f8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c00d2d70 0x7fc0c0155220 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 --2- 172.21.15.28:0/1396729483 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c00cf620 0x7fc0c01f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 >> 172.21.15.28:0/1396729483 conn(0x7fc0c00d4130 msgr2=0x7fc0c00b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 shutdown_connections 2024-09-10T20:42:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.601+0000 7fc0c79c2640 1 -- 172.21.15.28:0/1396729483 wait complete. 2024-09-10T20:42:49.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-09-10T20:42:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 -- 172.21.15.28:0/1354926750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c0d0c80 msgr2=0x7f3b1c15bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 --2- 172.21.15.28:0/1354926750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c0d0c80 0x7f3b1c15bc40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c005fd0 tx=0x7f3b0c02a530 comp rx=0 tx=0).stop 2024-09-10T20:42:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 -- 172.21.15.28:0/1354926750 shutdown_connections 2024-09-10T20:42:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 --2- 172.21.15.28:0/1354926750 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b1c070280 0x7f3b1c070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 --2- 172.21.15.28:0/1354926750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c0d0c80 0x7f3b1c15bc40 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 -- 172.21.15.28:0/1354926750 >> 172.21.15.28:0/1354926750 conn(0x7f3b1c06dbf0 msgr2=0x7f3b1c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 -- 172.21.15.28:0/1354926750 shutdown_connections 2024-09-10T20:42:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.717+0000 7f3b2089b640 1 -- 172.21.15.28:0/1354926750 wait complete. 2024-09-10T20:42:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.718+0000 7f3b2089b640 1 Processor -- start 2024-09-10T20:42:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.718+0000 7f3b2089b640 1 -- start start 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b2089b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c070280 0x7f3b1c145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b2089b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b1c0d0c80 0x7f3b1c06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b2089b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b1c0d1060 0x7f3b1c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b2089b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c15ef70 con 0x7f3b1c070280 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b2089b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c15edf0 con 0x7f3b1c0d1060 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b2089b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3b1c070bc0 con 0x7f3b1c0d0c80 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b1a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c070280 0x7f3b1c145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b1a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c070280 0x7f3b1c145320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57574/0 (socket says 172.21.15.28:57574) 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b1a59c640 1 -- 172.21.15.28:0/2089949540 learned_addr learned my addr 172.21.15.28:0/2089949540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b19d9b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b1c0d0c80 0x7f3b1c06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.719+0000 7f3b1ad9d640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b1c0d1060 0x7f3b1c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b1a59c640 1 -- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b1c0d0c80 msgr2=0x7f3b1c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b1a59c640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b1c0d0c80 0x7f3b1c06cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b1a59c640 1 -- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b1c0d1060 msgr2=0x7f3b1c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b1a59c640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b1c0d1060 0x7f3b1c06d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b1a59c640 1 -- 172.21.15.28:0/2089949540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c147450 con 0x7f3b1c070280 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b19d9b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b1c0d0c80 0x7f3b1c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b1ad9d640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b1c0d1060 0x7f3b1c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b1a59c640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c070280 0x7f3b1c145320 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009850 tx=0x7f3b0c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b12ffd640 1 -- 172.21.15.28:0/2089949540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c03f070 con 0x7f3b1c070280 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b12ffd640 1 -- 172.21.15.28:0/2089949540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b0c004890 con 0x7f3b1c070280 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c0cfc70 con 0x7f3b1c070280 2024-09-10T20:42:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b12ffd640 1 -- 172.21.15.28:0/2089949540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c035030 con 0x7f3b1c070280 2024-09-10T20:42:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.720+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c1f5090 con 0x7f3b1c070280 2024-09-10T20:42:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.721+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1c070680 con 0x7f3b1c070280 2024-09-10T20:42:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.722+0000 7f3b12ffd640 1 -- 172.21.15.28:0/2089949540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f3b0c004c60 con 0x7f3b1c070280 2024-09-10T20:42:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.722+0000 7f3b12ffd640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3afc078090 0x7f3afc07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:49.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.722+0000 7f3b12ffd640 1 -- 172.21.15.28:0/2089949540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7f3b0c0baf90 con 0x7f3b1c070280 2024-09-10T20:42:49.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.722+0000 7f3b19d9b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3afc078090 0x7f3afc07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:49.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.723+0000 7f3b19d9b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3afc078090 0x7f3afc07a550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b04006fd0 tx=0x7f3b04008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:49.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.727+0000 7f3b12ffd640 1 -- 172.21.15.28:0/2089949540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3b1c0cfc70 con 0x7f3b1c070280 2024-09-10T20:42:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:49.983+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7f3b1c0d0a60 con 0x7f3b1c070280 2024-09-10T20:42:50.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.967+0000 7f3b12ffd640 1 -- 172.21.15.28:0/2089949540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v259) ==== 152+0+0 (secure 0 0 0) 0x7f3b0c083b10 con 0x7f3b1c070280 2024-09-10T20:42:50.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3afc078090 msgr2=0x7f3afc07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3afc078090 0x7f3afc07a550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b04006fd0 tx=0x7f3b04008040 comp rx=0 tx=0).stop 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c070280 msgr2=0x7f3b1c145320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c070280 0x7f3b1c145320 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009850 tx=0x7f3b0c0039e0 comp rx=0 tx=0).stop 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 shutdown_connections 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3afc078090 0x7f3afc07a550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b1c0d1060 0x7f3b1c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b1c0d0c80 0x7f3b1c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 --2- 172.21.15.28:0/2089949540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b1c070280 0x7f3b1c145320 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.970+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 >> 172.21.15.28:0/2089949540 conn(0x7f3b1c06dbf0 msgr2=0x7f3b1c06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.971+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 shutdown_connections 2024-09-10T20:42:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:50.971+0000 7f3b2089b640 1 -- 172.21.15.28:0/2089949540 wait complete. 2024-09-10T20:42:50.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:42:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.096+0000 7f6dbb218640 1 -- 172.21.15.28:0/2563166867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 msgr2=0x7f6d9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.096+0000 7f6dbb218640 1 --2- 172.21.15.28:0/2563166867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 0x7f6d9c004c80 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6da800c2c0 tx=0x7f6d9c005830 comp rx=0 tx=0).stop 2024-09-10T20:42:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.096+0000 7f6db97b5640 1 -- 172.21.15.28:0/2563166867 reap_dead start 2024-09-10T20:42:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.096+0000 7f6dbb218640 1 -- 172.21.15.28:0/2563166867 shutdown_connections 2024-09-10T20:42:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.096+0000 7f6dbb218640 1 -- 172.21.15.28:0/2563166867 >> 172.21.15.28:0/2563166867 conn(0x7f6db40d3fb0 msgr2=0x7f6db40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.096+0000 7f6dbb218640 1 -- 172.21.15.28:0/2563166867 shutdown_connections 2024-09-10T20:42:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.096+0000 7f6dbb218640 1 -- 172.21.15.28:0/2563166867 wait complete. 2024-09-10T20:42:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.097+0000 7f6dbb218640 1 Processor -- start 2024-09-10T20:42:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.097+0000 7f6dbb218640 1 -- start start 2024-09-10T20:42:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6dbb218640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 0x7f6db41f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6dbb218640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6db41f4a70 0x7f6db40cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6dbb218640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6db4152650 0x7f6db40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6dbb218640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6db40cff80 con 0x7f6d9c004800 2024-09-10T20:42:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6dbb218640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6db40cfe00 con 0x7f6db4152650 2024-09-10T20:42:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6dbb218640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6db415e410 con 0x7f6db41f4a70 2024-09-10T20:42:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6db8fb4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 0x7f6db41f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6db8fb4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 0x7f6db41f8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49794/0 (socket says 172.21.15.28:49794) 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.098+0000 7f6db8fb4640 1 -- 172.21.15.28:0/1863926127 learned_addr learned my addr 172.21.15.28:0/1863926127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db3fff640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6db41f4a70 0x7f6db40cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db97b5640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6db4152650 0x7f6db40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db8fb4640 1 -- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6db41f4a70 msgr2=0x7f6db40cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db8fb4640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6db41f4a70 0x7f6db40cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db8fb4640 1 -- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6db4152650 msgr2=0x7f6db40cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db8fb4640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6db4152650 0x7f6db40cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db8fb4640 1 -- 172.21.15.28:0/1863926127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db41f4f30 con 0x7f6d9c004800 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db97b5640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6db4152650 0x7f6db40cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db8fb4640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 0x7f6db41f8a70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6dac011e70 tx=0x7f6dac00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db17fa640 1 -- 172.21.15.28:0/1863926127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac003970 con 0x7f6d9c004800 2024-09-10T20:42:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db17fa640 1 -- 172.21.15.28:0/1863926127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dac003b10 con 0x7f6d9c004800 2024-09-10T20:42:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6db17fa640 1 -- 172.21.15.28:0/1863926127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac004020 con 0x7f6d9c004800 2024-09-10T20:42:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db41f5210 con 0x7f6d9c004800 2024-09-10T20:42:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.099+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db41f57d0 con 0x7f6d9c004800 2024-09-10T20:42:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.100+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db406eb80 con 0x7f6d9c004800 2024-09-10T20:42:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.101+0000 7f6db17fa640 1 -- 172.21.15.28:0/1863926127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f6dac0041c0 con 0x7f6d9c004800 2024-09-10T20:42:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.102+0000 7f6db17fa640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6da00780c0 0x7f6da007a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.102+0000 7f6db3fff640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6da00780c0 0x7f6da007a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.102+0000 7f6db17fa640 1 -- 172.21.15.28:0/1863926127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7f6dac09ad00 con 0x7f6d9c004800 2024-09-10T20:42:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.102+0000 7f6db3fff640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6da00780c0 0x7f6da007a580 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6da8033000 tx=0x7f6da802d000 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.106+0000 7f6db17fa640 1 -- 172.21.15.28:0/1863926127 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6db406eb80 con 0x7f6d9c004800 2024-09-10T20:42:51.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.289+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f6db40d2b90 con 0x7f6d9c004800 2024-09-10T20:42:51.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.290+0000 7f6db17fa640 1 -- 172.21.15.28:0/1863926127 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v37) ==== 59+0+1998 (secure 0 0 0) 0x7f6dac063870 con 0x7f6d9c004800 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:42:51.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:42:51.297 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:42:51.297 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:42:51.297 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:42:51.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6da00780c0 msgr2=0x7f6da007a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6da00780c0 0x7f6da007a580 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6da8033000 tx=0x7f6da802d000 comp rx=0 tx=0).stop 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 msgr2=0x7f6db41f8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 0x7f6db41f8a70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6dac011e70 tx=0x7f6dac00f790 comp rx=0 tx=0).stop 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 shutdown_connections 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6da00780c0 0x7f6da007a580 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6db4152650 0x7f6db40cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6db41f4a70 0x7f6db40cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 --2- 172.21.15.28:0/1863926127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9c004800 0x7f6db41f8a70 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 >> 172.21.15.28:0/1863926127 conn(0x7f6db40d3fb0 msgr2=0x7f6db40b8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 shutdown_connections 2024-09-10T20:42:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.292+0000 7f6dbb218640 1 -- 172.21.15.28:0/1863926127 wait complete. 2024-09-10T20:42:51.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:42:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.411+0000 7f5351065640 1 -- 172.21.15.28:0/2182198616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334005910 msgr2=0x7f5334005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.411+0000 7f5351065640 1 --2- 172.21.15.28:0/2182198616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334005910 0x7f5334005d90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f5340001690 tx=0x7f5334006940 comp rx=0 tx=0).stop 2024-09-10T20:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.411+0000 7f5351065640 1 -- 172.21.15.28:0/2182198616 shutdown_connections 2024-09-10T20:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.411+0000 7f5351065640 1 --2- 172.21.15.28:0/2182198616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334005910 0x7f5334005d90 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.411+0000 7f5351065640 1 -- 172.21.15.28:0/2182198616 >> 172.21.15.28:0/2182198616 conn(0x7f534c06d110 msgr2=0x7f534c0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.411+0000 7f5351065640 1 -- 172.21.15.28:0/2182198616 shutdown_connections 2024-09-10T20:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.411+0000 7f5351065640 1 -- 172.21.15.28:0/2182198616 wait complete. 2024-09-10T20:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 Processor -- start 2024-09-10T20:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 -- start start 2024-09-10T20:42:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5334005910 0x7f534c155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f534c1445a0 0x7f534c159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534c144a00 0x7f534c159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f534c0b3330 con 0x7f534c144a00 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f534c0b31b0 con 0x7f5334005910 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f5351065640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f534c0b34b0 con 0x7f534c1445a0 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.413+0000 7f534b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534c144a00 0x7f534c159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534c144a00 0x7f534c159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49808/0 (socket says 172.21.15.28:49808) 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 -- 172.21.15.28:0/1723578240 learned_addr learned my addr 172.21.15.28:0/1723578240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534a59c640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f534c1445a0 0x7f534c159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534ad9d640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5334005910 0x7f534c155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 -- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f534c1445a0 msgr2=0x7f534c159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f534c1445a0 0x7f534c159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 -- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5334005910 msgr2=0x7f534c155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5334005910 0x7f534c155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 -- 172.21.15.28:0/1723578240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c151750 con 0x7f534c144a00 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f534b59e640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534c144a00 0x7f534c159760 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f5344005fd0 tx=0x7f534400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f53337fe640 1 -- 172.21.15.28:0/1723578240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344003d80 con 0x7f534c144a00 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f53337fe640 1 -- 172.21.15.28:0/1723578240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5344004b20 con 0x7f534c144a00 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c151a30 con 0x7f534c144a00 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f53337fe640 1 -- 172.21.15.28:0/1723578240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344004030 con 0x7f534c144a00 2024-09-10T20:42:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.414+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c151ff0 con 0x7f534c144a00 2024-09-10T20:42:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.415+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534c0746c0 con 0x7f534c144a00 2024-09-10T20:42:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.416+0000 7f53337fe640 1 -- 172.21.15.28:0/1723578240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f5344018030 con 0x7f534c144a00 2024-09-10T20:42:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.416+0000 7f53337fe640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5338078090 0x7f533807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.417+0000 7f534ad9d640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5338078090 0x7f533807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.417+0000 7f53337fe640 1 -- 172.21.15.28:0/1723578240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7f534409aa80 con 0x7f534c144a00 2024-09-10T20:42:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.417+0000 7f534ad9d640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5338078090 0x7f533807a550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f533c006cb0 tx=0x7f533c0067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:51.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.421+0000 7f53337fe640 1 -- 172.21.15.28:0/1723578240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5344063560 con 0x7f534c144a00 2024-09-10T20:42:51.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.607+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f534c157e10 con 0x7f534c144a00 2024-09-10T20:42:51.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.611+0000 7f53337fe640 1 -- 172.21.15.28:0/1723578240 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7f5344062e60 con 0x7f534c144a00 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5338078090 msgr2=0x7f533807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5338078090 0x7f533807a550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f533c006cb0 tx=0x7f533c0067a0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534c144a00 msgr2=0x7f534c159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534c144a00 0x7f534c159760 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f5344005fd0 tx=0x7f534400f790 comp rx=0 tx=0).stop 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 shutdown_connections 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5338078090 0x7f533807a550 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534c144a00 0x7f534c159760 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f534c1445a0 0x7f534c159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 --2- 172.21.15.28:0/1723578240 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5334005910 0x7f534c155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 >> 172.21.15.28:0/1723578240 conn(0x7f534c06d110 msgr2=0x7f534c1585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 shutdown_connections 2024-09-10T20:42:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.613+0000 7f5351065640 1 -- 172.21.15.28:0/1723578240 wait complete. 2024-09-10T20:42:51.629 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:42:51.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:51.666 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:42:51.709 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:42:51.709 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:51.709 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:51.725 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:42:51.725 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:51.725 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:51.769 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:42:51.769 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:42:51.769 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:51.784 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:42:51.785 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:42:51.785 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:42:51.827 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:42:51.827 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:42:51.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:42:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.929+0000 7ff302556640 1 -- 172.21.15.28:0/3358040814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2e4003f40 msgr2=0x7ff2e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.929+0000 7ff302556640 1 --2- 172.21.15.28:0/3358040814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2e4003f40 0x7ff2e4004800 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009b20 tx=0x7ff2e4005830 comp rx=0 tx=0).stop 2024-09-10T20:42:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.929+0000 7ff300af3640 1 -- 172.21.15.28:0/3358040814 reap_dead start 2024-09-10T20:42:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.929+0000 7ff302556640 1 -- 172.21.15.28:0/3358040814 shutdown_connections 2024-09-10T20:42:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.929+0000 7ff302556640 1 -- 172.21.15.28:0/3358040814 >> 172.21.15.28:0/3358040814 conn(0x7ff2fc0d0420 msgr2=0x7ff2fc0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.929+0000 7ff302556640 1 -- 172.21.15.28:0/3358040814 shutdown_connections 2024-09-10T20:42:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.929+0000 7ff302556640 1 -- 172.21.15.28:0/3358040814 wait complete. 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.930+0000 7ff302556640 1 Processor -- start 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.930+0000 7ff302556640 1 -- start start 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff302556640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2e4003f40 0x7ff2fc1f8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff302556640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2fc1f5540 0x7ff2fc14e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff302556640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2fc0b16f0 0x7ff2fc0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff302556640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc15f5b0 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff302556640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc15f430 con 0x7ff2fc1f5540 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff302556640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc15e7e0 con 0x7ff2e4003f40 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2fc0b16f0 0x7ff2fc0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2fc0b16f0 0x7ff2fc0c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49820/0 (socket says 172.21.15.28:49820) 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 -- 172.21.15.28:0/4165529186 learned_addr learned my addr 172.21.15.28:0/4165529186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff2fbfff640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2e4003f40 0x7ff2fc1f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff2fb7fe640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2fc1f5540 0x7ff2fc14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 -- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2e4003f40 msgr2=0x7ff2fc1f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2e4003f40 0x7ff2fc1f8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 -- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2fc1f5540 msgr2=0x7ff2fc14e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2fc1f5540 0x7ff2fc14e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.931+0000 7ff300af3640 1 -- 172.21.15.28:0/4165529186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc1f8d90 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.932+0000 7ff300af3640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2fc0b16f0 0x7ff2fc0c9770 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc0690c0 tx=0x7ff2f4001b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.932+0000 7ff2f8ff9640 1 -- 172.21.15.28:0/4165529186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4010890 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.932+0000 7ff2f8ff9640 1 -- 172.21.15.28:0/4165529186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2f4017070 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.932+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2fc0b1c70 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.932+0000 7ff2f8ff9640 1 -- 172.21.15.28:0/4165529186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4010de0 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.932+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2fc1f4d40 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.933+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2fc1f4850 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.933+0000 7ff2f8ff9640 1 -- 172.21.15.28:0/4165529186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7ff2f401c460 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.934+0000 7ff2f8ff9640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2e80780c0 0x7ff2e807a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:51.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.934+0000 7ff2f8ff9640 1 -- 172.21.15.28:0/4165529186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7ff2f409a4e0 con 0x7ff2fc0b16f0 2024-09-10T20:42:51.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.934+0000 7ff2fbfff640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2e80780c0 0x7ff2e807a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:51.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.934+0000 7ff2fbfff640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2e80780c0 0x7ff2e807a580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec000fd0 tx=0x7ff2ec006040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:51.938+0000 7ff2f8ff9640 1 -- 172.21.15.28:0/4165529186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff2fc1f8d90 con 0x7ff2fc0b16f0 2024-09-10T20:42:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.159+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2fc0738b0 con 0x7ff2fc0b16f0 2024-09-10T20:42:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.159+0000 7ff2f8ff9640 1 -- 172.21.15.28:0/4165529186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+3917 (secure 0 0 0) 0x7ff2f4062970 con 0x7ff2fc0b16f0 2024-09-10T20:42:52.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:52.164 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":60,"btime":"2024-09-10T20:42:45:952882+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2983894865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2983894865},{"type":"v1","addr":"172.21.15.159:6801","nonce":2983894865}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2105531171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2105531171},{"type":"v1","addr":"172.21.15.159:6835","nonce":2105531171}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1370248803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1370248803},{"type":"v1","addr":"172.21.15.28:6834","nonce":1370248803}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58},{"gid":6852,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3577035036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3577035036},{"type":"v1","addr":"172.21.15.28:6835","nonce":3577035036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[]} 2024-09-10T20:42:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 60 2024-09-10T20:42:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2e80780c0 msgr2=0x7ff2e807a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2e80780c0 0x7ff2e807a580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec000fd0 tx=0x7ff2ec006040 comp rx=0 tx=0).stop 2024-09-10T20:42:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2fc0b16f0 msgr2=0x7ff2fc0c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2fc0b16f0 0x7ff2fc0c9770 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc0690c0 tx=0x7ff2f4001b70 comp rx=0 tx=0).stop 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 shutdown_connections 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff2e80780c0 0x7ff2e807a580 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2fc0b16f0 0x7ff2fc0c9770 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2fc1f5540 0x7ff2fc14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 --2- 172.21.15.28:0/4165529186 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2e4003f40 0x7ff2fc1f8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 >> 172.21.15.28:0/4165529186 conn(0x7ff2fc0d0420 msgr2=0x7ff2fc151100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 shutdown_connections 2024-09-10T20:42:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.161+0000 7ff302556640 1 -- 172.21.15.28:0/4165529186 wait complete. 2024-09-10T20:42:52.177 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:42:52.178 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:42:52.178 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:42:52.178 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:52.178 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:42:52.178 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:42:52.178 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:42:52.178 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:52.179 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:42:52.179 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:42:52.179 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:42:52.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:52.179 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:42:52.179 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:42:52.179 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:42:52.180 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:42:52.190 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:42:52.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:42:52.191 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:42:52.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:42:52.192 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:42:52.192 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:42:52.193 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:42:52.193 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:42:52.232 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:42:52.232 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:52.233 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:42:52.233 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:52.234 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:42:52.235 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:52.235 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:42:52.235 DEBUG:teuthology.parallel:result is None 2024-09-10T20:42:52.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:42:52.310 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:42:52.310 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:42:52.317 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:42:52.318 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:42:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.341+0000 7fa48acc5640 1 -- 172.21.15.28:0/1085724038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4840d2b90 msgr2=0x7fa484161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.341+0000 7fa48acc5640 1 --2- 172.21.15.28:0/1085724038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4840d2b90 0x7fa484161b50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa470009830 tx=0x7fa47002a5c0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.342+0000 7fa48acc5640 1 -- 172.21.15.28:0/1085724038 shutdown_connections 2024-09-10T20:42:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.342+0000 7fa48acc5640 1 --2- 172.21.15.28:0/1085724038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4840d2b90 0x7fa484161b50 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.342+0000 7fa48acc5640 1 -- 172.21.15.28:0/1085724038 >> 172.21.15.28:0/1085724038 conn(0x7fa4840d3fb0 msgr2=0x7fa4840d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.342+0000 7fa48acc5640 1 -- 172.21.15.28:0/1085724038 shutdown_connections 2024-09-10T20:42:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.342+0000 7fa48acc5640 1 -- 172.21.15.28:0/1085724038 wait complete. 2024-09-10T20:42:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.343+0000 7fa48acc5640 1 Processor -- start 2024-09-10T20:42:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.343+0000 7fa48acc5640 1 -- start start 2024-09-10T20:42:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa48acc5640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa48415de70 0x7fa484155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa48acc5640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa484162090 0x7fa48415fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa48acc5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa48415e550 0x7fa48414c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa48acc5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa48415e3d0 con 0x7fa48415e550 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa48acc5640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa48415e250 con 0x7fa48415de70 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa48acc5640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa4840b7510 con 0x7fa484162090 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa48415e550 0x7fa48414c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa48415e550 0x7fa48414c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49888/0 (socket says 172.21.15.28:49888) 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 -- 172.21.15.28:0/2654770739 learned_addr learned my addr 172.21.15.28:0/2654770739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa488a61640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa48415de70 0x7fa484155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa483fff640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa484162090 0x7fa48415fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 -- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa484162090 msgr2=0x7fa48415fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa484162090 0x7fa48415fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 -- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa48415de70 msgr2=0x7fa484155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa48415de70 0x7fa484155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 -- 172.21.15.28:0/2654770739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48415eac0 con 0x7fa48415e550 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa488a61640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa48415de70 0x7fa484155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa483fff640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa484162090 0x7fa48415fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.344+0000 7fa489262640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa48415e550 0x7fa48414c340 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00c760 tx=0x7fa47c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.345+0000 7fa4817fa640 1 -- 172.21.15.28:0/2654770739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c003b70 con 0x7fa48415e550 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.345+0000 7fa4817fa640 1 -- 172.21.15.28:0/2654770739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa47c003d10 con 0x7fa48415e550 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.345+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa48415eda0 con 0x7fa48415e550 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.345+0000 7fa4817fa640 1 -- 172.21.15.28:0/2654770739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0052d0 con 0x7fa48415e550 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.345+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa48415f360 con 0x7fa48415e550 2024-09-10T20:42:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.346+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4840b79e0 con 0x7fa48415e550 2024-09-10T20:42:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.346+0000 7fa4817fa640 1 -- 172.21.15.28:0/2654770739 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fa47c005470 con 0x7fa48415e550 2024-09-10T20:42:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.347+0000 7fa4817fa640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa4680780c0 0x7fa46807a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.347+0000 7fa4817fa640 1 -- 172.21.15.28:0/2654770739 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5029+0+0 (secure 0 0 0) 0x7fa47c099b60 con 0x7fa48415e550 2024-09-10T20:42:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.347+0000 7fa488a61640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa4680780c0 0x7fa46807a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.348+0000 7fa488a61640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa4680780c0 0x7fa46807a580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa478007fd0 tx=0x7fa478009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.351+0000 7fa4817fa640 1 -- 172.21.15.28:0/2654770739 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa47c062640 con 0x7fa48415e550 2024-09-10T20:42:52.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.548+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa4840cf7c0 con 0x7fa48415e550 2024-09-10T20:42:52.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.978+0000 7fa4817fa640 1 -- 172.21.15.28:0/2654770739 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v94) ==== 97+0+0 (secure 0 0 0) 0x7fa47c061f40 con 0x7fa48415e550 2024-09-10T20:42:52.983 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa4680780c0 msgr2=0x7fa46807a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa4680780c0 0x7fa46807a580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa478007fd0 tx=0x7fa478009040 comp rx=0 tx=0).stop 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa48415e550 msgr2=0x7fa48414c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa48415e550 0x7fa48414c340 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00c760 tx=0x7fa47c00cb60 comp rx=0 tx=0).stop 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 shutdown_connections 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa4680780c0 0x7fa46807a580 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa48415e550 0x7fa48414c340 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa484162090 0x7fa48415fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 --2- 172.21.15.28:0/2654770739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa48415de70 0x7fa484155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 >> 172.21.15.28:0/2654770739 conn(0x7fa4840d3fb0 msgr2=0x7fa48406e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 shutdown_connections 2024-09-10T20:42:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:52.982+0000 7fa48acc5640 1 -- 172.21.15.28:0/2654770739 wait complete. 2024-09-10T20:42:53.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:42:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 -- 172.21.15.28:0/2708650075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c06d1f0 msgr2=0x7fbf3c06ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2708650075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c06d1f0 0x7fbf3c06ef30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fbf34005fd0 tx=0x7fbf3402aa00 comp rx=0 tx=0).stop 2024-09-10T20:42:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 -- 172.21.15.28:0/2708650075 shutdown_connections 2024-09-10T20:42:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2708650075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c06d1f0 0x7fbf3c06ef30 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2708650075 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf3c156130 0x7fbf3c0caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 -- 172.21.15.28:0/2708650075 >> 172.21.15.28:0/2708650075 conn(0x7fbf3c0d0420 msgr2=0x7fbf3c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 -- 172.21.15.28:0/2708650075 shutdown_connections 2024-09-10T20:42:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.102+0000 7fbf426b0640 1 -- 172.21.15.28:0/2708650075 wait complete. 2024-09-10T20:42:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.103+0000 7fbf426b0640 1 Processor -- start 2024-09-10T20:42:53.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.103+0000 7fbf426b0640 1 -- start start 2024-09-10T20:42:53.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf426b0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf3c06d1f0 0x7fbf3c0c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf426b0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf3c156130 0x7fbf3c0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf426b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c156510 0x7fbf3c0c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf426b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c0733b0 con 0x7fbf3c156510 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf426b0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c073230 con 0x7fbf3c06d1f0 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf426b0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3c073530 con 0x7fbf3c156130 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf40c4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c156510 0x7fbf3c0c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf40c4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c156510 0x7fbf3c0c9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49908/0 (socket says 172.21.15.28:49908) 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.104+0000 7fbf40c4d640 1 -- 172.21.15.28:0/2411120074 learned_addr learned my addr 172.21.15.28:0/2411120074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf40c4d640 1 -- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf3c156130 msgr2=0x7fbf3c0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf3b7fe640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf3c156130 0x7fbf3c0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf40c4d640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf3c156130 0x7fbf3c0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf3bfff640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf3c06d1f0 0x7fbf3c0c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf40c4d640 1 -- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf3c06d1f0 msgr2=0x7fbf3c0c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf40c4d640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf3c06d1f0 0x7fbf3c0c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf40c4d640 1 -- 172.21.15.28:0/2411120074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c0c6860 con 0x7fbf3c156510 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf3b7fe640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf3c156130 0x7fbf3c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf3bfff640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf3c06d1f0 0x7fbf3c0c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf40c4d640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c156510 0x7fbf3c0c9520 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf34003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf38ff9640 1 -- 172.21.15.28:0/2411120074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34047070 con 0x7fbf3c156510 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf38ff9640 1 -- 172.21.15.28:0/2411120074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf34004250 con 0x7fbf3c156510 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf3c0c6b40 con 0x7fbf3c156510 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf38ff9640 1 -- 172.21.15.28:0/2411120074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34033050 con 0x7fbf3c156510 2024-09-10T20:42:53.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.105+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf3c0c7100 con 0x7fbf3c156510 2024-09-10T20:42:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.106+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf3c06ef30 con 0x7fbf3c156510 2024-09-10T20:42:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.107+0000 7fbf38ff9640 1 -- 172.21.15.28:0/2411120074 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fbf340046a0 con 0x7fbf3c156510 2024-09-10T20:42:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.107+0000 7fbf38ff9640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf28078090 0x7fbf2807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.107+0000 7fbf38ff9640 1 -- 172.21.15.28:0/2411120074 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 4857+0+0 (secure 0 0 0) 0x7fbf340bb1f0 con 0x7fbf3c156510 2024-09-10T20:42:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.107+0000 7fbf3bfff640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf28078090 0x7fbf2807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.108+0000 7fbf3bfff640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf28078090 0x7fbf2807a550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0058c0 tx=0x7fbf2c005d30 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:53.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.111+0000 7fbf38ff9640 1 -- 172.21.15.28:0/2411120074 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbf34083e90 con 0x7fbf3c156510 2024-09-10T20:42:53.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.367+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fbf3c0d11c0 con 0x7fbf3c156510 2024-09-10T20:42:53.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.367+0000 7fbf38ff9640 1 -- 172.21.15.28:0/2411120074 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) ==== 80+0+5304 (secure 0 0 0) 0x7fbf34083790 con 0x7fbf3c156510 2024-09-10T20:42:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:53.372 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:42:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:42:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:42:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:42:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:42:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:42:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:42:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:42:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:42:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:42:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:42:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:42:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:42:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:42:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:42:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:42:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:42:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:42:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:42:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:42:53.390 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf28078090 msgr2=0x7fbf2807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf28078090 0x7fbf2807a550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0058c0 tx=0x7fbf2c005d30 comp rx=0 tx=0).stop 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c156510 msgr2=0x7fbf3c0c9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c156510 0x7fbf3c0c9520 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf34003f10 comp rx=0 tx=0).stop 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 shutdown_connections 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf28078090 0x7fbf2807a550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c156510 0x7fbf3c0c9520 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf3c156130 0x7fbf3c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 --2- 172.21.15.28:0/2411120074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf3c06d1f0 0x7fbf3c0c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 >> 172.21.15.28:0/2411120074 conn(0x7fbf3c0d0420 msgr2=0x7fbf3c15d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:53.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 shutdown_connections 2024-09-10T20:42:53.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.369+0000 7fbf426b0640 1 -- 172.21.15.28:0/2411120074 wait complete. 2024-09-10T20:42:53.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:42:53.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.494+0000 7f5cc41c6640 1 -- 172.21.15.28:0/4236649362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca8004800 msgr2=0x7f5ca8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.494+0000 7f5cc41c6640 1 --2- 172.21.15.28:0/4236649362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca8004800 0x7f5ca8004c80 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009b50 tx=0x7f5ca8005830 comp rx=0 tx=0).stop 2024-09-10T20:42:53.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.494+0000 7f5cc2763640 1 -- 172.21.15.28:0/4236649362 reap_dead start 2024-09-10T20:42:53.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.494+0000 7f5cc41c6640 1 -- 172.21.15.28:0/4236649362 shutdown_connections 2024-09-10T20:42:53.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.494+0000 7f5cc41c6640 1 -- 172.21.15.28:0/4236649362 >> 172.21.15.28:0/4236649362 conn(0x7f5cbc06cf00 msgr2=0x7f5cbc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:53.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.495+0000 7f5cc41c6640 1 -- 172.21.15.28:0/4236649362 shutdown_connections 2024-09-10T20:42:53.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.495+0000 7f5cc41c6640 1 -- 172.21.15.28:0/4236649362 wait complete. 2024-09-10T20:42:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.496+0000 7f5cc41c6640 1 Processor -- start 2024-09-10T20:42:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.496+0000 7f5cc41c6640 1 -- start start 2024-09-10T20:42:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.496+0000 7f5cc41c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ca8004800 0x7f5cbc1f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.496+0000 7f5cc41c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cbc0b0750 0x7f5cbc15d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc41c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cbc15de10 0x7f5cbc0cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc41c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc0d0090 con 0x7f5cbc0b0750 2024-09-10T20:42:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc41c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc0cd7b0 con 0x7f5ca8004800 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc41c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5cbc0d0210 con 0x7f5cbc15de10 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cbc0b0750 0x7f5cbc15d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cbc0b0750 0x7f5cbc15d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49922/0 (socket says 172.21.15.28:49922) 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 -- 172.21.15.28:0/1862538623 learned_addr learned my addr 172.21.15.28:0/1862538623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1f62640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ca8004800 0x7f5cbc1f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc2763640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cbc15de10 0x7f5cbc0cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 -- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cbc15de10 msgr2=0x7f5cbc0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cbc15de10 0x7f5cbc0cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 -- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ca8004800 msgr2=0x7f5cbc1f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ca8004800 0x7f5cbc1f7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 -- 172.21.15.28:0/1862538623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cbc1f8320 con 0x7f5cbc0b0750 2024-09-10T20:42:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc1761640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cbc0b0750 0x7f5cbc15d8d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009a70 tx=0x7f5cac033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cb27fc640 1 -- 172.21.15.28:0/1862538623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac043070 con 0x7f5cbc0b0750 2024-09-10T20:42:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cb27fc640 1 -- 172.21.15.28:0/1862538623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cac004230 con 0x7f5cbc0b0750 2024-09-10T20:42:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cb27fc640 1 -- 172.21.15.28:0/1862538623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac038040 con 0x7f5cbc0b0750 2024-09-10T20:42:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cbc1f8600 con 0x7f5cbc0b0750 2024-09-10T20:42:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.497+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cbc0ca6b0 con 0x7f5cbc0b0750 2024-09-10T20:42:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.498+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cbc0b2e80 con 0x7f5cbc0b0750 2024-09-10T20:42:53.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.499+0000 7f5cb27fc640 1 -- 172.21.15.28:0/1862538623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f5cac03b030 con 0x7f5cbc0b0750 2024-09-10T20:42:53.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.500+0000 7f5cb27fc640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ca0078090 0x7f5ca007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.500+0000 7f5cb27fc640 1 -- 172.21.15.28:0/1862538623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 4857+0+0 (secure 0 0 0) 0x7f5cac0b9db0 con 0x7f5cbc0b0750 2024-09-10T20:42:53.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.500+0000 7f5cc1f62640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ca0078090 0x7f5ca007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.500+0000 7f5cc1f62640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ca0078090 0x7f5ca007a550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8001010 tx=0x7f5cb8006040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:53.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.504+0000 7f5cb27fc640 1 -- 172.21.15.28:0/1862538623 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5cac082420 con 0x7f5cbc0b0750 2024-09-10T20:42:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.689+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5cbc15e1f0 con 0x7f5cbc0b0750 2024-09-10T20:42:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.690+0000 7f5cb27fc640 1 -- 172.21.15.28:0/1862538623 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) ==== 84+0+65 (secure 0 0 0) 0x7f5cac08b190 con 0x7f5cbc0b0750 2024-09-10T20:42:53.695 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:53.695 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:42:53.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:42:53.695 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:42:53.695 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:42:53.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:42:53.696 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:42:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ca0078090 msgr2=0x7f5ca007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ca0078090 0x7f5ca007a550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8001010 tx=0x7f5cb8006040 comp rx=0 tx=0).stop 2024-09-10T20:42:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cbc0b0750 msgr2=0x7f5cbc15d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cbc0b0750 0x7f5cbc15d8d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009a70 tx=0x7f5cac033670 comp rx=0 tx=0).stop 2024-09-10T20:42:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 shutdown_connections 2024-09-10T20:42:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ca0078090 0x7f5ca007a550 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cbc15de10 0x7f5cbc0cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cbc0b0750 0x7f5cbc15d8d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 --2- 172.21.15.28:0/1862538623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ca8004800 0x7f5cbc1f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 >> 172.21.15.28:0/1862538623 conn(0x7f5cbc06cf00 msgr2=0x7f5cbc0b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 shutdown_connections 2024-09-10T20:42:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.692+0000 7f5cc41c6640 1 -- 172.21.15.28:0/1862538623 wait complete. 2024-09-10T20:42:53.707 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:42:53.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:42:53.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 -- 172.21.15.28:0/2710299465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a815a350 msgr2=0x7fa5a80ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2710299465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a815a350 0x7fa5a80ca2d0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa5a806f7b0 tx=0x7fa59802a560 comp rx=0 tx=0).stop 2024-09-10T20:42:53.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 -- 172.21.15.28:0/2710299465 shutdown_connections 2024-09-10T20:42:53.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2710299465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a80cabc0 0x7fa5a806fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2710299465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a815a350 0x7fa5a80ca2d0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 -- 172.21.15.28:0/2710299465 >> 172.21.15.28:0/2710299465 conn(0x7fa5a80cfc80 msgr2=0x7fa5a80d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:53.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 -- 172.21.15.28:0/2710299465 shutdown_connections 2024-09-10T20:42:53.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.812+0000 7fa5aca95640 1 -- 172.21.15.28:0/2710299465 wait complete. 2024-09-10T20:42:53.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.813+0000 7fa5aca95640 1 Processor -- start 2024-09-10T20:42:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.813+0000 7fa5aca95640 1 -- start start 2024-09-10T20:42:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5aca95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a80cabc0 0x7fa5a80c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5aca95640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a815a350 0x7fa5a81f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5aca95640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a8151210 0x7fa5a80c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5aca95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a8072790 con 0x7fa5a80cabc0 2024-09-10T20:42:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5aca95640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a8072610 con 0x7fa5a8151210 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5aca95640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a8072910 con 0x7fa5a815a350 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a80cabc0 0x7fa5a80c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a80cabc0 0x7fa5a80c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49950/0 (socket says 172.21.15.28:49950) 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5a659c640 1 -- 172.21.15.28:0/2154901973 learned_addr learned my addr 172.21.15.28:0/2154901973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5a659c640 1 -- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a815a350 msgr2=0x7fa5a81f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5a5d9b640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a815a350 0x7fa5a81f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.814+0000 7fa5a6d9d640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a8151210 0x7fa5a80c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa5a659c640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a815a350 0x7fa5a81f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa5a659c640 1 -- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a8151210 msgr2=0x7fa5a80c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa5a659c640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a8151210 0x7fa5a80c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa5a659c640 1 -- 172.21.15.28:0/2154901973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a81f4240 con 0x7fa5a80cabc0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa5a659c640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a80cabc0 0x7fa5a80c8120 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa5980039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa596ffd640 1 -- 172.21.15.28:0/2154901973 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59803e030 con 0x7fa5a80cabc0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa596ffd640 1 -- 172.21.15.28:0/2154901973 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5980047a0 con 0x7fa5a80cabc0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a80c7080 con 0x7fa5a80cabc0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa596ffd640 1 -- 172.21.15.28:0/2154901973 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598035040 con 0x7fa5a80cabc0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.815+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a80c7640 con 0x7fa5a80cabc0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.816+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a8072e20 con 0x7fa5a80cabc0 2024-09-10T20:42:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.817+0000 7fa596ffd640 1 -- 172.21.15.28:0/2154901973 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fa598004b70 con 0x7fa5a80cabc0 2024-09-10T20:42:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.817+0000 7fa596ffd640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa588078090 0x7fa58807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.818+0000 7fa596ffd640 1 -- 172.21.15.28:0/2154901973 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 4857+0+0 (secure 0 0 0) 0x7fa5980ba7b0 con 0x7fa5a80cabc0 2024-09-10T20:42:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.818+0000 7fa5a5d9b640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa588078090 0x7fa58807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.818+0000 7fa5a5d9b640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa588078090 0x7fa58807a550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5a815d570 tx=0x7fa59c008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:53.822+0000 7fa596ffd640 1 -- 172.21.15.28:0/2154901973 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa5a81f4240 con 0x7fa5a80cabc0 2024-09-10T20:42:54.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.010+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa5a80ce7d0 con 0x7fa5a80cabc0 2024-09-10T20:42:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.988+0000 7fa596ffd640 1 -- 172.21.15.28:0/2154901973 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v97) ==== 138+0+0 (secure 0 0 0) 0x7fa598083340 con 0x7fa5a80cabc0 2024-09-10T20:42:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:42:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.991+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa588078090 msgr2=0x7fa58807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.991+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa588078090 0x7fa58807a550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5a815d570 tx=0x7fa59c008040 comp rx=0 tx=0).stop 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.991+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a80cabc0 msgr2=0x7fa5a80c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.991+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a80cabc0 0x7fa5a80c8120 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa5980039e0 comp rx=0 tx=0).stop 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.991+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 shutdown_connections 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.992+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa588078090 0x7fa58807a550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.992+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5a8151210 0x7fa5a80c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.992+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5a815a350 0x7fa5a81f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.992+0000 7fa5aca95640 1 --2- 172.21.15.28:0/2154901973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a80cabc0 0x7fa5a80c8120 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.992+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 >> 172.21.15.28:0/2154901973 conn(0x7fa5a80cfc80 msgr2=0x7fa5a806eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.992+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 shutdown_connections 2024-09-10T20:42:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:54.992+0000 7fa5aca95640 1 -- 172.21.15.28:0/2154901973 wait complete. 2024-09-10T20:42:55.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:42:55.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 -- 172.21.15.28:0/609572166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 msgr2=0x7fdfd80b46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:55.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/609572166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80b46e0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fdfd806dac0 tx=0x7fdfd002aa30 comp rx=0 tx=0).stop 2024-09-10T20:42:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 -- 172.21.15.28:0/609572166 shutdown_connections 2024-09-10T20:42:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/609572166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80b46e0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/609572166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfd80ad460 0x7fdfd806ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/609572166 >> v2:172.21.15.159:3300/0 conn(0x7fdfd80d00b0 0x7fdfd80d0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 -- 172.21.15.28:0/609572166 >> 172.21.15.28:0/609572166 conn(0x7fdfd806cf00 msgr2=0x7fdfd806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 -- 172.21.15.28:0/609572166 shutdown_connections 2024-09-10T20:42:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.127+0000 7fdfdd70e640 1 -- 172.21.15.28:0/609572166 wait complete. 2024-09-10T20:42:55.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.128+0000 7fdfdd70e640 1 Processor -- start 2024-09-10T20:42:55.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.128+0000 7fdfdd70e640 1 -- start start 2024-09-10T20:42:55.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfdd70e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfdd70e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfd80ad460 0x7fdfd815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfdd70e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfd80d00b0 0x7fdfd8157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfdd70e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd80adc50 con 0x7fdfd806f150 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfdd70e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd80adad0 con 0x7fdfd80d00b0 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfdd70e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd81555d0 con 0x7fdfd80ad460 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49978/0 (socket says 172.21.15.28:49978) 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2964265384 learned_addr learned my addr 172.21.15.28:0/2964265384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfce7fc640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfd80ad460 0x7fdfd815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.129+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfd80d00b0 0x7fdfd8157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfd80ad460 msgr2=0x7fdfd815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfd80ad460 0x7fdfd815a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfd80d00b0 msgr2=0x7fdfd8157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfd80d00b0 0x7fdfd8157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2964265384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd81f5cc0 con 0x7fdfd806f150 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfce7fc640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfd80ad460 0x7fdfd815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfd80d00b0 0x7fdfd8157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80cdc20 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800e790 tx=0x7fdfc800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfcffff640 1 -- 172.21.15.28:0/2964265384 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc800eea0 con 0x7fdfd806f150 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfcffff640 1 -- 172.21.15.28:0/2964265384 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfc800cdb0 con 0x7fdfd806f150 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd81f5fa0 con 0x7fdfd806f150 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfcffff640 1 -- 172.21.15.28:0/2964265384 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8010690 con 0x7fdfd806f150 2024-09-10T20:42:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.130+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd81f6560 con 0x7fdfd806f150 2024-09-10T20:42:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.131+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd80b4f90 con 0x7fdfd806f150 2024-09-10T20:42:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.132+0000 7fdfcffff640 1 -- 172.21.15.28:0/2964265384 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fdfc8010830 con 0x7fdfd806f150 2024-09-10T20:42:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.132+0000 7fdfcffff640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdfb8078090 0x7fdfb807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.132+0000 7fdfcffff640 1 -- 172.21.15.28:0/2964265384 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5227+0+0 (secure 0 0 0) 0x7fdfc809a840 con 0x7fdfd806f150 2024-09-10T20:42:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.132+0000 7fdfce7fc640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdfb8078090 0x7fdfb807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.133+0000 7fdfce7fc640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdfb8078090 0x7fdfb807a550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8154370 tx=0x7fdfc00094b0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:55.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.137+0000 7fdfcffff640 1 -- 172.21.15.28:0/2964265384 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdfc8063310 con 0x7fdfd806f150 2024-09-10T20:42:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.325+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fdfd815b140 con 0x7fdfd806f150 2024-09-10T20:42:55.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.990+0000 7fdfcffff640 1 -- 172.21.15.28:0/2964265384 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v98) ==== 170+0+0 (secure 0 0 0) 0x7fdfc8062c10 con 0x7fdfd806f150 2024-09-10T20:42:55.996 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:42:56.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdfb8078090 msgr2=0x7fdfb807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:56.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdfb8078090 0x7fdfb807a550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8154370 tx=0x7fdfc00094b0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 msgr2=0x7fdfd80cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:56.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80cdc20 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800e790 tx=0x7fdfc800ec50 comp rx=0 tx=0).stop 2024-09-10T20:42:56.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 shutdown_connections 2024-09-10T20:42:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdfb8078090 0x7fdfb807a550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfd80d00b0 0x7fdfd8157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfd80ad460 0x7fdfd815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 --2- 172.21.15.28:0/2964265384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd806f150 0x7fdfd80cdc20 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 >> 172.21.15.28:0/2964265384 conn(0x7fdfd806cf00 msgr2=0x7fdfd81543c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 shutdown_connections 2024-09-10T20:42:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:55.994+0000 7fdfdd70e640 1 -- 172.21.15.28:0/2964265384 wait complete. 2024-09-10T20:42:56.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:42:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 -- 172.21.15.28:0/4287616881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 msgr2=0x7f653c0b1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 --2- 172.21.15.28:0/4287616881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b1d70 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f6534005fd0 tx=0x7f653402a970 comp rx=0 tx=0).stop 2024-09-10T20:42:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 -- 172.21.15.28:0/4287616881 shutdown_connections 2024-09-10T20:42:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 --2- 172.21.15.28:0/4287616881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b1d70 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 --2- 172.21.15.28:0/4287616881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f653c0d27f0 0x7f653c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 -- 172.21.15.28:0/4287616881 >> 172.21.15.28:0/4287616881 conn(0x7f653c0d4070 msgr2=0x7f653c0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 -- 172.21.15.28:0/4287616881 shutdown_connections 2024-09-10T20:42:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.133+0000 7f6542fda640 1 -- 172.21.15.28:0/4287616881 wait complete. 2024-09-10T20:42:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.135+0000 7f6542fda640 1 Processor -- start 2024-09-10T20:42:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.135+0000 7f6542fda640 1 -- start start 2024-09-10T20:42:56.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6542fda640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f653c0d27f0 0x7f653c0b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:56.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6542fda640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f653c161770 0x7f653c0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6542fda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6542fda640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f653c0b2f50 con 0x7f653c161b50 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6542fda640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f653c0b2dd0 con 0x7f653c0d27f0 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6542fda640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f653c0b30d0 con 0x7f653c161770 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50006/0 (socket says 172.21.15.28:50006) 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 -- 172.21.15.28:0/1706062271 learned_addr learned my addr 172.21.15.28:0/1706062271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f653bfff640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f653c161770 0x7f653c0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6540d76640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f653c0d27f0 0x7f653c0b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 -- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f653c161770 msgr2=0x7f653c0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f653c161770 0x7f653c0c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 -- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f653c0d27f0 msgr2=0x7f653c0b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f653c0d27f0 0x7f653c0b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 -- 172.21.15.28:0/1706062271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c0c9c10 con 0x7f653c161b50 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f6541577640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b3320 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f6534006010 tx=0x7f6534003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.136+0000 7f65397fa640 1 -- 172.21.15.28:0/1706062271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6534047070 con 0x7f653c161b50 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.137+0000 7f65397fa640 1 -- 172.21.15.28:0/1706062271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65340045c0 con 0x7f653c161b50 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.137+0000 7f65397fa640 1 -- 172.21.15.28:0/1706062271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6534034070 con 0x7f653c161b50 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.137+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c1f82c0 con 0x7f653c161b50 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.137+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c0c9850 con 0x7f653c161b50 2024-09-10T20:42:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.138+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f653c0d20e0 con 0x7f653c161b50 2024-09-10T20:42:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.138+0000 7f65397fa640 1 -- 172.21.15.28:0/1706062271 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f6534041030 con 0x7f653c161b50 2024-09-10T20:42:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.139+0000 7f65397fa640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f65200780c0 0x7f652007a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.139+0000 7f65397fa640 1 -- 172.21.15.28:0/1706062271 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5609+0+0 (secure 0 0 0) 0x7f65340bae90 con 0x7f653c161b50 2024-09-10T20:42:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.139+0000 7f6540d76640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f65200780c0 0x7f652007a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.139+0000 7f6540d76640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f65200780c0 0x7f652007a580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6530004ea0 tx=0x7f6530009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.144+0000 7f65397fa640 1 -- 172.21.15.28:0/1706062271 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f653403f070 con 0x7f653c161b50 2024-09-10T20:42:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:56.369+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f653c06eb90 con 0x7f653c161b50 2024-09-10T20:42:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.013+0000 7f65397fa640 1 -- 172.21.15.28:0/1706062271 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v63) ==== 174+0+0 (secure 0 0 0) 0x7f6534083860 con 0x7f653c161b50 2024-09-10T20:42:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 16 and data pool 17 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.021+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f65200780c0 msgr2=0x7f652007a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.021+0000 7f6542fda640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f65200780c0 0x7f652007a580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6530004ea0 tx=0x7f6530009040 comp rx=0 tx=0).stop 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.021+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 msgr2=0x7f653c0b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.021+0000 7f6542fda640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b3320 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f6534006010 tx=0x7f6534003f90 comp rx=0 tx=0).stop 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.022+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 shutdown_connections 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.022+0000 7f6542fda640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f65200780c0 0x7f652007a580 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6530004ea0 tx=0x7f6530009040 comp rx=0 tx=0).stop 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.022+0000 7f6542fda640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c161b50 0x7f653c0b3320 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.022+0000 7f6542fda640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f653c161770 0x7f653c0c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.022+0000 7f6542fda640 1 --2- 172.21.15.28:0/1706062271 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f653c0d27f0 0x7f653c0b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.022+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 >> 172.21.15.28:0/1706062271 conn(0x7f653c0d4070 msgr2=0x7f653c0b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:57.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.024+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 shutdown_connections 2024-09-10T20:42:57.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.025+0000 7f6542fda640 1 -- 172.21.15.28:0/1706062271 wait complete. 2024-09-10T20:42:57.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 -- 172.21.15.28:0/1157323820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 msgr2=0x7fd5b80cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/1157323820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80cf0f0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009830 tx=0x7fd5a802a500 comp rx=0 tx=0).stop 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 -- 172.21.15.28:0/1157323820 shutdown_connections 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/1157323820 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b815eef0 0x7fd5b80cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/1157323820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80cf0f0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 -- 172.21.15.28:0/1157323820 >> 172.21.15.28:0/1157323820 conn(0x7fd5b80d0420 msgr2=0x7fd5b80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 -- 172.21.15.28:0/1157323820 shutdown_connections 2024-09-10T20:42:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.158+0000 7fd5bdd64640 1 -- 172.21.15.28:0/1157323820 wait complete. 2024-09-10T20:42:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.159+0000 7fd5bdd64640 1 Processor -- start 2024-09-10T20:42:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.159+0000 7fd5bdd64640 1 -- start start 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5bdd64640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b80738b0 0x7fd5b80b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5bdd64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5bdd64640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b815eef0 0x7fd5b80b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5bdd64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b8073e70 con 0x7fd5b8151cc0 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5bdd64640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b8073cf0 con 0x7fd5b80738b0 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5bdd64640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b8073ff0 con 0x7fd5b815eef0 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50014/0 (socket says 172.21.15.28:50014) 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b6ffd640 1 -- 172.21.15.28:0/728068726 learned_addr learned my addr 172.21.15.28:0/728068726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b77fe640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b80738b0 0x7fd5b80b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b7fff640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b815eef0 0x7fd5b80b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b6ffd640 1 -- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b815eef0 msgr2=0x7fd5b80b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b6ffd640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b815eef0 0x7fd5b80b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.160+0000 7fd5b6ffd640 1 -- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b80738b0 msgr2=0x7fd5b80b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd5b6ffd640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b80738b0 0x7fd5b80b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd5b6ffd640 1 -- 172.21.15.28:0/728068726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b81f7cd0 con 0x7fd5b8151cc0 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd5b7fff640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b815eef0 0x7fd5b80b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd5b6ffd640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80b1a60 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00c450 tx=0x7fd5ac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd59ffff640 1 -- 172.21.15.28:0/728068726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac016020 con 0x7fd5b8151cc0 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd59ffff640 1 -- 172.21.15.28:0/728068726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5ac012920 con 0x7fd5b8151cc0 2024-09-10T20:42:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b81f7fb0 con 0x7fd5b8151cc0 2024-09-10T20:42:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd59ffff640 1 -- 172.21.15.28:0/728068726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac012d70 con 0x7fd5b8151cc0 2024-09-10T20:42:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.161+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b81f8570 con 0x7fd5b8151cc0 2024-09-10T20:42:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.162+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b80cb9f0 con 0x7fd5b8151cc0 2024-09-10T20:42:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.163+0000 7fd59ffff640 1 -- 172.21.15.28:0/728068726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fd5ac01b400 con 0x7fd5b8151cc0 2024-09-10T20:42:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.163+0000 7fd59ffff640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd5a40780c0 0x7fd5a407a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.163+0000 7fd59ffff640 1 -- 172.21.15.28:0/728068726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fd5ac09a270 con 0x7fd5b8151cc0 2024-09-10T20:42:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.163+0000 7fd5b77fe640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd5a40780c0 0x7fd5a407a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.164+0000 7fd5b77fe640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd5a40780c0 0x7fd5a407a580 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8034000 tx=0x7fd5a8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.167+0000 7fd59ffff640 1 -- 172.21.15.28:0/728068726 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd5ac020020 con 0x7fd5b8151cc0 2024-09-10T20:42:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.356+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5b8068d10 con 0x7fd5b8151cc0 2024-09-10T20:42:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.357+0000 7fd59ffff640 1 -- 172.21.15.28:0/728068726 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+16745 (secure 0 0 0) 0x7fd5ac062b50 con 0x7fd5b8151cc0 2024-09-10T20:42:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:57.361 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:56.370992+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:54.016199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:55.330837+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd5a40780c0 msgr2=0x7fd5a407a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd5a40780c0 0x7fd5a407a580 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8034000 tx=0x7fd5a8004730 comp rx=0 tx=0).stop 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 msgr2=0x7fd5b80b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80b1a60 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00c450 tx=0x7fd5ac00c910 comp rx=0 tx=0).stop 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 shutdown_connections 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd5a40780c0 0x7fd5a407a580 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5b815eef0 0x7fd5b80b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b8151cc0 0x7fd5b80b1a60 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 --2- 172.21.15.28:0/728068726 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd5b80738b0 0x7fd5b80b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.359+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 >> 172.21.15.28:0/728068726 conn(0x7fd5b80d0420 msgr2=0x7fd5b80cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.360+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 shutdown_connections 2024-09-10T20:42:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.360+0000 7fd5bdd64640 1 -- 172.21.15.28:0/728068726 wait complete. 2024-09-10T20:42:57.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:42:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.477+0000 7f110430e640 1 -- 172.21.15.28:0/2632268425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10e80051c0 msgr2=0x7f10e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.477+0000 7f110430e640 1 --2- 172.21.15.28:0/2632268425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10e80051c0 0x7f10e800b9c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f10f400c640 tx=0x7f10e8005830 comp rx=0 tx=0).stop 2024-09-10T20:42:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.477+0000 7f110430e640 1 -- 172.21.15.28:0/2632268425 shutdown_connections 2024-09-10T20:42:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.477+0000 7f110430e640 1 --2- 172.21.15.28:0/2632268425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10e80051c0 0x7f10e800b9c0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.477+0000 7f110430e640 1 -- 172.21.15.28:0/2632268425 >> 172.21.15.28:0/2632268425 conn(0x7f10fc0d3fb0 msgr2=0x7f10fc0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.477+0000 7f110430e640 1 -- 172.21.15.28:0/2632268425 shutdown_connections 2024-09-10T20:42:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.477+0000 7f110430e640 1 -- 172.21.15.28:0/2632268425 wait complete. 2024-09-10T20:42:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.478+0000 7f110430e640 1 Processor -- start 2024-09-10T20:42:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.478+0000 7f110430e640 1 -- start start 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f110430e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10e80051c0 0x7f10fc15a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f110430e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10fc0a0430 0x7f10fc15e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f110430e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10fc0a0890 0x7f10fc0cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f110430e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc161b50 con 0x7f10fc0a0430 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f110430e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc15e460 con 0x7f10e80051c0 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f110430e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f10fc161cd0 con 0x7f10fc0a0890 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10fc0a0430 0x7f10fc15e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10fc0a0430 0x7f10fc15e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50024/0 (socket says 172.21.15.28:50024) 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 -- 172.21.15.28:0/2333671345 learned_addr learned my addr 172.21.15.28:0/2333671345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 -- 172.21.15.28:0/2333671345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10fc0a0890 msgr2=0x7f10fc0cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10fc0a0890 0x7f10fc0cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 -- 172.21.15.28:0/2333671345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10e80051c0 msgr2=0x7f10fc15a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10e80051c0 0x7f10fc15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.479+0000 7f11018a9640 1 -- 172.21.15.28:0/2333671345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10fc15f060 con 0x7f10fc0a0430 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.480+0000 7f11018a9640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10fc0a0430 0x7f10fc15e5e0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f10ec011ab0 tx=0x7f10ec011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.480+0000 7f10f2ffd640 1 -- 172.21.15.28:0/2333671345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec01e070 con 0x7f10fc0a0430 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.480+0000 7f10f2ffd640 1 -- 172.21.15.28:0/2333671345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10ec00fe70 con 0x7f10fc0a0430 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.480+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10fc15f340 con 0x7f10fc0a0430 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.480+0000 7f10f2ffd640 1 -- 172.21.15.28:0/2333671345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0042b0 con 0x7f10fc0a0430 2024-09-10T20:42:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.480+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10fc15fc00 con 0x7f10fc0a0430 2024-09-10T20:42:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.481+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10fc15f580 con 0x7f10fc0a0430 2024-09-10T20:42:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.481+0000 7f10f2ffd640 1 -- 172.21.15.28:0/2333671345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f10ec0035f0 con 0x7f10fc0a0430 2024-09-10T20:42:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.482+0000 7f10f2ffd640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10e0078090 0x7f10e007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.482+0000 7f10f2ffd640 1 -- 172.21.15.28:0/2333671345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7f10ec09ac30 con 0x7f10fc0a0430 2024-09-10T20:42:57.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.482+0000 7f11020aa640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10e0078090 0x7f10e007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.482+0000 7f11020aa640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10e0078090 0x7f10e007a550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10f8006850 tx=0x7f10f80067e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.486+0000 7f10f2ffd640 1 -- 172.21.15.28:0/2333671345 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f10fc15f580 con 0x7f10fc0a0430 2024-09-10T20:42:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.671+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10fc0cf860 con 0x7f10fc0a0430 2024-09-10T20:42:57.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.672+0000 7f10f2ffd640 1 -- 172.21.15.28:0/2333671345 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+16745 (secure 0 0 0) 0x7f10ec062f50 con 0x7f10fc0a0430 2024-09-10T20:42:57.676 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:42:57.676 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:56.370992+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:54.016199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:55.330837+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:42:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10e0078090 msgr2=0x7f10e007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10e0078090 0x7f10e007a550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10f8006850 tx=0x7f10f80067e0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10fc0a0430 msgr2=0x7f10fc15e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10fc0a0430 0x7f10fc15e5e0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f10ec011ab0 tx=0x7f10ec011f70 comp rx=0 tx=0).stop 2024-09-10T20:42:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 shutdown_connections 2024-09-10T20:42:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10e0078090 0x7f10e007a550 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10fc0a0890 0x7f10fc0cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10fc0a0430 0x7f10fc15e5e0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 --2- 172.21.15.28:0/2333671345 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f10e80051c0 0x7f10fc15a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 >> 172.21.15.28:0/2333671345 conn(0x7f10fc0d3fb0 msgr2=0x7f10fc075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 shutdown_connections 2024-09-10T20:42:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.674+0000 7f110430e640 1 -- 172.21.15.28:0/2333671345 wait complete. 2024-09-10T20:42:57.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:42:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 -- 172.21.15.28:0/174727598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 msgr2=0x7fd2540cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/174727598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2540cb830 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd254069560 tx=0x7fd24c02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 -- 172.21.15.28:0/174727598 shutdown_connections 2024-09-10T20:42:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/174727598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2540cb830 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/174727598 >> v2:172.21.15.159:3300/0 conn(0x7fd254152df0 0x7fd25415c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 -- 172.21.15.28:0/174727598 >> 172.21.15.28:0/174727598 conn(0x7fd2540d0d50 msgr2=0x7fd2540d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 -- 172.21.15.28:0/174727598 shutdown_connections 2024-09-10T20:42:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.794+0000 7fd25a7a2640 1 -- 172.21.15.28:0/174727598 wait complete. 2024-09-10T20:42:57.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.796+0000 7fd25a7a2640 1 Processor -- start 2024-09-10T20:42:57.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.796+0000 7fd25a7a2640 1 -- start start 2024-09-10T20:42:57.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd25a7a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2541f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd25a7a2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2540d1840 0x7fd2540c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd25a7a2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd254152df0 0x7fd2540b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd25a7a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd25406cae0 con 0x7fd2540cb3a0 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd25a7a2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd2541532c0 con 0x7fd2540d1840 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd25a7a2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd25406cc60 con 0x7fd254152df0 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2541f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2541f5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50040/0 (socket says 172.21.15.28:50040) 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 -- 172.21.15.28:0/392358898 learned_addr learned my addr 172.21.15.28:0/392358898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd258d3f640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd254152df0 0x7fd2540b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd2537fe640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2540d1840 0x7fd2540c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 -- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd254152df0 msgr2=0x7fd2540b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd254152df0 0x7fd2540b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 -- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2540d1840 msgr2=0x7fd2540c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2540d1840 0x7fd2540c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 -- 172.21.15.28:0/392358898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2540ca8f0 con 0x7fd2540cb3a0 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd253fff640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2541f5ac0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd24400d950 tx=0x7fd24400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd250ff9640 1 -- 172.21.15.28:0/392358898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd244017070 con 0x7fd2540cb3a0 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd250ff9640 1 -- 172.21.15.28:0/392358898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd244009d20 con 0x7fd2540cb3a0 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2540cabd0 con 0x7fd2540cb3a0 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.797+0000 7fd250ff9640 1 -- 172.21.15.28:0/392358898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2440106c0 con 0x7fd2540cb3a0 2024-09-10T20:42:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.798+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2540b3d60 con 0x7fd2540cb3a0 2024-09-10T20:42:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.798+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2540b3b80 con 0x7fd2540cb3a0 2024-09-10T20:42:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.799+0000 7fd250ff9640 1 -- 172.21.15.28:0/392358898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fd244010860 con 0x7fd2540cb3a0 2024-09-10T20:42:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.800+0000 7fd250ff9640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd2400780c0 0x7fd24007a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.800+0000 7fd250ff9640 1 -- 172.21.15.28:0/392358898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fd24409a260 con 0x7fd2540cb3a0 2024-09-10T20:42:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.800+0000 7fd2537fe640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd2400780c0 0x7fd24007a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:57.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.800+0000 7fd2537fe640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd2400780c0 0x7fd24007a580 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd25415f250 tx=0x7fd248007e90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:57.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:57.804+0000 7fd250ff9640 1 -- 172.21.15.28:0/392358898 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd2540b3b80 con 0x7fd2540cb3a0 2024-09-10T20:42:57.986 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-10T20:42:57.982+0000 7fa1ef19f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:42:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:58.029+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd254074870 con 0x7fd2540cb3a0 2024-09-10T20:42:59.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.025+0000 7fd250ff9640 1 -- 172.21.15.28:0/392358898 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) ==== 118+0+0 (secure 0 0 0) 0x7fd254074870 con 0x7fd2540cb3a0 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.028+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd2400780c0 msgr2=0x7fd24007a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.028+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd2400780c0 0x7fd24007a580 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd25415f250 tx=0x7fd248007e90 comp rx=0 tx=0).stop 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.028+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 msgr2=0x7fd2541f5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.028+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2541f5ac0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd24400d950 tx=0x7fd24400de10 comp rx=0 tx=0).stop 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 shutdown_connections 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd2400780c0 0x7fd24007a580 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd254152df0 0x7fd2540b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd2540d1840 0x7fd2540c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 --2- 172.21.15.28:0/392358898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2540cb3a0 0x7fd2541f5ac0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 >> 172.21.15.28:0/392358898 conn(0x7fd2540d0d50 msgr2=0x7fd25415e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 shutdown_connections 2024-09-10T20:42:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.029+0000 7fd25a7a2640 1 -- 172.21.15.28:0/392358898 wait complete. 2024-09-10T20:42:59.046 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:42:59.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:42:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.160+0000 7f51d5d17640 1 -- 172.21.15.28:0/2694431405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b8004d40 msgr2=0x7f51b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.160+0000 7f51d5d17640 1 --2- 172.21.15.28:0/2694431405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b8004d40 0x7f51b80051c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f51c8002230 tx=0x7f51b8005830 comp rx=0 tx=0).stop 2024-09-10T20:42:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.160+0000 7f51cffff640 1 -- 172.21.15.28:0/2694431405 reap_dead start 2024-09-10T20:42:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.160+0000 7f51d5d17640 1 -- 172.21.15.28:0/2694431405 shutdown_connections 2024-09-10T20:42:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.160+0000 7f51d5d17640 1 -- 172.21.15.28:0/2694431405 >> 172.21.15.28:0/2694431405 conn(0x7f51d00d1e60 msgr2=0x7f51d00d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:42:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.161+0000 7f51d5d17640 1 -- 172.21.15.28:0/2694431405 shutdown_connections 2024-09-10T20:42:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.161+0000 7f51d5d17640 1 -- 172.21.15.28:0/2694431405 wait complete. 2024-09-10T20:42:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.162+0000 7f51d5d17640 1 Processor -- start 2024-09-10T20:42:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.162+0000 7f51d5d17640 1 -- start start 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51d5d17640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51b8004d40 0x7f51d00b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51d5d17640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51d00c5990 0x7f51d00c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51d5d17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d01f2da0 0x7f51d01f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51d5d17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51d00b3410 con 0x7f51d01f2da0 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51d5d17640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f51d00b3290 con 0x7f51b8004d40 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51d5d17640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f51d00b3590 con 0x7f51d00c5990 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d01f2da0 0x7f51d01f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d01f2da0 0x7f51d01f3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50062/0 (socket says 172.21.15.28:50062) 2024-09-10T20:42:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51b8004d40 0x7f51d00b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 -- 172.21.15.28:0/850898352 learned_addr learned my addr 172.21.15.28:0/850898352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51ceffd640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51d00c5990 0x7f51d00c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 -- 172.21.15.28:0/850898352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51d00c5990 msgr2=0x7f51d00c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51d00c5990 0x7f51d00c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 -- 172.21.15.28:0/850898352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51b8004d40 msgr2=0x7f51d00b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51b8004d40 0x7f51d00b79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 -- 172.21.15.28:0/850898352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d00c78d0 con 0x7f51d01f2da0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.163+0000 7f51cffff640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d01f2da0 0x7f51d01f3230 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f51c802b4b0 tx=0x7f51c8001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.164+0000 7f51b7fff640 1 -- 172.21.15.28:0/850898352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c802bc10 con 0x7f51d01f2da0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.164+0000 7f51b7fff640 1 -- 172.21.15.28:0/850898352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51c80060a0 con 0x7f51d01f2da0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.164+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d00c7b70 con 0x7f51d01f2da0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.164+0000 7f51b7fff640 1 -- 172.21.15.28:0/850898352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c80064f0 con 0x7f51d01f2da0 2024-09-10T20:42:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.164+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d00c8130 con 0x7f51d01f2da0 2024-09-10T20:42:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.165+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d00cf140 con 0x7f51d01f2da0 2024-09-10T20:42:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.165+0000 7f51b7fff640 1 -- 172.21.15.28:0/850898352 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f51c8048060 con 0x7f51d01f2da0 2024-09-10T20:42:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.166+0000 7f51b7fff640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f51bc0780c0 0x7f51bc07a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:42:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.166+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f51bc0780c0 0x7f51bc07a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:42:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.166+0000 7f51b7fff640 1 -- 172.21.15.28:0/850898352 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f51c80b9ab0 con 0x7f51d01f2da0 2024-09-10T20:42:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.166+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f51bc0780c0 0x7f51bc07a580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51d00b53a0 tx=0x7f51c0005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:42:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.170+0000 7f51b7fff640 1 -- 172.21.15.28:0/850898352 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f51c8082320 con 0x7f51d01f2da0 2024-09-10T20:42:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:42:59.399+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f51d00b1220 con 0x7f51d01f2da0 2024-09-10T20:43:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.031+0000 7f51b7fff640 1 -- 172.21.15.28:0/850898352 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) ==== 122+0+0 (secure 0 0 0) 0x7f51c8081c20 con 0x7f51d01f2da0 2024-09-10T20:43:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f51bc0780c0 msgr2=0x7f51bc07a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f51bc0780c0 0x7f51bc07a580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51d00b53a0 tx=0x7f51c0005f90 comp rx=0 tx=0).stop 2024-09-10T20:43:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d01f2da0 msgr2=0x7f51d01f3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d01f2da0 0x7f51d01f3230 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f51c802b4b0 tx=0x7f51c8001aa0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 shutdown_connections 2024-09-10T20:43:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f51bc0780c0 0x7f51bc07a580 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d01f2da0 0x7f51d01f3230 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f51d00c5990 0x7f51d00c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 --2- 172.21.15.28:0/850898352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f51b8004d40 0x7f51d00b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 >> 172.21.15.28:0/850898352 conn(0x7f51d00d1e60 msgr2=0x7f51d01e8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 shutdown_connections 2024-09-10T20:43:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.035+0000 7f51d5d17640 1 -- 172.21.15.28:0/850898352 wait complete. 2024-09-10T20:43:00.052 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:43:00.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.160+0000 7f0423860640 1 -- 172.21.15.28:0/1070469622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 msgr2=0x7f041c0d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.160+0000 7f0423860640 1 --2- 172.21.15.28:0/1070469622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c0d2960 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f040c009830 tx=0x7f040c02a510 comp rx=0 tx=0).stop 2024-09-10T20:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.160+0000 7f0423860640 1 -- 172.21.15.28:0/1070469622 shutdown_connections 2024-09-10T20:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.160+0000 7f0423860640 1 --2- 172.21.15.28:0/1070469622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f041c0b5610 0x7f041c0b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.160+0000 7f0423860640 1 --2- 172.21.15.28:0/1070469622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c0d2960 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.160+0000 7f0423860640 1 -- 172.21.15.28:0/1070469622 >> 172.21.15.28:0/1070469622 conn(0x7f041c0d1e40 msgr2=0x7f041c0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.160+0000 7f0423860640 1 -- 172.21.15.28:0/1070469622 shutdown_connections 2024-09-10T20:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.161+0000 7f0423860640 1 -- 172.21.15.28:0/1070469622 wait complete. 2024-09-10T20:43:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 Processor -- start 2024-09-10T20:43:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 -- start start 2024-09-10T20:43:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f041c0b5610 0x7f041c1f56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c06edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f041c15bf30 0x7f041c06f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f041c0b5fc0 con 0x7f041c0d05d0 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f041c0d0d50 con 0x7f041c0b5610 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.162+0000 7f0423860640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f041c0b6140 con 0x7f041c15bf30 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c06edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c06edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50084/0 (socket says 172.21.15.28:50084) 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 -- 172.21.15.28:0/2882069464 learned_addr learned my addr 172.21.15.28:0/2882069464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f04215fc640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f041c0b5610 0x7f041c1f56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0421dfd640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f041c15bf30 0x7f041c06f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 -- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f041c15bf30 msgr2=0x7f041c06f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f041c15bf30 0x7f041c06f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 -- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f041c0b5610 msgr2=0x7f041c1f56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f041c0b5610 0x7f041c1f56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 -- 172.21.15.28:0/2882069464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041c06dd70 con 0x7f041c0d05d0 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0421dfd640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f041c15bf30 0x7f041c06f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0420dfb640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c06edf0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f040c009950 tx=0x7f040c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0411ffb640 1 -- 172.21.15.28:0/2882069464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c040070 con 0x7f041c0d05d0 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0411ffb640 1 -- 172.21.15.28:0/2882069464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f040c005310 con 0x7f041c0d05d0 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f041c06e030 con 0x7f041c0d05d0 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.163+0000 7f0411ffb640 1 -- 172.21.15.28:0/2882069464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c0047a0 con 0x7f041c0d05d0 2024-09-10T20:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.164+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f041c06e5f0 con 0x7f041c0d05d0 2024-09-10T20:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.165+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f041c0b65f0 con 0x7f041c0d05d0 2024-09-10T20:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.165+0000 7f0411ffb640 1 -- 172.21.15.28:0/2882069464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f040c04c030 con 0x7f041c0d05d0 2024-09-10T20:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.166+0000 7f0411ffb640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0400078090 0x7f040007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.166+0000 7f0411ffb640 1 -- 172.21.15.28:0/2882069464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f040c0bb0e0 con 0x7f041c0d05d0 2024-09-10T20:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.166+0000 7f04215fc640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0400078090 0x7f040007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.166+0000 7f04215fc640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0400078090 0x7f040007a550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0418000f60 tx=0x7f0418009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.170+0000 7f0411ffb640 1 -- 172.21.15.28:0/2882069464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f040c083920 con 0x7f041c0d05d0 2024-09-10T20:43:00.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.392+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f041c15a820 con 0x7f041c0d05d0 2024-09-10T20:43:00.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.393+0000 7f0411ffb640 1 -- 172.21.15.28:0/2882069464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5231 (secure 0 0 0) 0x7f040c083220 con 0x7f041c0d05d0 2024-09-10T20:43:00.397 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:00.397 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":66,"btime":"2024-09-10T20:43:00:026929+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4277647396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4277647396},{"type":"v1","addr":"172.21.15.28:6834","nonce":4277647396}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/4032655811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":4032655811},{"type":"v1","addr":"172.21.15.159:6835","nonce":4032655811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/3530092732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3530092732},{"type":"v1","addr":"172.21.15.159:6834","nonce":3530092732}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:56.370954+0000","modified":"2024-09-10T20:42:59.400466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1426218269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1426218269},{"type":"v1","addr":"172.21.15.28:6835","nonce":1426218269}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6936,"qdb_cluster":[6936]},"id":8}]} 2024-09-10T20:43:00.397 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 66 2024-09-10T20:43:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0400078090 msgr2=0x7f040007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0400078090 0x7f040007a550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0418000f60 tx=0x7f0418009040 comp rx=0 tx=0).stop 2024-09-10T20:43:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 msgr2=0x7f041c06edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c06edf0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f040c009950 tx=0x7f040c032c10 comp rx=0 tx=0).stop 2024-09-10T20:43:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 shutdown_connections 2024-09-10T20:43:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0400078090 0x7f040007a550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f041c15bf30 0x7f041c06f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041c0d05d0 0x7f041c06edf0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 --2- 172.21.15.28:0/2882069464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f041c0b5610 0x7f041c1f56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 >> 172.21.15.28:0/2882069464 conn(0x7f041c0d1e40 msgr2=0x7f041c0d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 shutdown_connections 2024-09-10T20:43:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.395+0000 7f0423860640 1 -- 172.21.15.28:0/2882069464 wait complete. 2024-09-10T20:43:00.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.517+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2608548025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf44003a00 msgr2=0x7fbf44003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.517+0000 7fbf5cb07640 1 --2- 172.21.15.28:0/2608548025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf44003a00 0x7fbf44003e20 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800bad0 tx=0x7fbf44005830 comp rx=0 tx=0).stop 2024-09-10T20:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.517+0000 7fbf56d9d640 1 -- 172.21.15.28:0/2608548025 reap_dead start 2024-09-10T20:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.517+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2608548025 shutdown_connections 2024-09-10T20:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.517+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2608548025 >> 172.21.15.28:0/2608548025 conn(0x7fbf5806cf00 msgr2=0x7fbf5806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.517+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2608548025 shutdown_connections 2024-09-10T20:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.517+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2608548025 wait complete. 2024-09-10T20:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 Processor -- start 2024-09-10T20:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 -- start start 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf44003a00 0x7fbf580afb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf581f2b60 0x7fbf580b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf580b1d40 0x7fbf5809cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5815bed0 con 0x7fbf580b1d40 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5815bd50 con 0x7fbf44003a00 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.519+0000 7fbf5cb07640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbf5815c050 con 0x7fbf581f2b60 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf580b1d40 0x7fbf5809cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf580b1d40 0x7fbf5809cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47442/0 (socket says 172.21.15.28:47442) 2024-09-10T20:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 -- 172.21.15.28:0/2873157937 learned_addr learned my addr 172.21.15.28:0/2873157937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf5659c640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf44003a00 0x7fbf580afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf581f2b60 0x7fbf580b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 -- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf581f2b60 msgr2=0x7fbf580b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf581f2b60 0x7fbf580b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 -- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf44003a00 msgr2=0x7fbf580afb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf44003a00 0x7fbf580afb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 -- 172.21.15.28:0/2873157937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf581f65a0 con 0x7fbf580b1d40 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf5659c640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf44003a00 0x7fbf580afb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf581f2b60 0x7fbf580b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf56d9d640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf580b1d40 0x7fbf5809cd10 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbf50005fd0 tx=0x7fbf5000ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf4effd640 1 -- 172.21.15.28:0/2873157937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf50016070 con 0x7fbf580b1d40 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf4effd640 1 -- 172.21.15.28:0/2873157937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf5000ccd0 con 0x7fbf580b1d40 2024-09-10T20:43:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf4effd640 1 -- 172.21.15.28:0/2873157937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5000cea0 con 0x7fbf580b1d40 2024-09-10T20:43:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf581f6880 con 0x7fbf580b1d40 2024-09-10T20:43:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.520+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf581f6e40 con 0x7fbf580b1d40 2024-09-10T20:43:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.521+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf580d05d0 con 0x7fbf580b1d40 2024-09-10T20:43:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.522+0000 7fbf4effd640 1 -- 172.21.15.28:0/2873157937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fbf5000f040 con 0x7fbf580b1d40 2024-09-10T20:43:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.522+0000 7fbf4effd640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf38078090 0x7fbf3807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.522+0000 7fbf4effd640 1 -- 172.21.15.28:0/2873157937 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fbf5009a360 con 0x7fbf580b1d40 2024-09-10T20:43:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.522+0000 7fbf5659c640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf38078090 0x7fbf3807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.523+0000 7fbf5659c640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf38078090 0x7fbf3807a550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf48036000 tx=0x7fbf480046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.527+0000 7fbf4effd640 1 -- 172.21.15.28:0/2873157937 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbf50062cd0 con 0x7fbf580b1d40 2024-09-10T20:43:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.711+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf5815d5c0 con 0x7fbf580b1d40 2024-09-10T20:43:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.712+0000 7fbf4effd640 1 -- 172.21.15.28:0/2873157937 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+16734 (secure 0 0 0) 0x7fbf500625d0 con 0x7fbf580b1d40 2024-09-10T20:43:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:00.716 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":100,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:42:58.012957+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:54.016199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:55.330837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:43:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf38078090 msgr2=0x7fbf3807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf38078090 0x7fbf3807a550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf48036000 tx=0x7fbf480046c0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf580b1d40 msgr2=0x7fbf5809cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf580b1d40 0x7fbf5809cd10 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbf50005fd0 tx=0x7fbf5000ed20 comp rx=0 tx=0).stop 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 shutdown_connections 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf38078090 0x7fbf3807a550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf580b1d40 0x7fbf5809cd10 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf581f2b60 0x7fbf580b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 --2- 172.21.15.28:0/2873157937 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf44003a00 0x7fbf580afb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 >> 172.21.15.28:0/2873157937 conn(0x7fbf5806cf00 msgr2=0x7fbf5815ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 shutdown_connections 2024-09-10T20:43:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.714+0000 7fbf5cb07640 1 -- 172.21.15.28:0/2873157937 wait complete. 2024-09-10T20:43:00.730 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:43:00.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:43:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.830+0000 7fb24e956640 1 -- 172.21.15.28:0/717860243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24815ed20 msgr2=0x7fb2480cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.830+0000 7fb24e956640 1 --2- 172.21.15.28:0/717860243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24815ed20 0x7fb2480cb210 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fb240005fd0 tx=0x7fb24002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.830+0000 7fb24e956640 1 -- 172.21.15.28:0/717860243 shutdown_connections 2024-09-10T20:43:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.830+0000 7fb24e956640 1 --2- 172.21.15.28:0/717860243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24815ed20 0x7fb2480cb210 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.830+0000 7fb24e956640 1 --2- 172.21.15.28:0/717860243 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2480cf0f0 0x7fb24815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.830+0000 7fb24e956640 1 -- 172.21.15.28:0/717860243 >> 172.21.15.28:0/717860243 conn(0x7fb2480d0420 msgr2=0x7fb2480d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.831+0000 7fb24e956640 1 -- 172.21.15.28:0/717860243 shutdown_connections 2024-09-10T20:43:00.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.831+0000 7fb24e956640 1 -- 172.21.15.28:0/717860243 wait complete. 2024-09-10T20:43:00.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.832+0000 7fb24e956640 1 Processor -- start 2024-09-10T20:43:00.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.832+0000 7fb24e956640 1 -- start start 2024-09-10T20:43:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2480cf0f0 0x7fb24813e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248151cc0 0x7fb248158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb24815ed20 0x7fb24815d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb24815f5b0 con 0x7fb248151cc0 2024-09-10T20:43:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb24815f430 con 0x7fb2480cf0f0 2024-09-10T20:43:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb2480cb750 con 0x7fb24815ed20 2024-09-10T20:43:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248151cc0 0x7fb248158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248151cc0 0x7fb248158570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47470/0 (socket says 172.21.15.28:47470) 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 -- 172.21.15.28:0/816367334 learned_addr learned my addr 172.21.15.28:0/816367334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24cef3640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb24815ed20 0x7fb24815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb247fff640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2480cf0f0 0x7fb24813e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 -- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb24815ed20 msgr2=0x7fb24815d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb24815ed20 0x7fb24815d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 -- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2480cf0f0 msgr2=0x7fb24813e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2480cf0f0 0x7fb24813e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 -- 172.21.15.28:0/816367334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2481577f0 con 0x7fb248151cc0 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb2477fe640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248151cc0 0x7fb248158570 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00c970 tx=0x7fb23c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb244ff9640 1 -- 172.21.15.28:0/816367334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c007c10 con 0x7fb248151cc0 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb244ff9640 1 -- 172.21.15.28:0/816367334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb23c018480 con 0x7fb248151cc0 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb248150530 con 0x7fb248151cc0 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb244ff9640 1 -- 172.21.15.28:0/816367334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0188d0 con 0x7fb248151cc0 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.833+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb248157d70 con 0x7fb248151cc0 2024-09-10T20:43:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.835+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2480745f0 con 0x7fb248151cc0 2024-09-10T20:43:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.835+0000 7fb244ff9640 1 -- 172.21.15.28:0/816367334 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fb23c018a70 con 0x7fb248151cc0 2024-09-10T20:43:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.836+0000 7fb244ff9640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb234078090 0x7fb23407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.836+0000 7fb247fff640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb234078090 0x7fb23407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.836+0000 7fb244ff9640 1 -- 172.21.15.28:0/816367334 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fb23c09aaa0 con 0x7fb248151cc0 2024-09-10T20:43:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.836+0000 7fb247fff640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb234078090 0x7fb23407a550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb238001700 tx=0x7fb238009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:00.840+0000 7fb244ff9640 1 -- 172.21.15.28:0/816367334 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb2480745f0 con 0x7fb248151cc0 2024-09-10T20:43:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.021+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2480cfe70 con 0x7fb234078090 2024-09-10T20:43:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.025+0000 7fb244ff9640 1 -- 172.21.15.28:0/816367334 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252167 (secure 0 0 0) 0x7fb2480cfe70 con 0x7fb234078090 2024-09-10T20:43:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:43:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb234078090 msgr2=0x7fb23407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb234078090 0x7fb23407a550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb238001700 tx=0x7fb238009040 comp rx=0 tx=0).stop 2024-09-10T20:43:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248151cc0 msgr2=0x7fb248158570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248151cc0 0x7fb248158570 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00c970 tx=0x7fb23c00ce30 comp rx=0 tx=0).stop 2024-09-10T20:43:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 shutdown_connections 2024-09-10T20:43:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb234078090 0x7fb23407a550 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb24815ed20 0x7fb24815d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248151cc0 0x7fb248158570 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 --2- 172.21.15.28:0/816367334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2480cf0f0 0x7fb24813e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 >> 172.21.15.28:0/816367334 conn(0x7fb2480d0420 msgr2=0x7fb2480ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 shutdown_connections 2024-09-10T20:43:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.028+0000 7fb24e956640 1 -- 172.21.15.28:0/816367334 wait complete. 2024-09-10T20:43:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-10T20:43:00.300529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277032,"kb_used_data":10864,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754697688,"statfs":{"total":773094113280,"available":772810432512,"internally_reserved":0,"allocated":11124736,"data_stored":5513684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.310072"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633866+0000","last_change":"2024-09-10T20:42:57.029855+0000","last_active":"2024-09-10T20:42:58.633866+0000","last_peered":"2024-09-10T20:42:58.633866+0000","last_clean":"2024-09-10T20:42:58.633866+0000","last_became_active":"2024-09-10T20:42:57.029307+0000","last_became_peered":"2024-09-10T20:42:57.029307+0000","last_unstale":"2024-09-10T20:42:58.633866+0000","last_undegraded":"2024-09-10T20:42:58.633866+0000","last_fullsized":"2024-09-10T20:42:58.633866+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:30.744380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776542+0000","last_change":"2024-09-10T20:42:57.020015+0000","last_active":"2024-09-10T20:42:58.776542+0000","last_peered":"2024-09-10T20:42:58.776542+0000","last_clean":"2024-09-10T20:42:58.776542+0000","last_became_active":"2024-09-10T20:42:57.019710+0000","last_became_peered":"2024-09-10T20:42:57.019710+0000","last_unstale":"2024-09-10T20:42:58.776542+0000","last_undegraded":"2024-09-10T20:42:58.776542+0000","last_fullsized":"2024-09-10T20:42:58.776542+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:58:50.532316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081889+0000","last_change":"2024-09-10T20:42:57.025166+0000","last_active":"2024-09-10T20:42:59.081889+0000","last_peered":"2024-09-10T20:42:59.081889+0000","last_clean":"2024-09-10T20:42:59.081889+0000","last_became_active":"2024-09-10T20:42:57.024892+0000","last_became_peered":"2024-09-10T20:42:57.024892+0000","last_unstale":"2024-09-10T20:42:59.081889+0000","last_undegraded":"2024-09-10T20:42:59.081889+0000","last_fullsized":"2024-09-10T20:42:59.081889+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:32.814547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019962+0000","last_change":"2024-09-10T20:42:57.023753+0000","last_active":"2024-09-10T20:42:58.019962+0000","last_peered":"2024-09-10T20:42:58.019962+0000","last_clean":"2024-09-10T20:42:58.019962+0000","last_became_active":"2024-09-10T20:42:57.023446+0000","last_became_peered":"2024-09-10T20:42:57.023446+0000","last_unstale":"2024-09-10T20:42:58.019962+0000","last_undegraded":"2024-09-10T20:42:58.019962+0000","last_fullsized":"2024-09-10T20:42:58.019962+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:06.298699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134708+0000","last_change":"2024-09-10T20:42:57.030708+0000","last_active":"2024-09-10T20:42:59.134708+0000","last_peered":"2024-09-10T20:42:59.134708+0000","last_clean":"2024-09-10T20:42:59.134708+0000","last_became_active":"2024-09-10T20:42:57.030431+0000","last_became_peered":"2024-09-10T20:42:57.030431+0000","last_unstale":"2024-09-10T20:42:59.134708+0000","last_undegraded":"2024-09-10T20:42:59.134708+0000","last_fullsized":"2024-09-10T20:42:59.134708+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:25:16.588271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017990+0000","last_change":"2024-09-10T20:42:57.023074+0000","last_active":"2024-09-10T20:42:58.017990+0000","last_peered":"2024-09-10T20:42:58.017990+0000","last_clean":"2024-09-10T20:42:58.017990+0000","last_became_active":"2024-09-10T20:42:57.020610+0000","last_became_peered":"2024-09-10T20:42:57.020610+0000","last_unstale":"2024-09-10T20:42:58.017990+0000","last_undegraded":"2024-09-10T20:42:58.017990+0000","last_fullsized":"2024-09-10T20:42:58.017990+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:00.371168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016201+0000","last_change":"2024-09-10T20:42:57.015140+0000","last_active":"2024-09-10T20:42:58.016201+0000","last_peered":"2024-09-10T20:42:58.016201+0000","last_clean":"2024-09-10T20:42:58.016201+0000","last_became_active":"2024-09-10T20:42:57.014823+0000","last_became_peered":"2024-09-10T20:42:57.014823+0000","last_unstale":"2024-09-10T20:42:58.016201+0000","last_undegraded":"2024-09-10T20:42:58.016201+0000","last_fullsized":"2024-09-10T20:42:58.016201+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:44.091183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016207+0000","last_change":"2024-09-10T20:42:57.022365+0000","last_active":"2024-09-10T20:42:58.016207+0000","last_peered":"2024-09-10T20:42:58.016207+0000","last_clean":"2024-09-10T20:42:58.016207+0000","last_became_active":"2024-09-10T20:42:57.021944+0000","last_became_peered":"2024-09-10T20:42:57.021944+0000","last_unstale":"2024-09-10T20:42:58.016207+0000","last_undegraded":"2024-09-10T20:42:58.016207+0000","last_fullsized":"2024-09-10T20:42:58.016207+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:48:06.357735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633926+0000","last_change":"2024-09-10T20:42:57.033265+0000","last_active":"2024-09-10T20:42:58.633926+0000","last_peered":"2024-09-10T20:42:58.633926+0000","last_clean":"2024-09-10T20:42:58.633926+0000","last_became_active":"2024-09-10T20:42:57.032956+0000","last_became_peered":"2024-09-10T20:42:57.032956+0000","last_unstale":"2024-09-10T20:42:58.633926+0000","last_undegraded":"2024-09-10T20:42:58.633926+0000","last_fullsized":"2024-09-10T20:42:58.633926+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:58.394937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134622+0000","last_change":"2024-09-10T20:42:57.029340+0000","last_active":"2024-09-10T20:42:59.134622+0000","last_peered":"2024-09-10T20:42:59.134622+0000","last_clean":"2024-09-10T20:42:59.134622+0000","last_became_active":"2024-09-10T20:42:57.029072+0000","last_became_peered":"2024-09-10T20:42:57.029072+0000","last_unstale":"2024-09-10T20:42:59.134622+0000","last_undegraded":"2024-09-10T20:42:59.134622+0000","last_fullsized":"2024-09-10T20:42:59.134622+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:34:01.259370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016517+0000","last_change":"2024-09-10T20:42:57.015630+0000","last_active":"2024-09-10T20:42:58.016517+0000","last_peered":"2024-09-10T20:42:58.016517+0000","last_clean":"2024-09-10T20:42:58.016517+0000","last_became_active":"2024-09-10T20:42:57.015356+0000","last_became_peered":"2024-09-10T20:42:57.015356+0000","last_unstale":"2024-09-10T20:42:58.016517+0000","last_undegraded":"2024-09-10T20:42:58.016517+0000","last_fullsized":"2024-09-10T20:42:58.016517+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:24.561913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134746+0000","last_change":"2024-09-10T20:42:57.025852+0000","last_active":"2024-09-10T20:42:59.134746+0000","last_peered":"2024-09-10T20:42:59.134746+0000","last_clean":"2024-09-10T20:42:59.134746+0000","last_became_active":"2024-09-10T20:42:57.025576+0000","last_became_peered":"2024-09-10T20:42:57.025576+0000","last_unstale":"2024-09-10T20:42:59.134746+0000","last_undegraded":"2024-09-10T20:42:59.134746+0000","last_fullsized":"2024-09-10T20:42:59.134746+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:47:57.927740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776413+0000","last_change":"2024-09-10T20:42:57.026175+0000","last_active":"2024-09-10T20:42:58.776413+0000","last_peered":"2024-09-10T20:42:58.776413+0000","last_clean":"2024-09-10T20:42:58.776413+0000","last_became_active":"2024-09-10T20:42:57.025856+0000","last_became_peered":"2024-09-10T20:42:57.025856+0000","last_unstale":"2024-09-10T20:42:58.776413+0000","last_undegraded":"2024-09-10T20:42:58.776413+0000","last_fullsized":"2024-09-10T20:42:58.776413+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:35:51.057064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133807+0000","last_change":"2024-09-10T20:42:57.020483+0000","last_active":"2024-09-10T20:42:59.133807+0000","last_peered":"2024-09-10T20:42:59.133807+0000","last_clean":"2024-09-10T20:42:59.133807+0000","last_became_active":"2024-09-10T20:42:57.020183+0000","last_became_peered":"2024-09-10T20:42:57.020183+0000","last_unstale":"2024-09-10T20:42:59.133807+0000","last_undegraded":"2024-09-10T20:42:59.133807+0000","last_fullsized":"2024-09-10T20:42:59.133807+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:54.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633437+0000","last_change":"2024-09-10T20:42:57.026600+0000","last_active":"2024-09-10T20:42:58.633437+0000","last_peered":"2024-09-10T20:42:58.633437+0000","last_clean":"2024-09-10T20:42:58.633437+0000","last_became_active":"2024-09-10T20:42:57.026275+0000","last_became_peered":"2024-09-10T20:42:57.026275+0000","last_unstale":"2024-09-10T20:42:58.633437+0000","last_undegraded":"2024-09-10T20:42:58.633437+0000","last_fullsized":"2024-09-10T20:42:58.633437+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:09.268660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134157+0000","last_change":"2024-09-10T20:42:57.032272+0000","last_active":"2024-09-10T20:42:59.134157+0000","last_peered":"2024-09-10T20:42:59.134157+0000","last_clean":"2024-09-10T20:42:59.134157+0000","last_became_active":"2024-09-10T20:42:57.031990+0000","last_became_peered":"2024-09-10T20:42:57.031990+0000","last_unstale":"2024-09-10T20:42:59.134157+0000","last_undegraded":"2024-09-10T20:42:59.134157+0000","last_fullsized":"2024-09-10T20:42:59.134157+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:26.631733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081997+0000","last_change":"2024-09-10T20:42:57.021382+0000","last_active":"2024-09-10T20:42:59.081997+0000","last_peered":"2024-09-10T20:42:59.081997+0000","last_clean":"2024-09-10T20:42:59.081997+0000","last_became_active":"2024-09-10T20:42:57.021091+0000","last_became_peered":"2024-09-10T20:42:57.021091+0000","last_unstale":"2024-09-10T20:42:59.081997+0000","last_undegraded":"2024-09-10T20:42:59.081997+0000","last_fullsized":"2024-09-10T20:42:59.081997+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:54.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016618+0000","last_change":"2024-09-10T20:42:57.028934+0000","last_active":"2024-09-10T20:42:58.016618+0000","last_peered":"2024-09-10T20:42:58.016618+0000","last_clean":"2024-09-10T20:42:58.016618+0000","last_became_active":"2024-09-10T20:42:57.028646+0000","last_became_peered":"2024-09-10T20:42:57.028646+0000","last_unstale":"2024-09-10T20:42:58.016618+0000","last_undegraded":"2024-09-10T20:42:58.016618+0000","last_fullsized":"2024-09-10T20:42:58.016618+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:22.269035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.082076+0000","last_change":"2024-09-10T20:42:57.027453+0000","last_active":"2024-09-10T20:42:59.082076+0000","last_peered":"2024-09-10T20:42:59.082076+0000","last_clean":"2024-09-10T20:42:59.082076+0000","last_became_active":"2024-09-10T20:42:57.026840+0000","last_became_peered":"2024-09-10T20:42:57.026840+0000","last_unstale":"2024-09-10T20:42:59.082076+0000","last_undegraded":"2024-09-10T20:42:59.082076+0000","last_fullsized":"2024-09-10T20:42:59.082076+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:01.914016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081702+0000","last_change":"2024-09-10T20:42:57.028940+0000","last_active":"2024-09-10T20:42:59.081702+0000","last_peered":"2024-09-10T20:42:59.081702+0000","last_clean":"2024-09-10T20:42:59.081702+0000","last_became_active":"2024-09-10T20:42:57.028656+0000","last_became_peered":"2024-09-10T20:42:57.028656+0000","last_unstale":"2024-09-10T20:42:59.081702+0000","last_undegraded":"2024-09-10T20:42:59.081702+0000","last_fullsized":"2024-09-10T20:42:59.081702+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:26.529675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016480+0000","last_change":"2024-09-10T20:42:57.021346+0000","last_active":"2024-09-10T20:42:58.016480+0000","last_peered":"2024-09-10T20:42:58.016480+0000","last_clean":"2024-09-10T20:42:58.016480+0000","last_became_active":"2024-09-10T20:42:57.021039+0000","last_became_peered":"2024-09-10T20:42:57.021039+0000","last_unstale":"2024-09-10T20:42:58.016480+0000","last_undegraded":"2024-09-10T20:42:58.016480+0000","last_fullsized":"2024-09-10T20:42:58.016480+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:50:32.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018184+0000","last_change":"2024-09-10T20:42:57.022255+0000","last_active":"2024-09-10T20:42:58.018184+0000","last_peered":"2024-09-10T20:42:58.018184+0000","last_clean":"2024-09-10T20:42:58.018184+0000","last_became_active":"2024-09-10T20:42:57.021985+0000","last_became_peered":"2024-09-10T20:42:57.021985+0000","last_unstale":"2024-09-10T20:42:58.018184+0000","last_undegraded":"2024-09-10T20:42:58.018184+0000","last_fullsized":"2024-09-10T20:42:58.018184+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:29:04.710331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134194+0000","last_change":"2024-09-10T20:42:57.024037+0000","last_active":"2024-09-10T20:42:59.134194+0000","last_peered":"2024-09-10T20:42:59.134194+0000","last_clean":"2024-09-10T20:42:59.134194+0000","last_became_active":"2024-09-10T20:42:57.023736+0000","last_became_peered":"2024-09-10T20:42:57.023736+0000","last_unstale":"2024-09-10T20:42:59.134194+0000","last_undegraded":"2024-09-10T20:42:59.134194+0000","last_fullsized":"2024-09-10T20:42:59.134194+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:55.620256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019012+0000","last_change":"2024-09-10T20:42:57.026621+0000","last_active":"2024-09-10T20:42:58.019012+0000","last_peered":"2024-09-10T20:42:58.019012+0000","last_clean":"2024-09-10T20:42:58.019012+0000","last_became_active":"2024-09-10T20:42:57.026323+0000","last_became_peered":"2024-09-10T20:42:57.026323+0000","last_unstale":"2024-09-10T20:42:58.019012+0000","last_undegraded":"2024-09-10T20:42:58.019012+0000","last_fullsized":"2024-09-10T20:42:58.019012+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:26.316832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016558+0000","last_change":"2024-09-10T20:42:57.023093+0000","last_active":"2024-09-10T20:42:58.016558+0000","last_peered":"2024-09-10T20:42:58.016558+0000","last_clean":"2024-09-10T20:42:58.016558+0000","last_became_active":"2024-09-10T20:42:57.022820+0000","last_became_peered":"2024-09-10T20:42:57.022820+0000","last_unstale":"2024-09-10T20:42:58.016558+0000","last_undegraded":"2024-09-10T20:42:58.016558+0000","last_fullsized":"2024-09-10T20:42:58.016558+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:48:55.929464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018281+0000","last_change":"2024-09-10T20:42:57.024636+0000","last_active":"2024-09-10T20:42:58.018281+0000","last_peered":"2024-09-10T20:42:58.018281+0000","last_clean":"2024-09-10T20:42:58.018281+0000","last_became_active":"2024-09-10T20:42:57.024367+0000","last_became_peered":"2024-09-10T20:42:57.024367+0000","last_unstale":"2024-09-10T20:42:58.018281+0000","last_undegraded":"2024-09-10T20:42:58.018281+0000","last_fullsized":"2024-09-10T20:42:58.018281+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:45.916803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019735+0000","last_change":"2024-09-10T20:42:57.025201+0000","last_active":"2024-09-10T20:42:58.019735+0000","last_peered":"2024-09-10T20:42:58.019735+0000","last_clean":"2024-09-10T20:42:58.019735+0000","last_became_active":"2024-09-10T20:42:57.024893+0000","last_became_peered":"2024-09-10T20:42:57.024893+0000","last_unstale":"2024-09-10T20:42:58.019735+0000","last_undegraded":"2024-09-10T20:42:58.019735+0000","last_fullsized":"2024-09-10T20:42:58.019735+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:14.838329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018291+0000","last_change":"2024-09-10T20:42:57.021572+0000","last_active":"2024-09-10T20:42:58.018291+0000","last_peered":"2024-09-10T20:42:58.018291+0000","last_clean":"2024-09-10T20:42:58.018291+0000","last_became_active":"2024-09-10T20:42:57.021305+0000","last_became_peered":"2024-09-10T20:42:57.021305+0000","last_unstale":"2024-09-10T20:42:58.018291+0000","last_undegraded":"2024-09-10T20:42:58.018291+0000","last_fullsized":"2024-09-10T20:42:58.018291+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:07.665375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134103+0000","last_change":"2024-09-10T20:42:57.017779+0000","last_active":"2024-09-10T20:42:59.134103+0000","last_peered":"2024-09-10T20:42:59.134103+0000","last_clean":"2024-09-10T20:42:59.134103+0000","last_became_active":"2024-09-10T20:42:57.017403+0000","last_became_peered":"2024-09-10T20:42:57.017403+0000","last_unstale":"2024-09-10T20:42:59.134103+0000","last_undegraded":"2024-09-10T20:42:59.134103+0000","last_fullsized":"2024-09-10T20:42:59.134103+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:11:15.214431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016317+0000","last_change":"2024-09-10T20:42:57.017619+0000","last_active":"2024-09-10T20:42:58.016317+0000","last_peered":"2024-09-10T20:42:58.016317+0000","last_clean":"2024-09-10T20:42:58.016317+0000","last_became_active":"2024-09-10T20:42:57.017337+0000","last_became_peered":"2024-09-10T20:42:57.017337+0000","last_unstale":"2024-09-10T20:42:58.016317+0000","last_undegraded":"2024-09-10T20:42:58.016317+0000","last_fullsized":"2024-09-10T20:42:58.016317+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:43:59.161863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633565+0000","last_change":"2024-09-10T20:42:57.028933+0000","last_active":"2024-09-10T20:42:58.633565+0000","last_peered":"2024-09-10T20:42:58.633565+0000","last_clean":"2024-09-10T20:42:58.633565+0000","last_became_active":"2024-09-10T20:42:57.028678+0000","last_became_peered":"2024-09-10T20:42:57.028678+0000","last_unstale":"2024-09-10T20:42:58.633565+0000","last_undegraded":"2024-09-10T20:42:58.633565+0000","last_fullsized":"2024-09-10T20:42:58.633565+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:32:00.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134144+0000","last_change":"2024-09-10T20:42:57.033433+0000","last_active":"2024-09-10T20:42:59.134144+0000","last_peered":"2024-09-10T20:42:59.134144+0000","last_clean":"2024-09-10T20:42:59.134144+0000","last_became_active":"2024-09-10T20:42:57.032798+0000","last_became_peered":"2024-09-10T20:42:57.032798+0000","last_unstale":"2024-09-10T20:42:59.134144+0000","last_undegraded":"2024-09-10T20:42:59.134144+0000","last_fullsized":"2024-09-10T20:42:59.134144+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:41.069604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633829+0000","last_change":"2024-09-10T20:42:57.033613+0000","last_active":"2024-09-10T20:42:58.633829+0000","last_peered":"2024-09-10T20:42:58.633829+0000","last_clean":"2024-09-10T20:42:58.633829+0000","last_became_active":"2024-09-10T20:42:57.033083+0000","last_became_peered":"2024-09-10T20:42:57.033083+0000","last_unstale":"2024-09-10T20:42:58.633829+0000","last_undegraded":"2024-09-10T20:42:58.633829+0000","last_fullsized":"2024-09-10T20:42:58.633829+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:55.535737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"99'4","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134230+0000","last_change":"2024-09-10T20:42:56.009858+0000","last_active":"2024-09-10T20:42:59.134230+0000","last_peered":"2024-09-10T20:42:59.134230+0000","last_clean":"2024-09-10T20:42:59.134230+0000","last_became_active":"2024-09-10T20:42:56.009575+0000","last_became_peered":"2024-09-10T20:42:56.009575+0000","last_unstale":"2024-09-10T20:42:59.134230+0000","last_undegraded":"2024-09-10T20:42:59.134230+0000","last_fullsized":"2024-09-10T20:42:59.134230+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:21.364713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134245+0000","last_change":"2024-09-10T20:42:57.027695+0000","last_active":"2024-09-10T20:42:59.134245+0000","last_peered":"2024-09-10T20:42:59.134245+0000","last_clean":"2024-09-10T20:42:59.134245+0000","last_became_active":"2024-09-10T20:42:57.026324+0000","last_became_peered":"2024-09-10T20:42:57.026324+0000","last_unstale":"2024-09-10T20:42:59.134245+0000","last_undegraded":"2024-09-10T20:42:59.134245+0000","last_fullsized":"2024-09-10T20:42:59.134245+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:01.246774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776692+0000","last_change":"2024-09-10T20:42:56.014671+0000","last_active":"2024-09-10T20:42:58.776692+0000","last_peered":"2024-09-10T20:42:58.776692+0000","last_clean":"2024-09-10T20:42:58.776692+0000","last_became_active":"2024-09-10T20:42:56.014382+0000","last_became_peered":"2024-09-10T20:42:56.014382+0000","last_unstale":"2024-09-10T20:42:58.776692+0000","last_undegraded":"2024-09-10T20:42:58.776692+0000","last_fullsized":"2024-09-10T20:42:58.776692+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:26.193994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018347+0000","last_change":"2024-09-10T20:42:57.023790+0000","last_active":"2024-09-10T20:42:58.018347+0000","last_peered":"2024-09-10T20:42:58.018347+0000","last_clean":"2024-09-10T20:42:58.018347+0000","last_became_active":"2024-09-10T20:42:57.023519+0000","last_became_peered":"2024-09-10T20:42:57.023519+0000","last_unstale":"2024-09-10T20:42:58.018347+0000","last_undegraded":"2024-09-10T20:42:58.018347+0000","last_fullsized":"2024-09-10T20:42:58.018347+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:29:25.621518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1d","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019791+0000","last_change":"2024-09-10T20:42:56.013868+0000","last_active":"2024-09-10T20:42:58.019791+0000","last_peered":"2024-09-10T20:42:58.019791+0000","last_clean":"2024-09-10T20:42:58.019791+0000","last_became_active":"2024-09-10T20:42:56.013535+0000","last_became_peered":"2024-09-10T20:42:56.013535+0000","last_unstale":"2024-09-10T20:42:58.019791+0000","last_undegraded":"2024-09-10T20:42:58.019791+0000","last_fullsized":"2024-09-10T20:42:58.019791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:37.865884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081947+0000","last_change":"2024-09-10T20:42:57.027102+0000","last_active":"2024-09-10T20:42:59.081947+0000","last_peered":"2024-09-10T20:42:59.081947+0000","last_clean":"2024-09-10T20:42:59.081947+0000","last_became_active":"2024-09-10T20:42:57.026760+0000","last_became_peered":"2024-09-10T20:42:57.026760+0000","last_unstale":"2024-09-10T20:42:59.081947+0000","last_undegraded":"2024-09-10T20:42:59.081947+0000","last_fullsized":"2024-09-10T20:42:59.081947+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:35.809111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019517+0000","last_change":"2024-09-10T20:42:56.008875+0000","last_active":"2024-09-10T20:42:58.019517+0000","last_peered":"2024-09-10T20:42:58.019517+0000","last_clean":"2024-09-10T20:42:58.019517+0000","last_became_active":"2024-09-10T20:42:56.008318+0000","last_became_peered":"2024-09-10T20:42:56.008318+0000","last_unstale":"2024-09-10T20:42:58.019517+0000","last_undegraded":"2024-09-10T20:42:58.019517+0000","last_fullsized":"2024-09-10T20:42:58.019517+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:24:47.050024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019902+0000","last_change":"2024-09-10T20:42:57.025899+0000","last_active":"2024-09-10T20:42:58.019902+0000","last_peered":"2024-09-10T20:42:58.019902+0000","last_clean":"2024-09-10T20:42:58.019902+0000","last_became_active":"2024-09-10T20:42:57.025642+0000","last_became_peered":"2024-09-10T20:42:57.025642+0000","last_unstale":"2024-09-10T20:42:58.019902+0000","last_undegraded":"2024-09-10T20:42:58.019902+0000","last_fullsized":"2024-09-10T20:42:58.019902+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:20.814654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1b","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081602+0000","last_change":"2024-09-10T20:42:56.016158+0000","last_active":"2024-09-10T20:42:59.081602+0000","last_peered":"2024-09-10T20:42:59.081602+0000","last_clean":"2024-09-10T20:42:59.081602+0000","last_became_active":"2024-09-10T20:42:56.015793+0000","last_became_peered":"2024-09-10T20:42:56.015793+0000","last_unstale":"2024-09-10T20:42:59.081602+0000","last_undegraded":"2024-09-10T20:42:59.081602+0000","last_fullsized":"2024-09-10T20:42:59.081602+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:39:09.435764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016528+0000","last_change":"2024-09-10T20:42:57.031474+0000","last_active":"2024-09-10T20:42:58.016528+0000","last_peered":"2024-09-10T20:42:58.016528+0000","last_clean":"2024-09-10T20:42:58.016528+0000","last_became_active":"2024-09-10T20:42:57.031188+0000","last_became_peered":"2024-09-10T20:42:57.031188+0000","last_unstale":"2024-09-10T20:42:58.016528+0000","last_undegraded":"2024-09-10T20:42:58.016528+0000","last_fullsized":"2024-09-10T20:42:58.016528+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:21.427198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633351+0000","last_change":"2024-09-10T20:42:56.012320+0000","last_active":"2024-09-10T20:42:58.633351+0000","last_peered":"2024-09-10T20:42:58.633351+0000","last_clean":"2024-09-10T20:42:58.633351+0000","last_became_active":"2024-09-10T20:42:56.012048+0000","last_became_peered":"2024-09-10T20:42:56.012048+0000","last_unstale":"2024-09-10T20:42:58.633351+0000","last_undegraded":"2024-09-10T20:42:58.633351+0000","last_fullsized":"2024-09-10T20:42:58.633351+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:44.676084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134593+0000","last_change":"2024-09-10T20:42:57.028906+0000","last_active":"2024-09-10T20:42:59.134593+0000","last_peered":"2024-09-10T20:42:59.134593+0000","last_clean":"2024-09-10T20:42:59.134593+0000","last_became_active":"2024-09-10T20:42:57.028622+0000","last_became_peered":"2024-09-10T20:42:57.028622+0000","last_unstale":"2024-09-10T20:42:59.134593+0000","last_undegraded":"2024-09-10T20:42:59.134593+0000","last_fullsized":"2024-09-10T20:42:59.134593+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:58:41.414994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016315+0000","last_change":"2024-09-10T20:42:56.014432+0000","last_active":"2024-09-10T20:42:58.016315+0000","last_peered":"2024-09-10T20:42:58.016315+0000","last_clean":"2024-09-10T20:42:58.016315+0000","last_became_active":"2024-09-10T20:42:56.014057+0000","last_became_peered":"2024-09-10T20:42:56.014057+0000","last_unstale":"2024-09-10T20:42:58.016315+0000","last_undegraded":"2024-09-10T20:42:58.016315+0000","last_fullsized":"2024-09-10T20:42:58.016315+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:09:35.683673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016215+0000","last_change":"2024-09-10T20:42:57.022290+0000","last_active":"2024-09-10T20:42:58.016215+0000","last_peered":"2024-09-10T20:42:58.016215+0000","last_clean":"2024-09-10T20:42:58.016215+0000","last_became_active":"2024-09-10T20:42:57.022006+0000","last_became_peered":"2024-09-10T20:42:57.022006+0000","last_unstale":"2024-09-10T20:42:58.016215+0000","last_undegraded":"2024-09-10T20:42:58.016215+0000","last_fullsized":"2024-09-10T20:42:58.016215+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:24.790256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134423+0000","last_change":"2024-09-10T20:42:56.015903+0000","last_active":"2024-09-10T20:42:59.134423+0000","last_peered":"2024-09-10T20:42:59.134423+0000","last_clean":"2024-09-10T20:42:59.134423+0000","last_became_active":"2024-09-10T20:42:56.015576+0000","last_became_peered":"2024-09-10T20:42:56.015576+0000","last_unstale":"2024-09-10T20:42:59.134423+0000","last_undegraded":"2024-09-10T20:42:59.134423+0000","last_fullsized":"2024-09-10T20:42:59.134423+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:45:42.481827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776639+0000","last_change":"2024-09-10T20:42:57.024977+0000","last_active":"2024-09-10T20:42:58.776639+0000","last_peered":"2024-09-10T20:42:58.776639+0000","last_clean":"2024-09-10T20:42:58.776639+0000","last_became_active":"2024-09-10T20:42:57.024638+0000","last_became_peered":"2024-09-10T20:42:57.024638+0000","last_unstale":"2024-09-10T20:42:58.776639+0000","last_undegraded":"2024-09-10T20:42:58.776639+0000","last_fullsized":"2024-09-10T20:42:58.776639+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:55.620256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019894+0000","last_change":"2024-09-10T20:42:56.014432+0000","last_active":"2024-09-10T20:42:58.019894+0000","last_peered":"2024-09-10T20:42:58.019894+0000","last_clean":"2024-09-10T20:42:58.019894+0000","last_became_active":"2024-09-10T20:42:56.014170+0000","last_became_peered":"2024-09-10T20:42:56.014170+0000","last_unstale":"2024-09-10T20:42:58.019894+0000","last_undegraded":"2024-09-10T20:42:58.019894+0000","last_fullsized":"2024-09-10T20:42:58.019894+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:49.280790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776608+0000","last_change":"2024-09-10T20:42:57.025302+0000","last_active":"2024-09-10T20:42:58.776608+0000","last_peered":"2024-09-10T20:42:58.776608+0000","last_clean":"2024-09-10T20:42:58.776608+0000","last_became_active":"2024-09-10T20:42:57.025012+0000","last_became_peered":"2024-09-10T20:42:57.025012+0000","last_unstale":"2024-09-10T20:42:58.776608+0000","last_undegraded":"2024-09-10T20:42:58.776608+0000","last_fullsized":"2024-09-10T20:42:58.776608+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:31.854297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134410+0000","last_change":"2024-09-10T20:42:56.016940+0000","last_active":"2024-09-10T20:42:59.134410+0000","last_peered":"2024-09-10T20:42:59.134410+0000","last_clean":"2024-09-10T20:42:59.134410+0000","last_became_active":"2024-09-10T20:42:56.016653+0000","last_became_peered":"2024-09-10T20:42:56.016653+0000","last_unstale":"2024-09-10T20:42:59.134410+0000","last_undegraded":"2024-09-10T20:42:59.134410+0000","last_fullsized":"2024-09-10T20:42:59.134410+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:03.974365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018454+0000","last_change":"2024-09-10T20:42:57.026750+0000","last_active":"2024-09-10T20:42:58.018454+0000","last_peered":"2024-09-10T20:42:58.018454+0000","last_clean":"2024-09-10T20:42:58.018454+0000","last_became_active":"2024-09-10T20:42:57.026411+0000","last_became_peered":"2024-09-10T20:42:57.026411+0000","last_unstale":"2024-09-10T20:42:58.018454+0000","last_undegraded":"2024-09-10T20:42:58.018454+0000","last_fullsized":"2024-09-10T20:42:58.018454+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:10.443908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.15","version":"99'3","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134549+0000","last_change":"2024-09-10T20:42:56.012469+0000","last_active":"2024-09-10T20:42:59.134549+0000","last_peered":"2024-09-10T20:42:59.134549+0000","last_clean":"2024-09-10T20:42:59.134549+0000","last_became_active":"2024-09-10T20:42:56.012115+0000","last_became_peered":"2024-09-10T20:42:56.012115+0000","last_unstale":"2024-09-10T20:42:59.134549+0000","last_undegraded":"2024-09-10T20:42:59.134549+0000","last_fullsized":"2024-09-10T20:42:59.134549+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:36.260836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016719+0000","last_change":"2024-09-10T20:42:57.022131+0000","last_active":"2024-09-10T20:42:58.016719+0000","last_peered":"2024-09-10T20:42:58.016719+0000","last_clean":"2024-09-10T20:42:58.016719+0000","last_became_active":"2024-09-10T20:42:57.021851+0000","last_became_peered":"2024-09-10T20:42:57.021851+0000","last_unstale":"2024-09-10T20:42:58.016719+0000","last_undegraded":"2024-09-10T20:42:58.016719+0000","last_fullsized":"2024-09-10T20:42:58.016719+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:44:58.275204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"99'6","reported_seq":21,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019684+0000","last_change":"2024-09-10T20:42:56.008582+0000","last_active":"2024-09-10T20:42:58.019684+0000","last_peered":"2024-09-10T20:42:58.019684+0000","last_clean":"2024-09-10T20:42:58.019684+0000","last_became_active":"2024-09-10T20:42:56.007992+0000","last_became_peered":"2024-09-10T20:42:56.007992+0000","last_unstale":"2024-09-10T20:42:58.019684+0000","last_undegraded":"2024-09-10T20:42:58.019684+0000","last_fullsized":"2024-09-10T20:42:58.019684+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:03:07.367121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134392+0000","last_change":"2024-09-10T20:42:57.031509+0000","last_active":"2024-09-10T20:42:59.134392+0000","last_peered":"2024-09-10T20:42:59.134392+0000","last_clean":"2024-09-10T20:42:59.134392+0000","last_became_active":"2024-09-10T20:42:57.031222+0000","last_became_peered":"2024-09-10T20:42:57.031222+0000","last_unstale":"2024-09-10T20:42:59.134392+0000","last_undegraded":"2024-09-10T20:42:59.134392+0000","last_fullsized":"2024-09-10T20:42:59.134392+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:49:30.963522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134558+0000","last_change":"2024-09-10T20:42:56.013564+0000","last_active":"2024-09-10T20:42:59.134558+0000","last_peered":"2024-09-10T20:42:59.134558+0000","last_clean":"2024-09-10T20:42:59.134558+0000","last_became_active":"2024-09-10T20:42:56.013247+0000","last_became_peered":"2024-09-10T20:42:56.013247+0000","last_unstale":"2024-09-10T20:42:59.134558+0000","last_undegraded":"2024-09-10T20:42:59.134558+0000","last_fullsized":"2024-09-10T20:42:59.134558+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:30.931318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019323+0000","last_change":"2024-09-10T20:42:57.025508+0000","last_active":"2024-09-10T20:42:58.019323+0000","last_peered":"2024-09-10T20:42:58.019323+0000","last_clean":"2024-09-10T20:42:58.019323+0000","last_became_active":"2024-09-10T20:42:57.025046+0000","last_became_peered":"2024-09-10T20:42:57.025046+0000","last_unstale":"2024-09-10T20:42:58.019323+0000","last_undegraded":"2024-09-10T20:42:58.019323+0000","last_fullsized":"2024-09-10T20:42:58.019323+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:37.933369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081772+0000","last_change":"2024-09-10T20:42:56.018247+0000","last_active":"2024-09-10T20:42:59.081772+0000","last_peered":"2024-09-10T20:42:59.081772+0000","last_clean":"2024-09-10T20:42:59.081772+0000","last_became_active":"2024-09-10T20:42:56.017976+0000","last_became_peered":"2024-09-10T20:42:56.017976+0000","last_unstale":"2024-09-10T20:42:59.081772+0000","last_undegraded":"2024-09-10T20:42:59.081772+0000","last_fullsized":"2024-09-10T20:42:59.081772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:29.699395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019205+0000","last_change":"2024-09-10T20:42:57.022140+0000","last_active":"2024-09-10T20:42:58.019205+0000","last_peered":"2024-09-10T20:42:58.019205+0000","last_clean":"2024-09-10T20:42:58.019205+0000","last_became_active":"2024-09-10T20:42:57.021872+0000","last_became_peered":"2024-09-10T20:42:57.021872+0000","last_unstale":"2024-09-10T20:42:58.019205+0000","last_undegraded":"2024-09-10T20:42:58.019205+0000","last_fullsized":"2024-09-10T20:42:58.019205+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:34.653250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"99'8","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633183+0000","last_change":"2024-09-10T20:42:56.014545+0000","last_active":"2024-09-10T20:42:58.633183+0000","last_peered":"2024-09-10T20:42:58.633183+0000","last_clean":"2024-09-10T20:42:58.633183+0000","last_became_active":"2024-09-10T20:42:56.014252+0000","last_became_peered":"2024-09-10T20:42:56.014252+0000","last_unstale":"2024-09-10T20:42:58.633183+0000","last_undegraded":"2024-09-10T20:42:58.633183+0000","last_fullsized":"2024-09-10T20:42:58.633183+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:25.524428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633896+0000","last_change":"2024-09-10T20:42:57.025794+0000","last_active":"2024-09-10T20:42:58.633896+0000","last_peered":"2024-09-10T20:42:58.633896+0000","last_clean":"2024-09-10T20:42:58.633896+0000","last_became_active":"2024-09-10T20:42:57.025518+0000","last_became_peered":"2024-09-10T20:42:57.025518+0000","last_unstale":"2024-09-10T20:42:58.633896+0000","last_undegraded":"2024-09-10T20:42:58.633896+0000","last_fullsized":"2024-09-10T20:42:58.633896+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:38.706038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.f","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017886+0000","last_change":"2024-09-10T20:42:56.012088+0000","last_active":"2024-09-10T20:42:58.017886+0000","last_peered":"2024-09-10T20:42:58.017886+0000","last_clean":"2024-09-10T20:42:58.017886+0000","last_became_active":"2024-09-10T20:42:56.011823+0000","last_became_peered":"2024-09-10T20:42:56.011823+0000","last_unstale":"2024-09-10T20:42:58.017886+0000","last_undegraded":"2024-09-10T20:42:58.017886+0000","last_fullsized":"2024-09-10T20:42:58.017886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:44.845617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.634005+0000","last_change":"2024-09-10T20:42:57.032358+0000","last_active":"2024-09-10T20:42:58.634005+0000","last_peered":"2024-09-10T20:42:58.634005+0000","last_clean":"2024-09-10T20:42:58.634005+0000","last_became_active":"2024-09-10T20:42:57.032057+0000","last_became_peered":"2024-09-10T20:42:57.032057+0000","last_unstale":"2024-09-10T20:42:58.634005+0000","last_undegraded":"2024-09-10T20:42:58.634005+0000","last_fullsized":"2024-09-10T20:42:58.634005+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:01:47.967682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.e","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134529+0000","last_change":"2024-09-10T20:42:56.012920+0000","last_active":"2024-09-10T20:42:59.134529+0000","last_peered":"2024-09-10T20:42:59.134529+0000","last_clean":"2024-09-10T20:42:59.134529+0000","last_became_active":"2024-09-10T20:42:56.012535+0000","last_became_peered":"2024-09-10T20:42:56.012535+0000","last_unstale":"2024-09-10T20:42:59.134529+0000","last_undegraded":"2024-09-10T20:42:59.134529+0000","last_fullsized":"2024-09-10T20:42:59.134529+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:33.459103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016481+0000","last_change":"2024-09-10T20:42:57.029859+0000","last_active":"2024-09-10T20:42:58.016481+0000","last_peered":"2024-09-10T20:42:58.016481+0000","last_clean":"2024-09-10T20:42:58.016481+0000","last_became_active":"2024-09-10T20:42:57.029524+0000","last_became_peered":"2024-09-10T20:42:57.029524+0000","last_unstale":"2024-09-10T20:42:58.016481+0000","last_undegraded":"2024-09-10T20:42:58.016481+0000","last_fullsized":"2024-09-10T20:42:58.016481+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:55:07.349407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134194+0000","last_change":"2024-09-10T20:42:56.003965+0000","last_active":"2024-09-10T20:42:59.134194+0000","last_peered":"2024-09-10T20:42:59.134194+0000","last_clean":"2024-09-10T20:42:59.134194+0000","last_became_active":"2024-09-10T20:42:56.003671+0000","last_became_peered":"2024-09-10T20:42:56.003671+0000","last_unstale":"2024-09-10T20:42:59.134194+0000","last_undegraded":"2024-09-10T20:42:59.134194+0000","last_fullsized":"2024-09-10T20:42:59.134194+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:04:54.457764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134100+0000","last_change":"2024-09-10T20:42:57.021217+0000","last_active":"2024-09-10T20:42:59.134100+0000","last_peered":"2024-09-10T20:42:59.134100+0000","last_clean":"2024-09-10T20:42:59.134100+0000","last_became_active":"2024-09-10T20:42:57.020799+0000","last_became_peered":"2024-09-10T20:42:57.020799+0000","last_unstale":"2024-09-10T20:42:59.134100+0000","last_undegraded":"2024-09-10T20:42:59.134100+0000","last_fullsized":"2024-09-10T20:42:59.134100+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:35:51.057064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133940+0000","last_change":"2024-09-10T20:42:56.002469+0000","last_active":"2024-09-10T20:42:59.133940+0000","last_peered":"2024-09-10T20:42:59.133940+0000","last_clean":"2024-09-10T20:42:59.133940+0000","last_became_active":"2024-09-10T20:42:56.002182+0000","last_became_peered":"2024-09-10T20:42:56.002182+0000","last_unstale":"2024-09-10T20:42:59.133940+0000","last_undegraded":"2024-09-10T20:42:59.133940+0000","last_fullsized":"2024-09-10T20:42:59.133940+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:27.829406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081633+0000","last_change":"2024-09-10T20:42:57.024378+0000","last_active":"2024-09-10T20:42:59.081633+0000","last_peered":"2024-09-10T20:42:59.081633+0000","last_clean":"2024-09-10T20:42:59.081633+0000","last_became_active":"2024-09-10T20:42:57.024104+0000","last_became_peered":"2024-09-10T20:42:57.024104+0000","last_unstale":"2024-09-10T20:42:59.081633+0000","last_undegraded":"2024-09-10T20:42:59.081633+0000","last_fullsized":"2024-09-10T20:42:59.081633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:53.388782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134666+0000","last_change":"2024-09-10T20:42:56.016899+0000","last_active":"2024-09-10T20:42:59.134666+0000","last_peered":"2024-09-10T20:42:59.134666+0000","last_clean":"2024-09-10T20:42:59.134666+0000","last_became_active":"2024-09-10T20:42:56.016591+0000","last_became_peered":"2024-09-10T20:42:56.016591+0000","last_unstale":"2024-09-10T20:42:59.134666+0000","last_undegraded":"2024-09-10T20:42:59.134666+0000","last_fullsized":"2024-09-10T20:42:59.134666+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:38:00.903905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633644+0000","last_change":"2024-09-10T20:42:57.027594+0000","last_active":"2024-09-10T20:42:58.633644+0000","last_peered":"2024-09-10T20:42:58.633644+0000","last_clean":"2024-09-10T20:42:58.633644+0000","last_became_active":"2024-09-10T20:42:57.027298+0000","last_became_peered":"2024-09-10T20:42:57.027298+0000","last_unstale":"2024-09-10T20:42:58.633644+0000","last_undegraded":"2024-09-10T20:42:58.633644+0000","last_fullsized":"2024-09-10T20:42:58.633644+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:45.311740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633389+0000","last_change":"2024-09-10T20:42:56.007507+0000","last_active":"2024-09-10T20:42:58.633389+0000","last_peered":"2024-09-10T20:42:58.633389+0000","last_clean":"2024-09-10T20:42:58.633389+0000","last_became_active":"2024-09-10T20:42:56.007245+0000","last_became_peered":"2024-09-10T20:42:56.007245+0000","last_unstale":"2024-09-10T20:42:58.633389+0000","last_undegraded":"2024-09-10T20:42:58.633389+0000","last_fullsized":"2024-09-10T20:42:58.633389+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:52.383535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133833+0000","last_change":"2024-09-10T20:42:57.025422+0000","last_active":"2024-09-10T20:42:59.133833+0000","last_peered":"2024-09-10T20:42:59.133833+0000","last_clean":"2024-09-10T20:42:59.133833+0000","last_became_active":"2024-09-10T20:42:57.024875+0000","last_became_peered":"2024-09-10T20:42:57.024875+0000","last_unstale":"2024-09-10T20:42:59.133833+0000","last_undegraded":"2024-09-10T20:42:59.133833+0000","last_fullsized":"2024-09-10T20:42:59.133833+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:01.914016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019282+0000","last_change":"2024-09-10T20:42:56.013936+0000","last_active":"2024-09-10T20:42:58.019282+0000","last_peered":"2024-09-10T20:42:58.019282+0000","last_clean":"2024-09-10T20:42:58.019282+0000","last_became_active":"2024-09-10T20:42:56.013414+0000","last_became_peered":"2024-09-10T20:42:56.013414+0000","last_unstale":"2024-09-10T20:42:58.019282+0000","last_undegraded":"2024-09-10T20:42:58.019282+0000","last_fullsized":"2024-09-10T20:42:58.019282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:16.702971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134248+0000","last_change":"2024-09-10T20:42:57.024432+0000","last_active":"2024-09-10T20:42:59.134248+0000","last_peered":"2024-09-10T20:42:59.134248+0000","last_clean":"2024-09-10T20:42:59.134248+0000","last_became_active":"2024-09-10T20:42:57.024158+0000","last_became_peered":"2024-09-10T20:42:57.024158+0000","last_unstale":"2024-09-10T20:42:59.134248+0000","last_undegraded":"2024-09-10T20:42:59.134248+0000","last_fullsized":"2024-09-10T20:42:59.134248+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:10:01.699054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019152+0000","last_change":"2024-09-10T20:42:56.014674+0000","last_active":"2024-09-10T20:42:58.019152+0000","last_peered":"2024-09-10T20:42:58.019152+0000","last_clean":"2024-09-10T20:42:58.019152+0000","last_became_active":"2024-09-10T20:42:56.014110+0000","last_became_peered":"2024-09-10T20:42:56.014110+0000","last_unstale":"2024-09-10T20:42:58.019152+0000","last_undegraded":"2024-09-10T20:42:58.019152+0000","last_fullsized":"2024-09-10T20:42:58.019152+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:49.404912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133927+0000","last_change":"2024-09-10T20:42:57.026077+0000","last_active":"2024-09-10T20:42:59.133927+0000","last_peered":"2024-09-10T20:42:59.133927+0000","last_clean":"2024-09-10T20:42:59.133927+0000","last_became_active":"2024-09-10T20:42:57.025807+0000","last_became_peered":"2024-09-10T20:42:57.025807+0000","last_unstale":"2024-09-10T20:42:59.133927+0000","last_undegraded":"2024-09-10T20:42:59.133927+0000","last_fullsized":"2024-09-10T20:42:59.133927+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:32:40.794936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.7","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019995+0000","last_change":"2024-09-10T20:42:56.015471+0000","last_active":"2024-09-10T20:42:58.019995+0000","last_peered":"2024-09-10T20:42:58.019995+0000","last_clean":"2024-09-10T20:42:58.019995+0000","last_became_active":"2024-09-10T20:42:56.015190+0000","last_became_peered":"2024-09-10T20:42:56.015190+0000","last_unstale":"2024-09-10T20:42:58.019995+0000","last_undegraded":"2024-09-10T20:42:58.019995+0000","last_fullsized":"2024-09-10T20:42:58.019995+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:19.148181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017963+0000","last_change":"2024-09-10T20:42:57.025806+0000","last_active":"2024-09-10T20:42:58.017963+0000","last_peered":"2024-09-10T20:42:58.017963+0000","last_clean":"2024-09-10T20:42:58.017963+0000","last_became_active":"2024-09-10T20:42:57.025445+0000","last_became_peered":"2024-09-10T20:42:57.025445+0000","last_unstale":"2024-09-10T20:42:58.017963+0000","last_undegraded":"2024-09-10T20:42:58.017963+0000","last_fullsized":"2024-09-10T20:42:58.017963+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:48:02.305241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"100'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.020563+0000","last_change":"2024-09-10T20:42:56.007680+0000","last_active":"2024-09-10T20:42:58.020563+0000","last_peered":"2024-09-10T20:42:58.020563+0000","last_clean":"2024-09-10T20:42:58.020563+0000","last_became_active":"2024-09-10T20:42:56.007392+0000","last_became_peered":"2024-09-10T20:42:56.007392+0000","last_unstale":"2024-09-10T20:42:58.020563+0000","last_undegraded":"2024-09-10T20:42:58.020563+0000","last_fullsized":"2024-09-10T20:42:58.020563+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:45:01.242317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633734+0000","last_change":"2024-09-10T20:42:57.025849+0000","last_active":"2024-09-10T20:42:58.633734+0000","last_peered":"2024-09-10T20:42:58.633734+0000","last_clean":"2024-09-10T20:42:58.633734+0000","last_became_active":"2024-09-10T20:42:57.025478+0000","last_became_peered":"2024-09-10T20:42:57.025478+0000","last_unstale":"2024-09-10T20:42:58.633734+0000","last_undegraded":"2024-09-10T20:42:58.633734+0000","last_fullsized":"2024-09-10T20:42:58.633734+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:58.645730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019902+0000","last_change":"2024-09-10T20:42:56.015660+0000","last_active":"2024-09-10T20:42:58.019902+0000","last_peered":"2024-09-10T20:42:58.019902+0000","last_clean":"2024-09-10T20:42:58.019902+0000","last_became_active":"2024-09-10T20:42:56.014362+0000","last_became_peered":"2024-09-10T20:42:56.014362+0000","last_unstale":"2024-09-10T20:42:58.019902+0000","last_undegraded":"2024-09-10T20:42:58.019902+0000","last_fullsized":"2024-09-10T20:42:58.019902+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:15.186008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134072+0000","last_change":"2024-09-10T20:42:57.030963+0000","last_active":"2024-09-10T20:42:59.134072+0000","last_peered":"2024-09-10T20:42:59.134072+0000","last_clean":"2024-09-10T20:42:59.134072+0000","last_became_active":"2024-09-10T20:42:57.030666+0000","last_became_peered":"2024-09-10T20:42:57.030666+0000","last_unstale":"2024-09-10T20:42:59.134072+0000","last_undegraded":"2024-09-10T20:42:59.134072+0000","last_fullsized":"2024-09-10T20:42:59.134072+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:05.228089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019795+0000","last_change":"2024-09-10T20:42:56.006037+0000","last_active":"2024-09-10T20:42:58.019795+0000","last_peered":"2024-09-10T20:42:58.019795+0000","last_clean":"2024-09-10T20:42:58.019795+0000","last_became_active":"2024-09-10T20:42:56.004553+0000","last_became_peered":"2024-09-10T20:42:56.004553+0000","last_unstale":"2024-09-10T20:42:58.019795+0000","last_undegraded":"2024-09-10T20:42:58.019795+0000","last_fullsized":"2024-09-10T20:42:58.019795+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:10:07.187475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081640+0000","last_change":"2024-09-10T20:42:57.028178+0000","last_active":"2024-09-10T20:42:59.081640+0000","last_peered":"2024-09-10T20:42:59.081640+0000","last_clean":"2024-09-10T20:42:59.081640+0000","last_became_active":"2024-09-10T20:42:57.027911+0000","last_became_peered":"2024-09-10T20:42:57.027911+0000","last_unstale":"2024-09-10T20:42:59.081640+0000","last_undegraded":"2024-09-10T20:42:59.081640+0000","last_fullsized":"2024-09-10T20:42:59.081640+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:10.882657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016569+0000","last_change":"2024-09-10T20:42:56.013071+0000","last_active":"2024-09-10T20:42:58.016569+0000","last_peered":"2024-09-10T20:42:58.016569+0000","last_clean":"2024-09-10T20:42:58.016569+0000","last_became_active":"2024-09-10T20:42:56.012762+0000","last_became_peered":"2024-09-10T20:42:56.012762+0000","last_unstale":"2024-09-10T20:42:58.016569+0000","last_undegraded":"2024-09-10T20:42:58.016569+0000","last_fullsized":"2024-09-10T20:42:58.016569+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:12:55.456247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134128+0000","last_change":"2024-09-10T20:42:57.030084+0000","last_active":"2024-09-10T20:42:59.134128+0000","last_peered":"2024-09-10T20:42:59.134128+0000","last_clean":"2024-09-10T20:42:59.134128+0000","last_became_active":"2024-09-10T20:42:57.029814+0000","last_became_peered":"2024-09-10T20:42:57.029814+0000","last_unstale":"2024-09-10T20:42:59.134128+0000","last_undegraded":"2024-09-10T20:42:59.134128+0000","last_fullsized":"2024-09-10T20:42:59.134128+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:31.854297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133821+0000","last_change":"2024-09-10T20:42:56.008409+0000","last_active":"2024-09-10T20:42:59.133821+0000","last_peered":"2024-09-10T20:42:59.133821+0000","last_clean":"2024-09-10T20:42:59.133821+0000","last_became_active":"2024-09-10T20:42:56.008156+0000","last_became_peered":"2024-09-10T20:42:56.008156+0000","last_unstale":"2024-09-10T20:42:59.133821+0000","last_undegraded":"2024-09-10T20:42:59.133821+0000","last_fullsized":"2024-09-10T20:42:59.133821+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.531748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133890+0000","last_change":"2024-09-10T20:42:57.025142+0000","last_active":"2024-09-10T20:42:59.133890+0000","last_peered":"2024-09-10T20:42:59.133890+0000","last_clean":"2024-09-10T20:42:59.133890+0000","last_became_active":"2024-09-10T20:42:57.024510+0000","last_became_peered":"2024-09-10T20:42:57.024510+0000","last_unstale":"2024-09-10T20:42:59.133890+0000","last_undegraded":"2024-09-10T20:42:59.133890+0000","last_fullsized":"2024-09-10T20:42:59.133890+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:53:10.990965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633644+0000","last_change":"2024-09-10T20:42:56.014524+0000","last_active":"2024-09-10T20:42:58.633644+0000","last_peered":"2024-09-10T20:42:58.633644+0000","last_clean":"2024-09-10T20:42:58.633644+0000","last_became_active":"2024-09-10T20:42:56.014209+0000","last_became_peered":"2024-09-10T20:42:56.014209+0000","last_unstale":"2024-09-10T20:42:58.633644+0000","last_undegraded":"2024-09-10T20:42:58.633644+0000","last_fullsized":"2024-09-10T20:42:58.633644+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:18.142897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017787+0000","last_change":"2024-09-10T20:42:57.023049+0000","last_active":"2024-09-10T20:42:58.017787+0000","last_peered":"2024-09-10T20:42:58.017787+0000","last_clean":"2024-09-10T20:42:58.017787+0000","last_became_active":"2024-09-10T20:42:57.022788+0000","last_became_peered":"2024-09-10T20:42:57.022788+0000","last_unstale":"2024-09-10T20:42:58.017787+0000","last_undegraded":"2024-09-10T20:42:58.017787+0000","last_fullsized":"2024-09-10T20:42:58.017787+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:39.719435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017649+0000","last_change":"2024-09-10T20:42:56.009317+0000","last_active":"2024-09-10T20:42:58.017649+0000","last_peered":"2024-09-10T20:42:58.017649+0000","last_clean":"2024-09-10T20:42:58.017649+0000","last_became_active":"2024-09-10T20:42:56.009038+0000","last_became_peered":"2024-09-10T20:42:56.009038+0000","last_unstale":"2024-09-10T20:42:58.017649+0000","last_undegraded":"2024-09-10T20:42:58.017649+0000","last_fullsized":"2024-09-10T20:42:58.017649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:02:53.603352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019895+0000","last_change":"2024-09-10T20:42:57.021758+0000","last_active":"2024-09-10T20:42:58.019895+0000","last_peered":"2024-09-10T20:42:58.019895+0000","last_clean":"2024-09-10T20:42:58.019895+0000","last_became_active":"2024-09-10T20:42:57.021337+0000","last_became_peered":"2024-09-10T20:42:57.021337+0000","last_unstale":"2024-09-10T20:42:58.019895+0000","last_undegraded":"2024-09-10T20:42:58.019895+0000","last_fullsized":"2024-09-10T20:42:58.019895+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:06.903753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776204+0000","last_change":"2024-09-10T20:42:56.009113+0000","last_active":"2024-09-10T20:42:58.776204+0000","last_peered":"2024-09-10T20:42:58.776204+0000","last_clean":"2024-09-10T20:42:58.776204+0000","last_became_active":"2024-09-10T20:42:56.008812+0000","last_became_peered":"2024-09-10T20:42:56.008812+0000","last_unstale":"2024-09-10T20:42:58.776204+0000","last_undegraded":"2024-09-10T20:42:58.776204+0000","last_fullsized":"2024-09-10T20:42:58.776204+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:12:13.797884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":283,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081469+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:42:59.081469+0000","last_peered":"2024-09-10T20:42:59.081469+0000","last_clean":"2024-09-10T20:42:59.081469+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:42:59.081469+0000","last_undegraded":"2024-09-10T20:42:59.081469+0000","last_fullsized":"2024-09-10T20:42:59.081469+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542207,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1257472,"data_stored":544327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542208,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1224704,"data_stored":536291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542206,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1839104,"data_stored":1133051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542206,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1253376,"data_stored":544293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509504,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":1236992,"data_stored":542591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509503,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36568,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335272,"statfs":{"total":96636764160,"available":96599318528,"internally_reserved":0,"allocated":1249280,"data_stored":543229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509503,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37132,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1818624,"data_stored":1126659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509503,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36568,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335272,"statfs":{"total":96636764160,"available":96599318528,"internally_reserved":0,"allocated":1245184,"data_stored":543243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:43:01.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:43:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.186+0000 7ff74e564640 1 -- 172.21.15.28:0/1458817677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff730004800 msgr2=0x7ff730004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.186+0000 7ff74e564640 1 --2- 172.21.15.28:0/1458817677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff730004800 0x7ff730004c80 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00c290 tx=0x7ff730005830 comp rx=0 tx=0).stop 2024-09-10T20:43:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.186+0000 7ff74cb01640 1 -- 172.21.15.28:0/1458817677 reap_dead start 2024-09-10T20:43:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.186+0000 7ff74e564640 1 -- 172.21.15.28:0/1458817677 shutdown_connections 2024-09-10T20:43:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.186+0000 7ff74e564640 1 -- 172.21.15.28:0/1458817677 >> 172.21.15.28:0/1458817677 conn(0x7ff7480d1e40 msgr2=0x7ff7480d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.186+0000 7ff74e564640 1 -- 172.21.15.28:0/1458817677 shutdown_connections 2024-09-10T20:43:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.186+0000 7ff74e564640 1 -- 172.21.15.28:0/1458817677 wait complete. 2024-09-10T20:43:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.187+0000 7ff74e564640 1 Processor -- start 2024-09-10T20:43:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.187+0000 7ff74e564640 1 -- start start 2024-09-10T20:43:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff74e564640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff730004800 0x7ff748155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff74e564640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff748154460 0x7ff7481f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff74e564640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7480c93a0 0x7ff7480c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff74e564640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7480d0cc0 con 0x7ff748154460 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff74e564640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff7480d0b40 con 0x7ff7480c93a0 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff74e564640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff7480d2960 con 0x7ff730004800 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff7477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff748154460 0x7ff7481f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff7477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff748154460 0x7ff7481f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47488/0 (socket says 172.21.15.28:47488) 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff7477fe640 1 -- 172.21.15.28:0/734835136 learned_addr learned my addr 172.21.15.28:0/734835136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff74cb01640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7480c93a0 0x7ff7480c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.188+0000 7ff747fff640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff730004800 0x7ff748155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff7477fe640 1 -- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff730004800 msgr2=0x7ff748155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff7477fe640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff730004800 0x7ff748155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff7477fe640 1 -- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7480c93a0 msgr2=0x7ff7480c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff7477fe640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7480c93a0 0x7ff7480c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff7477fe640 1 -- 172.21.15.28:0/734835136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff748155f70 con 0x7ff748154460 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff7477fe640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff748154460 0x7ff7481f5ee0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00c3b0 tx=0x7ff73c001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff744ff9640 1 -- 172.21.15.28:0/734835136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73c031040 con 0x7ff748154460 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff744ff9640 1 -- 172.21.15.28:0/734835136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff73c0035f0 con 0x7ff748154460 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7481533c0 con 0x7ff748154460 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff744ff9640 1 -- 172.21.15.28:0/734835136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73c003a40 con 0x7ff748154460 2024-09-10T20:43:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.189+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff748153c60 con 0x7ff748154460 2024-09-10T20:43:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.190+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7480d0300 con 0x7ff748154460 2024-09-10T20:43:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.191+0000 7ff744ff9640 1 -- 172.21.15.28:0/734835136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7ff73c048030 con 0x7ff748154460 2024-09-10T20:43:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.191+0000 7ff744ff9640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff734078090 0x7ff73407a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.191+0000 7ff744ff9640 1 -- 172.21.15.28:0/734835136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7ff73c0b9db0 con 0x7ff748154460 2024-09-10T20:43:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.191+0000 7ff747fff640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff734078090 0x7ff73407a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.192+0000 7ff747fff640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff734078090 0x7ff73407a550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff738009af0 tx=0x7ff738005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:01.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.196+0000 7ff744ff9640 1 -- 172.21.15.28:0/734835136 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff73c0be070 con 0x7ff748154460 2024-09-10T20:43:01.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.376+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7480b7870 con 0x7ff734078090 2024-09-10T20:43:01.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.380+0000 7ff744ff9640 1 -- 172.21.15.28:0/734835136 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252167 (secure 0 0 0) 0x7ff7480b7870 con 0x7ff734078090 2024-09-10T20:43:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:43:01.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff734078090 msgr2=0x7ff73407a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff734078090 0x7ff73407a550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff738009af0 tx=0x7ff738005f90 comp rx=0 tx=0).stop 2024-09-10T20:43:01.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff748154460 msgr2=0x7ff7481f5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff748154460 0x7ff7481f5ee0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00c3b0 tx=0x7ff73c001d40 comp rx=0 tx=0).stop 2024-09-10T20:43:01.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 shutdown_connections 2024-09-10T20:43:01.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff734078090 0x7ff73407a550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7480c93a0 0x7ff7480c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff748154460 0x7ff7481f5ee0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 --2- 172.21.15.28:0/734835136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff730004800 0x7ff748155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 >> 172.21.15.28:0/734835136 conn(0x7ff7480d1e40 msgr2=0x7ff7480b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:01.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 shutdown_connections 2024-09-10T20:43:01.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.383+0000 7ff74e564640 1 -- 172.21.15.28:0/734835136 wait complete. 2024-09-10T20:43:01.398 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-10T20:43:00.300529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277032,"kb_used_data":10864,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754697688,"statfs":{"total":773094113280,"available":772810432512,"internally_reserved":0,"allocated":11124736,"data_stored":5513684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.310072"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633866+0000","last_change":"2024-09-10T20:42:57.029855+0000","last_active":"2024-09-10T20:42:58.633866+0000","last_peered":"2024-09-10T20:42:58.633866+0000","last_clean":"2024-09-10T20:42:58.633866+0000","last_became_active":"2024-09-10T20:42:57.029307+0000","last_became_peered":"2024-09-10T20:42:57.029307+0000","last_unstale":"2024-09-10T20:42:58.633866+0000","last_undegraded":"2024-09-10T20:42:58.633866+0000","last_fullsized":"2024-09-10T20:42:58.633866+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:30.744380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776542+0000","last_change":"2024-09-10T20:42:57.020015+0000","last_active":"2024-09-10T20:42:58.776542+0000","last_peered":"2024-09-10T20:42:58.776542+0000","last_clean":"2024-09-10T20:42:58.776542+0000","last_became_active":"2024-09-10T20:42:57.019710+0000","last_became_peered":"2024-09-10T20:42:57.019710+0000","last_unstale":"2024-09-10T20:42:58.776542+0000","last_undegraded":"2024-09-10T20:42:58.776542+0000","last_fullsized":"2024-09-10T20:42:58.776542+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:58:50.532316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081889+0000","last_change":"2024-09-10T20:42:57.025166+0000","last_active":"2024-09-10T20:42:59.081889+0000","last_peered":"2024-09-10T20:42:59.081889+0000","last_clean":"2024-09-10T20:42:59.081889+0000","last_became_active":"2024-09-10T20:42:57.024892+0000","last_became_peered":"2024-09-10T20:42:57.024892+0000","last_unstale":"2024-09-10T20:42:59.081889+0000","last_undegraded":"2024-09-10T20:42:59.081889+0000","last_fullsized":"2024-09-10T20:42:59.081889+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:32.814547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019962+0000","last_change":"2024-09-10T20:42:57.023753+0000","last_active":"2024-09-10T20:42:58.019962+0000","last_peered":"2024-09-10T20:42:58.019962+0000","last_clean":"2024-09-10T20:42:58.019962+0000","last_became_active":"2024-09-10T20:42:57.023446+0000","last_became_peered":"2024-09-10T20:42:57.023446+0000","last_unstale":"2024-09-10T20:42:58.019962+0000","last_undegraded":"2024-09-10T20:42:58.019962+0000","last_fullsized":"2024-09-10T20:42:58.019962+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:06.298699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134708+0000","last_change":"2024-09-10T20:42:57.030708+0000","last_active":"2024-09-10T20:42:59.134708+0000","last_peered":"2024-09-10T20:42:59.134708+0000","last_clean":"2024-09-10T20:42:59.134708+0000","last_became_active":"2024-09-10T20:42:57.030431+0000","last_became_peered":"2024-09-10T20:42:57.030431+0000","last_unstale":"2024-09-10T20:42:59.134708+0000","last_undegraded":"2024-09-10T20:42:59.134708+0000","last_fullsized":"2024-09-10T20:42:59.134708+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:25:16.588271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017990+0000","last_change":"2024-09-10T20:42:57.023074+0000","last_active":"2024-09-10T20:42:58.017990+0000","last_peered":"2024-09-10T20:42:58.017990+0000","last_clean":"2024-09-10T20:42:58.017990+0000","last_became_active":"2024-09-10T20:42:57.020610+0000","last_became_peered":"2024-09-10T20:42:57.020610+0000","last_unstale":"2024-09-10T20:42:58.017990+0000","last_undegraded":"2024-09-10T20:42:58.017990+0000","last_fullsized":"2024-09-10T20:42:58.017990+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:00.371168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016201+0000","last_change":"2024-09-10T20:42:57.015140+0000","last_active":"2024-09-10T20:42:58.016201+0000","last_peered":"2024-09-10T20:42:58.016201+0000","last_clean":"2024-09-10T20:42:58.016201+0000","last_became_active":"2024-09-10T20:42:57.014823+0000","last_became_peered":"2024-09-10T20:42:57.014823+0000","last_unstale":"2024-09-10T20:42:58.016201+0000","last_undegraded":"2024-09-10T20:42:58.016201+0000","last_fullsized":"2024-09-10T20:42:58.016201+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:44.091183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016207+0000","last_change":"2024-09-10T20:42:57.022365+0000","last_active":"2024-09-10T20:42:58.016207+0000","last_peered":"2024-09-10T20:42:58.016207+0000","last_clean":"2024-09-10T20:42:58.016207+0000","last_became_active":"2024-09-10T20:42:57.021944+0000","last_became_peered":"2024-09-10T20:42:57.021944+0000","last_unstale":"2024-09-10T20:42:58.016207+0000","last_undegraded":"2024-09-10T20:42:58.016207+0000","last_fullsized":"2024-09-10T20:42:58.016207+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:48:06.357735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633926+0000","last_change":"2024-09-10T20:42:57.033265+0000","last_active":"2024-09-10T20:42:58.633926+0000","last_peered":"2024-09-10T20:42:58.633926+0000","last_clean":"2024-09-10T20:42:58.633926+0000","last_became_active":"2024-09-10T20:42:57.032956+0000","last_became_peered":"2024-09-10T20:42:57.032956+0000","last_unstale":"2024-09-10T20:42:58.633926+0000","last_undegraded":"2024-09-10T20:42:58.633926+0000","last_fullsized":"2024-09-10T20:42:58.633926+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:58.394937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134622+0000","last_change":"2024-09-10T20:42:57.029340+0000","last_active":"2024-09-10T20:42:59.134622+0000","last_peered":"2024-09-10T20:42:59.134622+0000","last_clean":"2024-09-10T20:42:59.134622+0000","last_became_active":"2024-09-10T20:42:57.029072+0000","last_became_peered":"2024-09-10T20:42:57.029072+0000","last_unstale":"2024-09-10T20:42:59.134622+0000","last_undegraded":"2024-09-10T20:42:59.134622+0000","last_fullsized":"2024-09-10T20:42:59.134622+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:34:01.259370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016517+0000","last_change":"2024-09-10T20:42:57.015630+0000","last_active":"2024-09-10T20:42:58.016517+0000","last_peered":"2024-09-10T20:42:58.016517+0000","last_clean":"2024-09-10T20:42:58.016517+0000","last_became_active":"2024-09-10T20:42:57.015356+0000","last_became_peered":"2024-09-10T20:42:57.015356+0000","last_unstale":"2024-09-10T20:42:58.016517+0000","last_undegraded":"2024-09-10T20:42:58.016517+0000","last_fullsized":"2024-09-10T20:42:58.016517+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:24.561913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134746+0000","last_change":"2024-09-10T20:42:57.025852+0000","last_active":"2024-09-10T20:42:59.134746+0000","last_peered":"2024-09-10T20:42:59.134746+0000","last_clean":"2024-09-10T20:42:59.134746+0000","last_became_active":"2024-09-10T20:42:57.025576+0000","last_became_peered":"2024-09-10T20:42:57.025576+0000","last_unstale":"2024-09-10T20:42:59.134746+0000","last_undegraded":"2024-09-10T20:42:59.134746+0000","last_fullsized":"2024-09-10T20:42:59.134746+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:47:57.927740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776413+0000","last_change":"2024-09-10T20:42:57.026175+0000","last_active":"2024-09-10T20:42:58.776413+0000","last_peered":"2024-09-10T20:42:58.776413+0000","last_clean":"2024-09-10T20:42:58.776413+0000","last_became_active":"2024-09-10T20:42:57.025856+0000","last_became_peered":"2024-09-10T20:42:57.025856+0000","last_unstale":"2024-09-10T20:42:58.776413+0000","last_undegraded":"2024-09-10T20:42:58.776413+0000","last_fullsized":"2024-09-10T20:42:58.776413+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:35:51.057064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133807+0000","last_change":"2024-09-10T20:42:57.020483+0000","last_active":"2024-09-10T20:42:59.133807+0000","last_peered":"2024-09-10T20:42:59.133807+0000","last_clean":"2024-09-10T20:42:59.133807+0000","last_became_active":"2024-09-10T20:42:57.020183+0000","last_became_peered":"2024-09-10T20:42:57.020183+0000","last_unstale":"2024-09-10T20:42:59.133807+0000","last_undegraded":"2024-09-10T20:42:59.133807+0000","last_fullsized":"2024-09-10T20:42:59.133807+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:54.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633437+0000","last_change":"2024-09-10T20:42:57.026600+0000","last_active":"2024-09-10T20:42:58.633437+0000","last_peered":"2024-09-10T20:42:58.633437+0000","last_clean":"2024-09-10T20:42:58.633437+0000","last_became_active":"2024-09-10T20:42:57.026275+0000","last_became_peered":"2024-09-10T20:42:57.026275+0000","last_unstale":"2024-09-10T20:42:58.633437+0000","last_undegraded":"2024-09-10T20:42:58.633437+0000","last_fullsized":"2024-09-10T20:42:58.633437+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:09.268660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134157+0000","last_change":"2024-09-10T20:42:57.032272+0000","last_active":"2024-09-10T20:42:59.134157+0000","last_peered":"2024-09-10T20:42:59.134157+0000","last_clean":"2024-09-10T20:42:59.134157+0000","last_became_active":"2024-09-10T20:42:57.031990+0000","last_became_peered":"2024-09-10T20:42:57.031990+0000","last_unstale":"2024-09-10T20:42:59.134157+0000","last_undegraded":"2024-09-10T20:42:59.134157+0000","last_fullsized":"2024-09-10T20:42:59.134157+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:26.631733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081997+0000","last_change":"2024-09-10T20:42:57.021382+0000","last_active":"2024-09-10T20:42:59.081997+0000","last_peered":"2024-09-10T20:42:59.081997+0000","last_clean":"2024-09-10T20:42:59.081997+0000","last_became_active":"2024-09-10T20:42:57.021091+0000","last_became_peered":"2024-09-10T20:42:57.021091+0000","last_unstale":"2024-09-10T20:42:59.081997+0000","last_undegraded":"2024-09-10T20:42:59.081997+0000","last_fullsized":"2024-09-10T20:42:59.081997+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:54.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016618+0000","last_change":"2024-09-10T20:42:57.028934+0000","last_active":"2024-09-10T20:42:58.016618+0000","last_peered":"2024-09-10T20:42:58.016618+0000","last_clean":"2024-09-10T20:42:58.016618+0000","last_became_active":"2024-09-10T20:42:57.028646+0000","last_became_peered":"2024-09-10T20:42:57.028646+0000","last_unstale":"2024-09-10T20:42:58.016618+0000","last_undegraded":"2024-09-10T20:42:58.016618+0000","last_fullsized":"2024-09-10T20:42:58.016618+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:22.269035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.082076+0000","last_change":"2024-09-10T20:42:57.027453+0000","last_active":"2024-09-10T20:42:59.082076+0000","last_peered":"2024-09-10T20:42:59.082076+0000","last_clean":"2024-09-10T20:42:59.082076+0000","last_became_active":"2024-09-10T20:42:57.026840+0000","last_became_peered":"2024-09-10T20:42:57.026840+0000","last_unstale":"2024-09-10T20:42:59.082076+0000","last_undegraded":"2024-09-10T20:42:59.082076+0000","last_fullsized":"2024-09-10T20:42:59.082076+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:01.914016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081702+0000","last_change":"2024-09-10T20:42:57.028940+0000","last_active":"2024-09-10T20:42:59.081702+0000","last_peered":"2024-09-10T20:42:59.081702+0000","last_clean":"2024-09-10T20:42:59.081702+0000","last_became_active":"2024-09-10T20:42:57.028656+0000","last_became_peered":"2024-09-10T20:42:57.028656+0000","last_unstale":"2024-09-10T20:42:59.081702+0000","last_undegraded":"2024-09-10T20:42:59.081702+0000","last_fullsized":"2024-09-10T20:42:59.081702+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:26.529675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016480+0000","last_change":"2024-09-10T20:42:57.021346+0000","last_active":"2024-09-10T20:42:58.016480+0000","last_peered":"2024-09-10T20:42:58.016480+0000","last_clean":"2024-09-10T20:42:58.016480+0000","last_became_active":"2024-09-10T20:42:57.021039+0000","last_became_peered":"2024-09-10T20:42:57.021039+0000","last_unstale":"2024-09-10T20:42:58.016480+0000","last_undegraded":"2024-09-10T20:42:58.016480+0000","last_fullsized":"2024-09-10T20:42:58.016480+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:50:32.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018184+0000","last_change":"2024-09-10T20:42:57.022255+0000","last_active":"2024-09-10T20:42:58.018184+0000","last_peered":"2024-09-10T20:42:58.018184+0000","last_clean":"2024-09-10T20:42:58.018184+0000","last_became_active":"2024-09-10T20:42:57.021985+0000","last_became_peered":"2024-09-10T20:42:57.021985+0000","last_unstale":"2024-09-10T20:42:58.018184+0000","last_undegraded":"2024-09-10T20:42:58.018184+0000","last_fullsized":"2024-09-10T20:42:58.018184+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:29:04.710331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134194+0000","last_change":"2024-09-10T20:42:57.024037+0000","last_active":"2024-09-10T20:42:59.134194+0000","last_peered":"2024-09-10T20:42:59.134194+0000","last_clean":"2024-09-10T20:42:59.134194+0000","last_became_active":"2024-09-10T20:42:57.023736+0000","last_became_peered":"2024-09-10T20:42:57.023736+0000","last_unstale":"2024-09-10T20:42:59.134194+0000","last_undegraded":"2024-09-10T20:42:59.134194+0000","last_fullsized":"2024-09-10T20:42:59.134194+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:55.620256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019012+0000","last_change":"2024-09-10T20:42:57.026621+0000","last_active":"2024-09-10T20:42:58.019012+0000","last_peered":"2024-09-10T20:42:58.019012+0000","last_clean":"2024-09-10T20:42:58.019012+0000","last_became_active":"2024-09-10T20:42:57.026323+0000","last_became_peered":"2024-09-10T20:42:57.026323+0000","last_unstale":"2024-09-10T20:42:58.019012+0000","last_undegraded":"2024-09-10T20:42:58.019012+0000","last_fullsized":"2024-09-10T20:42:58.019012+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:26.316832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016558+0000","last_change":"2024-09-10T20:42:57.023093+0000","last_active":"2024-09-10T20:42:58.016558+0000","last_peered":"2024-09-10T20:42:58.016558+0000","last_clean":"2024-09-10T20:42:58.016558+0000","last_became_active":"2024-09-10T20:42:57.022820+0000","last_became_peered":"2024-09-10T20:42:57.022820+0000","last_unstale":"2024-09-10T20:42:58.016558+0000","last_undegraded":"2024-09-10T20:42:58.016558+0000","last_fullsized":"2024-09-10T20:42:58.016558+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:48:55.929464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018281+0000","last_change":"2024-09-10T20:42:57.024636+0000","last_active":"2024-09-10T20:42:58.018281+0000","last_peered":"2024-09-10T20:42:58.018281+0000","last_clean":"2024-09-10T20:42:58.018281+0000","last_became_active":"2024-09-10T20:42:57.024367+0000","last_became_peered":"2024-09-10T20:42:57.024367+0000","last_unstale":"2024-09-10T20:42:58.018281+0000","last_undegraded":"2024-09-10T20:42:58.018281+0000","last_fullsized":"2024-09-10T20:42:58.018281+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:45.916803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019735+0000","last_change":"2024-09-10T20:42:57.025201+0000","last_active":"2024-09-10T20:42:58.019735+0000","last_peered":"2024-09-10T20:42:58.019735+0000","last_clean":"2024-09-10T20:42:58.019735+0000","last_became_active":"2024-09-10T20:42:57.024893+0000","last_became_peered":"2024-09-10T20:42:57.024893+0000","last_unstale":"2024-09-10T20:42:58.019735+0000","last_undegraded":"2024-09-10T20:42:58.019735+0000","last_fullsized":"2024-09-10T20:42:58.019735+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:14.838329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018291+0000","last_change":"2024-09-10T20:42:57.021572+0000","last_active":"2024-09-10T20:42:58.018291+0000","last_peered":"2024-09-10T20:42:58.018291+0000","last_clean":"2024-09-10T20:42:58.018291+0000","last_became_active":"2024-09-10T20:42:57.021305+0000","last_became_peered":"2024-09-10T20:42:57.021305+0000","last_unstale":"2024-09-10T20:42:58.018291+0000","last_undegraded":"2024-09-10T20:42:58.018291+0000","last_fullsized":"2024-09-10T20:42:58.018291+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:07.665375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134103+0000","last_change":"2024-09-10T20:42:57.017779+0000","last_active":"2024-09-10T20:42:59.134103+0000","last_peered":"2024-09-10T20:42:59.134103+0000","last_clean":"2024-09-10T20:42:59.134103+0000","last_became_active":"2024-09-10T20:42:57.017403+0000","last_became_peered":"2024-09-10T20:42:57.017403+0000","last_unstale":"2024-09-10T20:42:59.134103+0000","last_undegraded":"2024-09-10T20:42:59.134103+0000","last_fullsized":"2024-09-10T20:42:59.134103+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:11:15.214431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016317+0000","last_change":"2024-09-10T20:42:57.017619+0000","last_active":"2024-09-10T20:42:58.016317+0000","last_peered":"2024-09-10T20:42:58.016317+0000","last_clean":"2024-09-10T20:42:58.016317+0000","last_became_active":"2024-09-10T20:42:57.017337+0000","last_became_peered":"2024-09-10T20:42:57.017337+0000","last_unstale":"2024-09-10T20:42:58.016317+0000","last_undegraded":"2024-09-10T20:42:58.016317+0000","last_fullsized":"2024-09-10T20:42:58.016317+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:43:59.161863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633565+0000","last_change":"2024-09-10T20:42:57.028933+0000","last_active":"2024-09-10T20:42:58.633565+0000","last_peered":"2024-09-10T20:42:58.633565+0000","last_clean":"2024-09-10T20:42:58.633565+0000","last_became_active":"2024-09-10T20:42:57.028678+0000","last_became_peered":"2024-09-10T20:42:57.028678+0000","last_unstale":"2024-09-10T20:42:58.633565+0000","last_undegraded":"2024-09-10T20:42:58.633565+0000","last_fullsized":"2024-09-10T20:42:58.633565+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:32:00.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134144+0000","last_change":"2024-09-10T20:42:57.033433+0000","last_active":"2024-09-10T20:42:59.134144+0000","last_peered":"2024-09-10T20:42:59.134144+0000","last_clean":"2024-09-10T20:42:59.134144+0000","last_became_active":"2024-09-10T20:42:57.032798+0000","last_became_peered":"2024-09-10T20:42:57.032798+0000","last_unstale":"2024-09-10T20:42:59.134144+0000","last_undegraded":"2024-09-10T20:42:59.134144+0000","last_fullsized":"2024-09-10T20:42:59.134144+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:41.069604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633829+0000","last_change":"2024-09-10T20:42:57.033613+0000","last_active":"2024-09-10T20:42:58.633829+0000","last_peered":"2024-09-10T20:42:58.633829+0000","last_clean":"2024-09-10T20:42:58.633829+0000","last_became_active":"2024-09-10T20:42:57.033083+0000","last_became_peered":"2024-09-10T20:42:57.033083+0000","last_unstale":"2024-09-10T20:42:58.633829+0000","last_undegraded":"2024-09-10T20:42:58.633829+0000","last_fullsized":"2024-09-10T20:42:58.633829+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:55.535737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"99'4","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134230+0000","last_change":"2024-09-10T20:42:56.009858+0000","last_active":"2024-09-10T20:42:59.134230+0000","last_peered":"2024-09-10T20:42:59.134230+0000","last_clean":"2024-09-10T20:42:59.134230+0000","last_became_active":"2024-09-10T20:42:56.009575+0000","last_became_peered":"2024-09-10T20:42:56.009575+0000","last_unstale":"2024-09-10T20:42:59.134230+0000","last_undegraded":"2024-09-10T20:42:59.134230+0000","last_fullsized":"2024-09-10T20:42:59.134230+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:21.364713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134245+0000","last_change":"2024-09-10T20:42:57.027695+0000","last_active":"2024-09-10T20:42:59.134245+0000","last_peered":"2024-09-10T20:42:59.134245+0000","last_clean":"2024-09-10T20:42:59.134245+0000","last_became_active":"2024-09-10T20:42:57.026324+0000","last_became_peered":"2024-09-10T20:42:57.026324+0000","last_unstale":"2024-09-10T20:42:59.134245+0000","last_undegraded":"2024-09-10T20:42:59.134245+0000","last_fullsized":"2024-09-10T20:42:59.134245+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:01.246774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776692+0000","last_change":"2024-09-10T20:42:56.014671+0000","last_active":"2024-09-10T20:42:58.776692+0000","last_peered":"2024-09-10T20:42:58.776692+0000","last_clean":"2024-09-10T20:42:58.776692+0000","last_became_active":"2024-09-10T20:42:56.014382+0000","last_became_peered":"2024-09-10T20:42:56.014382+0000","last_unstale":"2024-09-10T20:42:58.776692+0000","last_undegraded":"2024-09-10T20:42:58.776692+0000","last_fullsized":"2024-09-10T20:42:58.776692+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:26.193994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018347+0000","last_change":"2024-09-10T20:42:57.023790+0000","last_active":"2024-09-10T20:42:58.018347+0000","last_peered":"2024-09-10T20:42:58.018347+0000","last_clean":"2024-09-10T20:42:58.018347+0000","last_became_active":"2024-09-10T20:42:57.023519+0000","last_became_peered":"2024-09-10T20:42:57.023519+0000","last_unstale":"2024-09-10T20:42:58.018347+0000","last_undegraded":"2024-09-10T20:42:58.018347+0000","last_fullsized":"2024-09-10T20:42:58.018347+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:29:25.621518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1d","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019791+0000","last_change":"2024-09-10T20:42:56.013868+0000","last_active":"2024-09-10T20:42:58.019791+0000","last_peered":"2024-09-10T20:42:58.019791+0000","last_clean":"2024-09-10T20:42:58.019791+0000","last_became_active":"2024-09-10T20:42:56.013535+0000","last_became_peered":"2024-09-10T20:42:56.013535+0000","last_unstale":"2024-09-10T20:42:58.019791+0000","last_undegraded":"2024-09-10T20:42:58.019791+0000","last_fullsized":"2024-09-10T20:42:58.019791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:37.865884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081947+0000","last_change":"2024-09-10T20:42:57.027102+0000","last_active":"2024-09-10T20:42:59.081947+0000","last_peered":"2024-09-10T20:42:59.081947+0000","last_clean":"2024-09-10T20:42:59.081947+0000","last_became_active":"2024-09-10T20:42:57.026760+0000","last_became_peered":"2024-09-10T20:42:57.026760+0000","last_unstale":"2024-09-10T20:42:59.081947+0000","last_undegraded":"2024-09-10T20:42:59.081947+0000","last_fullsized":"2024-09-10T20:42:59.081947+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:35.809111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019517+0000","last_change":"2024-09-10T20:42:56.008875+0000","last_active":"2024-09-10T20:42:58.019517+0000","last_peered":"2024-09-10T20:42:58.019517+0000","last_clean":"2024-09-10T20:42:58.019517+0000","last_became_active":"2024-09-10T20:42:56.008318+0000","last_became_peered":"2024-09-10T20:42:56.008318+0000","last_unstale":"2024-09-10T20:42:58.019517+0000","last_undegraded":"2024-09-10T20:42:58.019517+0000","last_fullsized":"2024-09-10T20:42:58.019517+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:24:47.050024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019902+0000","last_change":"2024-09-10T20:42:57.025899+0000","last_active":"2024-09-10T20:42:58.019902+0000","last_peered":"2024-09-10T20:42:58.019902+0000","last_clean":"2024-09-10T20:42:58.019902+0000","last_became_active":"2024-09-10T20:42:57.025642+0000","last_became_peered":"2024-09-10T20:42:57.025642+0000","last_unstale":"2024-09-10T20:42:58.019902+0000","last_undegraded":"2024-09-10T20:42:58.019902+0000","last_fullsized":"2024-09-10T20:42:58.019902+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:20.814654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1b","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081602+0000","last_change":"2024-09-10T20:42:56.016158+0000","last_active":"2024-09-10T20:42:59.081602+0000","last_peered":"2024-09-10T20:42:59.081602+0000","last_clean":"2024-09-10T20:42:59.081602+0000","last_became_active":"2024-09-10T20:42:56.015793+0000","last_became_peered":"2024-09-10T20:42:56.015793+0000","last_unstale":"2024-09-10T20:42:59.081602+0000","last_undegraded":"2024-09-10T20:42:59.081602+0000","last_fullsized":"2024-09-10T20:42:59.081602+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:39:09.435764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016528+0000","last_change":"2024-09-10T20:42:57.031474+0000","last_active":"2024-09-10T20:42:58.016528+0000","last_peered":"2024-09-10T20:42:58.016528+0000","last_clean":"2024-09-10T20:42:58.016528+0000","last_became_active":"2024-09-10T20:42:57.031188+0000","last_became_peered":"2024-09-10T20:42:57.031188+0000","last_unstale":"2024-09-10T20:42:58.016528+0000","last_undegraded":"2024-09-10T20:42:58.016528+0000","last_fullsized":"2024-09-10T20:42:58.016528+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:08:21.427198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633351+0000","last_change":"2024-09-10T20:42:56.012320+0000","last_active":"2024-09-10T20:42:58.633351+0000","last_peered":"2024-09-10T20:42:58.633351+0000","last_clean":"2024-09-10T20:42:58.633351+0000","last_became_active":"2024-09-10T20:42:56.012048+0000","last_became_peered":"2024-09-10T20:42:56.012048+0000","last_unstale":"2024-09-10T20:42:58.633351+0000","last_undegraded":"2024-09-10T20:42:58.633351+0000","last_fullsized":"2024-09-10T20:42:58.633351+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:44.676084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134593+0000","last_change":"2024-09-10T20:42:57.028906+0000","last_active":"2024-09-10T20:42:59.134593+0000","last_peered":"2024-09-10T20:42:59.134593+0000","last_clean":"2024-09-10T20:42:59.134593+0000","last_became_active":"2024-09-10T20:42:57.028622+0000","last_became_peered":"2024-09-10T20:42:57.028622+0000","last_unstale":"2024-09-10T20:42:59.134593+0000","last_undegraded":"2024-09-10T20:42:59.134593+0000","last_fullsized":"2024-09-10T20:42:59.134593+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:58:41.414994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016315+0000","last_change":"2024-09-10T20:42:56.014432+0000","last_active":"2024-09-10T20:42:58.016315+0000","last_peered":"2024-09-10T20:42:58.016315+0000","last_clean":"2024-09-10T20:42:58.016315+0000","last_became_active":"2024-09-10T20:42:56.014057+0000","last_became_peered":"2024-09-10T20:42:56.014057+0000","last_unstale":"2024-09-10T20:42:58.016315+0000","last_undegraded":"2024-09-10T20:42:58.016315+0000","last_fullsized":"2024-09-10T20:42:58.016315+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:09:35.683673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016215+0000","last_change":"2024-09-10T20:42:57.022290+0000","last_active":"2024-09-10T20:42:58.016215+0000","last_peered":"2024-09-10T20:42:58.016215+0000","last_clean":"2024-09-10T20:42:58.016215+0000","last_became_active":"2024-09-10T20:42:57.022006+0000","last_became_peered":"2024-09-10T20:42:57.022006+0000","last_unstale":"2024-09-10T20:42:58.016215+0000","last_undegraded":"2024-09-10T20:42:58.016215+0000","last_fullsized":"2024-09-10T20:42:58.016215+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:24.790256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134423+0000","last_change":"2024-09-10T20:42:56.015903+0000","last_active":"2024-09-10T20:42:59.134423+0000","last_peered":"2024-09-10T20:42:59.134423+0000","last_clean":"2024-09-10T20:42:59.134423+0000","last_became_active":"2024-09-10T20:42:56.015576+0000","last_became_peered":"2024-09-10T20:42:56.015576+0000","last_unstale":"2024-09-10T20:42:59.134423+0000","last_undegraded":"2024-09-10T20:42:59.134423+0000","last_fullsized":"2024-09-10T20:42:59.134423+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:45:42.481827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776639+0000","last_change":"2024-09-10T20:42:57.024977+0000","last_active":"2024-09-10T20:42:58.776639+0000","last_peered":"2024-09-10T20:42:58.776639+0000","last_clean":"2024-09-10T20:42:58.776639+0000","last_became_active":"2024-09-10T20:42:57.024638+0000","last_became_peered":"2024-09-10T20:42:57.024638+0000","last_unstale":"2024-09-10T20:42:58.776639+0000","last_undegraded":"2024-09-10T20:42:58.776639+0000","last_fullsized":"2024-09-10T20:42:58.776639+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:55.620256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019894+0000","last_change":"2024-09-10T20:42:56.014432+0000","last_active":"2024-09-10T20:42:58.019894+0000","last_peered":"2024-09-10T20:42:58.019894+0000","last_clean":"2024-09-10T20:42:58.019894+0000","last_became_active":"2024-09-10T20:42:56.014170+0000","last_became_peered":"2024-09-10T20:42:56.014170+0000","last_unstale":"2024-09-10T20:42:58.019894+0000","last_undegraded":"2024-09-10T20:42:58.019894+0000","last_fullsized":"2024-09-10T20:42:58.019894+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:49.280790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776608+0000","last_change":"2024-09-10T20:42:57.025302+0000","last_active":"2024-09-10T20:42:58.776608+0000","last_peered":"2024-09-10T20:42:58.776608+0000","last_clean":"2024-09-10T20:42:58.776608+0000","last_became_active":"2024-09-10T20:42:57.025012+0000","last_became_peered":"2024-09-10T20:42:57.025012+0000","last_unstale":"2024-09-10T20:42:58.776608+0000","last_undegraded":"2024-09-10T20:42:58.776608+0000","last_fullsized":"2024-09-10T20:42:58.776608+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:31.854297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134410+0000","last_change":"2024-09-10T20:42:56.016940+0000","last_active":"2024-09-10T20:42:59.134410+0000","last_peered":"2024-09-10T20:42:59.134410+0000","last_clean":"2024-09-10T20:42:59.134410+0000","last_became_active":"2024-09-10T20:42:56.016653+0000","last_became_peered":"2024-09-10T20:42:56.016653+0000","last_unstale":"2024-09-10T20:42:59.134410+0000","last_undegraded":"2024-09-10T20:42:59.134410+0000","last_fullsized":"2024-09-10T20:42:59.134410+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:03.974365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.018454+0000","last_change":"2024-09-10T20:42:57.026750+0000","last_active":"2024-09-10T20:42:58.018454+0000","last_peered":"2024-09-10T20:42:58.018454+0000","last_clean":"2024-09-10T20:42:58.018454+0000","last_became_active":"2024-09-10T20:42:57.026411+0000","last_became_peered":"2024-09-10T20:42:57.026411+0000","last_unstale":"2024-09-10T20:42:58.018454+0000","last_undegraded":"2024-09-10T20:42:58.018454+0000","last_fullsized":"2024-09-10T20:42:58.018454+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:10.443908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.15","version":"99'3","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134549+0000","last_change":"2024-09-10T20:42:56.012469+0000","last_active":"2024-09-10T20:42:59.134549+0000","last_peered":"2024-09-10T20:42:59.134549+0000","last_clean":"2024-09-10T20:42:59.134549+0000","last_became_active":"2024-09-10T20:42:56.012115+0000","last_became_peered":"2024-09-10T20:42:56.012115+0000","last_unstale":"2024-09-10T20:42:59.134549+0000","last_undegraded":"2024-09-10T20:42:59.134549+0000","last_fullsized":"2024-09-10T20:42:59.134549+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:36.260836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016719+0000","last_change":"2024-09-10T20:42:57.022131+0000","last_active":"2024-09-10T20:42:58.016719+0000","last_peered":"2024-09-10T20:42:58.016719+0000","last_clean":"2024-09-10T20:42:58.016719+0000","last_became_active":"2024-09-10T20:42:57.021851+0000","last_became_peered":"2024-09-10T20:42:57.021851+0000","last_unstale":"2024-09-10T20:42:58.016719+0000","last_undegraded":"2024-09-10T20:42:58.016719+0000","last_fullsized":"2024-09-10T20:42:58.016719+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:44:58.275204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"99'6","reported_seq":21,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019684+0000","last_change":"2024-09-10T20:42:56.008582+0000","last_active":"2024-09-10T20:42:58.019684+0000","last_peered":"2024-09-10T20:42:58.019684+0000","last_clean":"2024-09-10T20:42:58.019684+0000","last_became_active":"2024-09-10T20:42:56.007992+0000","last_became_peered":"2024-09-10T20:42:56.007992+0000","last_unstale":"2024-09-10T20:42:58.019684+0000","last_undegraded":"2024-09-10T20:42:58.019684+0000","last_fullsized":"2024-09-10T20:42:58.019684+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:03:07.367121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134392+0000","last_change":"2024-09-10T20:42:57.031509+0000","last_active":"2024-09-10T20:42:59.134392+0000","last_peered":"2024-09-10T20:42:59.134392+0000","last_clean":"2024-09-10T20:42:59.134392+0000","last_became_active":"2024-09-10T20:42:57.031222+0000","last_became_peered":"2024-09-10T20:42:57.031222+0000","last_unstale":"2024-09-10T20:42:59.134392+0000","last_undegraded":"2024-09-10T20:42:59.134392+0000","last_fullsized":"2024-09-10T20:42:59.134392+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:49:30.963522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134558+0000","last_change":"2024-09-10T20:42:56.013564+0000","last_active":"2024-09-10T20:42:59.134558+0000","last_peered":"2024-09-10T20:42:59.134558+0000","last_clean":"2024-09-10T20:42:59.134558+0000","last_became_active":"2024-09-10T20:42:56.013247+0000","last_became_peered":"2024-09-10T20:42:56.013247+0000","last_unstale":"2024-09-10T20:42:59.134558+0000","last_undegraded":"2024-09-10T20:42:59.134558+0000","last_fullsized":"2024-09-10T20:42:59.134558+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:30.931318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019323+0000","last_change":"2024-09-10T20:42:57.025508+0000","last_active":"2024-09-10T20:42:58.019323+0000","last_peered":"2024-09-10T20:42:58.019323+0000","last_clean":"2024-09-10T20:42:58.019323+0000","last_became_active":"2024-09-10T20:42:57.025046+0000","last_became_peered":"2024-09-10T20:42:57.025046+0000","last_unstale":"2024-09-10T20:42:58.019323+0000","last_undegraded":"2024-09-10T20:42:58.019323+0000","last_fullsized":"2024-09-10T20:42:58.019323+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:37.933369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081772+0000","last_change":"2024-09-10T20:42:56.018247+0000","last_active":"2024-09-10T20:42:59.081772+0000","last_peered":"2024-09-10T20:42:59.081772+0000","last_clean":"2024-09-10T20:42:59.081772+0000","last_became_active":"2024-09-10T20:42:56.017976+0000","last_became_peered":"2024-09-10T20:42:56.017976+0000","last_unstale":"2024-09-10T20:42:59.081772+0000","last_undegraded":"2024-09-10T20:42:59.081772+0000","last_fullsized":"2024-09-10T20:42:59.081772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:29.699395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019205+0000","last_change":"2024-09-10T20:42:57.022140+0000","last_active":"2024-09-10T20:42:58.019205+0000","last_peered":"2024-09-10T20:42:58.019205+0000","last_clean":"2024-09-10T20:42:58.019205+0000","last_became_active":"2024-09-10T20:42:57.021872+0000","last_became_peered":"2024-09-10T20:42:57.021872+0000","last_unstale":"2024-09-10T20:42:58.019205+0000","last_undegraded":"2024-09-10T20:42:58.019205+0000","last_fullsized":"2024-09-10T20:42:58.019205+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:34.653250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"99'8","reported_seq":23,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633183+0000","last_change":"2024-09-10T20:42:56.014545+0000","last_active":"2024-09-10T20:42:58.633183+0000","last_peered":"2024-09-10T20:42:58.633183+0000","last_clean":"2024-09-10T20:42:58.633183+0000","last_became_active":"2024-09-10T20:42:56.014252+0000","last_became_peered":"2024-09-10T20:42:56.014252+0000","last_unstale":"2024-09-10T20:42:58.633183+0000","last_undegraded":"2024-09-10T20:42:58.633183+0000","last_fullsized":"2024-09-10T20:42:58.633183+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:25.524428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633896+0000","last_change":"2024-09-10T20:42:57.025794+0000","last_active":"2024-09-10T20:42:58.633896+0000","last_peered":"2024-09-10T20:42:58.633896+0000","last_clean":"2024-09-10T20:42:58.633896+0000","last_became_active":"2024-09-10T20:42:57.025518+0000","last_became_peered":"2024-09-10T20:42:57.025518+0000","last_unstale":"2024-09-10T20:42:58.633896+0000","last_undegraded":"2024-09-10T20:42:58.633896+0000","last_fullsized":"2024-09-10T20:42:58.633896+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:38.706038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.f","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017886+0000","last_change":"2024-09-10T20:42:56.012088+0000","last_active":"2024-09-10T20:42:58.017886+0000","last_peered":"2024-09-10T20:42:58.017886+0000","last_clean":"2024-09-10T20:42:58.017886+0000","last_became_active":"2024-09-10T20:42:56.011823+0000","last_became_peered":"2024-09-10T20:42:56.011823+0000","last_unstale":"2024-09-10T20:42:58.017886+0000","last_undegraded":"2024-09-10T20:42:58.017886+0000","last_fullsized":"2024-09-10T20:42:58.017886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:44.845617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.634005+0000","last_change":"2024-09-10T20:42:57.032358+0000","last_active":"2024-09-10T20:42:58.634005+0000","last_peered":"2024-09-10T20:42:58.634005+0000","last_clean":"2024-09-10T20:42:58.634005+0000","last_became_active":"2024-09-10T20:42:57.032057+0000","last_became_peered":"2024-09-10T20:42:57.032057+0000","last_unstale":"2024-09-10T20:42:58.634005+0000","last_undegraded":"2024-09-10T20:42:58.634005+0000","last_fullsized":"2024-09-10T20:42:58.634005+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:01:47.967682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.e","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134529+0000","last_change":"2024-09-10T20:42:56.012920+0000","last_active":"2024-09-10T20:42:59.134529+0000","last_peered":"2024-09-10T20:42:59.134529+0000","last_clean":"2024-09-10T20:42:59.134529+0000","last_became_active":"2024-09-10T20:42:56.012535+0000","last_became_peered":"2024-09-10T20:42:56.012535+0000","last_unstale":"2024-09-10T20:42:59.134529+0000","last_undegraded":"2024-09-10T20:42:59.134529+0000","last_fullsized":"2024-09-10T20:42:59.134529+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:33.459103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016481+0000","last_change":"2024-09-10T20:42:57.029859+0000","last_active":"2024-09-10T20:42:58.016481+0000","last_peered":"2024-09-10T20:42:58.016481+0000","last_clean":"2024-09-10T20:42:58.016481+0000","last_became_active":"2024-09-10T20:42:57.029524+0000","last_became_peered":"2024-09-10T20:42:57.029524+0000","last_unstale":"2024-09-10T20:42:58.016481+0000","last_undegraded":"2024-09-10T20:42:58.016481+0000","last_fullsized":"2024-09-10T20:42:58.016481+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:55:07.349407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"99'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134194+0000","last_change":"2024-09-10T20:42:56.003965+0000","last_active":"2024-09-10T20:42:59.134194+0000","last_peered":"2024-09-10T20:42:59.134194+0000","last_clean":"2024-09-10T20:42:59.134194+0000","last_became_active":"2024-09-10T20:42:56.003671+0000","last_became_peered":"2024-09-10T20:42:56.003671+0000","last_unstale":"2024-09-10T20:42:59.134194+0000","last_undegraded":"2024-09-10T20:42:59.134194+0000","last_fullsized":"2024-09-10T20:42:59.134194+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:04:54.457764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134100+0000","last_change":"2024-09-10T20:42:57.021217+0000","last_active":"2024-09-10T20:42:59.134100+0000","last_peered":"2024-09-10T20:42:59.134100+0000","last_clean":"2024-09-10T20:42:59.134100+0000","last_became_active":"2024-09-10T20:42:57.020799+0000","last_became_peered":"2024-09-10T20:42:57.020799+0000","last_unstale":"2024-09-10T20:42:59.134100+0000","last_undegraded":"2024-09-10T20:42:59.134100+0000","last_fullsized":"2024-09-10T20:42:59.134100+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:35:51.057064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133940+0000","last_change":"2024-09-10T20:42:56.002469+0000","last_active":"2024-09-10T20:42:59.133940+0000","last_peered":"2024-09-10T20:42:59.133940+0000","last_clean":"2024-09-10T20:42:59.133940+0000","last_became_active":"2024-09-10T20:42:56.002182+0000","last_became_peered":"2024-09-10T20:42:56.002182+0000","last_unstale":"2024-09-10T20:42:59.133940+0000","last_undegraded":"2024-09-10T20:42:59.133940+0000","last_fullsized":"2024-09-10T20:42:59.133940+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:15:27.829406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081633+0000","last_change":"2024-09-10T20:42:57.024378+0000","last_active":"2024-09-10T20:42:59.081633+0000","last_peered":"2024-09-10T20:42:59.081633+0000","last_clean":"2024-09-10T20:42:59.081633+0000","last_became_active":"2024-09-10T20:42:57.024104+0000","last_became_peered":"2024-09-10T20:42:57.024104+0000","last_unstale":"2024-09-10T20:42:59.081633+0000","last_undegraded":"2024-09-10T20:42:59.081633+0000","last_fullsized":"2024-09-10T20:42:59.081633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:53.388782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134666+0000","last_change":"2024-09-10T20:42:56.016899+0000","last_active":"2024-09-10T20:42:59.134666+0000","last_peered":"2024-09-10T20:42:59.134666+0000","last_clean":"2024-09-10T20:42:59.134666+0000","last_became_active":"2024-09-10T20:42:56.016591+0000","last_became_peered":"2024-09-10T20:42:56.016591+0000","last_unstale":"2024-09-10T20:42:59.134666+0000","last_undegraded":"2024-09-10T20:42:59.134666+0000","last_fullsized":"2024-09-10T20:42:59.134666+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:38:00.903905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633644+0000","last_change":"2024-09-10T20:42:57.027594+0000","last_active":"2024-09-10T20:42:58.633644+0000","last_peered":"2024-09-10T20:42:58.633644+0000","last_clean":"2024-09-10T20:42:58.633644+0000","last_became_active":"2024-09-10T20:42:57.027298+0000","last_became_peered":"2024-09-10T20:42:57.027298+0000","last_unstale":"2024-09-10T20:42:58.633644+0000","last_undegraded":"2024-09-10T20:42:58.633644+0000","last_fullsized":"2024-09-10T20:42:58.633644+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:28:45.311740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633389+0000","last_change":"2024-09-10T20:42:56.007507+0000","last_active":"2024-09-10T20:42:58.633389+0000","last_peered":"2024-09-10T20:42:58.633389+0000","last_clean":"2024-09-10T20:42:58.633389+0000","last_became_active":"2024-09-10T20:42:56.007245+0000","last_became_peered":"2024-09-10T20:42:56.007245+0000","last_unstale":"2024-09-10T20:42:58.633389+0000","last_undegraded":"2024-09-10T20:42:58.633389+0000","last_fullsized":"2024-09-10T20:42:58.633389+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:52.383535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133833+0000","last_change":"2024-09-10T20:42:57.025422+0000","last_active":"2024-09-10T20:42:59.133833+0000","last_peered":"2024-09-10T20:42:59.133833+0000","last_clean":"2024-09-10T20:42:59.133833+0000","last_became_active":"2024-09-10T20:42:57.024875+0000","last_became_peered":"2024-09-10T20:42:57.024875+0000","last_unstale":"2024-09-10T20:42:59.133833+0000","last_undegraded":"2024-09-10T20:42:59.133833+0000","last_fullsized":"2024-09-10T20:42:59.133833+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:01.914016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019282+0000","last_change":"2024-09-10T20:42:56.013936+0000","last_active":"2024-09-10T20:42:58.019282+0000","last_peered":"2024-09-10T20:42:58.019282+0000","last_clean":"2024-09-10T20:42:58.019282+0000","last_became_active":"2024-09-10T20:42:56.013414+0000","last_became_peered":"2024-09-10T20:42:56.013414+0000","last_unstale":"2024-09-10T20:42:58.019282+0000","last_undegraded":"2024-09-10T20:42:58.019282+0000","last_fullsized":"2024-09-10T20:42:58.019282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:16.702971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134248+0000","last_change":"2024-09-10T20:42:57.024432+0000","last_active":"2024-09-10T20:42:59.134248+0000","last_peered":"2024-09-10T20:42:59.134248+0000","last_clean":"2024-09-10T20:42:59.134248+0000","last_became_active":"2024-09-10T20:42:57.024158+0000","last_became_peered":"2024-09-10T20:42:57.024158+0000","last_unstale":"2024-09-10T20:42:59.134248+0000","last_undegraded":"2024-09-10T20:42:59.134248+0000","last_fullsized":"2024-09-10T20:42:59.134248+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:10:01.699054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019152+0000","last_change":"2024-09-10T20:42:56.014674+0000","last_active":"2024-09-10T20:42:58.019152+0000","last_peered":"2024-09-10T20:42:58.019152+0000","last_clean":"2024-09-10T20:42:58.019152+0000","last_became_active":"2024-09-10T20:42:56.014110+0000","last_became_peered":"2024-09-10T20:42:56.014110+0000","last_unstale":"2024-09-10T20:42:58.019152+0000","last_undegraded":"2024-09-10T20:42:58.019152+0000","last_fullsized":"2024-09-10T20:42:58.019152+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:49.404912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133927+0000","last_change":"2024-09-10T20:42:57.026077+0000","last_active":"2024-09-10T20:42:59.133927+0000","last_peered":"2024-09-10T20:42:59.133927+0000","last_clean":"2024-09-10T20:42:59.133927+0000","last_became_active":"2024-09-10T20:42:57.025807+0000","last_became_peered":"2024-09-10T20:42:57.025807+0000","last_unstale":"2024-09-10T20:42:59.133927+0000","last_undegraded":"2024-09-10T20:42:59.133927+0000","last_fullsized":"2024-09-10T20:42:59.133927+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:32:40.794936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.7","version":"99'2","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019995+0000","last_change":"2024-09-10T20:42:56.015471+0000","last_active":"2024-09-10T20:42:58.019995+0000","last_peered":"2024-09-10T20:42:58.019995+0000","last_clean":"2024-09-10T20:42:58.019995+0000","last_became_active":"2024-09-10T20:42:56.015190+0000","last_became_peered":"2024-09-10T20:42:56.015190+0000","last_unstale":"2024-09-10T20:42:58.019995+0000","last_undegraded":"2024-09-10T20:42:58.019995+0000","last_fullsized":"2024-09-10T20:42:58.019995+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:19.148181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017963+0000","last_change":"2024-09-10T20:42:57.025806+0000","last_active":"2024-09-10T20:42:58.017963+0000","last_peered":"2024-09-10T20:42:58.017963+0000","last_clean":"2024-09-10T20:42:58.017963+0000","last_became_active":"2024-09-10T20:42:57.025445+0000","last_became_peered":"2024-09-10T20:42:57.025445+0000","last_unstale":"2024-09-10T20:42:58.017963+0000","last_undegraded":"2024-09-10T20:42:58.017963+0000","last_fullsized":"2024-09-10T20:42:58.017963+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:48:02.305241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"100'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.020563+0000","last_change":"2024-09-10T20:42:56.007680+0000","last_active":"2024-09-10T20:42:58.020563+0000","last_peered":"2024-09-10T20:42:58.020563+0000","last_clean":"2024-09-10T20:42:58.020563+0000","last_became_active":"2024-09-10T20:42:56.007392+0000","last_became_peered":"2024-09-10T20:42:56.007392+0000","last_unstale":"2024-09-10T20:42:58.020563+0000","last_undegraded":"2024-09-10T20:42:58.020563+0000","last_fullsized":"2024-09-10T20:42:58.020563+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:45:01.242317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633734+0000","last_change":"2024-09-10T20:42:57.025849+0000","last_active":"2024-09-10T20:42:58.633734+0000","last_peered":"2024-09-10T20:42:58.633734+0000","last_clean":"2024-09-10T20:42:58.633734+0000","last_became_active":"2024-09-10T20:42:57.025478+0000","last_became_peered":"2024-09-10T20:42:57.025478+0000","last_unstale":"2024-09-10T20:42:58.633734+0000","last_undegraded":"2024-09-10T20:42:58.633734+0000","last_fullsized":"2024-09-10T20:42:58.633734+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:58.645730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019902+0000","last_change":"2024-09-10T20:42:56.015660+0000","last_active":"2024-09-10T20:42:58.019902+0000","last_peered":"2024-09-10T20:42:58.019902+0000","last_clean":"2024-09-10T20:42:58.019902+0000","last_became_active":"2024-09-10T20:42:56.014362+0000","last_became_peered":"2024-09-10T20:42:56.014362+0000","last_unstale":"2024-09-10T20:42:58.019902+0000","last_undegraded":"2024-09-10T20:42:58.019902+0000","last_fullsized":"2024-09-10T20:42:58.019902+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:39:15.186008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134072+0000","last_change":"2024-09-10T20:42:57.030963+0000","last_active":"2024-09-10T20:42:59.134072+0000","last_peered":"2024-09-10T20:42:59.134072+0000","last_clean":"2024-09-10T20:42:59.134072+0000","last_became_active":"2024-09-10T20:42:57.030666+0000","last_became_peered":"2024-09-10T20:42:57.030666+0000","last_unstale":"2024-09-10T20:42:59.134072+0000","last_undegraded":"2024-09-10T20:42:59.134072+0000","last_fullsized":"2024-09-10T20:42:59.134072+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:05.228089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019795+0000","last_change":"2024-09-10T20:42:56.006037+0000","last_active":"2024-09-10T20:42:58.019795+0000","last_peered":"2024-09-10T20:42:58.019795+0000","last_clean":"2024-09-10T20:42:58.019795+0000","last_became_active":"2024-09-10T20:42:56.004553+0000","last_became_peered":"2024-09-10T20:42:56.004553+0000","last_unstale":"2024-09-10T20:42:58.019795+0000","last_undegraded":"2024-09-10T20:42:58.019795+0000","last_fullsized":"2024-09-10T20:42:58.019795+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:10:07.187475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081640+0000","last_change":"2024-09-10T20:42:57.028178+0000","last_active":"2024-09-10T20:42:59.081640+0000","last_peered":"2024-09-10T20:42:59.081640+0000","last_clean":"2024-09-10T20:42:59.081640+0000","last_became_active":"2024-09-10T20:42:57.027911+0000","last_became_peered":"2024-09-10T20:42:57.027911+0000","last_unstale":"2024-09-10T20:42:59.081640+0000","last_undegraded":"2024-09-10T20:42:59.081640+0000","last_fullsized":"2024-09-10T20:42:59.081640+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:10.882657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.016569+0000","last_change":"2024-09-10T20:42:56.013071+0000","last_active":"2024-09-10T20:42:58.016569+0000","last_peered":"2024-09-10T20:42:58.016569+0000","last_clean":"2024-09-10T20:42:58.016569+0000","last_became_active":"2024-09-10T20:42:56.012762+0000","last_became_peered":"2024-09-10T20:42:56.012762+0000","last_unstale":"2024-09-10T20:42:58.016569+0000","last_undegraded":"2024-09-10T20:42:58.016569+0000","last_fullsized":"2024-09-10T20:42:58.016569+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:12:55.456247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.134128+0000","last_change":"2024-09-10T20:42:57.030084+0000","last_active":"2024-09-10T20:42:59.134128+0000","last_peered":"2024-09-10T20:42:59.134128+0000","last_clean":"2024-09-10T20:42:59.134128+0000","last_became_active":"2024-09-10T20:42:57.029814+0000","last_became_peered":"2024-09-10T20:42:57.029814+0000","last_unstale":"2024-09-10T20:42:59.134128+0000","last_undegraded":"2024-09-10T20:42:59.134128+0000","last_fullsized":"2024-09-10T20:42:59.134128+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:31.854297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133821+0000","last_change":"2024-09-10T20:42:56.008409+0000","last_active":"2024-09-10T20:42:59.133821+0000","last_peered":"2024-09-10T20:42:59.133821+0000","last_clean":"2024-09-10T20:42:59.133821+0000","last_became_active":"2024-09-10T20:42:56.008156+0000","last_became_peered":"2024-09-10T20:42:56.008156+0000","last_unstale":"2024-09-10T20:42:59.133821+0000","last_undegraded":"2024-09-10T20:42:59.133821+0000","last_fullsized":"2024-09-10T20:42:59.133821+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.531748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.133890+0000","last_change":"2024-09-10T20:42:57.025142+0000","last_active":"2024-09-10T20:42:59.133890+0000","last_peered":"2024-09-10T20:42:59.133890+0000","last_clean":"2024-09-10T20:42:59.133890+0000","last_became_active":"2024-09-10T20:42:57.024510+0000","last_became_peered":"2024-09-10T20:42:57.024510+0000","last_unstale":"2024-09-10T20:42:59.133890+0000","last_undegraded":"2024-09-10T20:42:59.133890+0000","last_fullsized":"2024-09-10T20:42:59.133890+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:53:10.990965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.633644+0000","last_change":"2024-09-10T20:42:56.014524+0000","last_active":"2024-09-10T20:42:58.633644+0000","last_peered":"2024-09-10T20:42:58.633644+0000","last_clean":"2024-09-10T20:42:58.633644+0000","last_became_active":"2024-09-10T20:42:56.014209+0000","last_became_peered":"2024-09-10T20:42:56.014209+0000","last_unstale":"2024-09-10T20:42:58.633644+0000","last_undegraded":"2024-09-10T20:42:58.633644+0000","last_fullsized":"2024-09-10T20:42:58.633644+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:18.142897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017787+0000","last_change":"2024-09-10T20:42:57.023049+0000","last_active":"2024-09-10T20:42:58.017787+0000","last_peered":"2024-09-10T20:42:58.017787+0000","last_clean":"2024-09-10T20:42:58.017787+0000","last_became_active":"2024-09-10T20:42:57.022788+0000","last_became_peered":"2024-09-10T20:42:57.022788+0000","last_unstale":"2024-09-10T20:42:58.017787+0000","last_undegraded":"2024-09-10T20:42:58.017787+0000","last_fullsized":"2024-09-10T20:42:58.017787+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:39.719435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.017649+0000","last_change":"2024-09-10T20:42:56.009317+0000","last_active":"2024-09-10T20:42:58.017649+0000","last_peered":"2024-09-10T20:42:58.017649+0000","last_clean":"2024-09-10T20:42:58.017649+0000","last_became_active":"2024-09-10T20:42:56.009038+0000","last_became_peered":"2024-09-10T20:42:56.009038+0000","last_unstale":"2024-09-10T20:42:58.017649+0000","last_undegraded":"2024-09-10T20:42:58.017649+0000","last_fullsized":"2024-09-10T20:42:58.017649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:02:53.603352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.019895+0000","last_change":"2024-09-10T20:42:57.021758+0000","last_active":"2024-09-10T20:42:58.019895+0000","last_peered":"2024-09-10T20:42:58.019895+0000","last_clean":"2024-09-10T20:42:58.019895+0000","last_became_active":"2024-09-10T20:42:57.021337+0000","last_became_peered":"2024-09-10T20:42:57.021337+0000","last_unstale":"2024-09-10T20:42:58.019895+0000","last_undegraded":"2024-09-10T20:42:58.019895+0000","last_fullsized":"2024-09-10T20:42:58.019895+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:55.989846+0000","last_clean_scrub_stamp":"2024-09-10T20:42:55.989846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:06.903753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:58.776204+0000","last_change":"2024-09-10T20:42:56.009113+0000","last_active":"2024-09-10T20:42:58.776204+0000","last_peered":"2024-09-10T20:42:58.776204+0000","last_clean":"2024-09-10T20:42:58.776204+0000","last_became_active":"2024-09-10T20:42:56.008812+0000","last_became_peered":"2024-09-10T20:42:56.008812+0000","last_unstale":"2024-09-10T20:42:58.776204+0000","last_undegraded":"2024-09-10T20:42:58.776204+0000","last_fullsized":"2024-09-10T20:42:58.776204+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:42:54.984599+0000","last_clean_scrub_stamp":"2024-09-10T20:42:54.984599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:12:13.797884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":283,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-10T20:42:59.081469+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:42:59.081469+0000","last_peered":"2024-09-10T20:42:59.081469+0000","last_clean":"2024-09-10T20:42:59.081469+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:42:59.081469+0000","last_undegraded":"2024-09-10T20:42:59.081469+0000","last_fullsized":"2024-09-10T20:42:59.081469+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542207,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1257472,"data_stored":544327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542208,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1224704,"data_stored":536291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542206,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1839104,"data_stored":1133051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542206,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1253376,"data_stored":544293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509504,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":1236992,"data_stored":542591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509503,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36568,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335272,"statfs":{"total":96636764160,"available":96599318528,"internally_reserved":0,"allocated":1249280,"data_stored":543229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509503,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37132,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1818624,"data_stored":1126659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509503,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36568,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335272,"statfs":{"total":96636764160,"available":96599318528,"internally_reserved":0,"allocated":1245184,"data_stored":543243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:43:01.402 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:43:01.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:43:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.548+0000 7fa599fd3640 1 -- 172.21.15.28:0/3951948924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 msgr2=0x7fa594168d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.548+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3951948924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa594168d50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa58c005fd0 tx=0x7fa58c02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.548+0000 7fa599fd3640 1 -- 172.21.15.28:0/3951948924 shutdown_connections 2024-09-10T20:43:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.548+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3951948924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa594168d50 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.548+0000 7fa599fd3640 1 -- 172.21.15.28:0/3951948924 >> 172.21.15.28:0/3951948924 conn(0x7fa5940d4040 msgr2=0x7fa5940d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.548+0000 7fa599fd3640 1 -- 172.21.15.28:0/3951948924 shutdown_connections 2024-09-10T20:43:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.549+0000 7fa599fd3640 1 -- 172.21.15.28:0/3951948924 wait complete. 2024-09-10T20:43:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.550+0000 7fa599fd3640 1 Processor -- start 2024-09-10T20:43:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.550+0000 7fa599fd3640 1 -- start start 2024-09-10T20:43:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa59414d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5940d4b60 0x7fa59415adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa59415e130 0x7fa59414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa594169620 con 0x7fa5940cfc50 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941694a0 con 0x7fa5940d4b60 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa5941697a0 con 0x7fa59415e130 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa59414d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa59414d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47498/0 (socket says 172.21.15.28:47498) 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 -- 172.21.15.28:0/3711061397 learned_addr learned my addr 172.21.15.28:0/3711061397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa592ffd640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5940d4b60 0x7fa59415adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa593fff640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa59415e130 0x7fa59414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 -- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa59415e130 msgr2=0x7fa59414ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa59415e130 0x7fa59414ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 -- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5940d4b60 msgr2=0x7fa59415adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5940d4b60 0x7fa59415adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 -- 172.21.15.28:0/3711061397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5941f8700 con 0x7fa5940cfc50 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa592ffd640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5940d4b60 0x7fa59415adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa5937fe640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa59414d790 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c390 tx=0x7fa58400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa57bfff640 1 -- 172.21.15.28:0/3711061397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584011790 con 0x7fa5940cfc50 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa57bfff640 1 -- 172.21.15.28:0/3711061397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa584017070 con 0x7fa5940cfc50 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5941f89e0 con 0x7fa5940cfc50 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa57bfff640 1 -- 172.21.15.28:0/3711061397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584011ca0 con 0x7fa5940cfc50 2024-09-10T20:43:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.551+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59415b330 con 0x7fa5940cfc50 2024-09-10T20:43:01.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.553+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5940cf540 con 0x7fa5940cfc50 2024-09-10T20:43:01.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.553+0000 7fa57bfff640 1 -- 172.21.15.28:0/3711061397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fa58401c460 con 0x7fa5940cfc50 2024-09-10T20:43:01.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.554+0000 7fa57bfff640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa580078090 0x7fa58007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.554+0000 7fa592ffd640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa580078090 0x7fa58007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.554+0000 7fa57bfff640 1 -- 172.21.15.28:0/3711061397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fa584099ca0 con 0x7fa5940cfc50 2024-09-10T20:43:01.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.554+0000 7fa592ffd640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa580078090 0x7fa58007a550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa588000c00 tx=0x7fa588009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.558+0000 7fa57bfff640 1 -- 172.21.15.28:0/3711061397 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa584062610 con 0x7fa5940cfc50 2024-09-10T20:43:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.819+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fa59406dea0 con 0x7fa5940cfc50 2024-09-10T20:43:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.821+0000 7fa57bfff640 1 -- 172.21.15.28:0/3711061397 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v32) ==== 190+0+147 (secure 0 0 0) 0x7fa584061f10 con 0x7fa5940cfc50 2024-09-10T20:43:01.825 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:43:01.825 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:43:01.825 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:43:01.825 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:43:01.825 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:43:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:43:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa580078090 msgr2=0x7fa58007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa580078090 0x7fa58007a550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa588000c00 tx=0x7fa588009040 comp rx=0 tx=0).stop 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 msgr2=0x7fa59414d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa59414d790 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c390 tx=0x7fa58400c850 comp rx=0 tx=0).stop 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 shutdown_connections 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa580078090 0x7fa58007a550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa59415e130 0x7fa59414ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5940d4b60 0x7fa59415adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 --2- 172.21.15.28:0/3711061397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5940cfc50 0x7fa59414d790 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.823+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 >> 172.21.15.28:0/3711061397 conn(0x7fa5940d4040 msgr2=0x7fa59406d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.824+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 shutdown_connections 2024-09-10T20:43:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.824+0000 7fa599fd3640 1 -- 172.21.15.28:0/3711061397 wait complete. 2024-09-10T20:43:01.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:01.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:43:01.883 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:43:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.941+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4074007466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 msgr2=0x7f3cf0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.941+0000 7f3d0e1bb640 1 --2- 172.21.15.28:0/4074007466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 0x7f3cf0004c80 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc006fc0 tx=0x7f3cf0005830 comp rx=0 tx=0).stop 2024-09-10T20:43:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.941+0000 7f3d07fff640 1 -- 172.21.15.28:0/4074007466 reap_dead start 2024-09-10T20:43:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.941+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4074007466 shutdown_connections 2024-09-10T20:43:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.941+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4074007466 >> 172.21.15.28:0/4074007466 conn(0x7f3d080d3fb0 msgr2=0x7f3d080d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.941+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4074007466 shutdown_connections 2024-09-10T20:43:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.941+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4074007466 wait complete. 2024-09-10T20:43:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.942+0000 7f3d0e1bb640 1 Processor -- start 2024-09-10T20:43:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.942+0000 7f3d0e1bb640 1 -- start start 2024-09-10T20:43:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d0e1bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 0x7f3d081f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d0e1bb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080a0240 0x7f3d080a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d0e1bb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080cc0d0 0x7f3d080cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d0e1bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d080d2f20 con 0x7f3cf0004800 2024-09-10T20:43:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d0e1bb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3d080d2da0 con 0x7f3d080a0240 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d0e1bb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3d080cfbc0 con 0x7f3d080cc0d0 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 0x7f3d081f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 0x7f3d081f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47522/0 (socket says 172.21.15.28:47522) 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d077fe640 1 -- 172.21.15.28:0/4090176847 learned_addr learned my addr 172.21.15.28:0/4090176847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d07fff640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080cc0d0 0x7f3d080cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.943+0000 7f3d06ffd640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080a0240 0x7f3d080a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d077fe640 1 -- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080cc0d0 msgr2=0x7f3d080cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d077fe640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080cc0d0 0x7f3d080cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d077fe640 1 -- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080a0240 msgr2=0x7f3d080a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d077fe640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080a0240 0x7f3d080a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d077fe640 1 -- 172.21.15.28:0/4090176847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d080ccd90 con 0x7f3cf0004800 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d06ffd640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080a0240 0x7f3d080a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d077fe640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 0x7f3d081f4120 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8011ae0 tx=0x7f3cf8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3ceffff640 1 -- 172.21.15.28:0/4090176847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf801e070 con 0x7f3cf0004800 2024-09-10T20:43:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3ceffff640 1 -- 172.21.15.28:0/4090176847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cf800fe70 con 0x7f3cf0004800 2024-09-10T20:43:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d080cd070 con 0x7f3cf0004800 2024-09-10T20:43:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3ceffff640 1 -- 172.21.15.28:0/4090176847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8004290 con 0x7f3cf0004800 2024-09-10T20:43:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.944+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d080cd630 con 0x7f3cf0004800 2024-09-10T20:43:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.945+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d08161b50 con 0x7f3cf0004800 2024-09-10T20:43:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.946+0000 7f3ceffff640 1 -- 172.21.15.28:0/4090176847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f3cf80035f0 con 0x7f3cf0004800 2024-09-10T20:43:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.946+0000 7f3ceffff640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3cf40780c0 0x7f3cf407a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.946+0000 7f3ceffff640 1 -- 172.21.15.28:0/4090176847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f3cf809ad60 con 0x7f3cf0004800 2024-09-10T20:43:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.946+0000 7f3d06ffd640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3cf40780c0 0x7f3cf407a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.947+0000 7f3d06ffd640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3cf40780c0 0x7f3cf407a580 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc033000 tx=0x7f3cfc000fe0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:01.950+0000 7f3ceffff640 1 -- 172.21.15.28:0/4090176847 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3cf80630b0 con 0x7f3cf0004800 2024-09-10T20:43:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.171+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d080b9f30 con 0x7f3cf0004800 2024-09-10T20:43:02.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.171+0000 7f3ceffff640 1 -- 172.21.15.28:0/4090176847 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5231 (secure 0 0 0) 0x7f3cf8067ae0 con 0x7f3cf0004800 2024-09-10T20:43:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:02.175 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":66,"btime":"2024-09-10T20:43:00:026929+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4277647396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4277647396},{"type":"v1","addr":"172.21.15.28:6834","nonce":4277647396}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/4032655811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":4032655811},{"type":"v1","addr":"172.21.15.159:6835","nonce":4032655811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/3530092732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3530092732},{"type":"v1","addr":"172.21.15.159:6834","nonce":3530092732}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:56.370954+0000","modified":"2024-09-10T20:42:59.400466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1426218269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1426218269},{"type":"v1","addr":"172.21.15.28:6835","nonce":1426218269}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6936,"qdb_cluster":[6936]},"id":8}]} 2024-09-10T20:43:02.175 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 66 2024-09-10T20:43:02.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3cf40780c0 msgr2=0x7f3cf407a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3cf40780c0 0x7f3cf407a580 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc033000 tx=0x7f3cfc000fe0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 msgr2=0x7f3d081f4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 0x7f3d081f4120 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8011ae0 tx=0x7f3cf8009e90 comp rx=0 tx=0).stop 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 shutdown_connections 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3cf40780c0 0x7f3cf407a580 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d080cc0d0 0x7f3d080cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d080a0240 0x7f3d080a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 --2- 172.21.15.28:0/4090176847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf0004800 0x7f3d081f4120 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 >> 172.21.15.28:0/4090176847 conn(0x7f3d080d3fb0 msgr2=0x7f3d080d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.173+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 shutdown_connections 2024-09-10T20:43:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.174+0000 7f3d0e1bb640 1 -- 172.21.15.28:0/4090176847 wait complete. 2024-09-10T20:43:02.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:42:56.370954+0000', 'modified': '2024-09-10T20:42:59.400466+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6936}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6936': {'gid': 6936, 'name': 'a', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1426218269', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1426218269}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1426218269}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6936, 'qdb_cluster': [6936]} 2024-09-10T20:43:02.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:43:02.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-10T20:43:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.701+0000 7fd074191640 1 -- 172.21.15.28:0/2061353463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 msgr2=0x7fd058004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.701+0000 7fd074191640 1 --2- 172.21.15.28:0/2061353463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 0x7fd058004880 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00c260 tx=0x7fd058005830 comp rx=0 tx=0).stop 2024-09-10T20:43:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.701+0000 7fd07272e640 1 -- 172.21.15.28:0/2061353463 reap_dead start 2024-09-10T20:43:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.701+0000 7fd074191640 1 -- 172.21.15.28:0/2061353463 shutdown_connections 2024-09-10T20:43:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.701+0000 7fd074191640 1 -- 172.21.15.28:0/2061353463 >> 172.21.15.28:0/2061353463 conn(0x7fd06c0d0420 msgr2=0x7fd06c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.702+0000 7fd074191640 1 -- 172.21.15.28:0/2061353463 shutdown_connections 2024-09-10T20:43:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.702+0000 7fd074191640 1 -- 172.21.15.28:0/2061353463 wait complete. 2024-09-10T20:43:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.703+0000 7fd074191640 1 Processor -- start 2024-09-10T20:43:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.703+0000 7fd074191640 1 -- start start 2024-09-10T20:43:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 0x7fd06c1595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd06c0b1300 0x7fd06c1f4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd06c0b1780 0x7fd06c1f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c06f0b0 con 0x7fd0580043b0 2024-09-10T20:43:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c06ef30 con 0x7fd06c0b1780 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c06f230 con 0x7fd06c0b1300 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 0x7fd06c1595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 0x7fd06c1595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47548/0 (socket says 172.21.15.28:47548) 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 -- 172.21.15.28:0/2794190919 learned_addr learned my addr 172.21.15.28:0/2794190919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd07172c640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd06c0b1300 0x7fd06c1f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd07272e640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd06c0b1780 0x7fd06c1f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 -- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd06c0b1300 msgr2=0x7fd06c1f4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd06c0b1300 0x7fd06c1f4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 -- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd06c0b1780 msgr2=0x7fd06c1f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd06c0b1780 0x7fd06c1f4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 -- 172.21.15.28:0/2794190919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c158b40 con 0x7fd0580043b0 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd07172c640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd06c0b1300 0x7fd06c1f4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:02.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd07272e640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd06c0b1780 0x7fd06c1f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:02.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd071f2d640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 0x7fd06c1595e0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd068011ae0 tx=0x7fd068009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd0627fc640 1 -- 172.21.15.28:0/2794190919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06801e070 con 0x7fd0580043b0 2024-09-10T20:43:02.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd0627fc640 1 -- 172.21.15.28:0/2794190919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd068004460 con 0x7fd0580043b0 2024-09-10T20:43:02.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c0b6390 con 0x7fd0580043b0 2024-09-10T20:43:02.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd0627fc640 1 -- 172.21.15.28:0/2794190919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06800fc60 con 0x7fd0580043b0 2024-09-10T20:43:02.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.704+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c1f52d0 con 0x7fd0580043b0 2024-09-10T20:43:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.706+0000 7fd0627fc640 1 -- 172.21.15.28:0/2794190919 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fd0680035f0 con 0x7fd0580043b0 2024-09-10T20:43:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.707+0000 7fd0627fc640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd050078090 0x7fd05007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.707+0000 7fd0627fc640 1 -- 172.21.15.28:0/2794190919 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fd06809aac0 con 0x7fd0580043b0 2024-09-10T20:43:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.707+0000 7fd07172c640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd050078090 0x7fd05007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.707+0000 7fd07172c640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd050078090 0x7fd05007a550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd05c033000 tx=0x7fd05c02d000 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.707+0000 7fd077037740 1 Processor -- start 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd077037740 1 -- start start 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7ee0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd077037740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7ede5f0 0x55cde7edea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd077037740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cde7e55820 con 0x55cde7f28df0 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55cde7e44c50 con 0x55cde7f091b0 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55cde7e48a60 con 0x55cde7ede5f0 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd07172c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd07172c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f291f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47556/0 (socket says 172.21.15.28:47556) 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.708+0000 7fd07172c640 1 -- 172.21.15.28:0/2461335310 learned_addr learned my addr 172.21.15.28:0/2461335310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd07272e640 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7ede5f0 0x55cde7edea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd071f2d640 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7ee0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd07172c640 1 -- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7ede5f0 msgr2=0x55cde7edea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd07172c640 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7ede5f0 0x55cde7edea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd07172c640 1 -- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 msgr2=0x55cde7ee0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd07172c640 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7ee0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd07172c640 1 -- 172.21.15.28:0/2461335310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde7ba6850 con 0x55cde7f28df0 2024-09-10T20:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd07172c640 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f291f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000fb0 tx=0x7fd05c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d0823556e97891 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd04f7fe640 1 -- 172.21.15.28:0/2461335310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c031040 con 0x55cde7f28df0 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd04f7fe640 1 -- 172.21.15.28:0/2461335310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd05c0065a0 con 0x55cde7f28df0 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.709+0000 7fd04f7fe640 1 -- 172.21.15.28:0/2461335310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c003ce0 con 0x55cde7f28df0 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 -- 172.21.15.28:0/2461335310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 msgr2=0x55cde7f291f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f291f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000fb0 tx=0x7fd05c03d000 comp rx=0 tx=0).stop 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 -- 172.21.15.28:0/2461335310 shutdown_connections 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7ede5f0 0x55cde7edea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f291f0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 --2- 172.21.15.28:0/2461335310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7ee0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 -- 172.21.15.28:0/2461335310 >> 172.21.15.28:0/2461335310 conn(0x55cde7edf6a0 msgr2=0x55cde7ee30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 -- 172.21.15.28:0/2461335310 shutdown_connections 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.710+0000 7fd077037740 1 -- 172.21.15.28:0/2461335310 wait complete. 2024-09-10T20:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.711+0000 7fd077037740 1 -- start start 2024-09-10T20:43:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.711+0000 7fd077037740 1 Processor -- start 2024-09-10T20:43:02.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f269f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f26f30 0x55cde7ee1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cde7bb5620 con 0x55cde7f28df0 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55cde7dea5f0 con 0x55cde7f26f30 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55cde7edd890 con 0x55cde7f091b0 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 10 client.0 _ll_register_callbacks cb 0x55cde7dee0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f269f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47566/0 (socket says 172.21.15.28:47566) 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 -- 172.21.15.28:0/3502338488 learned_addr learned my addr 172.21.15.28:0/3502338488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd071f2d640 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07272e640 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f26f30 0x55cde7ee1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 -- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 msgr2=0x55cde7f3aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 -- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f26f30 msgr2=0x55cde7ee1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f26f30 0x55cde7ee1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 -- 172.21.15.28:0/3502338488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde7c1d6b0 con 0x55cde7f28df0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd07172c640 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f269f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd05c02b110 tx=0x7fd05c003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff36b677c533fe9c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd04cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd04cff9640 1 -- 172.21.15.28:0/3502338488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c057060 con 0x55cde7f28df0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd04cff9640 1 -- 172.21.15.28:0/3502338488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fd05c02c680 con 0x55cde7f28df0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.712+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd05c0065a0 con 0x55cde7f28df0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd04cff9640 1 -- 172.21.15.28:0/3502338488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7fd05c05d050 con 0x55cde7f28df0 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 10 client.7083 fetch_fsmap learned FSMap version 66 2024-09-10T20:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x55cde7f3bd90 con 0x55cde7f28df0 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd04cff9640 1 -- 172.21.15.28:0/3502338488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd05c0065a0 con 0x55cde7f28df0 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd04cff9640 1 -- 172.21.15.28:0/3502338488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 66) ==== 3502+0+0 (secure 0 0 0) 0x7fd05c060020 con 0x55cde7f28df0 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 10 client.7083 fetch_fsmap finished waiting for FSMap version 66 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 10 client.7083 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 20 client.7083 populate_metadata read hostname 'smithi028' 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7f221b0 0x55cde7f22590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 4 client.7083 mds_command: new command op to 6936 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 --> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cde7e57770 con 0x55cde7f221b0 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd071f2d640 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7f221b0 0x55cde7f22590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd071f2d640 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7f221b0 0x55cde7f22590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fffe8bafc1134999 server_cookie=c300cf21483d1b0d in_seq=0 out_seq=0 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.713+0000 7fd04cff9640 10 client.7083 ms_handle_connect on v2:172.21.15.28:6832/1426218269 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd04cff9640 1 -- 172.21.15.28:0/3502338488 <== mds.0 v2:172.21.15.28:6832/1426218269 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cde7e57770 con 0x55cde7f221b0 2024-09-10T20:43:02.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd04cff9640 10 client.7083 handle_command_reply: tid=0 2024-09-10T20:43:02.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 client.7083 shutdown 2024-09-10T20:43:02.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 msgr2=0x55cde7f269f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f269f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd05c02b110 tx=0x7fd05c003e30 comp rx=0 tx=0).stop 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 shutdown_connections 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7f221b0 0x55cde7f22590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f26f30 0x55cde7ee1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd04cff9640 0 client.7083 ms_handle_reset on v2:172.21.15.28:6832/1426218269 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f269f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 --2- 172.21.15.28:0/3502338488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 >> 172.21.15.28:0/3502338488 conn(0x55cde7ede5f0 msgr2=0x55cde7ee3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 shutdown_connections 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 1 -- 172.21.15.28:0/3502338488 wait complete. 2024-09-10T20:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.714+0000 7fd077037740 20 client.7083 trim_cache size 0 max 16384 2024-09-10T20:43:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.733+0000 7fd077037740 1 Processor -- start 2024-09-10T20:43:02.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.733+0000 7fd077037740 1 -- start start 2024-09-10T20:43:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.733+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 0x55cde7f217a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.733+0000 7fd077037740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f26f30 0x55cde7f4baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f28df0 0x55cde7f4c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd077037740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c2004f0 con 0x55cde7f26f30 2024-09-10T20:43:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1f84c0 con 0x55cde7f28df0 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c1ffea0 con 0x55cde7f091b0 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f26f30 0x55cde7f4baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f26f30 0x55cde7f4baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47574/0 (socket says 172.21.15.28:47574) 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 -- 172.21.15.28:0/1081672152 learned_addr learned my addr 172.21.15.28:0/1081672152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 -- 172.21.15.28:0/1081672152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 msgr2=0x55cde7f217a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 --2- 172.21.15.28:0/1081672152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 0x55cde7f217a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 -- 172.21.15.28:0/1081672152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f28df0 msgr2=0x55cde7f4c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 --2- 172.21.15.28:0/1081672152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f28df0 0x55cde7f4c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 -- 172.21.15.28:0/1081672152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde7f27890 con 0x55cde7f26f30 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd07172c640 1 --2- 172.21.15.28:0/1081672152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f26f30 0x55cde7f4baf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd05c004f30 tx=0x7fd05c004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f813bf2dd129ca server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd04effd640 1 -- 172.21.15.28:0/1081672152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c031040 con 0x55cde7f26f30 2024-09-10T20:43:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd04effd640 1 -- 172.21.15.28:0/1081672152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd05c02c680 con 0x55cde7f26f30 2024-09-10T20:43:02.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.734+0000 7fd04effd640 1 -- 172.21.15.28:0/1081672152 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c02c830 con 0x55cde7f26f30 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 -- 172.21.15.28:0/1081672152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f26f30 msgr2=0x55cde7f4baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 --2- 172.21.15.28:0/1081672152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f26f30 0x55cde7f4baf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd05c004f30 tx=0x7fd05c004f60 comp rx=0 tx=0).stop 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 -- 172.21.15.28:0/1081672152 shutdown_connections 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 --2- 172.21.15.28:0/1081672152 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f28df0 0x55cde7f4c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 --2- 172.21.15.28:0/1081672152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f26f30 0x55cde7f4baf0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 --2- 172.21.15.28:0/1081672152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f091b0 0x55cde7f217a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 -- 172.21.15.28:0/1081672152 >> 172.21.15.28:0/1081672152 conn(0x55cde7ede5f0 msgr2=0x55cde7f07d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 -- 172.21.15.28:0/1081672152 shutdown_connections 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 -- 172.21.15.28:0/1081672152 wait complete. 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 -- start start 2024-09-10T20:43:02.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.735+0000 7fd077037740 1 Processor -- start 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd077037740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f26f30 0x55cde7fad340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd077037740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f00b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd077037740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cde7bb5620 con 0x55cde7f28df0 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x55cde7dea5f0 con 0x55cde7f091b0 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd077037740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x55cde7f38fd0 con 0x55cde7f26f30 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd077037740 10 client.0 _ll_register_callbacks cb 0x7fd068005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.736+0000 7fd07272e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f00b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f00b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47584/0 (socket says 172.21.15.28:47584) 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 -- 172.21.15.28:0/1357462424 learned_addr learned my addr 172.21.15.28:0/1357462424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd071f2d640 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07172c640 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f26f30 0x55cde7fad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 -- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f26f30 msgr2=0x55cde7fad340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f26f30 0x55cde7fad340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 -- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 msgr2=0x55cde7f3aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 -- 172.21.15.28:0/1357462424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde7f377e0 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd07272e640 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f00b20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd064004270 tx=0x7fd064004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2274406d74f875 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd04cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd04cff9640 1 -- 172.21.15.28:0/1357462424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064022060 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd04cff9640 1 -- 172.21.15.28:0/1357462424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fd064010d20 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd05c02c680 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd04cff9640 1 -- 172.21.15.28:0/1357462424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7fd064028050 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd077037740 10 client.7089 fetch_fsmap learned FSMap version 66 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x55cde7f379f0 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.737+0000 7fd04cff9640 1 -- 172.21.15.28:0/1357462424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd05c02c680 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd04cff9640 1 -- 172.21.15.28:0/1357462424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 66) ==== 3502+0+0 (secure 0 0 0) 0x7fd06402b020 con 0x55cde7f28df0 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 10 client.7089 fetch_fsmap finished waiting for FSMap version 66 2024-09-10T20:43:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 10 client.7089 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 20 client.7089 populate_metadata read hostname 'smithi028' 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7fac310 0x55cde7f4a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 4 client.7089 mds_command: new command op to 6936 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 --> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] -- command(tid 0: {"prefix": "status"}) -- 0x55cde7e53220 con 0x55cde7fac310 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd071f2d640 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7fac310 0x55cde7f4a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd04cff9640 10 client.7089 ms_handle_connect on v2:172.21.15.28:6832/1426218269 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd071f2d640 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7fac310 0x55cde7f4a790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4cf1f3740dc39cb server_cookie=be5f8d09eccdb35b in_seq=0 out_seq=0 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd04cff9640 1 -- 172.21.15.28:0/1357462424 <== mds.0 v2:172.21.15.28:6832/1426218269 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cde7e53220 con 0x55cde7fac310 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd04cff9640 10 client.7089 handle_command_reply: tid=0 2024-09-10T20:43:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 client.7089 shutdown 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 msgr2=0x55cde7f00b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f00b20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd064004270 tx=0x7fd064004730 comp rx=0 tx=0).stop 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 shutdown_connections 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:6832/1426218269,v1:172.21.15.28:6835/1426218269] conn(0x55cde7fac310 0x55cde7f4a790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde7f28df0 0x55cde7f00b20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55cde7f26f30 0x55cde7fad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd04cff9640 0 client.7089 ms_handle_reset on v2:172.21.15.28:6832/1426218269 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 --2- 172.21.15.28:0/1357462424 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55cde7f091b0 0x55cde7f3aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.738+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 >> 172.21.15.28:0/1357462424 conn(0x55cde7ede5f0 msgr2=0x55cde7f291d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 shutdown_connections 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd077037740 1 -- 172.21.15.28:0/1357462424 wait complete. 2024-09-10T20:43:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd077037740 20 client.7089 trim_cache size 0 max 16384 2024-09-10T20:43:02.749 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:02.749 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:43:02.749 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:43:02.749 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6936, 2024-09-10T20:43:02.749 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:43:02.749 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:43:02.749 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:43:02.750 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7251354110000001, 2024-09-10T20:43:02.750 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 66, 2024-09-10T20:43:02.750 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 100, 2024-09-10T20:43:02.750 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 100, 2024-09-10T20:43:02.750 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.430604245 2024-09-10T20:43:02.750 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd050078090 msgr2=0x7fd05007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd050078090 0x7fd05007a550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd05c033000 tx=0x7fd05c02d000 comp rx=0 tx=0).stop 2024-09-10T20:43:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 msgr2=0x7fd06c1595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 0x7fd06c1595e0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd068011ae0 tx=0x7fd068009e40 comp rx=0 tx=0).stop 2024-09-10T20:43:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 shutdown_connections 2024-09-10T20:43:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd050078090 0x7fd05007a550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd06c0b1780 0x7fd06c1f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd06c0b1300 0x7fd06c1f4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 --2- 172.21.15.28:0/2794190919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0580043b0 0x7fd06c1595e0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 >> 172.21.15.28:0/2794190919 conn(0x7fd06c0d0420 msgr2=0x7fd06c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 shutdown_connections 2024-09-10T20:43:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.739+0000 7fd074191640 1 -- 172.21.15.28:0/2794190919 wait complete. 2024-09-10T20:43:02.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:43:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.854+0000 7f1591e6a640 1 -- 172.21.15.28:0/3879553408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 msgr2=0x7f158c0d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.854+0000 7f1591e6a640 1 --2- 172.21.15.28:0/3879553408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0d0530 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f1580005fd0 tx=0x7f158002a580 comp rx=0 tx=0).stop 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.854+0000 7f1591e6a640 1 -- 172.21.15.28:0/3879553408 shutdown_connections 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.854+0000 7f1591e6a640 1 --2- 172.21.15.28:0/3879553408 >> v2:172.21.15.159:3300/0 conn(0x7f158c0cec60 0x7f158c0cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.854+0000 7f1591e6a640 1 --2- 172.21.15.28:0/3879553408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0d0530 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.855+0000 7f1591e6a640 1 --2- 172.21.15.28:0/3879553408 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f158c155020 0x7f158c0b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.855+0000 7f1591e6a640 1 -- 172.21.15.28:0/3879553408 >> 172.21.15.28:0/3879553408 conn(0x7f158c06cf00 msgr2=0x7f158c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.855+0000 7f1591e6a640 1 -- 172.21.15.28:0/3879553408 shutdown_connections 2024-09-10T20:43:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.855+0000 7f1591e6a640 1 -- 172.21.15.28:0/3879553408 wait complete. 2024-09-10T20:43:02.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.856+0000 7f1591e6a640 1 Processor -- start 2024-09-10T20:43:02.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.856+0000 7f1591e6a640 1 -- start start 2024-09-10T20:43:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f1591e6a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f158c0cec60 0x7f158c09d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f1591e6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f1591e6a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f158c155020 0x7f158c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f1591e6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f158c0acee0 con 0x7f158c0d00f0 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f1591e6a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f158c0acd60 con 0x7f158c0cec60 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f1591e6a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f158c0ad060 con 0x7f158c155020 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47604/0 (socket says 172.21.15.28:47604) 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 -- 172.21.15.28:0/2815325952 learned_addr learned my addr 172.21.15.28:0/2815325952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158bfff640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f158c155020 0x7f158c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158b7fe640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f158c0cec60 0x7f158c09d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 -- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f158c155020 msgr2=0x7f158c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f158c155020 0x7f158c09ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 -- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f158c0cec60 msgr2=0x7f158c09d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f158c0cec60 0x7f158c09d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 -- 172.21.15.28:0/2815325952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158c09ddc0 con 0x7f158c0d00f0 2024-09-10T20:43:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.857+0000 7f158affd640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0b21e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1580009850 tx=0x7f15800039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.858+0000 7f1573fff640 1 -- 172.21.15.28:0/2815325952 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158003f070 con 0x7f158c0d00f0 2024-09-10T20:43:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.858+0000 7f1573fff640 1 -- 172.21.15.28:0/2815325952 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1580004890 con 0x7f158c0d00f0 2024-09-10T20:43:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.858+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158c09d800 con 0x7f158c0d00f0 2024-09-10T20:43:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.858+0000 7f1573fff640 1 -- 172.21.15.28:0/2815325952 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580035030 con 0x7f158c0d00f0 2024-09-10T20:43:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.858+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158c1567f0 con 0x7f158c0d00f0 2024-09-10T20:43:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.859+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f158c0d0530 con 0x7f158c0d00f0 2024-09-10T20:43:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.859+0000 7f1573fff640 1 -- 172.21.15.28:0/2815325952 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f1580004c60 con 0x7f158c0d00f0 2024-09-10T20:43:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.860+0000 7f1573fff640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1578078090 0x7f157807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.860+0000 7f1573fff640 1 -- 172.21.15.28:0/2815325952 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f15800bb1c0 con 0x7f158c0d00f0 2024-09-10T20:43:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.860+0000 7f158b7fe640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1578078090 0x7f157807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.860+0000 7f158b7fe640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1578078090 0x7f157807a550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f157c00a490 tx=0x7f157c00a420 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:02.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:02.864+0000 7f1573fff640 1 -- 172.21.15.28:0/2815325952 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f15800833d0 con 0x7f158c0d00f0 2024-09-10T20:43:03.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.086+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f158c154590 con 0x7f158c0d00f0 2024-09-10T20:43:03.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.086+0000 7f1573fff640 1 -- 172.21.15.28:0/2815325952 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) ==== 53+0+73 (secure 0 0 0) 0x7f1580087e00 con 0x7f158c0d00f0 2024-09-10T20:43:03.090 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.088+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1578078090 msgr2=0x7f157807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.088+0000 7f1591e6a640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1578078090 0x7f157807a550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f157c00a490 tx=0x7f157c00a420 comp rx=0 tx=0).stop 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.088+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 msgr2=0x7f158c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.088+0000 7f1591e6a640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0b21e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1580009850 tx=0x7f15800039e0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 shutdown_connections 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1578078090 0x7f157807a550 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f158c155020 0x7f158c09ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f158c0d00f0 0x7f158c0b21e0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 --2- 172.21.15.28:0/2815325952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f158c0cec60 0x7f158c09d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 >> 172.21.15.28:0/2815325952 conn(0x7f158c06cf00 msgr2=0x7f158c072a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 shutdown_connections 2024-09-10T20:43:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.089+0000 7f1591e6a640 1 -- 172.21.15.28:0/2815325952 wait complete. 2024-09-10T20:43:03.103 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:43:03.103 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:43:03.103 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:43:03.103 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:43:03.103 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:03.103 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:43:03.104 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:43:03.104 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:43:03.104 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:43:03.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:03.104 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:43:03.120 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:43:03.120 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:03.121 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42448sec preferred_lft 42448sec 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:03.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:03.124 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:43:03.185 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:43:03.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:03.186 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:43:03.243 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:43:03.243 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:43:03.243 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:43:03.243 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:43:03.244 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:43:03.244 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:43:03.244 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:43:03.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:43:03.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:43:03.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:43:03.244 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:43:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:43:03.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:03.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:03.372 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:43:03.388 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:43:03.388 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:43:03.388 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:43:03.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:03.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:43:03.454 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:43:03.454 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:43:03.510 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:43:03.510 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:03.566 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:43:03.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:03.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:03.650 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:03.651 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:43:03.651 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:43:03.659 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:03.659 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:43:03.660 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:43:03.660 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:43:03.660 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:43:03.666 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:03.666 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:43:03.666 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:43:03.666 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:43:03.666 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:43:03.666 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:43:03.666 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:43:03.667 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:43:03.667 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:43:03.667 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:43:03.667 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:43:03.667 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:43:03.667 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:43:03.668 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:43:03.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:03.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:03.694 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:43:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:43:03.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:03.795 INFO:teuthology.orchestra.run.smithi028.stdout:client6403 2024-09-10T20:43:03.795 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:03.796 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:03.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:43:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 -- 172.21.15.28:0/2544962573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415a190 msgr2=0x7ff0040b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 --2- 172.21.15.28:0/2544962573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415a190 0x7ff0040b2ba0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ff0040d09e0 tx=0x7feff402a540 comp rx=0 tx=0).stop 2024-09-10T20:43:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 -- 172.21.15.28:0/2544962573 shutdown_connections 2024-09-10T20:43:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 --2- 172.21.15.28:0/2544962573 >> v2:172.21.15.159:3300/0 conn(0x7ff0040b4d10 0x7ff0040ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 --2- 172.21.15.28:0/2544962573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415a190 0x7ff0040b2ba0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 -- 172.21.15.28:0/2544962573 >> 172.21.15.28:0/2544962573 conn(0x7ff00406cf00 msgr2=0x7ff00406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 -- 172.21.15.28:0/2544962573 shutdown_connections 2024-09-10T20:43:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.953+0000 7ff008b64640 1 -- 172.21.15.28:0/2544962573 wait complete. 2024-09-10T20:43:03.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.954+0000 7ff008b64640 1 Processor -- start 2024-09-10T20:43:03.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.954+0000 7ff008b64640 1 -- start start 2024-09-10T20:43:03.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff008b64640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0040b4d10 0x7ff004151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:03.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff008b64640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff00415a190 0x7ff004151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:03.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff008b64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415df70 0x7ff0041e8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:03.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff008b64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040d0270 con 0x7ff00415df70 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff008b64640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040d00f0 con 0x7ff00415a190 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff008b64640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff0040d03f0 con 0x7ff0040b4d10 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415df70 0x7ff0041e8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415df70 0x7ff0041e8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47614/0 (socket says 172.21.15.28:47614) 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 -- 172.21.15.28:0/1410313756 learned_addr learned my addr 172.21.15.28:0/1410313756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff00259c640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0040b4d10 0x7ff004151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff001d9b640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff00415a190 0x7ff004151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 -- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0040b4d10 msgr2=0x7ff004151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0040b4d10 0x7ff004151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 -- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff00415a190 msgr2=0x7ff004151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:03.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff00415a190 0x7ff004151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff002d9d640 1 -- 172.21.15.28:0/1410313756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00409b6d0 con 0x7ff00415df70 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.955+0000 7ff001d9b640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff00415a190 0x7ff004151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.956+0000 7ff002d9d640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415df70 0x7ff0041e8130 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7feffc00dd70 tx=0x7feffc00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.956+0000 7feff2ffd640 1 -- 172.21.15.28:0/1410313756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc004d20 con 0x7ff00415df70 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.956+0000 7feff2ffd640 1 -- 172.21.15.28:0/1410313756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feffc00be40 con 0x7ff00415df70 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.956+0000 7feff2ffd640 1 -- 172.21.15.28:0/1410313756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc00e8a0 con 0x7ff00415df70 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.956+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00409b9b0 con 0x7ff00415df70 2024-09-10T20:43:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.956+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff00409bf70 con 0x7ff00415df70 2024-09-10T20:43:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.957+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0040ce030 con 0x7ff00415df70 2024-09-10T20:43:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.957+0000 7feff2ffd640 1 -- 172.21.15.28:0/1410313756 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7feffc00b040 con 0x7ff00415df70 2024-09-10T20:43:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.958+0000 7feff2ffd640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fefe40780c0 0x7fefe407a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.958+0000 7ff00259c640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fefe40780c0 0x7fefe407a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.958+0000 7feff2ffd640 1 -- 172.21.15.28:0/1410313756 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7feffc09a980 con 0x7ff00415df70 2024-09-10T20:43:03.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.958+0000 7ff00259c640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fefe40780c0 0x7fefe407a580 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7feff4033000 tx=0x7feff4004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:03.962+0000 7feff2ffd640 1 -- 172.21.15.28:0/1410313756 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feffc063140 con 0x7ff00415df70 2024-09-10T20:43:04.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.187+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff00415b530 con 0x7ff00415df70 2024-09-10T20:43:04.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.187+0000 7feff2ffd640 1 -- 172.21.15.28:0/1410313756 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) ==== 53+0+73 (secure 0 0 0) 0x7feffc062a40 con 0x7ff00415df70 2024-09-10T20:43:04.191 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.189+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fefe40780c0 msgr2=0x7fefe407a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fefe40780c0 0x7fefe407a580 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7feff4033000 tx=0x7feff4004730 comp rx=0 tx=0).stop 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415df70 msgr2=0x7ff0041e8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415df70 0x7ff0041e8130 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7feffc00dd70 tx=0x7feffc00b730 comp rx=0 tx=0).stop 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 shutdown_connections 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fefe40780c0 0x7fefe407a580 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff00415df70 0x7ff0041e8130 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff00415a190 0x7ff004151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 --2- 172.21.15.28:0/1410313756 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0040b4d10 0x7ff004151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 >> 172.21.15.28:0/1410313756 conn(0x7ff00406cf00 msgr2=0x7ff004075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 shutdown_connections 2024-09-10T20:43:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:04.190+0000 7ff008b64640 1 -- 172.21.15.28:0/1410313756 wait complete. 2024-09-10T20:43:04.204 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:43:04.204 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:43:04.204 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:43:04.204 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:43:04.204 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:43:04.204 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:43:04.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:43:04.205 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:43:04.205 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:43:04.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:04.205 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:43:04.224 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:43:04.224 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:43:04.224 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:43:04.224 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:43:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42465sec preferred_lft 42465sec 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:43:04.226 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:04.227 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:04.227 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:43:04.227 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:43:04.227 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:04.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:04.227 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:43:04.293 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:43:04.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:04.295 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:43:04.353 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:43:04.353 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:43:04.354 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:43:04.354 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:43:04.354 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:43:04.354 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:43:04.354 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:43:04.354 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:43:04.354 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:43:04.354 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:43:04.354 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:43:04.446 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:43:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:43:04.480 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:43:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:04.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:04.486 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:43:04.502 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:43:04.502 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:43:04.502 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:43:04.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:04.502 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:43:04.567 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:43:04.567 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:43:04.623 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:43:04.624 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:43:04.624 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:43:04.679 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:43:04.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:04.679 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:43:04.762 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:43:04.762 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:43:04.762 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:43:04.771 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:43:04.771 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:43:04.771 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:43:04.771 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:43:04.771 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:43:04.777 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:43:04.777 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:43:04.777 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:43:04.777 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:43:04.777 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:43:04.777 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:43:04.778 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:43:04.778 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:43:04.778 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:43:04.778 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:43:04.778 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:43:04.778 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:43:04.778 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:43:04.780 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:43:04.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:04.780 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:43:04.845 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:43:04.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:43:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:43:04.935 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:43:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:04.940 INFO:teuthology.orchestra.run.smithi159.stdout:client7104 2024-09-10T20:43:04.941 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:43:04.941 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:04.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:43:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.079+0000 7fba24cbd640 1 -- 172.21.15.28:0/2156352415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200ce7b0 msgr2=0x7fba200cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.079+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2156352415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200ce7b0 0x7fba200cf970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fba10009830 tx=0x7fba1002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:43:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.079+0000 7fba24cbd640 1 -- 172.21.15.28:0/2156352415 shutdown_connections 2024-09-10T20:43:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.079+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2156352415 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200ce7b0 0x7fba200cf970 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.079+0000 7fba24cbd640 1 -- 172.21.15.28:0/2156352415 >> 172.21.15.28:0/2156352415 conn(0x7fba200d1da0 msgr2=0x7fba200d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.079+0000 7fba24cbd640 1 -- 172.21.15.28:0/2156352415 shutdown_connections 2024-09-10T20:43:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.079+0000 7fba24cbd640 1 -- 172.21.15.28:0/2156352415 wait complete. 2024-09-10T20:43:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.080+0000 7fba24cbd640 1 Processor -- start 2024-09-10T20:43:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.080+0000 7fba24cbd640 1 -- start start 2024-09-10T20:43:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba24cbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba200b58e0 0x7fba201512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba24cbd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba200ce7b0 0x7fba20070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba24cbd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200d28c0 0x7fba200714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba24cbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba200b6790 con 0x7fba200b58e0 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba24cbd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fba200b6610 con 0x7fba200d28c0 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba24cbd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fba200b6910 con 0x7fba200ce7b0 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba200b58e0 0x7fba201512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba200b58e0 0x7fba201512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47626/0 (socket says 172.21.15.28:47626) 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba1e59c640 1 -- 172.21.15.28:0/2785799090 learned_addr learned my addr 172.21.15.28:0/2785799090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.081+0000 7fba1ed9d640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200d28c0 0x7fba200714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1dd9b640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba200ce7b0 0x7fba20070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1e59c640 1 -- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba200ce7b0 msgr2=0x7fba20070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1e59c640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba200ce7b0 0x7fba20070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1e59c640 1 -- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200d28c0 msgr2=0x7fba200714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1e59c640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200d28c0 0x7fba200714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1e59c640 1 -- 172.21.15.28:0/2785799090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba200b1990 con 0x7fba200b58e0 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1ed9d640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200d28c0 0x7fba200714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba1e59c640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba200b58e0 0x7fba201512c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fba10009950 tx=0x7fba10003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba16ffd640 1 -- 172.21.15.28:0/2785799090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10005c90 con 0x7fba200b58e0 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba200b26e0 con 0x7fba200b58e0 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba16ffd640 1 -- 172.21.15.28:0/2785799090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba10005020 con 0x7fba200b58e0 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba16ffd640 1 -- 172.21.15.28:0/2785799090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10005450 con 0x7fba200b58e0 2024-09-10T20:43:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.082+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba200b1ee0 con 0x7fba200b58e0 2024-09-10T20:43:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.083+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba2006d750 con 0x7fba200b58e0 2024-09-10T20:43:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.084+0000 7fba16ffd640 1 -- 172.21.15.28:0/2785799090 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fba100317c0 con 0x7fba200b58e0 2024-09-10T20:43:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.084+0000 7fba16ffd640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fba00078090 0x7fba0007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.085+0000 7fba16ffd640 1 -- 172.21.15.28:0/2785799090 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fba100bb2a0 con 0x7fba200b58e0 2024-09-10T20:43:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.085+0000 7fba1dd9b640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fba00078090 0x7fba0007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.085+0000 7fba1dd9b640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fba00078090 0x7fba0007a550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fba200b4000 tx=0x7fba08009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.089+0000 7fba16ffd640 1 -- 172.21.15.28:0/2785799090 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fba10040030 con 0x7fba200b58e0 2024-09-10T20:43:05.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:05.311+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fba200cf970 con 0x7fba200b58e0 2024-09-10T20:43:06.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.055+0000 7fba16ffd640 1 -- 172.21.15.28:0/2785799090 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v67) ==== 136+0+0 (secure 0 0 0) 0x7fba10083b90 con 0x7fba200b58e0 2024-09-10T20:43:06.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fba00078090 msgr2=0x7fba0007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fba00078090 0x7fba0007a550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fba200b4000 tx=0x7fba08009040 comp rx=0 tx=0).stop 2024-09-10T20:43:06.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba200b58e0 msgr2=0x7fba201512c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba200b58e0 0x7fba201512c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fba10009950 tx=0x7fba10003c10 comp rx=0 tx=0).stop 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 shutdown_connections 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fba00078090 0x7fba0007a550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fba200d28c0 0x7fba200714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fba200ce7b0 0x7fba20070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 --2- 172.21.15.28:0/2785799090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba200b58e0 0x7fba201512c0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 >> 172.21.15.28:0/2785799090 conn(0x7fba200d1da0 msgr2=0x7fba200d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 shutdown_connections 2024-09-10T20:43:06.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.058+0000 7fba24cbd640 1 -- 172.21.15.28:0/2785799090 wait complete. 2024-09-10T20:43:06.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.185+0000 7fa3b27b4640 1 -- 172.21.15.28:0/474398248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac15de70 msgr2=0x7fa3ac0d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.185+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/474398248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac15de70 0x7fa3ac0d2b90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa39c005fd0 tx=0x7fa39c02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.186+0000 7fa3b27b4640 1 -- 172.21.15.28:0/474398248 shutdown_connections 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.186+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/474398248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac162260 0x7fa3ac0cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.186+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/474398248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac15de70 0x7fa3ac0d2b90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.186+0000 7fa3b27b4640 1 -- 172.21.15.28:0/474398248 >> 172.21.15.28:0/474398248 conn(0x7fa3ac0d3fb0 msgr2=0x7fa3ac0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.186+0000 7fa3b27b4640 1 -- 172.21.15.28:0/474398248 shutdown_connections 2024-09-10T20:43:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.186+0000 7fa3b27b4640 1 -- 172.21.15.28:0/474398248 wait complete. 2024-09-10T20:43:06.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.187+0000 7fa3b27b4640 1 Processor -- start 2024-09-10T20:43:06.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.187+0000 7fa3b27b4640 1 -- start start 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3b27b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac0cf780 0x7fa3ac15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3b27b4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac15de70 0x7fa3ac0a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3b27b4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac162260 0x7fa3ac0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3b27b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac0b7190 con 0x7fa3ac0cf780 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3b27b4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac0b7010 con 0x7fa3ac162260 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3b27b4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ac0b7310 con 0x7fa3ac15de70 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac0cf780 0x7fa3ac15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac0cf780 0x7fa3ac15aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47644/0 (socket says 172.21.15.28:47644) 2024-09-10T20:43:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 -- 172.21.15.28:0/483673442 learned_addr learned my addr 172.21.15.28:0/483673442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3ab7fe640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac15de70 0x7fa3ac0a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3b0d51640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac162260 0x7fa3ac0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 -- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac15de70 msgr2=0x7fa3ac0a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac15de70 0x7fa3ac0a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 -- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac162260 msgr2=0x7fa3ac0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac162260 0x7fa3ac0cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 -- 172.21.15.28:0/483673442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac0a0f60 con 0x7fa3ac0cf780 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.188+0000 7fa3abfff640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac0cf780 0x7fa3ac15aa10 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa39c000c00 tx=0x7fa39c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.189+0000 7fa3a8ff9640 1 -- 172.21.15.28:0/483673442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c03f070 con 0x7fa3ac0cf780 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.189+0000 7fa3a8ff9640 1 -- 172.21.15.28:0/483673442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa39c005600 con 0x7fa3ac0cf780 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.189+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ac1f56e0 con 0x7fa3ac0cf780 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.189+0000 7fa3a8ff9640 1 -- 172.21.15.28:0/483673442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c035030 con 0x7fa3ac0cf780 2024-09-10T20:43:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.189+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3ac1f5120 con 0x7fa3ac0cf780 2024-09-10T20:43:06.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.190+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3ac0b7830 con 0x7fa3ac0cf780 2024-09-10T20:43:06.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.190+0000 7fa3a8ff9640 1 -- 172.21.15.28:0/483673442 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fa39c004a00 con 0x7fa3ac0cf780 2024-09-10T20:43:06.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.191+0000 7fa3a8ff9640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa398078090 0x7fa39807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.191+0000 7fa3ab7fe640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa398078090 0x7fa39807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.191+0000 7fa3a8ff9640 1 -- 172.21.15.28:0/483673442 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fa39c0baf70 con 0x7fa3ac0cf780 2024-09-10T20:43:06.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.192+0000 7fa3ab7fe640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa398078090 0x7fa39807a550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0a1d00 tx=0x7fa3a0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:06.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.196+0000 7fa3a8ff9640 1 -- 172.21.15.28:0/483673442 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa39c083850 con 0x7fa3ac0cf780 2024-09-10T20:43:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.403+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fa3ac0b9f30 con 0x7fa398078090 2024-09-10T20:43:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.404+0000 7fa3a8ff9640 1 -- 172.21.15.28:0/483673442 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fa3ac0b9f30 con 0x7fa398078090 2024-09-10T20:43:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:43:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:43:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:43:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa398078090 msgr2=0x7fa39807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa398078090 0x7fa39807a550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0a1d00 tx=0x7fa3a0009040 comp rx=0 tx=0).stop 2024-09-10T20:43:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac0cf780 msgr2=0x7fa3ac15aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac0cf780 0x7fa3ac15aa10 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa39c000c00 tx=0x7fa39c0038a0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 shutdown_connections 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa398078090 0x7fa39807a550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3ac162260 0x7fa3ac0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ac15de70 0x7fa3ac0a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 --2- 172.21.15.28:0/483673442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ac0cf780 0x7fa3ac15aa10 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 >> 172.21.15.28:0/483673442 conn(0x7fa3ac0d3fb0 msgr2=0x7fa3ac0b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 shutdown_connections 2024-09-10T20:43:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.406+0000 7fa3b27b4640 1 -- 172.21.15.28:0/483673442 wait complete. 2024-09-10T20:43:06.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.524+0000 7f7fac7ca640 1 -- 172.21.15.28:0/2413814053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa4152e00 msgr2=0x7f7fa40b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.524+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/2413814053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa4152e00 0x7f7fa40b1f80 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009830 tx=0x7f7fa002a480 comp rx=0 tx=0).stop 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.525+0000 7f7fac7ca640 1 -- 172.21.15.28:0/2413814053 shutdown_connections 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.525+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/2413814053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa40b3580 0x7f7fa406e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.525+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/2413814053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa4152e00 0x7f7fa40b1f80 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.525+0000 7f7fac7ca640 1 -- 172.21.15.28:0/2413814053 >> 172.21.15.28:0/2413814053 conn(0x7f7fa406d110 msgr2=0x7f7fa40b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.525+0000 7f7fac7ca640 1 -- 172.21.15.28:0/2413814053 shutdown_connections 2024-09-10T20:43:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.525+0000 7f7fac7ca640 1 -- 172.21.15.28:0/2413814053 wait complete. 2024-09-10T20:43:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.526+0000 7f7fac7ca640 1 Processor -- start 2024-09-10T20:43:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.526+0000 7f7fac7ca640 1 -- start start 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.526+0000 7f7fac7ca640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa406e940 0x7f7fa4159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7fac7ca640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4152e00 0x7f7fa415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7fac7ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa415b3e0 0x7f7fa4140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7fac7ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa415f820 con 0x7f7fa415b3e0 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7fac7ca640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa40cf640 con 0x7f7fa406e940 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7fac7ca640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa415f9a0 con 0x7f7fa4152e00 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa415b3e0 0x7f7fa4140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa415b3e0 0x7f7fa4140410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47664/0 (socket says 172.21.15.28:47664) 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 -- 172.21.15.28:0/1508246661 learned_addr learned my addr 172.21.15.28:0/1508246661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7fa9d65640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4152e00 0x7f7fa415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faa566640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa406e940 0x7f7fa4159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 -- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4152e00 msgr2=0x7f7fa415aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4152e00 0x7f7fa415aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 -- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa406e940 msgr2=0x7f7fa4159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa406e940 0x7f7fa4159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 -- 172.21.15.28:0/1508246661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa415b920 con 0x7f7fa415b3e0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faa566640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa406e940 0x7f7fa4159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7faad67640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa415b3e0 0x7f7fa4140410 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00cbc0 tx=0x7f7f9c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7f9affd640 1 -- 172.21.15.28:0/1508246661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c016020 con 0x7f7fa415b3e0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7f9affd640 1 -- 172.21.15.28:0/1508246661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f9c00add0 con 0x7f7fa415b3e0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa415bc00 con 0x7f7fa415b3e0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.527+0000 7f7f9affd640 1 -- 172.21.15.28:0/1508246661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c012680 con 0x7f7fa415b3e0 2024-09-10T20:43:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.528+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa415c1c0 con 0x7f7fa415b3e0 2024-09-10T20:43:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.529+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa415fe70 con 0x7f7fa415b3e0 2024-09-10T20:43:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.529+0000 7f7f9affd640 1 -- 172.21.15.28:0/1508246661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f7f9c012820 con 0x7f7fa415b3e0 2024-09-10T20:43:06.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.529+0000 7f7f9affd640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7f88078090 0x7f7f8807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:06.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.530+0000 7f7f9affd640 1 -- 172.21.15.28:0/1508246661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f7f9c09a6b0 con 0x7f7fa415b3e0 2024-09-10T20:43:06.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.530+0000 7f7faa566640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7f88078090 0x7f7f8807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:06.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.530+0000 7f7faa566640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7f88078090 0x7f7f8807a550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0034000 tx=0x7f7fa0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:06.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.534+0000 7f7f9affd640 1 -- 172.21.15.28:0/1508246661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7f9c020030 con 0x7f7fa415b3e0 2024-09-10T20:43:06.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.717+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f7fa414e340 con 0x7f7fa415b3e0 2024-09-10T20:43:06.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.718+0000 7f7f9affd640 1 -- 172.21.15.28:0/1508246661 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7f7f9c062f90 con 0x7f7fa415b3e0 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7f88078090 msgr2=0x7f7f8807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7f88078090 0x7f7f8807a550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0034000 tx=0x7f7fa0004730 comp rx=0 tx=0).stop 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa415b3e0 msgr2=0x7f7fa4140410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa415b3e0 0x7f7fa4140410 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00cbc0 tx=0x7f7f9c00a430 comp rx=0 tx=0).stop 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 shutdown_connections 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7f88078090 0x7f7f8807a550 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fa415b3e0 0x7f7fa4140410 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fa4152e00 0x7f7fa415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 --2- 172.21.15.28:0/1508246661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fa406e940 0x7f7fa4159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 >> 172.21.15.28:0/1508246661 conn(0x7f7fa406d110 msgr2=0x7f7fa415e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:06.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 shutdown_connections 2024-09-10T20:43:06.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:06.720+0000 7f7fac7ca640 1 -- 172.21.15.28:0/1508246661 wait complete. 2024-09-10T20:43:06.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:06.735 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385) 2024-09-10T20:43:06.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:06.767 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/fake) 2024-09-10T20:43:06.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:06.840 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:43:06.896 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:43:06.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:06.896 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385) 2024-09-10T20:43:06.965 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:06.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol63210/.meta 2024-09-10T20:43:07.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol63210 2024-09-10T20:43:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.182+0000 7f0e11e62640 1 -- 172.21.15.28:0/4179781745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 msgr2=0x7f0df8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.182+0000 7f0e11e62640 1 --2- 172.21.15.28:0/4179781745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 0x7f0df8004c80 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f0df4001660 tx=0x7f0df8005830 comp rx=0 tx=0).stop 2024-09-10T20:43:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.182+0000 7f0e0bfff640 1 -- 172.21.15.28:0/4179781745 reap_dead start 2024-09-10T20:43:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.182+0000 7f0e11e62640 1 -- 172.21.15.28:0/4179781745 shutdown_connections 2024-09-10T20:43:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.182+0000 7f0e11e62640 1 -- 172.21.15.28:0/4179781745 >> 172.21.15.28:0/4179781745 conn(0x7f0e0c06d110 msgr2=0x7f0e0c0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.182+0000 7f0e11e62640 1 -- 172.21.15.28:0/4179781745 shutdown_connections 2024-09-10T20:43:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.182+0000 7f0e11e62640 1 -- 172.21.15.28:0/4179781745 wait complete. 2024-09-10T20:43:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.183+0000 7f0e11e62640 1 Processor -- start 2024-09-10T20:43:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.183+0000 7f0e11e62640 1 -- start start 2024-09-10T20:43:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e11e62640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 0x7f0e0c1f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e11e62640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e0c14da10 0x7f0e0c158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e11e62640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e0c158660 0x7f0e0c0731c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e11e62640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c0cf520 con 0x7f0df8004800 2024-09-10T20:43:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e11e62640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c0cf3a0 con 0x7f0e0c158660 2024-09-10T20:43:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e11e62640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0e0c1590a0 con 0x7f0e0c14da10 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 0x7f0e0c1f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 0x7f0e0c1f3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47680/0 (socket says 172.21.15.28:47680) 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e0b7fe640 1 -- 172.21.15.28:0/1600283672 learned_addr learned my addr 172.21.15.28:0/1600283672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e0bfff640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e0c158660 0x7f0e0c0731c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.184+0000 7f0e0affd640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e0c14da10 0x7f0e0c158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e0b7fe640 1 -- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e0c14da10 msgr2=0x7f0e0c158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e0b7fe640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e0c14da10 0x7f0e0c158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e0b7fe640 1 -- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e0c158660 msgr2=0x7f0e0c0731c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e0b7fe640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e0c158660 0x7f0e0c0731c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e0b7fe640 1 -- 172.21.15.28:0/1600283672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0c1f4490 con 0x7f0df8004800 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e0bfff640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e0c158660 0x7f0e0c0731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e0b7fe640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 0x7f0e0c1f3f50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc006310 tx=0x7f0dfc0067d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e03fff640 1 -- 172.21.15.28:0/1600283672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc015070 con 0x7f0df8004800 2024-09-10T20:43:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e03fff640 1 -- 172.21.15.28:0/1600283672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dfc00fce0 con 0x7f0df8004800 2024-09-10T20:43:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0c0c66c0 con 0x7f0df8004800 2024-09-10T20:43:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e03fff640 1 -- 172.21.15.28:0/1600283672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc011690 con 0x7f0df8004800 2024-09-10T20:43:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.185+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0c158dc0 con 0x7f0df8004800 2024-09-10T20:43:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.186+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e0c06e400 con 0x7f0df8004800 2024-09-10T20:43:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.187+0000 7f0e03fff640 1 -- 172.21.15.28:0/1600283672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f0dfc011830 con 0x7f0df8004800 2024-09-10T20:43:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.187+0000 7f0e03fff640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0dec078090 0x7f0dec07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.187+0000 7f0e0affd640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0dec078090 0x7f0dec07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.187+0000 7f0e03fff640 1 -- 172.21.15.28:0/1600283672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f0dfc09aea0 con 0x7f0df8004800 2024-09-10T20:43:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.188+0000 7f0e0affd640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0dec078090 0x7f0dec07a550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0df4032000 tx=0x7f0df4001190 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:07.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.192+0000 7f0e03fff640 1 -- 172.21.15.28:0/1600283672 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0e0c1f4490 con 0x7f0df8004800 2024-09-10T20:43:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.405+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol63210", "target": ["mon-mgr", ""]}) -- 0x7f0e0c150200 con 0x7f0dec078090 2024-09-10T20:43:07.413 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:07.409+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:07.413 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:07.409+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:07.413 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:07.409+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:07.414 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:07.409+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:07.414 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:07.410+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.441+0000 7f0e03fff640 1 -- 172.21.15.28:0/1600283672 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f0e0c150200 con 0x7f0dec078090 2024-09-10T20:43:07.445 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385 2024-09-10T20:43:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0dec078090 msgr2=0x7f0dec07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0dec078090 0x7f0dec07a550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0df4032000 tx=0x7f0df4001190 comp rx=0 tx=0).stop 2024-09-10T20:43:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 msgr2=0x7f0e0c1f3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 0x7f0e0c1f3f50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc006310 tx=0x7f0dfc0067d0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 shutdown_connections 2024-09-10T20:43:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0dec078090 0x7f0dec07a550 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0e0c158660 0x7f0e0c0731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e0c14da10 0x7f0e0c158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 --2- 172.21.15.28:0/1600283672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df8004800 0x7f0e0c1f3f50 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 >> 172.21.15.28:0/1600283672 conn(0x7f0e0c06d110 msgr2=0x7f0e0c15fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 shutdown_connections 2024-09-10T20:43:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.443+0000 7f0e11e62640 1 -- 172.21.15.28:0/1600283672 wait complete. 2024-09-10T20:43:07.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 -- 172.21.15.28:0/637764884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c240d1240 msgr2=0x7f1c2406f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/637764884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c240d1240 0x7f1c2406f8c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1c240b0810 tx=0x7f1c1c02aa30 comp rx=0 tx=0).stop 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 -- 172.21.15.28:0/637764884 shutdown_connections 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/637764884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c240d1240 0x7f1c2406f8c0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/637764884 >> v2:172.21.15.159:3300/0 conn(0x7f1c240b4fd0 0x7f1c240d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 -- 172.21.15.28:0/637764884 >> 172.21.15.28:0/637764884 conn(0x7f1c2406dbf0 msgr2=0x7f1c2406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 -- 172.21.15.28:0/637764884 shutdown_connections 2024-09-10T20:43:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.562+0000 7f1c2a01f640 1 -- 172.21.15.28:0/637764884 wait complete. 2024-09-10T20:43:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.563+0000 7f1c2a01f640 1 Processor -- start 2024-09-10T20:43:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.563+0000 7f1c2a01f640 1 -- start start 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c2a01f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c240b4fd0 0x7f1c241f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c2a01f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c240d1240 0x7f1c24152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c2a01f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24155250 0x7f1c240af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c2a01f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2415eae0 con 0x7f1c24155250 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c2a01f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2415e960 con 0x7f1c240b4fd0 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c2a01f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2406fe00 con 0x7f1c240d1240 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24155250 0x7f1c240af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24155250 0x7f1c240af6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47722/0 (socket says 172.21.15.28:47722) 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 -- 172.21.15.28:0/843278241 learned_addr learned my addr 172.21.15.28:0/843278241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c237fe640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c240b4fd0 0x7f1c241f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c22ffd640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c240d1240 0x7f1c24152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 -- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c240d1240 msgr2=0x7f1c24152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c240d1240 0x7f1c24152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 -- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c240b4fd0 msgr2=0x7f1c241f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c240b4fd0 0x7f1c241f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c23fff640 1 -- 172.21.15.28:0/843278241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c241f3600 con 0x7f1c24155250 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.564+0000 7f1c237fe640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c240b4fd0 0x7f1c241f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.565+0000 7f1c23fff640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24155250 0x7f1c240af6b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009d60 tx=0x7f1c1c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.565+0000 7f1c0bfff640 1 -- 172.21.15.28:0/843278241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c047070 con 0x7f1c24155250 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.565+0000 7f1c0bfff640 1 -- 172.21.15.28:0/843278241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c1c004250 con 0x7f1c24155250 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.565+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c240b0890 con 0x7f1c24155250 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.565+0000 7f1c0bfff640 1 -- 172.21.15.28:0/843278241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c033050 con 0x7f1c24155250 2024-09-10T20:43:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.565+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c240b0e10 con 0x7f1c24155250 2024-09-10T20:43:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.566+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c24070250 con 0x7f1c24155250 2024-09-10T20:43:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.567+0000 7f1c0bfff640 1 -- 172.21.15.28:0/843278241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f1c1c004680 con 0x7f1c24155250 2024-09-10T20:43:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.567+0000 7f1c0bfff640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c10078090 0x7f1c1007a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.567+0000 7f1c237fe640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c10078090 0x7f1c1007a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.567+0000 7f1c0bfff640 1 -- 172.21.15.28:0/843278241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f1c1c0bb720 con 0x7f1c24155250 2024-09-10T20:43:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.568+0000 7f1c237fe640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c10078090 0x7f1c1007a550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1c240b2350 tx=0x7f1c14008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.572+0000 7f1c0bfff640 1 -- 172.21.15.28:0/843278241 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c241f3600 con 0x7f1c24155250 2024-09-10T20:43:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.781+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) -- 0x7f1c240d0d00 con 0x7f1c10078090 2024-09-10T20:43:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.784+0000 7f1c0bfff640 1 -- 172.21.15.28:0/843278241 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f1c240d0d00 con 0x7f1c10078090 2024-09-10T20:43:07.788 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:43:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subvol63210" 2024-09-10T20:43:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:07.789 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c10078090 msgr2=0x7f1c1007a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c10078090 0x7f1c1007a550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1c240b2350 tx=0x7f1c14008040 comp rx=0 tx=0).stop 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24155250 msgr2=0x7f1c240af6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24155250 0x7f1c240af6b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009d60 tx=0x7f1c1c003f10 comp rx=0 tx=0).stop 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 shutdown_connections 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c10078090 0x7f1c1007a550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24155250 0x7f1c240af6b0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c240d1240 0x7f1c24152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 --2- 172.21.15.28:0/843278241 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c240b4fd0 0x7f1c241f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 >> 172.21.15.28:0/843278241 conn(0x7f1c2406dbf0 msgr2=0x7f1c2415b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 shutdown_connections 2024-09-10T20:43:07.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.787+0000 7f1c2a01f640 1 -- 172.21.15.28:0/843278241 wait complete. 2024-09-10T20:43:07.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol63210 2024-09-10T20:43:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.906+0000 7f55d7da3640 1 -- 172.21.15.28:0/1185559970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 msgr2=0x7f55b8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.906+0000 7f55d7da3640 1 --2- 172.21.15.28:0/1185559970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55b8003e70 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009ed0 tx=0x7f55b8005830 comp rx=0 tx=0).stop 2024-09-10T20:43:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.906+0000 7f55d7da3640 1 -- 172.21.15.28:0/1185559970 shutdown_connections 2024-09-10T20:43:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.906+0000 7f55d7da3640 1 --2- 172.21.15.28:0/1185559970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55b8003e70 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.906+0000 7f55d7da3640 1 -- 172.21.15.28:0/1185559970 >> 172.21.15.28:0/1185559970 conn(0x7f55d00d4130 msgr2=0x7f55d00d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:07.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.906+0000 7f55d7da3640 1 -- 172.21.15.28:0/1185559970 shutdown_connections 2024-09-10T20:43:07.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.906+0000 7f55d7da3640 1 -- 172.21.15.28:0/1185559970 wait complete. 2024-09-10T20:43:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.907+0000 7f55d7da3640 1 Processor -- start 2024-09-10T20:43:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.907+0000 7f55d7da3640 1 -- start start 2024-09-10T20:43:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d7da3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55d0157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d7da3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55d01f8d40 0x7f55d01f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d7da3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55d015ff50 0x7f55d01603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d7da3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55d00d2290 con 0x7f55b8003a50 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d7da3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f55d00d2110 con 0x7f55d015ff50 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d7da3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f55d00d2660 con 0x7f55d01f8d40 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55d0157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55d0157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47726/0 (socket says 172.21.15.28:47726) 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 -- 172.21.15.28:0/3392491589 learned_addr learned my addr 172.21.15.28:0/3392491589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d533e640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55d01f8d40 0x7f55d01f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d6340640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55d015ff50 0x7f55d01603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 -- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55d01f8d40 msgr2=0x7f55d01f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55d01f8d40 0x7f55d01f56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 -- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55d015ff50 msgr2=0x7f55d01603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55d015ff50 0x7f55d01603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.908+0000 7f55d5b3f640 1 -- 172.21.15.28:0/3392491589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d00cb9f0 con 0x7f55b8003a50 2024-09-10T20:43:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.909+0000 7f55d5b3f640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55d0157890 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f55c002aec0 tx=0x7f55c0033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.909+0000 7f55ce7fc640 1 -- 172.21.15.28:0/3392491589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c0038040 con 0x7f55b8003a50 2024-09-10T20:43:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.909+0000 7f55ce7fc640 1 -- 172.21.15.28:0/3392491589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55c0041070 con 0x7f55b8003a50 2024-09-10T20:43:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.909+0000 7f55ce7fc640 1 -- 172.21.15.28:0/3392491589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c003c690 con 0x7f55b8003a50 2024-09-10T20:43:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.909+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d01f6010 con 0x7f55b8003a50 2024-09-10T20:43:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.909+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d01f65d0 con 0x7f55b8003a50 2024-09-10T20:43:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.910+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d0161d00 con 0x7f55b8003a50 2024-09-10T20:43:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.910+0000 7f55ce7fc640 1 -- 172.21.15.28:0/3392491589 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f55c003b070 con 0x7f55b8003a50 2024-09-10T20:43:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.911+0000 7f55ce7fc640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55bc0780c0 0x7f55bc07a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.911+0000 7f55ce7fc640 1 -- 172.21.15.28:0/3392491589 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f55c00ba960 con 0x7f55b8003a50 2024-09-10T20:43:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.911+0000 7f55d533e640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55bc0780c0 0x7f55bc07a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.911+0000 7f55d533e640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55bc0780c0 0x7f55bc07a580 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f55c4001240 tx=0x7f55c4006720 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:07.915+0000 7f55ce7fc640 1 -- 172.21.15.28:0/3392491589 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f55c0082b60 con 0x7f55b8003a50 2024-09-10T20:43:08.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.129+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol63210", "target": ["mon-mgr", ""]}) -- 0x7f55d00caf00 con 0x7f55bc0780c0 2024-09-10T20:43:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.151+0000 7f55ce7fc640 1 -- 172.21.15.28:0/3392491589 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+686 (secure 0 0 0) 0x7f55d00caf00 con 0x7f55bc0780c0 2024-09-10T20:43:08.155 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "atime": "2024-09-10 20:43:06", 2024-09-10T20:43:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_pcent": "undefined", 2024-09-10T20:43:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_quota": "infinite", 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_used": 0, 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "created_at": "2024-09-10 20:43:06", 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "ctime": "2024-09-10 20:43:07", 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "data_pool": "cephfs_data", 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "features": [ 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot-clone", 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot-autoprotect" 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flavor": 1, 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 0, 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": 16877, 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_addrs": [ 2024-09-10T20:43:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6789", 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.159:6789", 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.159:6790" 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-09-10 20:43:06", 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385", 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_namespace": "", 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "complete", 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "subvolume", 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "uid": 0 2024-09-10T20:43:08.157 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55bc0780c0 msgr2=0x7f55bc07a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55bc0780c0 0x7f55bc07a580 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f55c4001240 tx=0x7f55c4006720 comp rx=0 tx=0).stop 2024-09-10T20:43:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 msgr2=0x7f55d0157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55d0157890 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f55c002aec0 tx=0x7f55c0033cb0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 shutdown_connections 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55bc0780c0 0x7f55bc07a580 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55d015ff50 0x7f55d01603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55d01f8d40 0x7f55d01f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 --2- 172.21.15.28:0/3392491589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b8003a50 0x7f55d0157890 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.153+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 >> 172.21.15.28:0/3392491589 conn(0x7f55d00d4130 msgr2=0x7f55d015d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.154+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 shutdown_connections 2024-09-10T20:43:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.154+0000 7f55d7da3640 1 -- 172.21.15.28:0/3392491589 wait complete. 2024-09-10T20:43:08.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize cephfs subvol63210 10485760 2024-09-10T20:43:08.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.274+0000 7fbddb56b640 1 -- 172.21.15.28:0/840037405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0004d40 msgr2=0x7fbdc00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.274+0000 7fbddb56b640 1 --2- 172.21.15.28:0/840037405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0004d40 0x7fbdc00051c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0013d0 tx=0x7fbdc0005830 comp rx=0 tx=0).stop 2024-09-10T20:43:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.274+0000 7fbdd9b08640 1 -- 172.21.15.28:0/840037405 reap_dead start 2024-09-10T20:43:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.274+0000 7fbddb56b640 1 -- 172.21.15.28:0/840037405 shutdown_connections 2024-09-10T20:43:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.274+0000 7fbddb56b640 1 -- 172.21.15.28:0/840037405 >> 172.21.15.28:0/840037405 conn(0x7fbdd40d1da0 msgr2=0x7fbdd40d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.274+0000 7fbddb56b640 1 -- 172.21.15.28:0/840037405 shutdown_connections 2024-09-10T20:43:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.275+0000 7fbddb56b640 1 -- 172.21.15.28:0/840037405 wait complete. 2024-09-10T20:43:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 Processor -- start 2024-09-10T20:43:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 -- start start 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc0004d40 0x7fbdd40c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdd4150bb0 0x7fbdd409bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd409c070 0x7fbdd409e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd40cfaf0 con 0x7fbdd409c070 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd40cf970 con 0x7fbdc0004d40 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.276+0000 7fbddb56b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd40cfc70 con 0x7fbdd4150bb0 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd409c070 0x7fbdd409e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd409c070 0x7fbdd409e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47736/0 (socket says 172.21.15.28:47736) 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 -- 172.21.15.28:0/3279793115 learned_addr learned my addr 172.21.15.28:0/3279793115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9307640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc0004d40 0x7fbdd40c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd8b06640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdd4150bb0 0x7fbdd409bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 -- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdd4150bb0 msgr2=0x7fbdd409bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdd4150bb0 0x7fbdd409bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 -- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc0004d40 msgr2=0x7fbdd40c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc0004d40 0x7fbdd40c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 -- 172.21.15.28:0/3279793115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd41515f0 con 0x7fbdd409c070 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdd9b08640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd409c070 0x7fbdd409e3d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00c6d0 tx=0x7fbdcc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdc9ffb640 1 -- 172.21.15.28:0/3279793115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc039070 con 0x7fbdd409c070 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdc9ffb640 1 -- 172.21.15.28:0/3279793115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdcc02ba90 con 0x7fbdd409c070 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd409f020 con 0x7fbdd409c070 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbdc9ffb640 1 -- 172.21.15.28:0/3279793115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc0061b0 con 0x7fbdd409c070 2024-09-10T20:43:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.277+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd40c6d50 con 0x7fbdd409c070 2024-09-10T20:43:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.278+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdd406d750 con 0x7fbdd409c070 2024-09-10T20:43:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.279+0000 7fbdc9ffb640 1 -- 172.21.15.28:0/3279793115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7fbdcc047060 con 0x7fbdd409c070 2024-09-10T20:43:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.280+0000 7fbdc9ffb640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbdb8078090 0x7fbdb807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.280+0000 7fbdc9ffb640 1 -- 172.21.15.28:0/3279793115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7fbdcc0b9db0 con 0x7fbdd409c070 2024-09-10T20:43:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.280+0000 7fbdd9307640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbdb8078090 0x7fbdb807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.280+0000 7fbdd9307640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbdb8078090 0x7fbdb807a550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0009c50 tx=0x7fbdd0005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.284+0000 7fbdc9ffb640 1 -- 172.21.15.28:0/3279793115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbdcc082190 con 0x7fbdd409c070 2024-09-10T20:43:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.506+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "cephfs", "sub_name": "subvol63210", "new_size": "10485760", "target": ["mon-mgr", ""]}) -- 0x7fbdd409f300 con 0x7fbdb8078090 2024-09-10T20:43:08.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.534+0000 7fbdc9ffb640 1 -- 172.21.15.28:0/3279793115 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+128 (secure 0 0 0) 0x7fbdd409f300 con 0x7fbdb8078090 2024-09-10T20:43:08.538 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:43:08.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:08.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_used": 0 2024-09-10T20:43:08.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:08.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:08.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_quota": 10485760 2024-09-10T20:43:08.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:08.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:08.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_pcent": "0.00" 2024-09-10T20:43:08.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:08.539 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.536+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbdb8078090 msgr2=0x7fbdb807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbdb8078090 0x7fbdb807a550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0009c50 tx=0x7fbdd0005f90 comp rx=0 tx=0).stop 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd409c070 msgr2=0x7fbdd409e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd409c070 0x7fbdd409e3d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00c6d0 tx=0x7fbdcc003ba0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 shutdown_connections 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbdb8078090 0x7fbdb807a550 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdd409c070 0x7fbdd409e3d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbdd4150bb0 0x7fbdd409bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 --2- 172.21.15.28:0/3279793115 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbdc0004d40 0x7fbdd40c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 >> 172.21.15.28:0/3279793115 conn(0x7fbdd40d1da0 msgr2=0x7fbdd415b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:08.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 shutdown_connections 2024-09-10T20:43:08.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.537+0000 7fbddb56b640 1 -- 172.21.15.28:0/3279793115 wait complete. 2024-09-10T20:43:08.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol63210 2024-09-10T20:43:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.655+0000 7f3ecfb05640 1 -- 172.21.15.28:0/3867249485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec815d150 msgr2=0x7f3ec80cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.655+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/3867249485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec815d150 0x7f3ec80cdf90 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80ccae0 tx=0x7f3eb802a560 comp rx=0 tx=0).stop 2024-09-10T20:43:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.656+0000 7f3ecfb05640 1 -- 172.21.15.28:0/3867249485 shutdown_connections 2024-09-10T20:43:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.656+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/3867249485 >> v2:172.21.15.159:3300/0 conn(0x7f3ec80ce4d0 0x7f3ec815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.656+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/3867249485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec815d150 0x7f3ec80cdf90 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.656+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/3867249485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ec80cee10 0x7f3ec80cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.656+0000 7f3ecfb05640 1 -- 172.21.15.28:0/3867249485 >> 172.21.15.28:0/3867249485 conn(0x7f3ec80d0090 msgr2=0x7f3ec80d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.656+0000 7f3ecfb05640 1 -- 172.21.15.28:0/3867249485 shutdown_connections 2024-09-10T20:43:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.656+0000 7f3ecfb05640 1 -- 172.21.15.28:0/3867249485 wait complete. 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.657+0000 7f3ecfb05640 1 Processor -- start 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.657+0000 7f3ecfb05640 1 -- start start 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec80ce4d0 0x7f3ec8159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ec80cee10 0x7f3ec8148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ec815d150 0x7f3ec8157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8160320 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec81601a0 con 0x7f3ec80cee10 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec81604a0 con 0x7f3ec815d150 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec80ce4d0 0x7f3ec8159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec80ce4d0 0x7f3ec8159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47758/0 (socket says 172.21.15.28:47758) 2024-09-10T20:43:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 -- 172.21.15.28:0/117547786 learned_addr learned my addr 172.21.15.28:0/117547786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd0a0640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ec80cee10 0x7f3ec8148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ece0a2640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ec815d150 0x7f3ec8157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 -- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ec815d150 msgr2=0x7f3ec8157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ec815d150 0x7f3ec8157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 -- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ec80cee10 msgr2=0x7f3ec8148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ec80cee10 0x7f3ec8148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 -- 172.21.15.28:0/117547786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec8156cc0 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ece0a2640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ec815d150 0x7f3ec8157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecd8a1640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec80ce4d0 0x7f3ec8159e60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400d7a0 tx=0x7f3ec400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ebe7fc640 1 -- 172.21.15.28:0/117547786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4017070 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ebe7fc640 1 -- 172.21.15.28:0/117547786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ec4009cc0 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec8156fa0 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ebe7fc640 1 -- 172.21.15.28:0/117547786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4005280 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.658+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec8157560 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.660+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec80cf1f0 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.660+0000 7f3ebe7fc640 1 -- 172.21.15.28:0/117547786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f3ec4005420 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.661+0000 7f3ebe7fc640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3eac078090 0x7f3eac07a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.661+0000 7f3ebe7fc640 1 -- 172.21.15.28:0/117547786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f3ec409a140 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.661+0000 7f3ecd0a0640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3eac078090 0x7f3eac07a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.661+0000 7f3ecd0a0640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3eac078090 0x7f3eac07a550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8033000 tx=0x7f3eb8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.665+0000 7f3ebe7fc640 1 -- 172.21.15.28:0/117547786 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3ec4062a20 con 0x7f3ec80ce4d0 2024-09-10T20:43:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.875+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol63210", "target": ["mon-mgr", ""]}) -- 0x7f3ec8069d50 con 0x7f3eac078090 2024-09-10T20:43:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.896+0000 7f3ebe7fc640 1 -- 172.21.15.28:0/117547786 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+679 (secure 0 0 0) 0x7f3ec8069d50 con 0x7f3eac078090 2024-09-10T20:43:08.900 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "atime": "2024-09-10 20:43:06", 2024-09-10T20:43:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_pcent": "0.00", 2024-09-10T20:43:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_quota": 10485760, 2024-09-10T20:43:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_used": 0, 2024-09-10T20:43:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "created_at": "2024-09-10 20:43:06", 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "ctime": "2024-09-10 20:43:08", 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "data_pool": "cephfs_data", 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "features": [ 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot-clone", 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot-autoprotect" 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flavor": 1, 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 0, 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": 16877, 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_addrs": [ 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6789", 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.159:6789", 2024-09-10T20:43:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.159:6790" 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-09-10 20:43:06", 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385", 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_namespace": "", 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "complete", 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "subvolume", 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "uid": 0 2024-09-10T20:43:08.902 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3eac078090 msgr2=0x7f3eac07a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3eac078090 0x7f3eac07a550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8033000 tx=0x7f3eb8004730 comp rx=0 tx=0).stop 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec80ce4d0 msgr2=0x7f3ec8159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec80ce4d0 0x7f3ec8159e60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400d7a0 tx=0x7f3ec400dc60 comp rx=0 tx=0).stop 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 shutdown_connections 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3eac078090 0x7f3eac07a550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ec815d150 0x7f3ec8157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ec80cee10 0x7f3ec8148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 --2- 172.21.15.28:0/117547786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec80ce4d0 0x7f3ec8159e60 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 >> 172.21.15.28:0/117547786 conn(0x7f3ec80d0090 msgr2=0x7f3ec80b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 shutdown_connections 2024-09-10T20:43:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:08.898+0000 7f3ecfb05640 1 -- 172.21.15.28:0/117547786 wait complete. 2024-09-10T20:43:08.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol63210 --mode 777 --uid 1000 --gid 1000 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 -- 172.21.15.28:0/1362953241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 msgr2=0x7f9b1c06e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 --2- 172.21.15.28:0/1362953241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c06e490 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009830 tx=0x7f9b1002a480 comp rx=0 tx=0).stop 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 -- 172.21.15.28:0/1362953241 shutdown_connections 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 --2- 172.21.15.28:0/1362953241 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b1c06e9d0 0x7f9b1c0ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 --2- 172.21.15.28:0/1362953241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c06e490 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 -- 172.21.15.28:0/1362953241 >> 172.21.15.28:0/1362953241 conn(0x7f9b1c06cf00 msgr2=0x7f9b1c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 -- 172.21.15.28:0/1362953241 shutdown_connections 2024-09-10T20:43:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.025+0000 7f9b22060640 1 -- 172.21.15.28:0/1362953241 wait complete. 2024-09-10T20:43:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.026+0000 7f9b22060640 1 Processor -- start 2024-09-10T20:43:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.026+0000 7f9b22060640 1 -- start start 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b22060640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b1c06e9d0 0x7f9b1c1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b22060640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c0aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b22060640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b1c155360 0x7f9b1c0b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b22060640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c072890 con 0x7f9b1c0cf700 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b22060640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c072710 con 0x7f9b1c155360 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b22060640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1c072a10 con 0x7f9b1c06e9d0 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c0aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c0aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47772/0 (socket says 172.21.15.28:47772) 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 -- 172.21.15.28:0/3158499988 learned_addr learned my addr 172.21.15.28:0/3158499988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1b7fe640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b1c06e9d0 0x7f9b1c1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1bfff640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b1c155360 0x7f9b1c0b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 -- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b1c06e9d0 msgr2=0x7f9b1c1547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b1c06e9d0 0x7f9b1c1547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 -- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b1c155360 msgr2=0x7f9b1c0b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b1c155360 0x7f9b1c0b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.027+0000 7f9b1affd640 1 -- 172.21.15.28:0/3158499988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c1557d0 con 0x7f9b1c0cf700 2024-09-10T20:43:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.028+0000 7f9b1affd640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c0aff40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009950 tx=0x7f9b10032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.028+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b10040070 con 0x7f9b1c0cf700 2024-09-10T20:43:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.028+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b10032e70 con 0x7f9b1c0cf700 2024-09-10T20:43:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.028+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1c155ab0 con 0x7f9b1c0cf700 2024-09-10T20:43:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.028+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b100055b0 con 0x7f9b1c0cf700 2024-09-10T20:43:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.028+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1c156070 con 0x7f9b1c0cf700 2024-09-10T20:43:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.029+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b1c072e60 con 0x7f9b1c0cf700 2024-09-10T20:43:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.029+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 100703+0+0 (secure 0 0 0) 0x7f9b1004c030 con 0x7f9b1c0cf700 2024-09-10T20:43:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.030+0000 7f9b03fff640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9b08078090 0x7f9b0807a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.030+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5709+0+0 (secure 0 0 0) 0x7f9b100baf00 con 0x7f9b1c0cf700 2024-09-10T20:43:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.030+0000 7f9b1b7fe640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9b08078090 0x7f9b0807a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.030+0000 7f9b1b7fe640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9b08078090 0x7f9b0807a550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0cdd10 tx=0x7f9b0c007e50 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.034+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9b10083220 con 0x7f9b1c0cf700 2024-09-10T20:43:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.076+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 17) ==== 100754+0+0 (secure 0 0 0) 0x7f9b100b9420 con 0x7f9b1c0cf700 2024-09-10T20:43:09.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.253+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol63210", "mode": "777", "uid": 1000, "gid": 1000, "target": ["mon-mgr", ""]}) -- 0x7f9b1c06d620 con 0x7f9b08078090 2024-09-10T20:43:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.276+0000 7f9b03fff640 1 -- 172.21.15.28:0/3158499988 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9b1c06d620 con 0x7f9b08078090 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9b08078090 msgr2=0x7f9b0807a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9b08078090 0x7f9b0807a550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0cdd10 tx=0x7f9b0c007e50 comp rx=0 tx=0).stop 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 msgr2=0x7f9b1c0aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c0aff40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009950 tx=0x7f9b10032c80 comp rx=0 tx=0).stop 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 shutdown_connections 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9b08078090 0x7f9b0807a550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b1c155360 0x7f9b1c0b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b1c0cf700 0x7f9b1c0aff40 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 --2- 172.21.15.28:0/3158499988 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b1c06e9d0 0x7f9b1c1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.278+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 >> 172.21.15.28:0/3158499988 conn(0x7f9b1c06cf00 msgr2=0x7f9b1c0b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.279+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 shutdown_connections 2024-09-10T20:43:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.279+0000 7f9b22060640 1 -- 172.21.15.28:0/3158499988 wait complete. 2024-09-10T20:43:09.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol63210 2024-09-10T20:43:09.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 -- 172.21.15.28:0/1747924641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839815bfe0 msgr2=0x7f83980cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 --2- 172.21.15.28:0/1747924641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839815bfe0 0x7f83980cf140 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f8388009830 tx=0x7f838802a480 comp rx=0 tx=0).stop 2024-09-10T20:43:09.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 -- 172.21.15.28:0/1747924641 shutdown_connections 2024-09-10T20:43:09.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 --2- 172.21.15.28:0/1747924641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83980cecb0 0x7f83980b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 --2- 172.21.15.28:0/1747924641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839815bfe0 0x7f83980cf140 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 -- 172.21.15.28:0/1747924641 >> 172.21.15.28:0/1747924641 conn(0x7f83980d1e60 msgr2=0x7f83980d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:09.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 -- 172.21.15.28:0/1747924641 shutdown_connections 2024-09-10T20:43:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.398+0000 7f839d016640 1 -- 172.21.15.28:0/1747924641 wait complete. 2024-09-10T20:43:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.399+0000 7f839d016640 1 Processor -- start 2024-09-10T20:43:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.399+0000 7f839d016640 1 -- start start 2024-09-10T20:43:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839d016640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83980cecb0 0x7f8398159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839d016640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83980d0440 0x7f83980c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839d016640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839815bfe0 0x7f8398151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839d016640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83980b2b50 con 0x7f83980cecb0 2024-09-10T20:43:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839d016640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f83980b29d0 con 0x7f83980d0440 2024-09-10T20:43:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839d016640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f83980b2cd0 con 0x7f839815bfe0 2024-09-10T20:43:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f8396d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83980cecb0 0x7f8398159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f8396d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83980cecb0 0x7f8398159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47778/0 (socket says 172.21.15.28:47778) 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f8396d9d640 1 -- 172.21.15.28:0/2574229664 learned_addr learned my addr 172.21.15.28:0/2574229664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839759e640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839815bfe0 0x7f8398151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.400+0000 7f839659c640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83980d0440 0x7f83980c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f8396d9d640 1 -- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839815bfe0 msgr2=0x7f8398151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f8396d9d640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839815bfe0 0x7f8398151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f8396d9d640 1 -- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83980d0440 msgr2=0x7f83980c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f8396d9d640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83980d0440 0x7f83980c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f8396d9d640 1 -- 172.21.15.28:0/2574229664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83980c6d40 con 0x7f83980cecb0 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f8396d9d640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83980cecb0 0x7f8398159fe0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f838802aa70 tx=0x7f8388033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f837f7fe640 1 -- 172.21.15.28:0/2574229664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388031d60 con 0x7f83980cecb0 2024-09-10T20:43:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f837f7fe640 1 -- 172.21.15.28:0/2574229664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8388033e10 con 0x7f83980cecb0 2024-09-10T20:43:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83980c7020 con 0x7f83980cecb0 2024-09-10T20:43:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f837f7fe640 1 -- 172.21.15.28:0/2574229664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388005260 con 0x7f83980cecb0 2024-09-10T20:43:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.401+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83981f2b70 con 0x7f83980cecb0 2024-09-10T20:43:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.402+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83980b3260 con 0x7f83980cecb0 2024-09-10T20:43:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.403+0000 7f837f7fe640 1 -- 172.21.15.28:0/2574229664 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 100754+0+0 (secure 0 0 0) 0x7f8388005400 con 0x7f83980cecb0 2024-09-10T20:43:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.403+0000 7f837f7fe640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8384078120 0x7f838407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.403+0000 7f837f7fe640 1 -- 172.21.15.28:0/2574229664 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f83880bb210 con 0x7f83980cecb0 2024-09-10T20:43:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.403+0000 7f839659c640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8384078120 0x7f838407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.404+0000 7f839659c640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8384078120 0x7f838407a5e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f838c007fd0 tx=0x7f838c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.407+0000 7f837f7fe640 1 -- 172.21.15.28:0/2574229664 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f838803f030 con 0x7f83980cecb0 2024-09-10T20:43:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.617+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol63210", "target": ["mon-mgr", ""]}) -- 0x7f8398159850 con 0x7f8384078120 2024-09-10T20:43:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.649+0000 7f837f7fe640 1 -- 172.21.15.28:0/2574229664 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f8398159850 con 0x7f8384078120 2024-09-10T20:43:09.653 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385 2024-09-10T20:43:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8384078120 msgr2=0x7f838407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8384078120 0x7f838407a5e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f838c007fd0 tx=0x7f838c009040 comp rx=0 tx=0).stop 2024-09-10T20:43:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83980cecb0 msgr2=0x7f8398159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83980cecb0 0x7f8398159fe0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f838802aa70 tx=0x7f8388033c20 comp rx=0 tx=0).stop 2024-09-10T20:43:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 shutdown_connections 2024-09-10T20:43:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8384078120 0x7f838407a5e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f839815bfe0 0x7f8398151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83980d0440 0x7f83980c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 --2- 172.21.15.28:0/2574229664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83980cecb0 0x7f8398159fe0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 >> 172.21.15.28:0/2574229664 conn(0x7f83980d1e60 msgr2=0x7f8398071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 shutdown_connections 2024-09-10T20:43:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:09.651+0000 7f839d016640 1 -- 172.21.15.28:0/2574229664 wait complete. 2024-09-10T20:43:09.666 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol63210 with 8 files each 1MB size under directory volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385 2024-09-10T20:43:09.666 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:09.666 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:09.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:09.682 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106276 s, 98.7 MB/s 2024-09-10T20:43:09.755 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:09.755 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:09.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:09.812 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910499 s, 115 MB/s 2024-09-10T20:43:09.878 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:09.878 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:09.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:09.935 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0085323 s, 123 MB/s 2024-09-10T20:43:10.002 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:10.002 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:10.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:10.058 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00781224 s, 134 MB/s 2024-09-10T20:43:10.124 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:10.124 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:10.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:10.180 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:10.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:10.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:10.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00866129 s, 121 MB/s 2024-09-10T20:43:10.247 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:10.247 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:10.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:10.303 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:10.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:10.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:10.371 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00923647 s, 114 MB/s 2024-09-10T20:43:10.371 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:10.371 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:10.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:10.427 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00911886 s, 115 MB/s 2024-09-10T20:43:10.494 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:10.494 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:10.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:10.551 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2024-09-10T20:43:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T20:43:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T20:43:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00706409 s, 148 MB/s 2024-09-10T20:43:10.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol63210 subvol_snap85350 2024-09-10T20:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 -- 172.21.15.28:0/1516078304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc0702e0 msgr2=0x7f2fbc0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 --2- 172.21.15.28:0/1516078304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc0702e0 0x7f2fbc0cda70 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc06f7b0 tx=0x7f2fac02a820 comp rx=0 tx=0).stop 2024-09-10T20:43:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 -- 172.21.15.28:0/1516078304 shutdown_connections 2024-09-10T20:43:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 --2- 172.21.15.28:0/1516078304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc0702e0 0x7f2fbc0cda70 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 --2- 172.21.15.28:0/1516078304 >> v2:172.21.15.159:3300/0 conn(0x7f2fbc0cabc0 0x7f2fbc06fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 -- 172.21.15.28:0/1516078304 >> 172.21.15.28:0/1516078304 conn(0x7f2fbc0cfc80 msgr2=0x7f2fbc0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 -- 172.21.15.28:0/1516078304 shutdown_connections 2024-09-10T20:43:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.761+0000 7f2fc4016640 1 -- 172.21.15.28:0/1516078304 wait complete. 2024-09-10T20:43:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.762+0000 7f2fc4016640 1 Processor -- start 2024-09-10T20:43:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc4016640 1 -- start start 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc4016640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fbc0cabc0 0x7f2fbc148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc4016640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fbc15a350 0x7f2fbc1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc4016640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc159400 0x7f2fbc155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc4016640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc06e510 con 0x7f2fbc159400 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc4016640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc06e390 con 0x7f2fbc0cabc0 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc4016640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2fbc06e690 con 0x7f2fbc15a350 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc25b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc159400 0x7f2fbc155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc25b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc159400 0x7f2fbc155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43390/0 (socket says 172.21.15.28:43390) 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.763+0000 7f2fc25b3640 1 -- 172.21.15.28:0/3411906824 learned_addr learned my addr 172.21.15.28:0/3411906824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc1db2640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fbc0cabc0 0x7f2fbc148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc15b1640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fbc15a350 0x7f2fbc1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc25b3640 1 -- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fbc15a350 msgr2=0x7f2fbc1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc25b3640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fbc15a350 0x7f2fbc1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc25b3640 1 -- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fbc0cabc0 msgr2=0x7f2fbc148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc25b3640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fbc0cabc0 0x7f2fbc148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc25b3640 1 -- 172.21.15.28:0/3411906824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc159900 con 0x7f2fbc159400 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc25b3640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc159400 0x7f2fbc155fa0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2fac005fd0 tx=0x7f2fac032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fba7fc640 1 -- 172.21.15.28:0/3411906824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac03f020 con 0x7f2fbc159400 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fba7fc640 1 -- 172.21.15.28:0/3411906824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fac004040 con 0x7f2fbc159400 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fbc1f8430 con 0x7f2fbc159400 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fba7fc640 1 -- 172.21.15.28:0/3411906824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac037040 con 0x7f2fbc159400 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.764+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fbc1f89f0 con 0x7f2fbc159400 2024-09-10T20:43:10.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.765+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fbc1f8bc0 con 0x7f2fbc159400 2024-09-10T20:43:10.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.766+0000 7f2fba7fc640 1 -- 172.21.15.28:0/3411906824 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 100754+0+0 (secure 0 0 0) 0x7f2fac03a030 con 0x7f2fbc159400 2024-09-10T20:43:10.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.767+0000 7f2fba7fc640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fa8078120 0x7f2fa807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:10.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.767+0000 7f2fba7fc640 1 -- 172.21.15.28:0/3411906824 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f2fac0ba690 con 0x7f2fbc159400 2024-09-10T20:43:10.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.767+0000 7f2fc1db2640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fa8078120 0x7f2fa807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:10.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.767+0000 7f2fc1db2640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fa8078120 0x7f2fa807a5e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4001610 tx=0x7f2fb4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:10.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.771+0000 7f2fba7fc640 1 -- 172.21.15.28:0/3411906824 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2fac049030 con 0x7f2fbc159400 2024-09-10T20:43:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:10.987+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol63210", "snap_name": "subvol_snap85350", "target": ["mon-mgr", ""]}) -- 0x7f2fbc0cce20 con 0x7f2fa8078120 2024-09-10T20:43:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.012+0000 7f2fba7fc640 1 -- 172.21.15.28:0/3411906824 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2fbc0cce20 con 0x7f2fa8078120 2024-09-10T20:43:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fa8078120 msgr2=0x7f2fa807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fa8078120 0x7f2fa807a5e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4001610 tx=0x7f2fb4009040 comp rx=0 tx=0).stop 2024-09-10T20:43:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc159400 msgr2=0x7f2fbc155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc159400 0x7f2fbc155fa0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2fac005fd0 tx=0x7f2fac032c90 comp rx=0 tx=0).stop 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 shutdown_connections 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fa8078120 0x7f2fa807a5e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fbc159400 0x7f2fbc155fa0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fbc15a350 0x7f2fbc1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 --2- 172.21.15.28:0/3411906824 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fbc0cabc0 0x7f2fbc148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 >> 172.21.15.28:0/3411906824 conn(0x7f2fbc0cfc80 msgr2=0x7f2fbc0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 shutdown_connections 2024-09-10T20:43:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.014+0000 7f2fc4016640 1 -- 172.21.15.28:0/3411906824 wait complete. 2024-09-10T20:43:11.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol63210 subvol_snap85350 subvol_clone92450 2024-09-10T20:43:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.138+0000 7fd8e75ad640 1 -- 172.21.15.28:0/1577889590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d26f0 msgr2=0x7fd8e00b55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.138+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/1577889590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d26f0 0x7fd8e00b55f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009830 tx=0x7fd8d002a510 comp rx=0 tx=0).stop 2024-09-10T20:43:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.138+0000 7fd8e75ad640 1 -- 172.21.15.28:0/1577889590 shutdown_connections 2024-09-10T20:43:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.139+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/1577889590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d26f0 0x7fd8e00b55f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.139+0000 7fd8e75ad640 1 -- 172.21.15.28:0/1577889590 >> 172.21.15.28:0/1577889590 conn(0x7fd8e00d4070 msgr2=0x7fd8e00d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.139+0000 7fd8e75ad640 1 -- 172.21.15.28:0/1577889590 shutdown_connections 2024-09-10T20:43:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.139+0000 7fd8e75ad640 1 -- 172.21.15.28:0/1577889590 wait complete. 2024-09-10T20:43:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.140+0000 7fd8e75ad640 1 Processor -- start 2024-09-10T20:43:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.140+0000 7fd8e75ad640 1 -- start start 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e75ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d1260 0x7fd8e0153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e75ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8e00d1640 0x7fd8e01f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e75ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e00d26f0 0x7fd8e01f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e75ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e00b18c0 con 0x7fd8e00d1260 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e75ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e00b1740 con 0x7fd8e00d1640 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e75ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e00b1a40 con 0x7fd8e00d26f0 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d1260 0x7fd8e0153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d1260 0x7fd8e0153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43398/0 (socket says 172.21.15.28:43398) 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 -- 172.21.15.28:0/514732473 learned_addr learned my addr 172.21.15.28:0/514732473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e4b48640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8e00d1640 0x7fd8e01f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 -- 172.21.15.28:0/514732473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e00d26f0 msgr2=0x7fd8e01f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e00d26f0 0x7fd8e01f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 -- 172.21.15.28:0/514732473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8e00d1640 msgr2=0x7fd8e01f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8e00d1640 0x7fd8e01f7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 -- 172.21.15.28:0/514732473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e00b7040 con 0x7fd8e00d1260 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e5349640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d1260 0x7fd8e0153470 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00c430 tx=0x7fd8dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8d5ffb640 1 -- 172.21.15.28:0/514732473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc011790 con 0x7fd8e00d1260 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8d5ffb640 1 -- 172.21.15.28:0/514732473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8dc017070 con 0x7fd8e00d1260 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8d5ffb640 1 -- 172.21.15.28:0/514732473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc011d00 con 0x7fd8e00d1260 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.141+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e00b98b0 con 0x7fd8e00d1260 2024-09-10T20:43:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.142+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e00b72e0 con 0x7fd8e00d1260 2024-09-10T20:43:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.143+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e00b5db0 con 0x7fd8e00d1260 2024-09-10T20:43:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.143+0000 7fd8d5ffb640 1 -- 172.21.15.28:0/514732473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 100754+0+0 (secure 0 0 0) 0x7fd8dc01c460 con 0x7fd8e00d1260 2024-09-10T20:43:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.144+0000 7fd8d5ffb640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8c4078120 0x7fd8c407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.144+0000 7fd8d5ffb640 1 -- 172.21.15.28:0/514732473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7fd8dc09a090 con 0x7fd8e00d1260 2024-09-10T20:43:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.144+0000 7fd8e4b48640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8c4078120 0x7fd8c407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.144+0000 7fd8e4b48640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8c4078120 0x7fd8c407a5e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0034000 tx=0x7fd8d0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.148+0000 7fd8d5ffb640 1 -- 172.21.15.28:0/514732473 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd8dc0627b0 con 0x7fd8e00d1260 2024-09-10T20:43:11.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.366+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol63210", "snap_name": "subvol_snap85350", "target_sub_name": "subvol_clone92450", "target": ["mon-mgr", ""]}) -- 0x7fd8e00b7520 con 0x7fd8c4078120 2024-09-10T20:43:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.453+0000 7fd8d5ffb640 1 -- 172.21.15.28:0/514732473 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd8d0004440 con 0x7fd8c4078120 2024-09-10T20:43:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8c4078120 msgr2=0x7fd8c407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8c4078120 0x7fd8c407a5e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0034000 tx=0x7fd8d0004730 comp rx=0 tx=0).stop 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d1260 msgr2=0x7fd8e0153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d1260 0x7fd8e0153470 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00c430 tx=0x7fd8dc00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 shutdown_connections 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8c4078120 0x7fd8c407a5e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e00d26f0 0x7fd8e01f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8e00d1640 0x7fd8e01f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 --2- 172.21.15.28:0/514732473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e00d1260 0x7fd8e0153470 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.455+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 >> 172.21.15.28:0/514732473 conn(0x7fd8e00d4070 msgr2=0x7fd8e00b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.456+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 shutdown_connections 2024-09-10T20:43:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.456+0000 7fd8e75ad640 1 -- 172.21.15.28:0/514732473 wait complete. 2024-09-10T20:43:11.461 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.456+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.461 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.456+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.461 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.456+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.461 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.456+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.461 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.456+0000 7f93d4409640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone92450 2024-09-10T20:43:11.475 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.471+0000 7f93d2c06640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.475 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.471+0000 7f93d2c06640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.475 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.471+0000 7f93d2c06640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.475 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.471+0000 7f93d2c06640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.475 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:11.471+0000 7f93d2c06640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:11.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 -- 172.21.15.28:0/2652701487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540d4b90 msgr2=0x7f2a540d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 --2- 172.21.15.28:0/2652701487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540d4b90 0x7f2a540d2620 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c005fd0 tx=0x7f2a4c02a820 comp rx=0 tx=0).stop 2024-09-10T20:43:11.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 -- 172.21.15.28:0/2652701487 shutdown_connections 2024-09-10T20:43:11.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 --2- 172.21.15.28:0/2652701487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540d4b90 0x7f2a540d2620 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 --2- 172.21.15.28:0/2652701487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54161770 0x7f2a540d20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 -- 172.21.15.28:0/2652701487 >> 172.21.15.28:0/2652701487 conn(0x7f2a540d4070 msgr2=0x7f2a540d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:11.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 -- 172.21.15.28:0/2652701487 shutdown_connections 2024-09-10T20:43:11.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.591+0000 7f2a59240640 1 -- 172.21.15.28:0/2652701487 wait complete. 2024-09-10T20:43:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.592+0000 7f2a59240640 1 Processor -- start 2024-09-10T20:43:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.592+0000 7f2a59240640 1 -- start start 2024-09-10T20:43:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a59240640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a540d4b90 0x7f2a54155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a59240640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54161770 0x7f2a541556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a59240640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54161b50 0x7f2a541f8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a59240640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540b2f50 con 0x7f2a54161770 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a59240640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540b2dd0 con 0x7f2a540d4b90 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a59240640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2a540b30d0 con 0x7f2a54161b50 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54161770 0x7f2a541556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54161770 0x7f2a541556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43460/0 (socket says 172.21.15.28:43460) 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a4bfff640 1 -- 172.21.15.28:0/1868355624 learned_addr learned my addr 172.21.15.28:0/1868355624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a52d9d640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a540d4b90 0x7f2a54155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.593+0000 7f2a5359e640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54161b50 0x7f2a541f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4bfff640 1 -- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54161b50 msgr2=0x7f2a541f8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4bfff640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54161b50 0x7f2a541f8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4bfff640 1 -- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a540d4b90 msgr2=0x7f2a54155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4bfff640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a540d4b90 0x7f2a54155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4bfff640 1 -- 172.21.15.28:0/1868355624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a54155be0 con 0x7f2a54161770 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4bfff640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54161770 0x7f2a541556a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00c3c0 tx=0x7f2a3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4b7fe640 1 -- 172.21.15.28:0/1868355624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c017070 con 0x7f2a54161770 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a540b9ee0 con 0x7f2a54161770 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4b7fe640 1 -- 172.21.15.28:0/1868355624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a3c0035f0 con 0x7f2a54161770 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a4b7fe640 1 -- 172.21.15.28:0/1868355624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c003a80 con 0x7f2a54161770 2024-09-10T20:43:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.594+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5415dc00 con 0x7f2a54161770 2024-09-10T20:43:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.595+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a540b3870 con 0x7f2a54161770 2024-09-10T20:43:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.596+0000 7f2a4b7fe640 1 -- 172.21.15.28:0/1868355624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 100754+0+0 (secure 0 0 0) 0x7f2a3c003c20 con 0x7f2a54161770 2024-09-10T20:43:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.597+0000 7f2a4b7fe640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a34078120 0x7f2a3407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.597+0000 7f2a4b7fe640 1 -- 172.21.15.28:0/1868355624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f2a3c0995f0 con 0x7f2a54161770 2024-09-10T20:43:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.597+0000 7f2a52d9d640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a34078120 0x7f2a3407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.597+0000 7f2a52d9d640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a34078120 0x7f2a3407a5e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400bbe0 tx=0x7f2a4400b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:11.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.602+0000 7f2a4b7fe640 1 -- 172.21.15.28:0/1868355624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2a3c061850 con 0x7f2a54161770 2024-09-10T20:43:11.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:11.843+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone92450", "target": ["mon-mgr", ""]}) -- 0x7f2a540d3400 con 0x7f2a34078120 2024-09-10T20:43:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.321+0000 7f2a4b7fe640 1 -- 172.21.15.28:0/1868355624 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f2a540d3400 con 0x7f2a34078120 2024-09-10T20:43:12.325 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:12.326 INFO:teuthology.orchestra.run.smithi028.stdout: "status": { 2024-09-10T20:43:12.326 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "complete" 2024-09-10T20:43:12.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:12.326 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a34078120 msgr2=0x7f2a3407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a34078120 0x7f2a3407a5e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400bbe0 tx=0x7f2a4400b390 comp rx=0 tx=0).stop 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54161770 msgr2=0x7f2a541556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54161770 0x7f2a541556a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00c3c0 tx=0x7f2a3c00c880 comp rx=0 tx=0).stop 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 shutdown_connections 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a34078120 0x7f2a3407a5e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54161b50 0x7f2a541f8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54161770 0x7f2a541556a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 --2- 172.21.15.28:0/1868355624 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a540d4b90 0x7f2a54155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 >> 172.21.15.28:0/1868355624 conn(0x7f2a540d4070 msgr2=0x7f2a540d1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 shutdown_connections 2024-09-10T20:43:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.324+0000 7f2a59240640 1 -- 172.21.15.28:0/1868355624 wait complete. 2024-09-10T20:43:12.339 DEBUG:tasks.cephfs.test_volumes:current clone state = complete 2024-09-10T20:43:12.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:12.340 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol_clone92450/.meta) 2024-09-10T20:43:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:43:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:version = 2 2024-09-10T20:43:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:type = clone 2024-09-10T20:43:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51 2024-09-10T20:43:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:43:12.407 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:12.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol63210 2024-09-10T20:43:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 -- 172.21.15.28:0/1667190037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9840b3580 msgr2=0x7fe98406e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 --2- 172.21.15.28:0/1667190037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9840b3580 0x7fe98406e400 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe9840cd440 tx=0x7fe96c02a4a0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 -- 172.21.15.28:0/1667190037 shutdown_connections 2024-09-10T20:43:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 --2- 172.21.15.28:0/1667190037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9840b3580 0x7fe98406e400 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 --2- 172.21.15.28:0/1667190037 >> v2:172.21.15.159:3300/0 conn(0x7fe984152e00 0x7fe9840b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 -- 172.21.15.28:0/1667190037 >> 172.21.15.28:0/1667190037 conn(0x7fe98406d110 msgr2=0x7fe9840b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 -- 172.21.15.28:0/1667190037 shutdown_connections 2024-09-10T20:43:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.554+0000 7fe98907f640 1 -- 172.21.15.28:0/1667190037 wait complete. 2024-09-10T20:43:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.555+0000 7fe98907f640 1 Processor -- start 2024-09-10T20:43:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.555+0000 7fe98907f640 1 -- start start 2024-09-10T20:43:12.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe98907f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98406e940 0x7fe9841f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe98907f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe984152e00 0x7fe9841f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe98907f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe984153350 0x7fe9841519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe98907f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe98415f820 con 0x7fe98406e940 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe98907f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe9840cf640 con 0x7fe984152e00 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe98907f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe98415f9a0 con 0x7fe984153350 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe982d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98406e940 0x7fe9841f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe982d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98406e940 0x7fe9841f4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43478/0 (socket says 172.21.15.28:43478) 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.556+0000 7fe982d9d640 1 -- 172.21.15.28:0/1988376184 learned_addr learned my addr 172.21.15.28:0/1988376184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe98259c640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe984152e00 0x7fe9841f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe98359e640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe984153350 0x7fe9841519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe982d9d640 1 -- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe984153350 msgr2=0x7fe9841519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe982d9d640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe984153350 0x7fe9841519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe982d9d640 1 -- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe984152e00 msgr2=0x7fe9841f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe982d9d640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe984152e00 0x7fe9841f8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe982d9d640 1 -- 172.21.15.28:0/1988376184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9841f5190 con 0x7fe98406e940 2024-09-10T20:43:12.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe98259c640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe984152e00 0x7fe9841f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe982d9d640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98406e940 0x7fe9841f4b80 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d740 tx=0x7fe97c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe97b7fe640 1 -- 172.21.15.28:0/1988376184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c017070 con 0x7fe98406e940 2024-09-10T20:43:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe97b7fe640 1 -- 172.21.15.28:0/1988376184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe97c009d20 con 0x7fe98406e940 2024-09-10T20:43:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9841f5470 con 0x7fe98406e940 2024-09-10T20:43:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe97b7fe640 1 -- 172.21.15.28:0/1988376184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0106c0 con 0x7fe98406e940 2024-09-10T20:43:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.557+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe98414bf90 con 0x7fe98406e940 2024-09-10T20:43:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.558+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe98415fe50 con 0x7fe98406e940 2024-09-10T20:43:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.559+0000 7fe97b7fe640 1 -- 172.21.15.28:0/1988376184 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 100754+0+0 (secure 0 0 0) 0x7fe97c010860 con 0x7fe98406e940 2024-09-10T20:43:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.559+0000 7fe97b7fe640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe964078120 0x7fe96407a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.559+0000 7fe97b7fe640 1 -- 172.21.15.28:0/1988376184 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7fe97c09a550 con 0x7fe98406e940 2024-09-10T20:43:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.559+0000 7fe98259c640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe964078120 0x7fe96407a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.560+0000 7fe98259c640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe964078120 0x7fe96407a5e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe96c034000 tx=0x7fe96c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.564+0000 7fe97b7fe640 1 -- 172.21.15.28:0/1988376184 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe97c062d10 con 0x7fe98406e940 2024-09-10T20:43:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.776+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol63210", "target": ["mon-mgr", ""]}) -- 0x7fe98414f010 con 0x7fe964078120 2024-09-10T20:43:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.789+0000 7fe97b7fe640 1 -- 172.21.15.28:0/1988376184 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7fe98414f010 con 0x7fe964078120 2024-09-10T20:43:12.792 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe964078120 msgr2=0x7fe96407a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe964078120 0x7fe96407a5e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe96c034000 tx=0x7fe96c004730 comp rx=0 tx=0).stop 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98406e940 msgr2=0x7fe9841f4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98406e940 0x7fe9841f4b80 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d740 tx=0x7fe97c00dc00 comp rx=0 tx=0).stop 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 shutdown_connections 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe964078120 0x7fe96407a5e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe984153350 0x7fe9841519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe984152e00 0x7fe9841f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 --2- 172.21.15.28:0/1988376184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98406e940 0x7fe9841f4b80 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 >> 172.21.15.28:0/1988376184 conn(0x7fe98406d110 msgr2=0x7fe98414f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 shutdown_connections 2024-09-10T20:43:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.791+0000 7fe98907f640 1 -- 172.21.15.28:0/1988376184 wait complete. 2024-09-10T20:43:12.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone92450 2024-09-10T20:43:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.911+0000 7f46927d3640 1 -- 172.21.15.28:0/144393534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0cb3a0 msgr2=0x7f468c0cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.911+0000 7f46927d3640 1 --2- 172.21.15.28:0/144393534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0cb3a0 0x7f468c0cb830 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f4684005fd0 tx=0x7f468402ab40 comp rx=0 tx=0).stop 2024-09-10T20:43:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.911+0000 7f46927d3640 1 -- 172.21.15.28:0/144393534 shutdown_connections 2024-09-10T20:43:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.911+0000 7f46927d3640 1 --2- 172.21.15.28:0/144393534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0cb3a0 0x7f468c0cb830 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.911+0000 7f46927d3640 1 -- 172.21.15.28:0/144393534 >> 172.21.15.28:0/144393534 conn(0x7f468c0d0d50 msgr2=0x7f468c0d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.911+0000 7f46927d3640 1 -- 172.21.15.28:0/144393534 shutdown_connections 2024-09-10T20:43:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.911+0000 7f46927d3640 1 -- 172.21.15.28:0/144393534 wait complete. 2024-09-10T20:43:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.912+0000 7f46927d3640 1 Processor -- start 2024-09-10T20:43:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.912+0000 7f46927d3640 1 -- start start 2024-09-10T20:43:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f46927d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f468c0cb3a0 0x7f468c0c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f46927d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0d1840 0x7f468c0c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f46927d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f468c152df0 0x7f468c0c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f46927d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f468c06ce70 con 0x7f468c0d1840 2024-09-10T20:43:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f46927d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f468c06ccf0 con 0x7f468c152df0 2024-09-10T20:43:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f46927d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f468c06cff0 con 0x7f468c0cb3a0 2024-09-10T20:43:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0d1840 0x7f468c0c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0d1840 0x7f468c0c4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43496/0 (socket says 172.21.15.28:43496) 2024-09-10T20:43:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 -- 172.21.15.28:0/531110412 learned_addr learned my addr 172.21.15.28:0/531110412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468bfff640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f468c0cb3a0 0x7f468c0c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f4690d70640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f468c152df0 0x7f468c0c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 -- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f468c0cb3a0 msgr2=0x7f468c0c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f468c0cb3a0 0x7f468c0c4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 -- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f468c152df0 msgr2=0x7f468c0c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f468c152df0 0x7f468c0c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.913+0000 7f468b7fe640 1 -- 172.21.15.28:0/531110412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c15b960 con 0x7f468c0d1840 2024-09-10T20:43:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.914+0000 7f468b7fe640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0d1840 0x7f468c0c4bb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f468000c530 tx=0x7f468000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.914+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46800052d0 con 0x7f468c0d1840 2024-09-10T20:43:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.914+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4680005470 con 0x7f468c0d1840 2024-09-10T20:43:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.914+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680012870 con 0x7f468c0d1840 2024-09-10T20:43:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.914+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468c0c5890 con 0x7f468c0d1840 2024-09-10T20:43:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.914+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468c0c5e50 con 0x7f468c0d1840 2024-09-10T20:43:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.915+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468c0cb830 con 0x7f468c0d1840 2024-09-10T20:43:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.916+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 100754+0+0 (secure 0 0 0) 0x7f46800035f0 con 0x7f468c0d1840 2024-09-10T20:43:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.916+0000 7f4688ff9640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4678078120 0x7f467807a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.916+0000 7f468bfff640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4678078120 0x7f467807a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.916+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f4680099be0 con 0x7f468c0d1840 2024-09-10T20:43:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.917+0000 7f468bfff640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4678078120 0x7f467807a5e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f467c0017c0 tx=0x7f467c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:12.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:12.921+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4680061d50 con 0x7f468c0d1840 2024-09-10T20:43:13.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.094+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 18) ==== 100856+0+0 (secure 0 0 0) 0x7f468005dfc0 con 0x7f468c0d1840 2024-09-10T20:43:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.138+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone92450", "target": ["mon-mgr", ""]}) -- 0x7f468c15c4c0 con 0x7f4678078120 2024-09-10T20:43:13.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.142+0000 7f4688ff9640 1 -- 172.21.15.28:0/531110412 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+73 (secure 0 0 0) 0x7f468c15c4c0 con 0x7f4678078120 2024-09-10T20:43:13.147 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51 2024-09-10T20:43:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.144+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4678078120 msgr2=0x7f467807a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4678078120 0x7f467807a5e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f467c0017c0 tx=0x7f467c009040 comp rx=0 tx=0).stop 2024-09-10T20:43:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0d1840 msgr2=0x7f468c0c4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0d1840 0x7f468c0c4bb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f468000c530 tx=0x7f468000c9f0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 shutdown_connections 2024-09-10T20:43:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4678078120 0x7f467807a5e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f468c152df0 0x7f468c0c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f468c0d1840 0x7f468c0c4bb0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 --2- 172.21.15.28:0/531110412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f468c0cb3a0 0x7f468c0c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 >> 172.21.15.28:0/531110412 conn(0x7f468c0d0d50 msgr2=0x7f468c15e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 shutdown_connections 2024-09-10T20:43:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.145+0000 7f46927d3640 1 -- 172.21.15.28:0/531110412 wait complete. 2024-09-10T20:43:13.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.159 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:13.177 INFO:teuthology.orchestra.run.smithi028.stdout:9 2024-09-10T20:43:13.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.178 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51) 2024-09-10T20:43:13.234 INFO:teuthology.orchestra.run.smithi028.stdout:9 2024-09-10T20:43:13.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone92450 2024-09-10T20:43:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.379+0000 7f80581a3640 1 -- 172.21.15.28:0/3657123925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80500af700 msgr2=0x7f80500d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.379+0000 7f80581a3640 1 --2- 172.21.15.28:0/3657123925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80500af700 0x7f80500d0860 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f8040009830 tx=0x7f804002a510 comp rx=0 tx=0).stop 2024-09-10T20:43:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.379+0000 7f80581a3640 1 -- 172.21.15.28:0/3657123925 shutdown_connections 2024-09-10T20:43:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.379+0000 7f80581a3640 1 --2- 172.21.15.28:0/3657123925 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80500afb90 0x7f80500d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.379+0000 7f80581a3640 1 --2- 172.21.15.28:0/3657123925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80500af700 0x7f80500d0860 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.379+0000 7f80581a3640 1 -- 172.21.15.28:0/3657123925 >> 172.21.15.28:0/3657123925 conn(0x7f805006cf00 msgr2=0x7f805006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.380+0000 7f80581a3640 1 -- 172.21.15.28:0/3657123925 shutdown_connections 2024-09-10T20:43:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.380+0000 7f80581a3640 1 -- 172.21.15.28:0/3657123925 wait complete. 2024-09-10T20:43:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 Processor -- start 2024-09-10T20:43:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 -- start start 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80500af700 0x7f805014ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80500afb90 0x7f80500b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80501560c0 0x7f80500b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f805015c880 con 0x7f80501560c0 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f805015c700 con 0x7f80500af700 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.381+0000 7f80581a3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f805015ca00 con 0x7f80500afb90 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80501560c0 0x7f80500b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80501560c0 0x7f80500b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43504/0 (socket says 172.21.15.28:43504) 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 -- 172.21.15.28:0/569362759 learned_addr learned my addr 172.21.15.28:0/569362759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f805573e640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80500afb90 0x7f80500b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8055f3f640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80500af700 0x7f805014ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 -- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80500afb90 msgr2=0x7f80500b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80500afb90 0x7f80500b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 -- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80500af700 msgr2=0x7f805014ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80500af700 0x7f805014ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 -- 172.21.15.28:0/569362759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8050156910 con 0x7f80501560c0 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f8056740640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80501560c0 0x7f80500b15d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f804800cce0 tx=0x7f804800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f80467fc640 1 -- 172.21.15.28:0/569362759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804800f450 con 0x7f80501560c0 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f80467fc640 1 -- 172.21.15.28:0/569362759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f804800f700 con 0x7f80501560c0 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8050156bf0 con 0x7f80501560c0 2024-09-10T20:43:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f80467fc640 1 -- 172.21.15.28:0/569362759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804800fb50 con 0x7f80501560c0 2024-09-10T20:43:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.382+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f805014f410 con 0x7f80501560c0 2024-09-10T20:43:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.383+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f805015ceb0 con 0x7f80501560c0 2024-09-10T20:43:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.384+0000 7f80467fc640 1 -- 172.21.15.28:0/569362759 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 100856+0+0 (secure 0 0 0) 0x7f80480213f0 con 0x7f80501560c0 2024-09-10T20:43:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.384+0000 7f80467fc640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8034078240 0x7f803407a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.384+0000 7f8055f3f640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8034078240 0x7f803407a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.384+0000 7f80467fc640 1 -- 172.21.15.28:0/569362759 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f804809af70 con 0x7f80501560c0 2024-09-10T20:43:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.385+0000 7f8055f3f640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8034078240 0x7f803407a700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f80500b30a0 tx=0x7f804c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.388+0000 7f80467fc640 1 -- 172.21.15.28:0/569362759 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8048014030 con 0x7f80501560c0 2024-09-10T20:43:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.602+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone92450", "target": ["mon-mgr", ""]}) -- 0x7f8050156690 con 0x7f8034078240 2024-09-10T20:43:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.610+0000 7f80467fc640 1 -- 172.21.15.28:0/569362759 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+724 (secure 0 0 0) 0x7f8050156690 con 0x7f8034078240 2024-09-10T20:43:13.613 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "atime": "2024-09-10 20:43:06", 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_pcent": "80.00", 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_quota": 10485760, 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "bytes_used": 8388608, 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "created_at": "2024-09-10 20:43:11", 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "ctime": "2024-09-10 20:43:11", 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "data_pool": "cephfs_data", 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "features": [ 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot-clone", 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot-autoprotect", 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot-retention" 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flavor": 2, 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 1000, 2024-09-10T20:43:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": 16895, 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_addrs": [ 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6789", 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.159:6789", 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.159:6790" 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "mtime": "2024-09-10 20:43:10", 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51", 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_namespace": "", 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "complete", 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "clone", 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "uid": 1000 2024-09-10T20:43:13.615 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8034078240 msgr2=0x7f803407a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8034078240 0x7f803407a700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f80500b30a0 tx=0x7f804c009040 comp rx=0 tx=0).stop 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80501560c0 msgr2=0x7f80500b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80501560c0 0x7f80500b15d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f804800cce0 tx=0x7f804800a620 comp rx=0 tx=0).stop 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 shutdown_connections 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8034078240 0x7f803407a700 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80501560c0 0x7f80500b15d0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80500afb90 0x7f80500b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 --2- 172.21.15.28:0/569362759 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80500af700 0x7f805014ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 >> 172.21.15.28:0/569362759 conn(0x7f805006cf00 msgr2=0x7f80500b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 shutdown_connections 2024-09-10T20:43:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:13.612+0000 7f80581a3640 1 -- 172.21.15.28:0/569362759 wait complete. 2024-09-10T20:43:13.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.627 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:43:13.643 INFO:teuthology.orchestra.run.smithi028.stderr:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350: ceph.quota.max_bytes: No such attribute 2024-09-10T20:43:13.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.644 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:13.700 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:43:13.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.700 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:13.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.757 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:13.815 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350 2024-09-10T20:43:13.815 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.2 2024-09-10T20:43:13.815 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.5 2024-09-10T20:43:13.815 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.4 2024-09-10T20:43:13.815 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.7 2024-09-10T20:43:13.816 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.6 2024-09-10T20:43:13.816 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.0 2024-09-10T20:43:13.816 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.3 2024-09-10T20:43:13.816 INFO:teuthology.orchestra.run.smithi028.stdout:volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.1 2024-09-10T20:43:13.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.816 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:13.872 INFO:teuthology.orchestra.run.smithi028.stdout:41ff 2024-09-10T20:43:13.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.872 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/) 2024-09-10T20:43:13.928 INFO:teuthology.orchestra.run.smithi028.stdout:41ff 2024-09-10T20:43:13.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.929 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:13.984 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:13.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:13.985 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/) 2024-09-10T20:43:14.041 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:14.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.041 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:14.098 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:14.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.098 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/) 2024-09-10T20:43:14.155 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:14.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.156 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350) 2024-09-10T20:43:14.211 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:14.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.212 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/) 2024-09-10T20:43:14.267 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:14.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.268 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.2) 2024-09-10T20:43:14.325 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:14.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.325 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.2) 2024-09-10T20:43:14.385 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:14.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.387 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.2) 2024-09-10T20:43:14.442 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:14.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.443 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.2) 2024-09-10T20:43:14.498 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:14.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.499 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.2) 2024-09-10T20:43:14.554 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:14.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.555 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.2) 2024-09-10T20:43:14.610 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:14.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.610 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.2) 2024-09-10T20:43:14.667 INFO:teuthology.orchestra.run.smithi028.stdout:1726000989 2024-09-10T20:43:14.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.667 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.2) 2024-09-10T20:43:14.724 INFO:teuthology.orchestra.run.smithi028.stdout:1726000989 2024-09-10T20:43:14.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.724 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.5) 2024-09-10T20:43:14.782 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:14.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.782 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.5) 2024-09-10T20:43:14.842 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:14.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.842 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.5) 2024-09-10T20:43:14.898 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:14.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.899 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.5) 2024-09-10T20:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:14.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:14.954 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.5) 2024-09-10T20:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:15.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.011 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.5) 2024-09-10T20:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:15.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.067 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.5) 2024-09-10T20:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:15.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.124 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.5) 2024-09-10T20:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:15.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.181 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.4) 2024-09-10T20:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:15.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.240 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.4) 2024-09-10T20:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:15.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.299 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.4) 2024-09-10T20:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:15.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.355 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.4) 2024-09-10T20:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:15.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.412 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.4) 2024-09-10T20:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:15.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.467 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.4) 2024-09-10T20:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:15.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:15.524 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.4) 2024-09-10T20:43:17.318 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:17.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.319 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.4) 2024-09-10T20:43:17.339 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:17.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.340 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.7) 2024-09-10T20:43:17.399 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:17.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.400 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.7) 2024-09-10T20:43:17.462 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:17.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.463 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.7) 2024-09-10T20:43:17.521 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:17.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.521 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.7) 2024-09-10T20:43:17.578 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:17.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.579 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.7) 2024-09-10T20:43:17.637 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:17.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.638 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.7) 2024-09-10T20:43:17.695 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:17.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.696 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.7) 2024-09-10T20:43:17.753 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:17.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.754 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.7) 2024-09-10T20:43:17.812 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:17.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.813 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.6) 2024-09-10T20:43:17.870 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:17.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.871 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.6) 2024-09-10T20:43:17.931 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:17.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.932 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.6) 2024-09-10T20:43:17.989 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:17.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:17.989 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.6) 2024-09-10T20:43:18.045 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:18.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.046 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.6) 2024-09-10T20:43:18.102 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:18.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.102 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.6) 2024-09-10T20:43:18.159 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:18.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.159 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.6) 2024-09-10T20:43:18.216 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:18.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.217 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.6) 2024-09-10T20:43:18.273 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:18.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.274 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.0) 2024-09-10T20:43:18.331 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:18.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.332 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.0) 2024-09-10T20:43:18.389 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:18.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.390 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.0) 2024-09-10T20:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:18.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.446 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.0) 2024-09-10T20:43:18.502 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:18.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.502 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.0) 2024-09-10T20:43:18.557 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:18.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.558 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.0) 2024-09-10T20:43:18.613 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:18.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.614 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.0) 2024-09-10T20:43:18.672 INFO:teuthology.orchestra.run.smithi028.stdout:1726000989 2024-09-10T20:43:18.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.673 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.0) 2024-09-10T20:43:18.728 INFO:teuthology.orchestra.run.smithi028.stdout:1726000989 2024-09-10T20:43:18.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.729 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.3) 2024-09-10T20:43:18.786 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:18.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.786 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.3) 2024-09-10T20:43:18.842 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:18.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.843 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.3) 2024-09-10T20:43:18.899 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:18.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.900 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.3) 2024-09-10T20:43:18.955 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:18.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:18.956 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.3) 2024-09-10T20:43:19.011 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:19.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.012 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.3) 2024-09-10T20:43:19.067 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:19.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.067 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.3) 2024-09-10T20:43:19.125 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:19.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.126 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.3) 2024-09-10T20:43:19.183 INFO:teuthology.orchestra.run.smithi028.stdout:1726000990 2024-09-10T20:43:19.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.183 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.1) 2024-09-10T20:43:19.239 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:19.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.240 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.1) 2024-09-10T20:43:19.296 INFO:teuthology.orchestra.run.smithi028.stdout:81a4 2024-09-10T20:43:19.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.296 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.1) 2024-09-10T20:43:19.352 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:19.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.353 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.1) 2024-09-10T20:43:19.409 INFO:teuthology.orchestra.run.smithi028.stdout:1000 2024-09-10T20:43:19.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.409 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.1) 2024-09-10T20:43:19.464 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:19.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.465 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.1) 2024-09-10T20:43:19.520 INFO:teuthology.orchestra.run.smithi028.stdout:1333 2024-09-10T20:43:19.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.521 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385/.snap/subvol_snap85350/subvolume_file.1) 2024-09-10T20:43:19.579 INFO:teuthology.orchestra.run.smithi028.stdout:1726000989 2024-09-10T20:43:19.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:19.579 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone92450/00d17032-2357-45f4-83c9-354784c57e51/subvolume_file.1) 2024-09-10T20:43:19.635 INFO:teuthology.orchestra.run.smithi028.stdout:1726000989 2024-09-10T20:43:19.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol63210 fake subvol_clone48531 2024-09-10T20:43:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.783+0000 7f8e1e216640 1 -- 172.21.15.28:0/2363074006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 msgr2=0x7f8e04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.783+0000 7f8e1e216640 1 --2- 172.21.15.28:0/2363074006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 0x7f8e04004c80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009b50 tx=0x7f8e04005830 comp rx=0 tx=0).stop 2024-09-10T20:43:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.783+0000 7f8e17fff640 1 -- 172.21.15.28:0/2363074006 reap_dead start 2024-09-10T20:43:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.783+0000 7f8e1e216640 1 -- 172.21.15.28:0/2363074006 shutdown_connections 2024-09-10T20:43:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.783+0000 7f8e1e216640 1 -- 172.21.15.28:0/2363074006 >> 172.21.15.28:0/2363074006 conn(0x7f8e1806d110 msgr2=0x7f8e180b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.783+0000 7f8e1e216640 1 -- 172.21.15.28:0/2363074006 shutdown_connections 2024-09-10T20:43:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.783+0000 7f8e1e216640 1 -- 172.21.15.28:0/2363074006 wait complete. 2024-09-10T20:43:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 Processor -- start 2024-09-10T20:43:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 -- start start 2024-09-10T20:43:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 0x7f8e18144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e1814b750 0x7f8e18147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e1814bbd0 0x7f8e181482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1806e580 con 0x7f8e04004800 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1806e400 con 0x7f8e1814bbd0 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.785+0000 7f8e1e216640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8e1806e700 con 0x7f8e1814b750 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 0x7f8e18144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 0x7f8e18144510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43518/0 (socket says 172.21.15.28:43518) 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 -- 172.21.15.28:0/2878182249 learned_addr learned my addr 172.21.15.28:0/2878182249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e17fff640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e1814bbd0 0x7f8e181482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e0ffff640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e1814b750 0x7f8e18147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 -- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e1814b750 msgr2=0x7f8e18147d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e1814b750 0x7f8e18147d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 -- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e1814bbd0 msgr2=0x7f8e181482b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e1814bbd0 0x7f8e181482b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 -- 172.21.15.28:0/2878182249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e18144a50 con 0x7f8e04004800 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e177fe640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 0x7f8e18144510 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8e08006680 tx=0x7f8e080066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e14ff9640 1 -- 172.21.15.28:0/2878182249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08015070 con 0x7f8e04004800 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e14ff9640 1 -- 172.21.15.28:0/2878182249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e08011900 con 0x7f8e04004800 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1814b1e0 con 0x7f8e04004800 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e14ff9640 1 -- 172.21.15.28:0/2878182249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08011d50 con 0x7f8e04004800 2024-09-10T20:43:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.786+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1815c4a0 con 0x7f8e04004800 2024-09-10T20:43:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.787+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e1814f030 con 0x7f8e04004800 2024-09-10T20:43:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.788+0000 7f8e14ff9640 1 -- 172.21.15.28:0/2878182249 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 100856+0+0 (secure 0 0 0) 0x7f8e080103c0 con 0x7f8e04004800 2024-09-10T20:43:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.788+0000 7f8e14ff9640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8df8078240 0x7f8df807a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.788+0000 7f8e14ff9640 1 -- 172.21.15.28:0/2878182249 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f8e0809b350 con 0x7f8e04004800 2024-09-10T20:43:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.788+0000 7f8e0ffff640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8df8078240 0x7f8df807a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.789+0000 7f8e0ffff640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8df8078240 0x7f8df807a700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e00035000 tx=0x7f8e0002d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:19.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:19.793+0000 7f8e14ff9640 1 -- 172.21.15.28:0/2878182249 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8e08063a70 con 0x7f8e04004800 2024-09-10T20:43:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.010+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol63210", "snap_name": "fake", "target_sub_name": "subvol_clone48531", "target": ["mon-mgr", ""]}) -- 0x7f8e180b3500 con 0x7f8df8078240 2024-09-10T20:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.070+0000 7f8e14ff9640 1 -- 172.21.15.28:0/2878182249 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8e180b3500 con 0x7f8df8078240 2024-09-10T20:43:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.073+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8df8078240 msgr2=0x7f8df807a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.073+0000 7f8e1e216640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8df8078240 0x7f8df807a700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e00035000 tx=0x7f8e0002d040 comp rx=0 tx=0).stop 2024-09-10T20:43:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 msgr2=0x7f8e18144510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 0x7f8e18144510 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8e08006680 tx=0x7f8e080066b0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 shutdown_connections 2024-09-10T20:43:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8df8078240 0x7f8df807a700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8e1814bbd0 0x7f8e181482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8e1814b750 0x7f8e18147d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 --2- 172.21.15.28:0/2878182249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e04004800 0x7f8e18144510 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 >> 172.21.15.28:0/2878182249 conn(0x7f8e1806d110 msgr2=0x7f8e1815d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:20.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 shutdown_connections 2024-09-10T20:43:20.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.074+0000 7f8e1e216640 1 -- 172.21.15.28:0/2878182249 wait complete. 2024-09-10T20:43:20.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48531 2024-09-10T20:43:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 -- 172.21.15.28:0/445903110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc161770 msgr2=0x7f5ecc0d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 --2- 172.21.15.28:0/445903110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc161770 0x7f5ecc0d20e0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0d1e80 tx=0x7f5ebc02a4a0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 -- 172.21.15.28:0/445903110 shutdown_connections 2024-09-10T20:43:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 --2- 172.21.15.28:0/445903110 >> v2:172.21.15.159:3300/0 conn(0x7f5ecc0d4b90 0x7f5ecc0d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 --2- 172.21.15.28:0/445903110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc161770 0x7f5ecc0d20e0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 -- 172.21.15.28:0/445903110 >> 172.21.15.28:0/445903110 conn(0x7f5ecc0d4070 msgr2=0x7f5ecc0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 -- 172.21.15.28:0/445903110 shutdown_connections 2024-09-10T20:43:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.195+0000 7f5ed4471640 1 -- 172.21.15.28:0/445903110 wait complete. 2024-09-10T20:43:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 Processor -- start 2024-09-10T20:43:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 -- start start 2024-09-10T20:43:20.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc0d4b90 0x7f5ecc15d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:20.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ecc161770 0x7f5ecc1f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ecc161b50 0x7f5ecc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc0b2f50 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc0b2dd0 con 0x7f5ecc161770 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.197+0000 7f5ed4471640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5ecc0b30d0 con 0x7f5ecc161b50 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc0d4b90 0x7f5ecc15d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc0d4b90 0x7f5ecc15d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43526/0 (socket says 172.21.15.28:43526) 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 -- 172.21.15.28:0/2885599606 learned_addr learned my addr 172.21.15.28:0/2885599606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed1a0c640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ecc161770 0x7f5ecc1f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed2a0e640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ecc161b50 0x7f5ecc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 -- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ecc161b50 msgr2=0x7f5ecc06d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ecc161b50 0x7f5ecc06d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 -- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ecc161770 msgr2=0x7f5ecc1f8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ecc161770 0x7f5ecc1f8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 -- 172.21.15.28:0/2885599606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ecc1f9020 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed220d640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc0d4b90 0x7f5ecc15d370 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009780 tx=0x7f5ebc032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ecaffd640 1 -- 172.21.15.28:0/2885599606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc040070 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ecaffd640 1 -- 172.21.15.28:0/2885599606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ebc005310 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ecc1f4c80 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ecaffd640 1 -- 172.21.15.28:0/2885599606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc003800 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.198+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ecc1f5240 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.199+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ecc0b3870 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.200+0000 7f5ecaffd640 1 -- 172.21.15.28:0/2885599606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 100856+0+0 (secure 0 0 0) 0x7f5ebc0039a0 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.201+0000 7f5ecaffd640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5eb8078240 0x7f5eb807a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.201+0000 7f5ed1a0c640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5eb8078240 0x7f5eb807a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.201+0000 7f5ecaffd640 1 -- 172.21.15.28:0/2885599606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f5ebc0bb8b0 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.201+0000 7f5ed1a0c640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5eb8078240 0x7f5eb807a700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0c7a10 tx=0x7f5ec0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.205+0000 7f5ecaffd640 1 -- 172.21.15.28:0/2885599606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ecc1f9020 con 0x7f5ecc0d4b90 2024-09-10T20:43:20.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:20.427+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48531", "target": ["mon-mgr", ""]}) -- 0x7f5ecc1eab60 con 0x7f5eb8078240 2024-09-10T20:43:22.328 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:22.324+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:22.329 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:22.324+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:22.329 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:22.324+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:22.329 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:22.324+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:22.329 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:22.324+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.338+0000 7f5ecaffd640 1 -- 172.21.15.28:0/2885599606 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+153 (secure 0 0 0) 0x7f5ecc1eab60 con 0x7f5eb8078240 2024-09-10T20:43:22.343 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:22.343 INFO:teuthology.orchestra.run.smithi028.stdout: "status": { 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "pending", 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout: "source": { 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout: "volume": "cephfs", 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout: "subvolume": "subvol63210", 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot": "fake" 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:22.344 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5eb8078240 msgr2=0x7f5eb807a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5eb8078240 0x7f5eb807a700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0c7a10 tx=0x7f5ec0009040 comp rx=0 tx=0).stop 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc0d4b90 msgr2=0x7f5ecc15d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc0d4b90 0x7f5ecc15d370 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009780 tx=0x7f5ebc032d50 comp rx=0 tx=0).stop 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 shutdown_connections 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5eb8078240 0x7f5eb807a700 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ecc161b50 0x7f5ecc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ecc161770 0x7f5ecc1f8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 --2- 172.21.15.28:0/2885599606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc0d4b90 0x7f5ecc15d370 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.342+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 >> 172.21.15.28:0/2885599606 conn(0x7f5ecc0d4070 msgr2=0x7f5ecc06ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.343+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 shutdown_connections 2024-09-10T20:43:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:22.343+0000 7f5ed4471640 1 -- 172.21.15.28:0/2885599606 wait complete. 2024-09-10T20:43:22.362 DEBUG:tasks.cephfs.test_volumes:current clone state = pending 2024-09-10T20:43:23.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48531 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.482+0000 7f48975da640 1 -- 172.21.15.28:0/3818628214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 msgr2=0x7f48900b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.482+0000 7f48975da640 1 --2- 172.21.15.28:0/3818628214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48900b1830 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4880005fd0 tx=0x7f488002a4a0 comp rx=0 tx=0).stop 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.482+0000 7f48975da640 1 -- 172.21.15.28:0/3818628214 shutdown_connections 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.482+0000 7f48975da640 1 --2- 172.21.15.28:0/3818628214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4890161b50 0x7f48900b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.482+0000 7f48975da640 1 --2- 172.21.15.28:0/3818628214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48900b1830 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.482+0000 7f48975da640 1 -- 172.21.15.28:0/3818628214 >> 172.21.15.28:0/3818628214 conn(0x7f48900d4070 msgr2=0x7f48900d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.483+0000 7f48975da640 1 -- 172.21.15.28:0/3818628214 shutdown_connections 2024-09-10T20:43:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.483+0000 7f48975da640 1 -- 172.21.15.28:0/3818628214 wait complete. 2024-09-10T20:43:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.484+0000 7f48975da640 1 Processor -- start 2024-09-10T20:43:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.484+0000 7f48975da640 1 -- start start 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f48975da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48901f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f48975da640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4890161770 0x7f489015b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f48975da640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4890161b50 0x7f4890157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f48975da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48900b2dd0 con 0x7f48900d27f0 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f48975da640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f48900d4e60 con 0x7f4890161770 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f48975da640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f48900b2f50 con 0x7f4890161b50 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48901f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48901f54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45800/0 (socket says 172.21.15.28:45800) 2024-09-10T20:43:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 -- 172.21.15.28:0/1536950456 learned_addr learned my addr 172.21.15.28:0/1536950456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4894b75640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4890161770 0x7f489015b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895b77640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4890161b50 0x7f4890157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 -- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4890161b50 msgr2=0x7f4890157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4890161b50 0x7f4890157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 -- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4890161770 msgr2=0x7f489015b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4890161770 0x7f489015b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 -- 172.21.15.28:0/1536950456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489015c740 con 0x7f48900d27f0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4895376640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48901f54a0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c7c0 tx=0x7f488c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4885ffb640 1 -- 172.21.15.28:0/1536950456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c016020 con 0x7f48900d27f0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4885ffb640 1 -- 172.21.15.28:0/1536950456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f488c00ab40 con 0x7f48900d27f0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f4885ffb640 1 -- 172.21.15.28:0/1536950456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c012420 con 0x7f48900d27f0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.485+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489015ca20 con 0x7f48900d27f0 2024-09-10T20:43:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.486+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48901f4f80 con 0x7f48900d27f0 2024-09-10T20:43:23.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.487+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48900d20e0 con 0x7f48900d27f0 2024-09-10T20:43:23.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.487+0000 7f4885ffb640 1 -- 172.21.15.28:0/1536950456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7f488c01a3b0 con 0x7f48900d27f0 2024-09-10T20:43:23.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.488+0000 7f4885ffb640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48740782d0 0x7f487407a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:23.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.488+0000 7f4885ffb640 1 -- 172.21.15.28:0/1536950456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f488c09a370 con 0x7f48900d27f0 2024-09-10T20:43:23.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.488+0000 7f4894b75640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48740782d0 0x7f487407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:23.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.489+0000 7f4894b75640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48740782d0 0x7f487407a790 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4880033000 tx=0x7f4880004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.492+0000 7f4885ffb640 1 -- 172.21.15.28:0/1536950456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f488c020070 con 0x7f48900d27f0 2024-09-10T20:43:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:23.716+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48531", "target": ["mon-mgr", ""]}) -- 0x7f48900cfe60 con 0x7f48740782d0 2024-09-10T20:43:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.320+0000 7f4885ffb640 1 -- 172.21.15.28:0/1536950456 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f48900cfe60 con 0x7f48740782d0 2024-09-10T20:43:27.326 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "status": { 2024-09-10T20:43:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "complete" 2024-09-10T20:43:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:27.326 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48740782d0 msgr2=0x7f487407a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48740782d0 0x7f487407a790 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4880033000 tx=0x7f4880004730 comp rx=0 tx=0).stop 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 msgr2=0x7f48901f54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48901f54a0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c7c0 tx=0x7f488c00cc80 comp rx=0 tx=0).stop 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 shutdown_connections 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48740782d0 0x7f487407a790 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4890161b50 0x7f4890157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4890161770 0x7f489015b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 --2- 172.21.15.28:0/1536950456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48900d27f0 0x7f48901f54a0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.323+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 >> 172.21.15.28:0/1536950456 conn(0x7f48900d4070 msgr2=0x7f48900b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.324+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 shutdown_connections 2024-09-10T20:43:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.324+0000 7f48975da640 1 -- 172.21.15.28:0/1536950456 wait complete. 2024-09-10T20:43:27.342 DEBUG:tasks.cephfs.test_volumes:current clone state = complete 2024-09-10T20:43:27.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:27.343 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol_clone48531/.meta) 2024-09-10T20:43:27.371 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:43:27.372 INFO:teuthology.orchestra.run.smithi028.stdout:version = 2 2024-09-10T20:43:27.372 INFO:teuthology.orchestra.run.smithi028.stdout:type = clone 2024-09-10T20:43:27.372 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol_clone48531/76d75643-7272-4c0f-8646-6d3f6b6f21da 2024-09-10T20:43:27.372 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:43:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:27.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot info cephfs subvol63210 subvol_snap85350 2024-09-10T20:43:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 -- 172.21.15.28:0/3181021152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7815a350 msgr2=0x7f1f780ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 --2- 172.21.15.28:0/3181021152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7815a350 0x7f1f780ca2d0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1f68005fd0 tx=0x7f1f6802a9c0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 -- 172.21.15.28:0/3181021152 shutdown_connections 2024-09-10T20:43:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 --2- 172.21.15.28:0/3181021152 >> v2:172.21.15.159:3300/0 conn(0x7f1f780702e0 0x7f1f780cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 --2- 172.21.15.28:0/3181021152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f780cabc0 0x7f1f7806fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 --2- 172.21.15.28:0/3181021152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7815a350 0x7f1f780ca2d0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 -- 172.21.15.28:0/3181021152 >> 172.21.15.28:0/3181021152 conn(0x7f1f780cfc80 msgr2=0x7f1f780d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 -- 172.21.15.28:0/3181021152 shutdown_connections 2024-09-10T20:43:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.524+0000 7f1f7d116640 1 -- 172.21.15.28:0/3181021152 wait complete. 2024-09-10T20:43:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.525+0000 7f1f7d116640 1 Processor -- start 2024-09-10T20:43:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.525+0000 7f1f7d116640 1 -- start start 2024-09-10T20:43:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7d116640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f780cabc0 0x7f1f78149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7d116640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7815a350 0x7f1f7814a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7d116640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f78166400 0x7f1f7814ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7d116640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f780ce3c0 con 0x7f1f78166400 2024-09-10T20:43:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7d116640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1f780ce240 con 0x7f1f780cabc0 2024-09-10T20:43:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7d116640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7806e390 con 0x7f1f7815a350 2024-09-10T20:43:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f78166400 0x7f1f7814ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f78166400 0x7f1f7814ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45830/0 (socket says 172.21.15.28:45830) 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7759e640 1 -- 172.21.15.28:0/331823208 learned_addr learned my addr 172.21.15.28:0/331823208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f7659c640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7815a350 0x7f1f7814a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.526+0000 7f1f76d9d640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f780cabc0 0x7f1f78149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7759e640 1 -- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7815a350 msgr2=0x7f1f7814a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7759e640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7815a350 0x7f1f7814a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7759e640 1 -- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f780cabc0 msgr2=0x7f1f78149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7759e640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f780cabc0 0x7f1f78149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7759e640 1 -- 172.21.15.28:0/331823208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7814fb40 con 0x7f1f78166400 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7759e640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f78166400 0x7f1f7814ad00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000d6a0 tx=0x7f1f7000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/331823208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70017070 con 0x7f1f78166400 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/331823208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f70005310 con 0x7f1f78166400 2024-09-10T20:43:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/331823208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70009820 con 0x7f1f78166400 2024-09-10T20:43:27.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f78149fc0 con 0x7f1f78166400 2024-09-10T20:43:27.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.527+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7814fde0 con 0x7f1f78166400 2024-09-10T20:43:27.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.528+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7806e7e0 con 0x7f1f78166400 2024-09-10T20:43:27.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.529+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/331823208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7f1f700099c0 con 0x7f1f78166400 2024-09-10T20:43:27.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.529+0000 7f1f6f7fe640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f580782d0 0x7f1f5807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.529+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/331823208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f1f7009a5b0 con 0x7f1f78166400 2024-09-10T20:43:27.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.529+0000 7f1f76d9d640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f580782d0 0x7f1f5807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.530+0000 7f1f76d9d640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f580782d0 0x7f1f5807a790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1f68035000 tx=0x7f1f68004ab0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:27.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.533+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/331823208 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1f70066db0 con 0x7f1f78166400 2024-09-10T20:43:27.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.749+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot info", "vol_name": "cephfs", "sub_name": "subvol63210", "snap_name": "subvol_snap85350", "target": ["mon-mgr", ""]}) -- 0x7f1f780ce7d0 con 0x7f1f580782d0 2024-09-10T20:43:27.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.762+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/331823208 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+115 (secure 0 0 0) 0x7f1f780ce7d0 con 0x7f1f580782d0 2024-09-10T20:43:27.767 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "created_at": "2024-09-10 20:43:11.007832", 2024-09-10T20:43:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "data_pool": "cephfs_data", 2024-09-10T20:43:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "has_pending_clones": "no" 2024-09-10T20:43:27.767 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f580782d0 msgr2=0x7f1f5807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f580782d0 0x7f1f5807a790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1f68035000 tx=0x7f1f68004ab0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f78166400 msgr2=0x7f1f7814ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f78166400 0x7f1f7814ad00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000d6a0 tx=0x7f1f7000db60 comp rx=0 tx=0).stop 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 shutdown_connections 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f580782d0 0x7f1f5807a790 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f78166400 0x7f1f7814ad00 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7815a350 0x7f1f7814a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 --2- 172.21.15.28:0/331823208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f780cabc0 0x7f1f78149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 >> 172.21.15.28:0/331823208 conn(0x7f1f780cfc80 msgr2=0x7f1f780b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 shutdown_connections 2024-09-10T20:43:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.764+0000 7f1f7d116640 1 -- 172.21.15.28:0/331823208 wait complete. 2024-09-10T20:43:27.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs subvol63210 2024-09-10T20:43:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.886+0000 7fa405705640 1 -- 172.21.15.28:0/2912845492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ec004800 msgr2=0x7fa3ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.886+0000 7fa405705640 1 --2- 172.21.15.28:0/2912845492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ec004800 0x7fa3ec004c80 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80016c0 tx=0x7fa3ec005830 comp rx=0 tx=0).stop 2024-09-10T20:43:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.886+0000 7fa405705640 1 -- 172.21.15.28:0/2912845492 shutdown_connections 2024-09-10T20:43:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.886+0000 7fa405705640 1 --2- 172.21.15.28:0/2912845492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3ec004800 0x7fa3ec004c80 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.886+0000 7fa405705640 1 -- 172.21.15.28:0/2912845492 >> 172.21.15.28:0/2912845492 conn(0x7fa40009d260 msgr2=0x7fa4000d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.886+0000 7fa405705640 1 -- 172.21.15.28:0/2912845492 shutdown_connections 2024-09-10T20:43:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.886+0000 7fa405705640 1 -- 172.21.15.28:0/2912845492 wait complete. 2024-09-10T20:43:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.887+0000 7fa405705640 1 Processor -- start 2024-09-10T20:43:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.887+0000 7fa405705640 1 -- start start 2024-09-10T20:43:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa405705640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ec004800 0x7fa4000c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa405705640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4000b2170 0x7fa4001602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa405705640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4001607e0 0x7fa4000c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa405705640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4000d2660 con 0x7fa4001607e0 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa405705640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa4000d4c50 con 0x7fa4000b2170 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa405705640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa4000d27e0 con 0x7fa3ec004800 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4001607e0 0x7fa4000c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4001607e0 0x7fa4000c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45852/0 (socket says 172.21.15.28:45852) 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3feffd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ec004800 0x7fa4000c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 -- 172.21.15.28:0/472863269 learned_addr learned my addr 172.21.15.28:0/472863269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3f67fc640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4000b2170 0x7fa4001602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 -- 172.21.15.28:0/472863269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ec004800 msgr2=0x7fa4000c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ec004800 0x7fa4000c7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 -- 172.21.15.28:0/472863269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4000b2170 msgr2=0x7fa4001602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4000b2170 0x7fa4001602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.888+0000 7fa3ff7fe640 1 -- 172.21.15.28:0/472863269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4001f5e00 con 0x7fa4001607e0 2024-09-10T20:43:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.889+0000 7fa3ff7fe640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4001607e0 0x7fa4000c9230 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8000f80 tx=0x7fa3f800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.889+0000 7fa3f7fff640 1 -- 172.21.15.28:0/472863269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f800cc20 con 0x7fa4001607e0 2024-09-10T20:43:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.889+0000 7fa3f7fff640 1 -- 172.21.15.28:0/472863269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3f8018070 con 0x7fa4001607e0 2024-09-10T20:43:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.889+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4001f5280 con 0x7fa4001607e0 2024-09-10T20:43:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.889+0000 7fa3f7fff640 1 -- 172.21.15.28:0/472863269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80116c0 con 0x7fa4001607e0 2024-09-10T20:43:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.889+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4001f5840 con 0x7fa4001607e0 2024-09-10T20:43:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.890+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa40008d630 con 0x7fa4001607e0 2024-09-10T20:43:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.890+0000 7fa3f7fff640 1 -- 172.21.15.28:0/472863269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7fa3f8011860 con 0x7fa4001607e0 2024-09-10T20:43:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.891+0000 7fa3f7fff640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3e00782d0 0x7fa3e007a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.891+0000 7fa3f7fff640 1 -- 172.21.15.28:0/472863269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7fa3f809a680 con 0x7fa4001607e0 2024-09-10T20:43:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.891+0000 7fa3feffd640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3e00782d0 0x7fa3e007a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.891+0000 7fa3feffd640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3e00782d0 0x7fa3e007a790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00037e0 tx=0x7fa3f000b260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:27.895+0000 7fa3f7fff640 1 -- 172.21.15.28:0/472863269 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa40008d630 con 0x7fa4001607e0 2024-09-10T20:43:28.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.112+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "subvol63210", "target": ["mon-mgr", ""]}) -- 0x7fa4000d3670 con 0x7fa3e00782d0 2024-09-10T20:43:28.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.124+0000 7fa3f7fff640 1 -- 172.21.15.28:0/472863269 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+87 (secure 0 0 0) 0x7fa4000d3670 con 0x7fa3e00782d0 2024-09-10T20:43:28.129 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:43:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fake" 2024-09-10T20:43:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subvol_snap85350" 2024-09-10T20:43:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:28.130 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3e00782d0 msgr2=0x7fa3e007a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3e00782d0 0x7fa3e007a790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00037e0 tx=0x7fa3f000b260 comp rx=0 tx=0).stop 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4001607e0 msgr2=0x7fa4000c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4001607e0 0x7fa4000c9230 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8000f80 tx=0x7fa3f800ed10 comp rx=0 tx=0).stop 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 shutdown_connections 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3e00782d0 0x7fa3e007a790 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4001607e0 0x7fa4000c9230 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4000b2170 0x7fa4001602a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 --2- 172.21.15.28:0/472863269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3ec004800 0x7fa4000c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.126+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 >> 172.21.15.28:0/472863269 conn(0x7fa40009d260 msgr2=0x7fa4000cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.127+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 shutdown_connections 2024-09-10T20:43:28.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.127+0000 7fa405705640 1 -- 172.21.15.28:0/472863269 wait complete. 2024-09-10T20:43:28.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol63210 subvol_snap85350 2024-09-10T20:43:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 -- 172.21.15.28:0/3422113403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54072db0 msgr2=0x7f2a5406fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/3422113403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54072db0 0x7f2a5406fd30 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009860 tx=0x7f2a4802a540 comp rx=0 tx=0).stop 2024-09-10T20:43:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 -- 172.21.15.28:0/3422113403 shutdown_connections 2024-09-10T20:43:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/3422113403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54072db0 0x7f2a5406fd30 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/3422113403 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54151cf0 0x7f2a5415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 -- 172.21.15.28:0/3422113403 >> 172.21.15.28:0/3422113403 conn(0x7f2a540cfc80 msgr2=0x7f2a540d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 -- 172.21.15.28:0/3422113403 shutdown_connections 2024-09-10T20:43:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.247+0000 7f2a5aa71640 1 -- 172.21.15.28:0/3422113403 wait complete. 2024-09-10T20:43:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.248+0000 7f2a5aa71640 1 Processor -- start 2024-09-10T20:43:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.248+0000 7f2a5aa71640 1 -- start start 2024-09-10T20:43:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5aa71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54070270 0x7f2a540a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5aa71640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54072db0 0x7f2a54150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5aa71640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a54151cf0 0x7f2a5414c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5aa71640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5415eed0 con 0x7f2a54070270 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5aa71640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540cb840 con 0x7f2a54151cf0 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5aa71640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2a5415f050 con 0x7f2a54072db0 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5880d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54070270 0x7f2a540a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5880d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54070270 0x7f2a540a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45878/0 (socket says 172.21.15.28:45878) 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.249+0000 7f2a5880d640 1 -- 172.21.15.28:0/2639613064 learned_addr learned my addr 172.21.15.28:0/2639613064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5880d640 1 -- 172.21.15.28:0/2639613064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54072db0 msgr2=0x7f2a54150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5880d640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54072db0 0x7f2a54150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5880d640 1 -- 172.21.15.28:0/2639613064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a54151cf0 msgr2=0x7f2a5414c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5880d640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a54151cf0 0x7f2a5414c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5880d640 1 -- 172.21.15.28:0/2639613064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a541f6910 con 0x7f2a54070270 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5880d640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54070270 0x7f2a540a94a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400d740 tx=0x7f2a4400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a517fa640 1 -- 172.21.15.28:0/2639613064 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44017070 con 0x7f2a54070270 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a517fa640 1 -- 172.21.15.28:0/2639613064 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a44009d20 con 0x7f2a54070270 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a517fa640 1 -- 172.21.15.28:0/2639613064 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440106c0 con 0x7f2a54070270 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a541501f0 con 0x7f2a54070270 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.250+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a541f65c0 con 0x7f2a54070270 2024-09-10T20:43:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.251+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5415f500 con 0x7f2a54070270 2024-09-10T20:43:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.252+0000 7f2a517fa640 1 -- 172.21.15.28:0/2639613064 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7f2a44010860 con 0x7f2a54070270 2024-09-10T20:43:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.252+0000 7f2a517fa640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a40078300 0x7f2a4007a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.252+0000 7f2a517fa640 1 -- 172.21.15.28:0/2639613064 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f2a4409a820 con 0x7f2a54070270 2024-09-10T20:43:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.252+0000 7f2a53fff640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a40078300 0x7f2a4007a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.253+0000 7f2a53fff640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a40078300 0x7f2a4007a7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2a48034000 tx=0x7f2a480046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.257+0000 7f2a517fa640 1 -- 172.21.15.28:0/2639613064 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2a44062d50 con 0x7f2a54070270 2024-09-10T20:43:28.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.477+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol63210", "snap_name": "subvol_snap85350", "target": ["mon-mgr", ""]}) -- 0x7f2a540ccbb0 con 0x7f2a40078300 2024-09-10T20:43:28.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.503+0000 7f2a517fa640 1 -- 172.21.15.28:0/2639613064 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2a540ccbb0 con 0x7f2a40078300 2024-09-10T20:43:28.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a40078300 msgr2=0x7f2a4007a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a40078300 0x7f2a4007a7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2a48034000 tx=0x7f2a480046c0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54070270 msgr2=0x7f2a540a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54070270 0x7f2a540a94a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400d740 tx=0x7f2a4400dc00 comp rx=0 tx=0).stop 2024-09-10T20:43:28.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 shutdown_connections 2024-09-10T20:43:28.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a40078300 0x7f2a4007a7c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2a54151cf0 0x7f2a5414c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2a54072db0 0x7f2a54150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 --2- 172.21.15.28:0/2639613064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a54070270 0x7f2a540a94a0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 >> 172.21.15.28:0/2639613064 conn(0x7f2a540cfc80 msgr2=0x7f2a540ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:28.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 shutdown_connections 2024-09-10T20:43:28.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.505+0000 7f2a5aa71640 1 -- 172.21.15.28:0/2639613064 wait complete. 2024-09-10T20:43:28.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol63210 fake 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.626+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/2750043066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c806e5e0 msgr2=0x7fc0c806e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.626+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/2750043066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c806e5e0 0x7fc0c806e9c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0005fd0 tx=0x7fc0c002a540 comp rx=0 tx=0).stop 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.626+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/2750043066 shutdown_connections 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.627+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/2750043066 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c80d0eb0 0x7fc0c80b6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.627+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/2750043066 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c806e5e0 0x7fc0c806e9c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.627+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/2750043066 >> 172.21.15.28:0/2750043066 conn(0x7fc0c80d1da0 msgr2=0x7fc0c80d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.627+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/2750043066 shutdown_connections 2024-09-10T20:43:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.627+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/2750043066 wait complete. 2024-09-10T20:43:28.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.628+0000 7fc0cfe2a640 1 Processor -- start 2024-09-10T20:43:28.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.628+0000 7fc0cfe2a640 1 -- start start 2024-09-10T20:43:28.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.628+0000 7fc0cfe2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c806e5e0 0x7fc0c80b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cfe2a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c80b2b40 0x7fc0c809f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cfe2a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c80d0eb0 0x7fc0c80c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cfe2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c8162b10 con 0x7fc0c806e5e0 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cfe2a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c8162990 con 0x7fc0c80b2b40 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cfe2a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c80b6ca0 con 0x7fc0c80d0eb0 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c806e5e0 0x7fc0c80b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c806e5e0 0x7fc0c80b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45882/0 (socket says 172.21.15.28:45882) 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 -- 172.21.15.28:0/1960063601 learned_addr learned my addr 172.21.15.28:0/1960063601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cd3c5640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c80b2b40 0x7fc0c809f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0ce3c7640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c80d0eb0 0x7fc0c80c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 -- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c80d0eb0 msgr2=0x7fc0c80c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c80d0eb0 0x7fc0c80c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 -- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c80b2b40 msgr2=0x7fc0c809f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c80b2b40 0x7fc0c809f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 -- 172.21.15.28:0/1960063601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c80ae9b0 con 0x7fc0c806e5e0 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cdbc6640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c806e5e0 0x7fc0c80b10b0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0000c00 tx=0x7fc0c00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0c67fc640 1 -- 172.21.15.28:0/1960063601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c003f070 con 0x7fc0c806e5e0 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0c67fc640 1 -- 172.21.15.28:0/1960063601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0c00047a0 con 0x7fc0c806e5e0 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c80aec90 con 0x7fc0c806e5e0 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.629+0000 7fc0c67fc640 1 -- 172.21.15.28:0/1960063601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0035030 con 0x7fc0c806e5e0 2024-09-10T20:43:28.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.630+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c80af250 con 0x7fc0c806e5e0 2024-09-10T20:43:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.630+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c806d750 con 0x7fc0c806e5e0 2024-09-10T20:43:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.631+0000 7fc0c67fc640 1 -- 172.21.15.28:0/1960063601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7fc0c0004bf0 con 0x7fc0c806e5e0 2024-09-10T20:43:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.632+0000 7fc0c67fc640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078240 0x7fc0b407a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.632+0000 7fc0c67fc640 1 -- 172.21.15.28:0/1960063601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7fc0c00bb680 con 0x7fc0c806e5e0 2024-09-10T20:43:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.632+0000 7fc0cd3c5640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078240 0x7fc0b407a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.632+0000 7fc0cd3c5640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078240 0x7fc0b407a700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b32b0 tx=0x7fc0bc009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.636+0000 7fc0c67fc640 1 -- 172.21.15.28:0/1960063601 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc0c00836c0 con 0x7fc0c806e5e0 2024-09-10T20:43:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.854+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol63210", "snap_name": "fake", "target": ["mon-mgr", ""]}) -- 0x7fc0c80b71e0 con 0x7fc0b4078240 2024-09-10T20:43:28.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.880+0000 7fc0c67fc640 1 -- 172.21.15.28:0/1960063601 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc0c80b71e0 con 0x7fc0b4078240 2024-09-10T20:43:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078240 msgr2=0x7fc0b407a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078240 0x7fc0b407a700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b32b0 tx=0x7fc0bc009040 comp rx=0 tx=0).stop 2024-09-10T20:43:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c806e5e0 msgr2=0x7fc0c80b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c806e5e0 0x7fc0c80b10b0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0000c00 tx=0x7fc0c00038a0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 shutdown_connections 2024-09-10T20:43:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078240 0x7fc0b407a700 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c80d0eb0 0x7fc0c80c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c80b2b40 0x7fc0c809f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 --2- 172.21.15.28:0/1960063601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c806e5e0 0x7fc0c80b10b0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 >> 172.21.15.28:0/1960063601 conn(0x7fc0c80d1da0 msgr2=0x7fc0c80b6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 shutdown_connections 2024-09-10T20:43:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:28.882+0000 7fc0cfe2a640 1 -- 172.21.15.28:0/1960063601 wait complete. 2024-09-10T20:43:28.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:28.898 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol63210/.meta) 2024-09-10T20:43:28.924 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:43:28.924 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:43:28.925 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:43:28.925 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol63210/ef91703b-0c8c-46b7-ad31-f031bbf48385 2024-09-10T20:43:28.925 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:43:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:28.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol63210 2024-09-10T20:43:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.072+0000 7fd9033ec640 1 -- 172.21.15.28:0/216629955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8003a80 msgr2=0x7fd8e8003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.072+0000 7fd9033ec640 1 --2- 172.21.15.28:0/216629955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8003a80 0x7fd8e8003ea0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8001660 tx=0x7fd8e8005830 comp rx=0 tx=0).stop 2024-09-10T20:43:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.072+0000 7fd901989640 1 -- 172.21.15.28:0/216629955 reap_dead start 2024-09-10T20:43:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.072+0000 7fd9033ec640 1 -- 172.21.15.28:0/216629955 shutdown_connections 2024-09-10T20:43:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.072+0000 7fd9033ec640 1 -- 172.21.15.28:0/216629955 >> 172.21.15.28:0/216629955 conn(0x7fd8fc06d110 msgr2=0x7fd8fc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.073+0000 7fd9033ec640 1 -- 172.21.15.28:0/216629955 shutdown_connections 2024-09-10T20:43:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.073+0000 7fd9033ec640 1 -- 172.21.15.28:0/216629955 wait complete. 2024-09-10T20:43:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.074+0000 7fd9033ec640 1 Processor -- start 2024-09-10T20:43:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.074+0000 7fd9033ec640 1 -- start start 2024-09-10T20:43:29.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.074+0000 7fd9033ec640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e8003a80 0x7fd8fc152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.074+0000 7fd9033ec640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8fc072a70 0x7fd8fc072e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.074+0000 7fd9033ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8fc1538e0 0x7fd8fc153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.074+0000 7fd9033ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc06f8a0 con 0x7fd8fc1538e0 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.074+0000 7fd9033ec640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc06f720 con 0x7fd8fc072a70 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd9033ec640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd8fc15d2e0 con 0x7fd8e8003a80 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8fc1538e0 0x7fd8fc153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8fc1538e0 0x7fd8fc153d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45904/0 (socket says 172.21.15.28:45904) 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901188640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e8003a80 0x7fd8fc152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 -- 172.21.15.28:0/931277564 learned_addr learned my addr 172.21.15.28:0/931277564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd900987640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8fc072a70 0x7fd8fc072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 -- 172.21.15.28:0/931277564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e8003a80 msgr2=0x7fd8fc152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e8003a80 0x7fd8fc152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 -- 172.21.15.28:0/931277564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8fc072a70 msgr2=0x7fd8fc072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8fc072a70 0x7fd8fc072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 -- 172.21.15.28:0/931277564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8fc0737e0 con 0x7fd8fc1538e0 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd901989640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8fc1538e0 0x7fd8fc153d70 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0b3150 tx=0x7fd8f400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd8f1ffb640 1 -- 172.21.15.28:0/931277564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4015070 con 0x7fd8fc1538e0 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd8f1ffb640 1 -- 172.21.15.28:0/931277564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8f400cd30 con 0x7fd8fc1538e0 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8fc073ac0 con 0x7fd8fc1538e0 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.075+0000 7fd8f1ffb640 1 -- 172.21.15.28:0/931277564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40116c0 con 0x7fd8fc1538e0 2024-09-10T20:43:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.076+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8fc0ad9d0 con 0x7fd8fc1538e0 2024-09-10T20:43:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.077+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8fc15cbd0 con 0x7fd8fc1538e0 2024-09-10T20:43:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.077+0000 7fd8f1ffb640 1 -- 172.21.15.28:0/931277564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7fd8f4011860 con 0x7fd8fc1538e0 2024-09-10T20:43:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.078+0000 7fd8f1ffb640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8e0078300 0x7fd8e007a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.078+0000 7fd8f1ffb640 1 -- 172.21.15.28:0/931277564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7fd8f409ac30 con 0x7fd8fc1538e0 2024-09-10T20:43:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.078+0000 7fd901188640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8e0078300 0x7fd8e007a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.078+0000 7fd901188640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8e0078300 0x7fd8e007a7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8032000 tx=0x7fd8f8001190 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.082+0000 7fd8f1ffb640 1 -- 172.21.15.28:0/931277564 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd8f4062b10 con 0x7fd8fc1538e0 2024-09-10T20:43:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.294+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol63210", "target": ["mon-mgr", ""]}) -- 0x7fd8fc0b2540 con 0x7fd8e0078300 2024-09-10T20:43:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.330+0000 7fd8f1ffb640 1 -- 172.21.15.28:0/931277564 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd8fc0b2540 con 0x7fd8e0078300 2024-09-10T20:43:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8e0078300 msgr2=0x7fd8e007a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8e0078300 0x7fd8e007a7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8032000 tx=0x7fd8f8001190 comp rx=0 tx=0).stop 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8fc1538e0 msgr2=0x7fd8fc153d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8fc1538e0 0x7fd8fc153d70 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0b3150 tx=0x7fd8f400ed90 comp rx=0 tx=0).stop 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 shutdown_connections 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8e0078300 0x7fd8e007a7c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8fc1538e0 0x7fd8fc153d70 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8fc072a70 0x7fd8fc072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 --2- 172.21.15.28:0/931277564 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8e8003a80 0x7fd8fc152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 >> 172.21.15.28:0/931277564 conn(0x7fd8fc06d110 msgr2=0x7fd8fc1597c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 shutdown_connections 2024-09-10T20:43:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.332+0000 7fd9033ec640 1 -- 172.21.15.28:0/931277564 wait complete. 2024-09-10T20:43:29.339 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.334+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.339 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.334+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.339 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.334+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.339 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.334+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.339 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.334+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone92450 2024-09-10T20:43:29.356 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.349+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.356 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.349+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.356 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.349+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.356 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.349+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.356 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:43:29.349+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:43:29.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 -- 172.21.15.28:0/1810345448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b3110 msgr2=0x7fb02c0cec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 --2- 172.21.15.28:0/1810345448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b3110 0x7fb02c0cec90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fb02c073bc0 tx=0x7fb02002a560 comp rx=0 tx=0).stop 2024-09-10T20:43:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 -- 172.21.15.28:0/1810345448 shutdown_connections 2024-09-10T20:43:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 --2- 172.21.15.28:0/1810345448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb02c0b4d60 0x7fb02c0b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 --2- 172.21.15.28:0/1810345448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b3110 0x7fb02c0cec90 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 --2- 172.21.15.28:0/1810345448 >> v2:172.21.15.159:3300/0 conn(0x7fb02c15f580 0x7fb02c0ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 -- 172.21.15.28:0/1810345448 >> 172.21.15.28:0/1810345448 conn(0x7fb02c0d1f30 msgr2=0x7fb02c0d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 -- 172.21.15.28:0/1810345448 shutdown_connections 2024-09-10T20:43:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.463+0000 7fb034216640 1 -- 172.21.15.28:0/1810345448 wait complete. 2024-09-10T20:43:29.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.464+0000 7fb034216640 1 Processor -- start 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb034216640 1 -- start start 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb034216640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb02c0b3110 0x7fb02c0c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb034216640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b4d60 0x7fb02c0cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb034216640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb02c15f580 0x7fb02c0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb034216640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c0d1030 con 0x7fb02c0b4d60 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb034216640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c0d0eb0 con 0x7fb02c0b3110 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb034216640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb02c0b56f0 con 0x7fb02c15f580 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb0317b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b4d60 0x7fb02c0cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb0317b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b4d60 0x7fb02c0cbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45946/0 (socket says 172.21.15.28:45946) 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb0317b1640 1 -- 172.21.15.28:0/2118964730 learned_addr learned my addr 172.21.15.28:0/2118964730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb031fb2640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb02c0b3110 0x7fb02c0c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.465+0000 7fb0327b3640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb02c15f580 0x7fb02c0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb0317b1640 1 -- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb02c15f580 msgr2=0x7fb02c0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb0317b1640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb02c15f580 0x7fb02c0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb0317b1640 1 -- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb02c0b3110 msgr2=0x7fb02c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb0317b1640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb02c0b3110 0x7fb02c0c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb0317b1640 1 -- 172.21.15.28:0/2118964730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02c06dcc0 con 0x7fb02c0b4d60 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb0317b1640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b4d60 0x7fb02c0cbb90 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb0200099c0 tx=0x7fb0200039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb02a7fc640 1 -- 172.21.15.28:0/2118964730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02003f070 con 0x7fb02c0b4d60 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb02a7fc640 1 -- 172.21.15.28:0/2118964730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb020004890 con 0x7fb02c0b4d60 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02c06dfc0 con 0x7fb02c0b4d60 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb02a7fc640 1 -- 172.21.15.28:0/2118964730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020035030 con 0x7fb02c0b4d60 2024-09-10T20:43:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.466+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb02c06e800 con 0x7fb02c0b4d60 2024-09-10T20:43:29.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.467+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb02c0b5c60 con 0x7fb02c0b4d60 2024-09-10T20:43:29.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.468+0000 7fb02a7fc640 1 -- 172.21.15.28:0/2118964730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7fb020004ce0 con 0x7fb02c0b4d60 2024-09-10T20:43:29.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.469+0000 7fb02a7fc640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0180782d0 0x7fb01807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.469+0000 7fb02a7fc640 1 -- 172.21.15.28:0/2118964730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7fb0200bb660 con 0x7fb02c0b4d60 2024-09-10T20:43:29.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.469+0000 7fb031fb2640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0180782d0 0x7fb01807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.469+0000 7fb031fb2640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0180782d0 0x7fb01807a790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb0240098e0 tx=0x7fb024009060 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:29.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.473+0000 7fb02a7fc640 1 -- 172.21.15.28:0/2118964730 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb02c06dfc0 con 0x7fb02c0b4d60 2024-09-10T20:43:29.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.685+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone92450", "target": ["mon-mgr", ""]}) -- 0x7fb02c06e1a0 con 0x7fb0180782d0 2024-09-10T20:43:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.699+0000 7fb02a7fc640 1 -- 172.21.15.28:0/2118964730 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb02c06e1a0 con 0x7fb0180782d0 2024-09-10T20:43:29.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0180782d0 msgr2=0x7fb01807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0180782d0 0x7fb01807a790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb0240098e0 tx=0x7fb024009060 comp rx=0 tx=0).stop 2024-09-10T20:43:29.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b4d60 msgr2=0x7fb02c0cbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b4d60 0x7fb02c0cbb90 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb0200099c0 tx=0x7fb0200039e0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 shutdown_connections 2024-09-10T20:43:29.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb0180782d0 0x7fb01807a790 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb02c15f580 0x7fb02c0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb02c0b4d60 0x7fb02c0cbb90 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 --2- 172.21.15.28:0/2118964730 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb02c0b3110 0x7fb02c0c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 >> 172.21.15.28:0/2118964730 conn(0x7fb02c0d1f30 msgr2=0x7fb02c0b6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:29.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 shutdown_connections 2024-09-10T20:43:29.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.702+0000 7fb034216640 1 -- 172.21.15.28:0/2118964730 wait complete. 2024-09-10T20:43:29.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone48531 2024-09-10T20:43:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 -- 172.21.15.28:0/2452735538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae80d27f0 msgr2=0x7f1ae80b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/2452735538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae80d27f0 0x7f1ae80b1830 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f1adc005fd0 tx=0x7f1adc02a540 comp rx=0 tx=0).stop 2024-09-10T20:43:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 -- 172.21.15.28:0/2452735538 shutdown_connections 2024-09-10T20:43:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/2452735538 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae8161b50 0x7f1ae80b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/2452735538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae80d27f0 0x7f1ae80b1830 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 -- 172.21.15.28:0/2452735538 >> 172.21.15.28:0/2452735538 conn(0x7f1ae80d4070 msgr2=0x7f1ae80d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 -- 172.21.15.28:0/2452735538 shutdown_connections 2024-09-10T20:43:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.864+0000 7f1aecdbb640 1 -- 172.21.15.28:0/2452735538 wait complete. 2024-09-10T20:43:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 Processor -- start 2024-09-10T20:43:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 -- start start 2024-09-10T20:43:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80d27f0 0x7f1ae8157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae8161770 0x7f1ae80b9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae8161b50 0x7f1ae80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae80b2dd0 con 0x7f1ae8161b50 2024-09-10T20:43:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae80d4e60 con 0x7f1ae8161770 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1aecdbb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae80b2f50 con 0x7f1ae80d27f0 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.866+0000 7f1ae6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae8161b50 0x7f1ae80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae8161b50 0x7f1ae80ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45964/0 (socket says 172.21.15.28:45964) 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 -- 172.21.15.28:0/1967237260 learned_addr learned my addr 172.21.15.28:0/1967237260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae659c640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80d27f0 0x7f1ae8157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae5d9b640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae8161770 0x7f1ae80b9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 -- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80d27f0 msgr2=0x7f1ae8157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80d27f0 0x7f1ae8157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 -- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae8161770 msgr2=0x7f1ae80b9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae8161770 0x7f1ae80b9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 -- 172.21.15.28:0/1967237260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae81f8630 con 0x7f1ae8161b50 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae659c640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80d27f0 0x7f1ae8157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae5d9b640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae8161770 0x7f1ae80b9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ae6d9d640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae8161b50 0x7f1ae80ccdb0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000cad0 tx=0x7f1ae000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ad6ffd640 1 -- 172.21.15.28:0/1967237260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000ae20 con 0x7f1ae8161b50 2024-09-10T20:43:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ad6ffd640 1 -- 172.21.15.28:0/1967237260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ae000f450 con 0x7f1ae8161b50 2024-09-10T20:43:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae80b8bf0 con 0x7f1ae8161b50 2024-09-10T20:43:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1ad6ffd640 1 -- 172.21.15.28:0/1967237260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000f8c0 con 0x7f1ae8161b50 2024-09-10T20:43:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.867+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae80b8ed0 con 0x7f1ae8161b50 2024-09-10T20:43:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.868+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae80d20e0 con 0x7f1ae8161b50 2024-09-10T20:43:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.869+0000 7f1ad6ffd640 1 -- 172.21.15.28:0/1967237260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 100907+0+0 (secure 0 0 0) 0x7f1ae000fa60 con 0x7f1ae8161b50 2024-09-10T20:43:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.870+0000 7f1ad6ffd640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1ac80782d0 0x7f1ac807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.870+0000 7f1ae659c640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1ac80782d0 0x7f1ac807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.870+0000 7f1ad6ffd640 1 -- 172.21.15.28:0/1967237260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f1ae009b070 con 0x7f1ae8161b50 2024-09-10T20:43:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.870+0000 7f1ae659c640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1ac80782d0 0x7f1ac807a790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8001740 tx=0x7f1ad8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:29.874+0000 7f1ad6ffd640 1 -- 172.21.15.28:0/1967237260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1ae0014030 con 0x7f1ae8161b50 2024-09-10T20:43:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.084+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone48531", "target": ["mon-mgr", ""]}) -- 0x7f1ae80d0c90 con 0x7f1ac80782d0 2024-09-10T20:43:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.098+0000 7f1ad6ffd640 1 -- 172.21.15.28:0/1967237260 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1ae80d0c90 con 0x7f1ac80782d0 2024-09-10T20:43:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1ac80782d0 msgr2=0x7f1ac807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1ac80782d0 0x7f1ac807a790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8001740 tx=0x7f1ad8009040 comp rx=0 tx=0).stop 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae8161b50 msgr2=0x7f1ae80ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae8161b50 0x7f1ae80ccdb0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000cad0 tx=0x7f1ae000cf90 comp rx=0 tx=0).stop 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 shutdown_connections 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1ac80782d0 0x7f1ac807a790 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae8161b50 0x7f1ae80ccdb0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1ae8161770 0x7f1ae80b9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 --2- 172.21.15.28:0/1967237260 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1ae80d27f0 0x7f1ae8157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 >> 172.21.15.28:0/1967237260 conn(0x7f1ae80d4070 msgr2=0x7f1ae80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 shutdown_connections 2024-09-10T20:43:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.100+0000 7f1aecdbb640 1 -- 172.21.15.28:0/1967237260 wait complete. 2024-09-10T20:43:30.118 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-10T20:43:30.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:43:30.174 INFO:teuthology.orchestra.run.smithi028.stdout:2 2024-09-10T20:43:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:30.180 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:43:30.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:43:30.222 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:43:30.272 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:43:30.273 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:43:30.273 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:30.273 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:30.289 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:30.289 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:30.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:43:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 -- 172.21.15.28:0/3655132654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead40b4d10 msgr2=0x7fead40ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 --2- 172.21.15.28:0/3655132654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead40b4d10 0x7fead40ce030 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7feacc005fd0 tx=0x7feacc02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 -- 172.21.15.28:0/3655132654 shutdown_connections 2024-09-10T20:43:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 --2- 172.21.15.28:0/3655132654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fead415df70 0x7fead40b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 --2- 172.21.15.28:0/3655132654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead40b4d10 0x7fead40ce030 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 -- 172.21.15.28:0/3655132654 >> 172.21.15.28:0/3655132654 conn(0x7fead406cf00 msgr2=0x7fead406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 -- 172.21.15.28:0/3655132654 shutdown_connections 2024-09-10T20:43:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.489+0000 7fead9ee6640 1 -- 172.21.15.28:0/3655132654 wait complete. 2024-09-10T20:43:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.490+0000 7fead9ee6640 1 Processor -- start 2024-09-10T20:43:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.491+0000 7fead9ee6640 1 -- start start 2024-09-10T20:43:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.491+0000 7fead9ee6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead40b4d10 0x7fead414b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.491+0000 7fead9ee6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead415a190 0x7fead4152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.491+0000 7fead9ee6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fead415df70 0x7fead415cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.491+0000 7fead9ee6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fead40d0270 con 0x7fead40b4d10 2024-09-10T20:43:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.491+0000 7fead9ee6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fead40d00f0 con 0x7fead415a190 2024-09-10T20:43:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.491+0000 7fead9ee6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fead40d03f0 con 0x7fead415df70 2024-09-10T20:43:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead40b4d10 0x7fead414b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead40b4d10 0x7fead414b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53056/0 (socket says 172.21.15.28:53056) 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 -- 172.21.15.28:0/815181279 learned_addr learned my addr 172.21.15.28:0/815181279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead2ffd640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead415a190 0x7fead4152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead3fff640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fead415df70 0x7fead415cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 -- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fead415df70 msgr2=0x7fead415cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fead415df70 0x7fead415cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 -- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead415a190 msgr2=0x7fead4152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead415a190 0x7fead4152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 -- 172.21.15.28:0/815181279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead406e440 con 0x7fead40b4d10 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead37fe640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead40b4d10 0x7fead414b8b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7feac400c4c0 tx=0x7feac400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7feabbfff640 1 -- 172.21.15.28:0/815181279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4016020 con 0x7fead40b4d10 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7feabbfff640 1 -- 172.21.15.28:0/815181279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feac4012940 con 0x7fead40b4d10 2024-09-10T20:43:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead406e720 con 0x7fead40b4d10 2024-09-10T20:43:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7feabbfff640 1 -- 172.21.15.28:0/815181279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4012d90 con 0x7fead40b4d10 2024-09-10T20:43:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.492+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead4152760 con 0x7fead40b4d10 2024-09-10T20:43:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.493+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead40ce030 con 0x7fead40b4d10 2024-09-10T20:43:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.494+0000 7feabbfff640 1 -- 172.21.15.28:0/815181279 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7feac401b400 con 0x7fead40b4d10 2024-09-10T20:43:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.494+0000 7feabbfff640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feac00783f0 0x7feac007a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.495+0000 7fead2ffd640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feac00783f0 0x7feac007a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.495+0000 7feabbfff640 1 -- 172.21.15.28:0/815181279 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7feac409a930 con 0x7fead40b4d10 2024-09-10T20:43:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.495+0000 7fead2ffd640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feac00783f0 0x7feac007a8b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feac80095f0 tx=0x7feac8005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.499+0000 7feabbfff640 1 -- 172.21.15.28:0/815181279 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feac4020020 con 0x7fead40b4d10 2024-09-10T20:43:30.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.696+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fead41546d0 con 0x7fead40b4d10 2024-09-10T20:43:30.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.696+0000 7feabbfff640 1 -- 172.21.15.28:0/815181279 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v102) ==== 80+0+0 (secure 0 0 0) 0x7feac4062510 con 0x7fead40b4d10 2024-09-10T20:43:30.702 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:43:30.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.698+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feac00783f0 msgr2=0x7feac007a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.698+0000 7fead9ee6640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feac00783f0 0x7feac007a8b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feac80095f0 tx=0x7feac8005f90 comp rx=0 tx=0).stop 2024-09-10T20:43:30.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.698+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead40b4d10 msgr2=0x7fead414b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.698+0000 7fead9ee6640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead40b4d10 0x7fead414b8b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7feac400c4c0 tx=0x7feac400c980 comp rx=0 tx=0).stop 2024-09-10T20:43:30.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 shutdown_connections 2024-09-10T20:43:30.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feac00783f0 0x7feac007a8b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fead415df70 0x7fead415cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fead415a190 0x7fead4152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 --2- 172.21.15.28:0/815181279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead40b4d10 0x7fead414b8b0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 >> 172.21.15.28:0/815181279 conn(0x7fead406cf00 msgr2=0x7fead40aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:30.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 shutdown_connections 2024-09-10T20:43:30.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.699+0000 7fead9ee6640 1 -- 172.21.15.28:0/815181279 wait complete. 2024-09-10T20:43:30.714 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:43:30.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:30.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:30.756 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:43:30.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:30.757 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:43:30.813 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:43:30.813 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:43:30.813 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:30.831 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:43:30.831 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:30.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:43:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.990+0000 7f4798a16640 1 -- 172.21.15.28:0/2696075675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 msgr2=0x7f477c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.990+0000 7f4798a16640 1 --2- 172.21.15.28:0/2696075675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f477c00b9c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f478c001dc0 tx=0x7f477c005830 comp rx=0 tx=0).stop 2024-09-10T20:43:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.990+0000 7f4798a16640 1 -- 172.21.15.28:0/2696075675 shutdown_connections 2024-09-10T20:43:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.990+0000 7f4798a16640 1 --2- 172.21.15.28:0/2696075675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f477c00b9c0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.990+0000 7f4798a16640 1 -- 172.21.15.28:0/2696075675 >> 172.21.15.28:0/2696075675 conn(0x7f479406cf00 msgr2=0x7f479406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.990+0000 7f4798a16640 1 -- 172.21.15.28:0/2696075675 shutdown_connections 2024-09-10T20:43:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.990+0000 7f4798a16640 1 -- 172.21.15.28:0/2696075675 wait complete. 2024-09-10T20:43:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.991+0000 7f4798a16640 1 Processor -- start 2024-09-10T20:43:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.991+0000 7f4798a16640 1 -- start start 2024-09-10T20:43:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4798a16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f47940b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4798a16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47940afa40 0x7f47940ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4798a16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47940ad600 0x7f47941f8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4798a16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47940d0210 con 0x7f477c005290 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4798a16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f47940d0090 con 0x7f47940ad600 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4798a16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f47940d0390 con 0x7f47940afa40 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f47940b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f47940b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53076/0 (socket says 172.21.15.28:53076) 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 -- 172.21.15.28:0/3777226485 learned_addr learned my addr 172.21.15.28:0/3777226485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4791d9b640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47940afa40 0x7f47940ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4792d9d640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47940ad600 0x7f47941f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 -- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47940afa40 msgr2=0x7f47940ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47940afa40 0x7f47940ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 -- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47940ad600 msgr2=0x7f47941f8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47940ad600 0x7f47941f8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 -- 172.21.15.28:0/3777226485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4794148480 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f4791d9b640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47940afa40 0x7f47940ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.992+0000 7f479259c640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f47940b11a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f478400e640 tx=0x7f478400eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.993+0000 7f4782ffd640 1 -- 172.21.15.28:0/3777226485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784015070 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.993+0000 7f4782ffd640 1 -- 172.21.15.28:0/3777226485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f478400c9a0 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.993+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4794148760 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.993+0000 7f4782ffd640 1 -- 172.21.15.28:0/3777226485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478400cdd0 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.993+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4794148d20 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.994+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47940ada10 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.995+0000 7f4782ffd640 1 -- 172.21.15.28:0/3777226485 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f4784011460 con 0x7f477c005290 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.995+0000 7f4782ffd640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4774078420 0x7f477407a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.995+0000 7f4791d9b640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4774078420 0x7f477407a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.995+0000 7f4782ffd640 1 -- 172.21.15.28:0/3777226485 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f478409ace0 con 0x7f477c005290 2024-09-10T20:43:31.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.996+0000 7f4791d9b640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4774078420 0x7f477407a8e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4788005fd0 tx=0x7f478800eea0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:30.999+0000 7f4782ffd640 1 -- 172.21.15.28:0/3777226485 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f47940ada10 con 0x7f477c005290 2024-09-10T20:43:31.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.198+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f47940b4630 con 0x7f477c005290 2024-09-10T20:43:31.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.198+0000 7f4782ffd640 1 -- 172.21.15.28:0/3777226485 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v102) ==== 80+0+0 (secure 0 0 0) 0x7f47940d0a00 con 0x7f477c005290 2024-09-10T20:43:31.203 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:43:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4774078420 msgr2=0x7f477407a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4774078420 0x7f477407a8e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4788005fd0 tx=0x7f478800eea0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 msgr2=0x7f47940b11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f47940b11a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f478400e640 tx=0x7f478400eb00 comp rx=0 tx=0).stop 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 shutdown_connections 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4774078420 0x7f477407a8e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47940ad600 0x7f47941f8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47940afa40 0x7f47940ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 --2- 172.21.15.28:0/3777226485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f477c005290 0x7f47940b11a0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 >> 172.21.15.28:0/3777226485 conn(0x7f479406cf00 msgr2=0x7f479415ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.200+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 shutdown_connections 2024-09-10T20:43:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.201+0000 7f4798a16640 1 -- 172.21.15.28:0/3777226485 wait complete. 2024-09-10T20:43:31.216 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:43:31.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:31.216 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:43:31.247 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:43:31.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:31.247 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:43:31.303 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:43:31.303 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:31.304 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:31.320 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:43:31.320 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:43:31.320 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:43:31.320 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:31.361 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:43:31.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 -- 172.21.15.28:0/3739619687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 msgr2=0x7f780c0caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 --2- 172.21.15.28:0/3739619687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c0caa70 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f7800005fd0 tx=0x7f780002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 -- 172.21.15.28:0/3739619687 shutdown_connections 2024-09-10T20:43:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 --2- 172.21.15.28:0/3739619687 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f780c06d1f0 0x7f780c06ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 --2- 172.21.15.28:0/3739619687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c0caa70 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 -- 172.21.15.28:0/3739619687 >> 172.21.15.28:0/3739619687 conn(0x7f780c0d0420 msgr2=0x7f780c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 -- 172.21.15.28:0/3739619687 shutdown_connections 2024-09-10T20:43:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.462+0000 7f7812cf6640 1 -- 172.21.15.28:0/3739619687 wait complete. 2024-09-10T20:43:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.463+0000 7f7812cf6640 1 Processor -- start 2024-09-10T20:43:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.463+0000 7f7812cf6640 1 -- start start 2024-09-10T20:43:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7812cf6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f780c06d1f0 0x7f780c154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7812cf6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7812cf6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f780c156510 0x7f780c0c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7812cf6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f780c073400 con 0x7f780c156130 2024-09-10T20:43:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7812cf6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f780c0afa60 con 0x7f780c06d1f0 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7812cf6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f780c073580 con 0x7f780c156510 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c144ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53090/0 (socket says 172.21.15.28:53090) 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 -- 172.21.15.28:0/195152917 learned_addr learned my addr 172.21.15.28:0/195152917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7810a92640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f780c06d1f0 0x7f780c154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f7811293640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f780c156510 0x7f780c0c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 -- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f780c156510 msgr2=0x7f780c0c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f780c156510 0x7f780c0c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 -- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f780c06d1f0 msgr2=0x7f780c154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f780c06d1f0 0x7f780c154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.464+0000 7f780bfff640 1 -- 172.21.15.28:0/195152917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c156a10 con 0x7f780c156130 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.465+0000 7f780bfff640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c144ad0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f77f800c370 tx=0x7f77f800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.465+0000 7f78097fa640 1 -- 172.21.15.28:0/195152917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8017070 con 0x7f780c156130 2024-09-10T20:43:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.465+0000 7f78097fa640 1 -- 172.21.15.28:0/195152917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77f80053c0 con 0x7f780c156130 2024-09-10T20:43:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.465+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780c156cf0 con 0x7f780c156130 2024-09-10T20:43:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.465+0000 7f78097fa640 1 -- 172.21.15.28:0/195152917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8003850 con 0x7f780c156130 2024-09-10T20:43:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.465+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f780c145590 con 0x7f780c156130 2024-09-10T20:43:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.466+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f780c06ef30 con 0x7f780c156130 2024-09-10T20:43:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.467+0000 7f78097fa640 1 -- 172.21.15.28:0/195152917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f77f80039f0 con 0x7f780c156130 2024-09-10T20:43:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.467+0000 7f78097fa640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f77f0078420 0x7f77f007a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.467+0000 7f78097fa640 1 -- 172.21.15.28:0/195152917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f77f8099c00 con 0x7f780c156130 2024-09-10T20:43:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.467+0000 7f7810a92640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f77f0078420 0x7f77f007a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.468+0000 7f7810a92640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f77f0078420 0x7f77f007a8e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7800033000 tx=0x7f7800004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.471+0000 7f78097fa640 1 -- 172.21.15.28:0/195152917 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f77f8061930 con 0x7f780c156130 2024-09-10T20:43:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.696+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f780c145760 con 0x7f780c156130 2024-09-10T20:43:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.697+0000 7f78097fa640 1 -- 172.21.15.28:0/195152917 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5231 (secure 0 0 0) 0x7f77f8066360 con 0x7f780c156130 2024-09-10T20:43:31.701 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:31.701 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"btime":"2024-09-10T20:43:06:048867+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4277647396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4277647396},{"type":"v1","addr":"172.21.15.28:6834","nonce":4277647396}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/4032655811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":4032655811},{"type":"v1","addr":"172.21.15.159:6835","nonce":4032655811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/3530092732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3530092732},{"type":"v1","addr":"172.21.15.159:6834","nonce":3530092732}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:56.370954+0000","modified":"2024-09-10T20:42:59.400466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1426218269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1426218269},{"type":"v1","addr":"172.21.15.28:6835","nonce":1426218269}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6936,"qdb_cluster":[6936]},"id":8}]} 2024-09-10T20:43:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-09-10T20:43:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f77f0078420 msgr2=0x7f77f007a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f77f0078420 0x7f77f007a8e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7800033000 tx=0x7f7800004730 comp rx=0 tx=0).stop 2024-09-10T20:43:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 msgr2=0x7f780c144ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c144ad0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f77f800c370 tx=0x7f77f800c830 comp rx=0 tx=0).stop 2024-09-10T20:43:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 shutdown_connections 2024-09-10T20:43:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f77f0078420 0x7f77f007a8e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f780c156510 0x7f780c0c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780c156130 0x7f780c144ad0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 --2- 172.21.15.28:0/195152917 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f780c06d1f0 0x7f780c154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 >> 172.21.15.28:0/195152917 conn(0x7f780c0d0420 msgr2=0x7f780c15e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 shutdown_connections 2024-09-10T20:43:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.699+0000 7f7812cf6640 1 -- 172.21.15.28:0/195152917 wait complete. 2024-09-10T20:43:31.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 -- 172.21.15.28:0/349405671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 msgr2=0x7f0d080d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 --2- 172.21.15.28:0/349405671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080d0090 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc005fd0 tx=0x7f0cfc02a620 comp rx=0 tx=0).stop 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 -- 172.21.15.28:0/349405671 shutdown_connections 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 --2- 172.21.15.28:0/349405671 >> v2:172.21.15.159:3300/0 conn(0x7f0d080d05d0 0x7f0d080b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 --2- 172.21.15.28:0/349405671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080d0090 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 -- 172.21.15.28:0/349405671 >> 172.21.15.28:0/349405671 conn(0x7f0d0806cf00 msgr2=0x7f0d0806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 -- 172.21.15.28:0/349405671 shutdown_connections 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.814+0000 7f0d0de23640 1 -- 172.21.15.28:0/349405671 wait complete. 2024-09-10T20:43:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.816+0000 7f0d0de23640 1 Processor -- start 2024-09-10T20:43:31.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.816+0000 7f0d0de23640 1 -- start start 2024-09-10T20:43:31.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.816+0000 7f0d0de23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.816+0000 7f0d0de23640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d080cd3d0 0x7f0d081f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.816+0000 7f0d0de23640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d080d05d0 0x7f0d080afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.816+0000 7f0d0de23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d080b33c0 con 0x7f0d080cc570 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d0de23640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0d080cd7b0 con 0x7f0d080d05d0 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d0de23640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0d080b3540 con 0x7f0d080cd3d0 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53102/0 (socket says 172.21.15.28:53102) 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 -- 172.21.15.28:0/3891839262 learned_addr learned my addr 172.21.15.28:0/3891839262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d07fff640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d080d05d0 0x7f0d080afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d06ffd640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d080cd3d0 0x7f0d081f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 -- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d080cd3d0 msgr2=0x7f0d081f6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d080cd3d0 0x7f0d081f6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 -- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d080d05d0 msgr2=0x7f0d080afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d080d05d0 0x7f0d080afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 -- 172.21.15.28:0/3891839262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d081f3580 con 0x7f0d080cc570 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d077fe640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080cb710 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c450 tx=0x7f0cf800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0ceffff640 1 -- 172.21.15.28:0/3891839262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf80052d0 con 0x7f0d080cc570 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0ceffff640 1 -- 172.21.15.28:0/3891839262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cf8005470 con 0x7f0d080cc570 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d081f2ce0 con 0x7f0d080cc570 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0ceffff640 1 -- 172.21.15.28:0/3891839262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8012870 con 0x7f0d080cc570 2024-09-10T20:43:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.817+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d081f32a0 con 0x7f0d080cc570 2024-09-10T20:43:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.818+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d080b3a40 con 0x7f0d080cc570 2024-09-10T20:43:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.819+0000 7f0ceffff640 1 -- 172.21.15.28:0/3891839262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f0cf8012a10 con 0x7f0d080cc570 2024-09-10T20:43:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.819+0000 7f0ceffff640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cf40783f0 0x7f0cf407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.820+0000 7f0ceffff640 1 -- 172.21.15.28:0/3891839262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7f0cf809a110 con 0x7f0d080cc570 2024-09-10T20:43:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.820+0000 7f0d06ffd640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cf40783f0 0x7f0cf407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.820+0000 7f0d06ffd640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cf40783f0 0x7f0cf407a8b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc033000 tx=0x7f0cfc004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:31.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:31.824+0000 7f0ceffff640 1 -- 172.21.15.28:0/3891839262 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0d080d0a00 con 0x7f0d080cc570 2024-09-10T20:43:31.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:43:32.000 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:43:32.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.050+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d080b3090 con 0x7f0d080cc570 2024-09-10T20:43:32.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.051+0000 7f0ceffff640 1 -- 172.21.15.28:0/3891839262 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5231 (secure 0 0 0) 0x7f0cf8062530 con 0x7f0d080cc570 2024-09-10T20:43:32.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:32.055 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"btime":"2024-09-10T20:43:06:048867+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4277647396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4277647396},{"type":"v1","addr":"172.21.15.28:6834","nonce":4277647396}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/4032655811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":4032655811},{"type":"v1","addr":"172.21.15.159:6835","nonce":4032655811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/3530092732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3530092732},{"type":"v1","addr":"172.21.15.159:6834","nonce":3530092732}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:56.370954+0000","modified":"2024-09-10T20:42:59.400466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1426218269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1426218269},{"type":"v1","addr":"172.21.15.28:6835","nonce":1426218269}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6936,"qdb_cluster":[6936]},"id":8}]} 2024-09-10T20:43:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cf40783f0 msgr2=0x7f0cf407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cf40783f0 0x7f0cf407a8b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc033000 tx=0x7f0cfc004730 comp rx=0 tx=0).stop 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 msgr2=0x7f0d080cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080cb710 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c450 tx=0x7f0cf800c910 comp rx=0 tx=0).stop 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 shutdown_connections 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cf40783f0 0x7f0cf407a8b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d080d05d0 0x7f0d080afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d080cd3d0 0x7f0d081f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 --2- 172.21.15.28:0/3891839262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d080cc570 0x7f0d080cb710 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 >> 172.21.15.28:0/3891839262 conn(0x7f0d0806cf00 msgr2=0x7f0d08070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 shutdown_connections 2024-09-10T20:43:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.053+0000 7f0d0de23640 1 -- 172.21.15.28:0/3891839262 wait complete. 2024-09-10T20:43:32.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:43:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.209+0000 7fed505e2640 1 -- 172.21.15.28:0/2540366894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed340051c0 msgr2=0x7fed3400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.209+0000 7fed505e2640 1 --2- 172.21.15.28:0/2540366894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed340051c0 0x7fed3400b9c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fed4000c610 tx=0x7fed34005830 comp rx=0 tx=0).stop 2024-09-10T20:43:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.210+0000 7fed505e2640 1 -- 172.21.15.28:0/2540366894 shutdown_connections 2024-09-10T20:43:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.210+0000 7fed505e2640 1 --2- 172.21.15.28:0/2540366894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed340051c0 0x7fed3400b9c0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.210+0000 7fed505e2640 1 -- 172.21.15.28:0/2540366894 >> 172.21.15.28:0/2540366894 conn(0x7fed4806cf00 msgr2=0x7fed4806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.210+0000 7fed505e2640 1 -- 172.21.15.28:0/2540366894 shutdown_connections 2024-09-10T20:43:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.210+0000 7fed505e2640 1 -- 172.21.15.28:0/2540366894 wait complete. 2024-09-10T20:43:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.211+0000 7fed505e2640 1 Processor -- start 2024-09-10T20:43:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.211+0000 7fed505e2640 1 -- start start 2024-09-10T20:43:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed340051c0 0x7fed481503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed48157aa0 0x7fed48157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed481f46b0 0x7fed481f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed480b4780 con 0x7fed48157aa0 2024-09-10T20:43:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fed480b4600 con 0x7fed340051c0 2024-09-10T20:43:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fed480b4900 con 0x7fed481f46b0 2024-09-10T20:43:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed48157aa0 0x7fed48157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed48157aa0 0x7fed48157f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53132/0 (socket says 172.21.15.28:53132) 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 -- 172.21.15.28:0/2507508983 learned_addr learned my addr 172.21.15.28:0/2507508983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4e37e640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed340051c0 0x7fed481503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4eb7f640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed481f46b0 0x7fed481f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 -- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed481f46b0 msgr2=0x7fed481f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed481f46b0 0x7fed481f4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 -- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed340051c0 msgr2=0x7fed481503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed340051c0 0x7fed481503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 -- 172.21.15.28:0/2507508983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed48152470 con 0x7fed48157aa0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed4db7d640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed48157aa0 0x7fed48157f20 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fed3800eaa0 tx=0x7fed3800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed3effd640 1 -- 172.21.15.28:0/2507508983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3800dd20 con 0x7fed48157aa0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed3effd640 1 -- 172.21.15.28:0/2507508983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed38004050 con 0x7fed48157aa0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed48152750 con 0x7fed48157aa0 2024-09-10T20:43:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed3effd640 1 -- 172.21.15.28:0/2507508983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380044a0 con 0x7fed48157aa0 2024-09-10T20:43:32.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.212+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed48152100 con 0x7fed48157aa0 2024-09-10T20:43:32.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.213+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed480b3c20 con 0x7fed48157aa0 2024-09-10T20:43:32.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.214+0000 7fed3effd640 1 -- 172.21.15.28:0/2507508983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fed3800b040 con 0x7fed48157aa0 2024-09-10T20:43:32.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.215+0000 7fed3effd640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed2c078420 0x7fed2c07a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.215+0000 7fed3effd640 1 -- 172.21.15.28:0/2507508983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5767+0+0 (secure 0 0 0) 0x7fed3809af60 con 0x7fed48157aa0 2024-09-10T20:43:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.215+0000 7fed4e37e640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed2c078420 0x7fed2c07a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.215+0000 7fed4e37e640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed2c078420 0x7fed2c07a8e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed44006e40 tx=0x7fed44006090 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.219+0000 7fed3effd640 1 -- 172.21.15.28:0/2507508983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fed480b3c20 con 0x7fed48157aa0 2024-09-10T20:43:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.410+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed480d0260 con 0x7fed48157aa0 2024-09-10T20:43:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.411+0000 7fed3effd640 1 -- 172.21.15.28:0/2507508983 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16932 (secure 0 0 0) 0x7fed38091e20 con 0x7fed48157aa0 2024-09-10T20:43:32.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:32.415 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":103,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:43:32.355983+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:54.016199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:55.330837+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.4","mappings":[{"from":4,"to":3}]},{"pgid":"17.21","mappings":[{"from":6,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:43:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed2c078420 msgr2=0x7fed2c07a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed2c078420 0x7fed2c07a8e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed44006e40 tx=0x7fed44006090 comp rx=0 tx=0).stop 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed48157aa0 msgr2=0x7fed48157f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed48157aa0 0x7fed48157f20 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fed3800eaa0 tx=0x7fed3800ef60 comp rx=0 tx=0).stop 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 shutdown_connections 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed2c078420 0x7fed2c07a8e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed481f46b0 0x7fed481f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed48157aa0 0x7fed48157f20 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 --2- 172.21.15.28:0/2507508983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed340051c0 0x7fed481503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.413+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 >> 172.21.15.28:0/2507508983 conn(0x7fed4806cf00 msgr2=0x7fed48070450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.414+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 shutdown_connections 2024-09-10T20:43:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.414+0000 7fed505e2640 1 -- 172.21.15.28:0/2507508983 wait complete. 2024-09-10T20:43:32.428 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:43:32.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 -- 172.21.15.28:0/2348283239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 msgr2=0x7f48bc0d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/2348283239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc0d2960 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f48ac005fd0 tx=0x7f48ac02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 -- 172.21.15.28:0/2348283239 shutdown_connections 2024-09-10T20:43:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/2348283239 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc0b6620 0x7f48bc0d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/2348283239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc0d2960 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 -- 172.21.15.28:0/2348283239 >> 172.21.15.28:0/2348283239 conn(0x7f48bc0d1e40 msgr2=0x7f48bc0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 -- 172.21.15.28:0/2348283239 shutdown_connections 2024-09-10T20:43:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.532+0000 7f48c0cb8640 1 -- 172.21.15.28:0/2348283239 wait complete. 2024-09-10T20:43:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.533+0000 7f48c0cb8640 1 Processor -- start 2024-09-10T20:43:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.533+0000 7f48c0cb8640 1 -- start start 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48c0cb8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0b6620 0x7f48bc152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48c0cb8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc0cfec0 0x7f48bc157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48c0cb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48c0cb8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc0d0cc0 con 0x7f48bc15bf30 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48c0cb8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc0d0b40 con 0x7f48bc0b6620 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48c0cb8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f48bc0ced00 con 0x7f48bc0cfec0 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc153d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53150/0 (socket says 172.21.15.28:53150) 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.534+0000 7f48bad9d640 1 -- 172.21.15.28:0/1785239046 learned_addr learned my addr 172.21.15.28:0/1785239046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48b9d9b640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc0cfec0 0x7f48bc157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48ba59c640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0b6620 0x7f48bc152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48bad9d640 1 -- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc0cfec0 msgr2=0x7f48bc157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48bad9d640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc0cfec0 0x7f48bc157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48bad9d640 1 -- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0b6620 msgr2=0x7f48bc152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48bad9d640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0b6620 0x7f48bc152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48bad9d640 1 -- 172.21.15.28:0/1785239046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc156210 con 0x7f48bc15bf30 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48ba59c640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0b6620 0x7f48bc152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48b9d9b640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc0cfec0 0x7f48bc157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48bad9d640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc153d80 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f48b400cd10 tx=0x7f48b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48aaffd640 1 -- 172.21.15.28:0/1785239046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b400f450 con 0x7f48bc15bf30 2024-09-10T20:43:32.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48aaffd640 1 -- 172.21.15.28:0/1785239046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48b400f6e0 con 0x7f48bc15bf30 2024-09-10T20:43:32.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48bc0c94f0 con 0x7f48bc15bf30 2024-09-10T20:43:32.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48aaffd640 1 -- 172.21.15.28:0/1785239046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b400fb30 con 0x7f48bc15bf30 2024-09-10T20:43:32.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.535+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48bc0c9ab0 con 0x7f48bc15bf30 2024-09-10T20:43:32.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.536+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48bc0cf0d0 con 0x7f48bc15bf30 2024-09-10T20:43:32.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.537+0000 7f48aaffd640 1 -- 172.21.15.28:0/1785239046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f48b400fcd0 con 0x7f48bc15bf30 2024-09-10T20:43:32.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.537+0000 7f48aaffd640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f489c0783f0 0x7f489c07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.537+0000 7f48ba59c640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f489c0783f0 0x7f489c07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.537+0000 7f48aaffd640 1 -- 172.21.15.28:0/1785239046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5823+0+0 (secure 0 0 0) 0x7f48b409b230 con 0x7f48bc15bf30 2024-09-10T20:43:32.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.538+0000 7f48ba59c640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f489c0783f0 0x7f489c07a8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f48ac033000 tx=0x7f48ac004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.542+0000 7f48aaffd640 1 -- 172.21.15.28:0/1785239046 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f48b4014030 con 0x7f48bc15bf30 2024-09-10T20:43:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.768+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48bc0b7cc0 con 0x7f48bc15bf30 2024-09-10T20:43:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.769+0000 7f48aaffd640 1 -- 172.21.15.28:0/1785239046 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5231 (secure 0 0 0) 0x7f48b4063580 con 0x7f48bc15bf30 2024-09-10T20:43:32.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:32.773 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"btime":"2024-09-10T20:43:06:048867+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4277647396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4277647396},{"type":"v1","addr":"172.21.15.28:6834","nonce":4277647396}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/4032655811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":4032655811},{"type":"v1","addr":"172.21.15.159:6835","nonce":4032655811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/3530092732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3530092732},{"type":"v1","addr":"172.21.15.159:6834","nonce":3530092732}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:56.370954+0000","modified":"2024-09-10T20:42:59.400466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1426218269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1426218269},{"type":"v1","addr":"172.21.15.28:6835","nonce":1426218269}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6936,"qdb_cluster":[6936]},"id":8}]} 2024-09-10T20:43:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f489c0783f0 msgr2=0x7f489c07a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f489c0783f0 0x7f489c07a8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f48ac033000 tx=0x7f48ac004730 comp rx=0 tx=0).stop 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 msgr2=0x7f48bc153d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc153d80 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f48b400cd10 tx=0x7f48b400a620 comp rx=0 tx=0).stop 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 shutdown_connections 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f489c0783f0 0x7f489c07a8b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48bc15bf30 0x7f48bc153d80 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f48bc0cfec0 0x7f48bc157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 --2- 172.21.15.28:0/1785239046 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f48bc0b6620 0x7f48bc152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 >> 172.21.15.28:0/1785239046 conn(0x7f48bc0d1e40 msgr2=0x7f48bc071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 shutdown_connections 2024-09-10T20:43:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.771+0000 7f48c0cb8640 1 -- 172.21.15.28:0/1785239046 wait complete. 2024-09-10T20:43:32.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 -- 172.21.15.28:0/2447272557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6384155100 msgr2=0x7f638415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 --2- 172.21.15.28:0/2447272557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6384155100 0x7f638415bd50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6380009830 tx=0x7f638002a510 comp rx=0 tx=0).stop 2024-09-10T20:43:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 -- 172.21.15.28:0/2447272557 shutdown_connections 2024-09-10T20:43:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 --2- 172.21.15.28:0/2447272557 >> v2:172.21.15.159:3300/0 conn(0x7f63840d0140 0x7f63840d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 --2- 172.21.15.28:0/2447272557 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6384155100 0x7f638415bd50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 -- 172.21.15.28:0/2447272557 >> 172.21.15.28:0/2447272557 conn(0x7f638406cf00 msgr2=0x7f638406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 -- 172.21.15.28:0/2447272557 shutdown_connections 2024-09-10T20:43:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.887+0000 7f638c023640 1 -- 172.21.15.28:0/2447272557 wait complete. 2024-09-10T20:43:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.888+0000 7f638c023640 1 Processor -- start 2024-09-10T20:43:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.888+0000 7f638c023640 1 -- start start 2024-09-10T20:43:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f638c023640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840d0140 0x7f6384157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f638c023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6384155100 0x7f63841f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f638c023640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63841554e0 0x7f638409ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f638c023640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6384072f60 con 0x7f6384155100 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f638c023640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6384072de0 con 0x7f63841554e0 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f638c023640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f63840730e0 con 0x7f63840d0140 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6384155100 0x7f63841f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6384155100 0x7f63841f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53154/0 (socket says 172.21.15.28:53154) 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 -- 172.21.15.28:0/3560817314 learned_addr learned my addr 172.21.15.28:0/3560817314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f638a5c0640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63841554e0 0x7f638409ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f6389dbf640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840d0140 0x7f6384157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 -- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840d0140 msgr2=0x7f6384157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840d0140 0x7f6384157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 -- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63841554e0 msgr2=0x7f638409ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63841554e0 0x7f638409ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.889+0000 7f63895be640 1 -- 172.21.15.28:0/3560817314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638409de80 con 0x7f6384155100 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.890+0000 7f63895be640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6384155100 0x7f63841f26a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f637400c500 tx=0x7f637400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.890+0000 7f637a7fc640 1 -- 172.21.15.28:0/3560817314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63740052d0 con 0x7f6384155100 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.890+0000 7f637a7fc640 1 -- 172.21.15.28:0/3560817314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6374005470 con 0x7f6384155100 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.890+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638409c110 con 0x7f6384155100 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.890+0000 7f637a7fc640 1 -- 172.21.15.28:0/3560817314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374012870 con 0x7f6384155100 2024-09-10T20:43:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.890+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f638409c6d0 con 0x7f6384155100 2024-09-10T20:43:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.891+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6384073600 con 0x7f6384155100 2024-09-10T20:43:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.892+0000 7f637a7fc640 1 -- 172.21.15.28:0/3560817314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f63740035f0 con 0x7f6384155100 2024-09-10T20:43:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.892+0000 7f637a7fc640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63680783f0 0x7f636807a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.892+0000 7f637a7fc640 1 -- 172.21.15.28:0/3560817314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5823+0+0 (secure 0 0 0) 0x7f637409a000 con 0x7f6384155100 2024-09-10T20:43:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.892+0000 7f6389dbf640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63680783f0 0x7f636807a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.893+0000 7f6389dbf640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63680783f0 0x7f636807a8b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6380034000 tx=0x7f6380004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:32.896+0000 7f637a7fc640 1 -- 172.21.15.28:0/3560817314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f638415c290 con 0x7f6384155100 2024-09-10T20:43:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.119+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f638406ad20 con 0x7f6384155100 2024-09-10T20:43:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.119+0000 7f637a7fc640 1 -- 172.21.15.28:0/3560817314 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5231 (secure 0 0 0) 0x7f6374061db0 con 0x7f6384155100 2024-09-10T20:43:33.123 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:33.124 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"btime":"2024-09-10T20:43:06:048867+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4277647396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4277647396},{"type":"v1","addr":"172.21.15.28:6834","nonce":4277647396}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/4032655811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":4032655811},{"type":"v1","addr":"172.21.15.159:6835","nonce":4032655811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/3530092732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3530092732},{"type":"v1","addr":"172.21.15.159:6834","nonce":3530092732}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:42:56.370954+0000","modified":"2024-09-10T20:42:59.400466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1426218269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1426218269},{"type":"v1","addr":"172.21.15.28:6835","nonce":1426218269}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6936,"qdb_cluster":[6936]},"id":8}]} 2024-09-10T20:43:33.124 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63680783f0 msgr2=0x7f636807a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63680783f0 0x7f636807a8b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6380034000 tx=0x7f6380004730 comp rx=0 tx=0).stop 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6384155100 msgr2=0x7f63841f26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6384155100 0x7f63841f26a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f637400c500 tx=0x7f637400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 shutdown_connections 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63680783f0 0x7f636807a8b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63841554e0 0x7f638409ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6384155100 0x7f63841f26a0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 --2- 172.21.15.28:0/3560817314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63840d0140 0x7f6384157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 >> 172.21.15.28:0/3560817314 conn(0x7f638406cf00 msgr2=0x7f638415e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 shutdown_connections 2024-09-10T20:43:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.122+0000 7f638c023640 1 -- 172.21.15.28:0/3560817314 wait complete. 2024-09-10T20:43:33.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 -- 172.21.15.28:0/2712252656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2015bf30 msgr2=0x7efe200d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 --2- 172.21.15.28:0/2712252656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2015bf30 0x7efe200d2960 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7efe200d1a20 tx=0x7efe1002a560 comp rx=0 tx=0).stop 2024-09-10T20:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 -- 172.21.15.28:0/2712252656 shutdown_connections 2024-09-10T20:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 --2- 172.21.15.28:0/2712252656 >> v2:172.21.15.159:3300/0 conn(0x7efe200cfec0 0x7efe200d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 --2- 172.21.15.28:0/2712252656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2015bf30 0x7efe200d2960 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 -- 172.21.15.28:0/2712252656 >> 172.21.15.28:0/2712252656 conn(0x7efe200d1e40 msgr2=0x7efe200d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 -- 172.21.15.28:0/2712252656 shutdown_connections 2024-09-10T20:43:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.237+0000 7efe267ad640 1 -- 172.21.15.28:0/2712252656 wait complete. 2024-09-10T20:43:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.238+0000 7efe267ad640 1 Processor -- start 2024-09-10T20:43:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.238+0000 7efe267ad640 1 -- start start 2024-09-10T20:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe267ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe200ce5f0 0x7efe2014a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe267ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe200cfec0 0x7efe200c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe267ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe2015bf30 0x7efe201555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe267ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe200d0cc0 con 0x7efe200cfec0 2024-09-10T20:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe267ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe200d0b40 con 0x7efe2015bf30 2024-09-10T20:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe267ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe200b6620 con 0x7efe200ce5f0 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe200cfec0 0x7efe200c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe200cfec0 0x7efe200c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53172/0 (socket says 172.21.15.28:53172) 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 -- 172.21.15.28:0/2949066626 learned_addr learned my addr 172.21.15.28:0/2949066626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe24d4a640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe2015bf30 0x7efe201555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1ffff640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe200ce5f0 0x7efe2014a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 -- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe200ce5f0 msgr2=0x7efe2014a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe200ce5f0 0x7efe2014a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 -- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe2015bf30 msgr2=0x7efe201555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe2015bf30 0x7efe201555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.239+0000 7efe1f7fe640 1 -- 172.21.15.28:0/2949066626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe200ca2c0 con 0x7efe200cfec0 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.240+0000 7efe1f7fe640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe200cfec0 0x7efe200c9d80 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c9a0 tx=0x7efe1400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.240+0000 7efe1cff9640 1 -- 172.21.15.28:0/2949066626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14007bb0 con 0x7efe200cfec0 2024-09-10T20:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.240+0000 7efe1cff9640 1 -- 172.21.15.28:0/2949066626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe14005020 con 0x7efe200cfec0 2024-09-10T20:43:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.240+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe200ca570 con 0x7efe200cfec0 2024-09-10T20:43:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.240+0000 7efe1cff9640 1 -- 172.21.15.28:0/2949066626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14007e60 con 0x7efe200cfec0 2024-09-10T20:43:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.240+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe200cab30 con 0x7efe200cfec0 2024-09-10T20:43:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.241+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe200d0330 con 0x7efe200cfec0 2024-09-10T20:43:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.242+0000 7efe1cff9640 1 -- 172.21.15.28:0/2949066626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7efe14005470 con 0x7efe200cfec0 2024-09-10T20:43:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.242+0000 7efe1cff9640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe0c0783f0 0x7efe0c07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.242+0000 7efe1cff9640 1 -- 172.21.15.28:0/2949066626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5823+0+0 (secure 0 0 0) 0x7efe1409ac10 con 0x7efe200cfec0 2024-09-10T20:43:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.242+0000 7efe1ffff640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe0c0783f0 0x7efe0c07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.243+0000 7efe1ffff640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe0c0783f0 0x7efe0c07a8b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe10033000 tx=0x7efe10004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.246+0000 7efe1cff9640 1 -- 172.21.15.28:0/2949066626 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efe14014030 con 0x7efe200cfec0 2024-09-10T20:43:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.433+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe200b2f20 con 0x7efe200cfec0 2024-09-10T20:43:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.434+0000 7efe1cff9640 1 -- 172.21.15.28:0/2949066626 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16932 (secure 0 0 0) 0x7efe14062880 con 0x7efe200cfec0 2024-09-10T20:43:33.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:33.439 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":103,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:43:32.355983+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:42:54.016199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-10T20:42:55.330837+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.4","mappings":[{"from":4,"to":3}]},{"pgid":"17.21","mappings":[{"from":6,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe0c0783f0 msgr2=0x7efe0c07a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe0c0783f0 0x7efe0c07a8b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe10033000 tx=0x7efe10004730 comp rx=0 tx=0).stop 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe200cfec0 msgr2=0x7efe200c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe200cfec0 0x7efe200c9d80 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c9a0 tx=0x7efe1400ce60 comp rx=0 tx=0).stop 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 shutdown_connections 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe0c0783f0 0x7efe0c07a8b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe2015bf30 0x7efe201555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe200cfec0 0x7efe200c9d80 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 --2- 172.21.15.28:0/2949066626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe200ce5f0 0x7efe2014a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 >> 172.21.15.28:0/2949066626 conn(0x7efe200d1e40 msgr2=0x7efe200d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 shutdown_connections 2024-09-10T20:43:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.437+0000 7efe267ad640 1 -- 172.21.15.28:0/2949066626 wait complete. 2024-09-10T20:43:33.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:43:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 -- 172.21.15.28:0/3384008680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 msgr2=0x7f5ff00d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 --2- 172.21.15.28:0/3384008680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00d2b90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00d4d80 tx=0x7f5fe002a560 comp rx=0 tx=0).stop 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 -- 172.21.15.28:0/3384008680 shutdown_connections 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 --2- 172.21.15.28:0/3384008680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ff0161b50 0x7f5ff0161f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 --2- 172.21.15.28:0/3384008680 >> v2:172.21.15.159:3300/0 conn(0x7f5ff00cf780 0x7f5ff00cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 --2- 172.21.15.28:0/3384008680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00d2b90 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 -- 172.21.15.28:0/3384008680 >> 172.21.15.28:0/3384008680 conn(0x7f5ff00d3fb0 msgr2=0x7f5ff00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 -- 172.21.15.28:0/3384008680 shutdown_connections 2024-09-10T20:43:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.554+0000 7f5ff634b640 1 -- 172.21.15.28:0/3384008680 wait complete. 2024-09-10T20:43:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.556+0000 7f5ff634b640 1 Processor -- start 2024-09-10T20:43:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.556+0000 7f5ff634b640 1 -- start start 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ff00cf780 0x7f5ff00b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ff0161b50 0x7f5ff0152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff00b6e00 con 0x7f5ff015de70 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff015e460 con 0x7f5ff00cf780 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5ff00b6f80 con 0x7f5ff0161b50 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53188/0 (socket says 172.21.15.28:53188) 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 -- 172.21.15.28:0/4165619648 learned_addr learned my addr 172.21.15.28:0/4165619648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5feffff640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ff00cf780 0x7f5ff00b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff48e8640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ff0161b50 0x7f5ff0152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 -- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ff0161b50 msgr2=0x7f5ff0152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ff0161b50 0x7f5ff0152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 -- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ff00cf780 msgr2=0x7f5ff00b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ff00cf780 0x7f5ff00b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 -- 172.21.15.28:0/4165619648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff00b46d0 con 0x7f5ff015de70 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff48e8640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ff0161b50 0x7f5ff0152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fef7fe640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00b2720 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400b4a0 tx=0x7f5fe400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fecff9640 1 -- 172.21.15.28:0/4165619648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4014070 con 0x7f5ff015de70 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff00b0c80 con 0x7f5ff015de70 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fecff9640 1 -- 172.21.15.28:0/4165619648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fe4010700 con 0x7f5ff015de70 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5fecff9640 1 -- 172.21.15.28:0/4165619648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4010b50 con 0x7f5ff015de70 2024-09-10T20:43:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.557+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff00b1240 con 0x7f5ff015de70 2024-09-10T20:43:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.559+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff00cd7a0 con 0x7f5ff015de70 2024-09-10T20:43:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.559+0000 7f5fecff9640 1 -- 172.21.15.28:0/4165619648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f5fe4017360 con 0x7f5ff015de70 2024-09-10T20:43:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.560+0000 7f5fecff9640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fdc0783f0 0x7f5fdc07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.560+0000 7f5feffff640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fdc0783f0 0x7f5fdc07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.560+0000 7f5fecff9640 1 -- 172.21.15.28:0/4165619648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5823+0+0 (secure 0 0 0) 0x7f5fe409a4c0 con 0x7f5ff015de70 2024-09-10T20:43:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.560+0000 7f5feffff640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fdc0783f0 0x7f5fdc07a8b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0033000 tx=0x7f5fe0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.564+0000 7f5fecff9640 1 -- 172.21.15.28:0/4165619648 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ff00cd7a0 con 0x7f5ff015de70 2024-09-10T20:43:33.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.787+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5ff00d2b90 con 0x7f5ff015de70 2024-09-10T20:43:33.791 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:43:33.787+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:43:33.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.789+0000 7f5fecff9640 1 -- 172.21.15.28:0/4165619648 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) ==== 161+0+0 (secure 0 0 0) 0x7f5fe40620b0 con 0x7f5ff015de70 2024-09-10T20:43:33.793 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fdc0783f0 msgr2=0x7f5fdc07a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fdc0783f0 0x7f5fdc07a8b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0033000 tx=0x7f5fe0004730 comp rx=0 tx=0).stop 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 msgr2=0x7f5ff00b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00b2720 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400b4a0 tx=0x7f5fe400b960 comp rx=0 tx=0).stop 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 shutdown_connections 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fdc0783f0 0x7f5fdc07a8b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ff0161b50 0x7f5ff0152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ff015de70 0x7f5ff00b2720 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 --2- 172.21.15.28:0/4165619648 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ff00cf780 0x7f5ff00b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 >> 172.21.15.28:0/4165619648 conn(0x7f5ff00d3fb0 msgr2=0x7f5ff00cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 shutdown_connections 2024-09-10T20:43:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.793+0000 7f5ff634b640 1 -- 172.21.15.28:0/4165619648 wait complete. 2024-09-10T20:43:33.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:43:33.833 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:43:33.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 -- 172.21.15.28:0/3874793056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 msgr2=0x7fa3d80b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/3874793056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80b4fd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fa3d80b0810 tx=0x7fa3d002a560 comp rx=0 tx=0).stop 2024-09-10T20:43:33.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 -- 172.21.15.28:0/3874793056 shutdown_connections 2024-09-10T20:43:33.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/3874793056 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80d1240 0x7fa3d806f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/3874793056 >> v2:172.21.15.159:3300/0 conn(0x7fa3d815bbe0 0x7fa3d80d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/3874793056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80b4fd0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 -- 172.21.15.28:0/3874793056 >> 172.21.15.28:0/3874793056 conn(0x7fa3d806dbf0 msgr2=0x7fa3d806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:33.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 -- 172.21.15.28:0/3874793056 shutdown_connections 2024-09-10T20:43:33.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.912+0000 7fa3dfe16640 1 -- 172.21.15.28:0/3874793056 wait complete. 2024-09-10T20:43:33.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.913+0000 7fa3dfe16640 1 Processor -- start 2024-09-10T20:43:33.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.913+0000 7fa3dfe16640 1 -- start start 2024-09-10T20:43:33.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dfe16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80d1240 0x7fa3d814f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dfe16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dfe16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d815bbe0 0x7fa3d80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dfe16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d815c040 con 0x7fa3d8155250 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dfe16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d815e960 con 0x7fa3d80d1240 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dfe16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d80b1c80 con 0x7fa3d815bbe0 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dd3b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dd3b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53222/0 (socket says 172.21.15.28:53222) 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3dd3b1640 1 -- 172.21.15.28:0/2686805836 learned_addr learned my addr 172.21.15.28:0/2686805836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3ddbb2640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80d1240 0x7fa3d814f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.914+0000 7fa3de3b3640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d815bbe0 0x7fa3d80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dd3b1640 1 -- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d815bbe0 msgr2=0x7fa3d80cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dd3b1640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d815bbe0 0x7fa3d80cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dd3b1640 1 -- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80d1240 msgr2=0x7fa3d814f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dd3b1640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80d1240 0x7fa3d814f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:33.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dd3b1640 1 -- 172.21.15.28:0/2686805836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d80ca910 con 0x7fa3d8155250 2024-09-10T20:43:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3de3b3640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d815bbe0 0x7fa3d80cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dd3b1640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80c9fa0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00c970 tx=0x7fa3cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3d67fc640 1 -- 172.21.15.28:0/2686805836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc007bb0 con 0x7fa3d8155250 2024-09-10T20:43:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3d67fc640 1 -- 172.21.15.28:0/2686805836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3cc007e40 con 0x7fa3d8155250 2024-09-10T20:43:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d80cabf0 con 0x7fa3d8155250 2024-09-10T20:43:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3d67fc640 1 -- 172.21.15.28:0/2686805836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00f690 con 0x7fa3d8155250 2024-09-10T20:43:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.915+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d80cb1b0 con 0x7fa3d8155250 2024-09-10T20:43:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.916+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d80b4fb0 con 0x7fa3d8155250 2024-09-10T20:43:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.916+0000 7fa3d67fc640 1 -- 172.21.15.28:0/2686805836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fa3cc020030 con 0x7fa3d8155250 2024-09-10T20:43:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.917+0000 7fa3d67fc640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3c4078420 0x7fa3c407a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.917+0000 7fa3d67fc640 1 -- 172.21.15.28:0/2686805836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5881+0+0 (secure 0 0 0) 0x7fa3cc09aa90 con 0x7fa3d8155250 2024-09-10T20:43:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.917+0000 7fa3ddbb2640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3c4078420 0x7fa3c407a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.918+0000 7fa3ddbb2640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3c4078420 0x7fa3c407a8e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0033000 tx=0x7fa3d0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:33.921+0000 7fa3d67fc640 1 -- 172.21.15.28:0/2686805836 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa3cc09ce20 con 0x7fa3d8155250 2024-09-10T20:43:34.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.147+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa3d80d0110 con 0x7fa3d8155250 2024-09-10T20:43:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.796+0000 7fa3d67fc640 1 -- 172.21.15.28:0/2686805836 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) ==== 104+0+0 (secure 0 0 0) 0x7fa3cc014030 con 0x7fa3d8155250 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3c4078420 msgr2=0x7fa3c407a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3c4078420 0x7fa3c407a8e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0033000 tx=0x7fa3d0004730 comp rx=0 tx=0).stop 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 msgr2=0x7fa3d80c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80c9fa0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00c970 tx=0x7fa3cc00ce30 comp rx=0 tx=0).stop 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 shutdown_connections 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3c4078420 0x7fa3c407a8e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d815bbe0 0x7fa3d80cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3d8155250 0x7fa3d80c9fa0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 --2- 172.21.15.28:0/2686805836 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3d80d1240 0x7fa3d814f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 >> 172.21.15.28:0/2686805836 conn(0x7fa3d806dbf0 msgr2=0x7fa3d80af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.799+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 shutdown_connections 2024-09-10T20:43:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.800+0000 7fa3dfe16640 1 -- 172.21.15.28:0/2686805836 wait complete. 2024-09-10T20:43:34.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:43:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 -- 172.21.15.28:0/860188674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700cf3b0 msgr2=0x7fec700ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 --2- 172.21.15.28:0/860188674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700cf3b0 0x7fec700ca2d0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fec70051530 tx=0x7fec6002b1e0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 -- 172.21.15.28:0/860188674 shutdown_connections 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 --2- 172.21.15.28:0/860188674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec7006fff0 0x7fec700ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 --2- 172.21.15.28:0/860188674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700cf3b0 0x7fec700ca2d0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 --2- 172.21.15.28:0/860188674 >> v2:172.21.15.159:3300/0 conn(0x7fec70159f20 0x7fec70072610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 -- 172.21.15.28:0/860188674 >> 172.21.15.28:0/860188674 conn(0x7fec700cfc80 msgr2=0x7fec700d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 -- 172.21.15.28:0/860188674 shutdown_connections 2024-09-10T20:43:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.974+0000 7fec7747f640 1 -- 172.21.15.28:0/860188674 wait complete. 2024-09-10T20:43:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 Processor -- start 2024-09-10T20:43:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 -- start start 2024-09-10T20:43:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec7006fff0 0x7fec7014b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec700cf3b0 0x7fec701ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec70159f20 0x7fec701efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec70070880 con 0x7fec70159f20 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fec70070700 con 0x7fec7006fff0 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.976+0000 7fec7747f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fec7006f2f0 con 0x7fec700cf3b0 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec70159f20 0x7fec701efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec7521b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec7006fff0 0x7fec7014b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec70159f20 0x7fec701efe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53228/0 (socket says 172.21.15.28:53228) 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 -- 172.21.15.28:0/3934637949 learned_addr learned my addr 172.21.15.28:0/3934637949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec74a1a640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec700cf3b0 0x7fec701ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 -- 172.21.15.28:0/3934637949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec700cf3b0 msgr2=0x7fec701ef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec700cf3b0 0x7fec701ef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 -- 172.21.15.28:0/3934637949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec7006fff0 msgr2=0x7fec7014b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec7006fff0 0x7fec7014b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 -- 172.21.15.28:0/3934637949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7014aea0 con 0x7fec70159f20 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec75a1c640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec70159f20 0x7fec701efe50 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fec6800cab0 tx=0x7fec6800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec65ffb640 1 -- 172.21.15.28:0/3934637949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68007d90 con 0x7fec70159f20 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec65ffb640 1 -- 172.21.15.28:0/3934637949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec68003a80 con 0x7fec70159f20 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec700c8120 con 0x7fec70159f20 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.977+0000 7fec65ffb640 1 -- 172.21.15.28:0/3934637949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68004020 con 0x7fec70159f20 2024-09-10T20:43:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.978+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec700c86e0 con 0x7fec70159f20 2024-09-10T20:43:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.979+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec70072610 con 0x7fec70159f20 2024-09-10T20:43:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.979+0000 7fec65ffb640 1 -- 172.21.15.28:0/3934637949 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fec680041c0 con 0x7fec70159f20 2024-09-10T20:43:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.980+0000 7fec65ffb640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec540783f0 0x7fec5407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.980+0000 7fec65ffb640 1 -- 172.21.15.28:0/3934637949 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5881+0+0 (secure 0 0 0) 0x7fec6809b0d0 con 0x7fec70159f20 2024-09-10T20:43:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.980+0000 7fec7521b640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec540783f0 0x7fec5407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.980+0000 7fec7521b640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec540783f0 0x7fec5407a8b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00a490 tx=0x7fec6c00a420 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:34.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:34.984+0000 7fec65ffb640 1 -- 172.21.15.28:0/3934637949 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fec7014aea0 con 0x7fec70159f20 2024-09-10T20:43:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.171+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fec7015e8d0 con 0x7fec70159f20 2024-09-10T20:43:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.798+0000 7fec65ffb640 1 -- 172.21.15.28:0/3934637949 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) ==== 181+0+0 (secure 0 0 0) 0x7fec68014070 con 0x7fec70159f20 2024-09-10T20:43:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:43:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec540783f0 msgr2=0x7fec5407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec540783f0 0x7fec5407a8b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00a490 tx=0x7fec6c00a420 comp rx=0 tx=0).stop 2024-09-10T20:43:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec70159f20 msgr2=0x7fec701efe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec70159f20 0x7fec701efe50 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fec6800cab0 tx=0x7fec6800cf70 comp rx=0 tx=0).stop 2024-09-10T20:43:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 shutdown_connections 2024-09-10T20:43:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec540783f0 0x7fec5407a8b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec70159f20 0x7fec701efe50 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec700cf3b0 0x7fec701ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 --2- 172.21.15.28:0/3934637949 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec7006fff0 0x7fec7014b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 >> 172.21.15.28:0/3934637949 conn(0x7fec700cfc80 msgr2=0x7fec7015c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 shutdown_connections 2024-09-10T20:43:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.802+0000 7fec7747f640 1 -- 172.21.15.28:0/3934637949 wait complete. 2024-09-10T20:43:35.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:43:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/679597171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78070270 msgr2=0x7f1e780ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/679597171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78070270 0x7f1e780ca620 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f1e70005fd0 tx=0x7f1e7002aa80 comp rx=0 tx=0).stop 2024-09-10T20:43:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/679597171 shutdown_connections 2024-09-10T20:43:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/679597171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78070270 0x7f1e780ca620 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/679597171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e7815bd70 0x7f1e7806fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/679597171 >> 172.21.15.28:0/679597171 conn(0x7f1e780cfc80 msgr2=0x7f1e780d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/679597171 shutdown_connections 2024-09-10T20:43:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.932+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/679597171 wait complete. 2024-09-10T20:43:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.933+0000 7f1e7f0d4640 1 Processor -- start 2024-09-10T20:43:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.933+0000 7f1e7f0d4640 1 -- start start 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7f0d4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e78070270 0x7f1e781522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7f0d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78151cf0 0x7f1e781f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7f0d4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e7815bd70 0x7f1e781f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7f0d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e780caef0 con 0x7f1e78151cf0 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7f0d4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1e780cad70 con 0x7f1e78070270 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7f0d4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1e780cb070 con 0x7f1e7815bd70 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78151cf0 0x7f1e781f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78151cf0 0x7f1e781f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53250/0 (socket says 172.21.15.28:53250) 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 -- 172.21.15.28:0/1145508385 learned_addr learned my addr 172.21.15.28:0/1145508385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7ce70640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e78070270 0x7f1e781522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e7d671640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e7815bd70 0x7f1e781f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 -- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e7815bd70 msgr2=0x7f1e781f60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e7815bd70 0x7f1e781f60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 -- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e78070270 msgr2=0x7f1e781522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e78070270 0x7f1e781522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.934+0000 7f1e6ffff640 1 -- 172.21.15.28:0/1145508385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e78152830 con 0x7f1e78151cf0 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.935+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78151cf0 0x7f1e781f5b80 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800c9a0 tx=0x7f1e6800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.935+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/1145508385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68007bb0 con 0x7f1e78151cf0 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.935+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/1145508385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e68007e40 con 0x7f1e78151cf0 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.935+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/1145508385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6800f690 con 0x7f1e78151cf0 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.935+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e78154c90 con 0x7f1e78151cf0 2024-09-10T20:43:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.935+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e78155250 con 0x7f1e78151cf0 2024-09-10T20:43:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.936+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e780ca620 con 0x7f1e78151cf0 2024-09-10T20:43:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.937+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/1145508385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f1e6800f830 con 0x7f1e78151cf0 2024-09-10T20:43:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.937+0000 7f1e6d7fa640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e640783f0 0x7f1e6407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.937+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/1145508385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5414+0+0 (secure 0 0 0) 0x7f1e6809ae00 con 0x7f1e78151cf0 2024-09-10T20:43:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.937+0000 7f1e7ce70640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e640783f0 0x7f1e6407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.938+0000 7f1e7ce70640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e640783f0 0x7f1e6407a8b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e74001220 tx=0x7f1e74009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:35.941+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/1145508385 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1e68014030 con 0x7f1e78151cf0 2024-09-10T20:43:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.128+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1e78155650 con 0x7f1e78151cf0 2024-09-10T20:43:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.803+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/1145508385 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) ==== 169+0+0 (secure 0 0 0) 0x7f1e680631d0 con 0x7f1e78151cf0 2024-09-10T20:43:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:43:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.807+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e640783f0 msgr2=0x7f1e6407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.807+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e640783f0 0x7f1e6407a8b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e74001220 tx=0x7f1e74009040 comp rx=0 tx=0).stop 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.807+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78151cf0 msgr2=0x7f1e781f5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.807+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78151cf0 0x7f1e781f5b80 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800c9a0 tx=0x7f1e6800ce60 comp rx=0 tx=0).stop 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 shutdown_connections 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e640783f0 0x7f1e6407a8b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e7815bd70 0x7f1e781f60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e78151cf0 0x7f1e781f5b80 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 --2- 172.21.15.28:0/1145508385 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e78070270 0x7f1e781522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 >> 172.21.15.28:0/1145508385 conn(0x7f1e780cfc80 msgr2=0x7f1e78072d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 shutdown_connections 2024-09-10T20:43:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.808+0000 7f1e7f0d4640 1 -- 172.21.15.28:0/1145508385 wait complete. 2024-09-10T20:43:36.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:43:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.943+0000 7ffb162bd640 1 -- 172.21.15.28:0/1141702414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffaf8004c70 msgr2=0x7ffaf8005070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.943+0000 7ffb162bd640 1 --2- 172.21.15.28:0/1141702414 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffaf8004c70 0x7ffaf8005070 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ffb00001eb0 tx=0x7ffaf8005830 comp rx=0 tx=0).stop 2024-09-10T20:43:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.944+0000 7ffb1485a640 1 -- 172.21.15.28:0/1141702414 reap_dead start 2024-09-10T20:43:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.944+0000 7ffb162bd640 1 -- 172.21.15.28:0/1141702414 shutdown_connections 2024-09-10T20:43:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.944+0000 7ffb162bd640 1 -- 172.21.15.28:0/1141702414 >> 172.21.15.28:0/1141702414 conn(0x7ffb100d1ed0 msgr2=0x7ffb100d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.944+0000 7ffb162bd640 1 -- 172.21.15.28:0/1141702414 shutdown_connections 2024-09-10T20:43:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.944+0000 7ffb162bd640 1 -- 172.21.15.28:0/1141702414 wait complete. 2024-09-10T20:43:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.945+0000 7ffb162bd640 1 Processor -- start 2024-09-10T20:43:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.946+0000 7ffb162bd640 1 -- start start 2024-09-10T20:43:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.946+0000 7ffb162bd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaf8004c70 0x7ffb100b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.946+0000 7ffb162bd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb101f8190 0x7ffb1015f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.946+0000 7ffb162bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb100aeab0 0x7ffb100c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.946+0000 7ffb162bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1006df80 con 0x7ffb100aeab0 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.946+0000 7ffb162bd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1006de00 con 0x7ffb101f8190 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.946+0000 7ffb162bd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffb1006e100 con 0x7ffaf8004c70 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb100aeab0 0x7ffb100c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb100aeab0 0x7ffb100c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53264/0 (socket says 172.21.15.28:53264) 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 -- 172.21.15.28:0/1343706070 learned_addr learned my addr 172.21.15.28:0/1343706070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb0f7fe640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb101f8190 0x7ffb1015f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb0ffff640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaf8004c70 0x7ffb100b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 -- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaf8004c70 msgr2=0x7ffb100b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaf8004c70 0x7ffb100b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 -- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb101f8190 msgr2=0x7ffb1015f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb101f8190 0x7ffb1015f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 -- 172.21.15.28:0/1343706070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb100b0e30 con 0x7ffb100aeab0 2024-09-10T20:43:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb0f7fe640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb101f8190 0x7ffb1015f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb0ffff640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaf8004c70 0x7ffb100b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb1485a640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb100aeab0 0x7ffb100c7060 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ffb04000c00 tx=0x7ffb04001750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb0cff9640 1 -- 172.21.15.28:0/1343706070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04015020 con 0x7ffb100aeab0 2024-09-10T20:43:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb0cff9640 1 -- 172.21.15.28:0/1343706070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb04011750 con 0x7ffb100aeab0 2024-09-10T20:43:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb100b1110 con 0x7ffb100aeab0 2024-09-10T20:43:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb0cff9640 1 -- 172.21.15.28:0/1343706070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb08009be0 con 0x7ffb100aeab0 2024-09-10T20:43:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.947+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb100c6b40 con 0x7ffb100aeab0 2024-09-10T20:43:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.948+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb100d0210 con 0x7ffb100aeab0 2024-09-10T20:43:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.949+0000 7ffb0cff9640 1 -- 172.21.15.28:0/1343706070 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7ffb04011be0 con 0x7ffb100aeab0 2024-09-10T20:43:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.949+0000 7ffb0cff9640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffafc0783f0 0x7ffafc07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.950+0000 7ffb0cff9640 1 -- 172.21.15.28:0/1343706070 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7ffb0409aa50 con 0x7ffb100aeab0 2024-09-10T20:43:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.950+0000 7ffb0ffff640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffafc0783f0 0x7ffafc07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.950+0000 7ffb0ffff640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffafc0783f0 0x7ffafc07a8b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80057e0 tx=0x7ffb0002d000 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:36.954+0000 7ffb0cff9640 1 -- 172.21.15.28:0/1343706070 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffb0401f030 con 0x7ffb100aeab0 2024-09-10T20:43:37.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.138+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7ffb100b36b0 con 0x7ffb100aeab0 2024-09-10T20:43:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.142+0000 7ffb0cff9640 1 -- 172.21.15.28:0/1343706070 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v39) ==== 71+0+0 (secure 0 0 0) 0x7ffb04063030 con 0x7ffb100aeab0 2024-09-10T20:43:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffafc0783f0 msgr2=0x7ffafc07a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffafc0783f0 0x7ffafc07a8b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80057e0 tx=0x7ffb0002d000 comp rx=0 tx=0).stop 2024-09-10T20:43:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb100aeab0 msgr2=0x7ffb100c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb100aeab0 0x7ffb100c7060 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ffb04000c00 tx=0x7ffb04001750 comp rx=0 tx=0).stop 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 shutdown_connections 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffafc0783f0 0x7ffafc07a8b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb100aeab0 0x7ffb100c7060 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffb101f8190 0x7ffb1015f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 --2- 172.21.15.28:0/1343706070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaf8004c70 0x7ffb100b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 >> 172.21.15.28:0/1343706070 conn(0x7ffb100d1ed0 msgr2=0x7ffb100b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 shutdown_connections 2024-09-10T20:43:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.144+0000 7ffb162bd640 1 -- 172.21.15.28:0/1343706070 wait complete. 2024-09-10T20:43:37.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-09-10T20:43:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.260+0000 7f3a07ddb640 1 -- 172.21.15.28:0/3520745780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a0006d1f0 msgr2=0x7f3a000caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.260+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/3520745780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a0006d1f0 0x7f3a000caa70 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f39f0005fd0 tx=0x7f39f002a530 comp rx=0 tx=0).stop 2024-09-10T20:43:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.260+0000 7f3a07ddb640 1 -- 172.21.15.28:0/3520745780 shutdown_connections 2024-09-10T20:43:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.260+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/3520745780 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a00156510 0x7f3a0006ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.260+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/3520745780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a0006d1f0 0x7f3a000caa70 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.260+0000 7f3a07ddb640 1 -- 172.21.15.28:0/3520745780 >> 172.21.15.28:0/3520745780 conn(0x7f3a000d0420 msgr2=0x7f3a000d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.261+0000 7f3a07ddb640 1 -- 172.21.15.28:0/3520745780 shutdown_connections 2024-09-10T20:43:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.261+0000 7f3a07ddb640 1 -- 172.21.15.28:0/3520745780 wait complete. 2024-09-10T20:43:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.262+0000 7f3a07ddb640 1 Processor -- start 2024-09-10T20:43:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.262+0000 7f3a07ddb640 1 -- start start 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a0006d1f0 0x7f3a000b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a00156130 0x7f3a0014f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a00156510 0x7f3a001f8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a00073370 con 0x7f3a00156510 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3a000731f0 con 0x7f3a00156130 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3a000734f0 con 0x7f3a0006d1f0 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a00156510 0x7f3a001f8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a00156510 0x7f3a001f8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53284/0 (socket says 172.21.15.28:53284) 2024-09-10T20:43:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 -- 172.21.15.28:0/851347785 learned_addr learned my addr 172.21.15.28:0/851347785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a05b77640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a0006d1f0 0x7f3a000b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a05376640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a00156130 0x7f3a0014f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 -- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a0006d1f0 msgr2=0x7f3a000b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a0006d1f0 0x7f3a000b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 -- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a00156130 msgr2=0x7f3a0014f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a00156130 0x7f3a0014f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 -- 172.21.15.28:0/851347785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a000ca3c0 con 0x7f3a00156510 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a05376640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a00156130 0x7f3a0014f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a06378640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a00156510 0x7f3a001f8a10 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f39f800e990 tx=0x7f39f800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f39f67fc640 1 -- 172.21.15.28:0/851347785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8004d20 con 0x7f3a00156510 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f39f67fc640 1 -- 172.21.15.28:0/851347785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39f800caa0 con 0x7f3a00156510 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a000b1e20 con 0x7f3a00156510 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f39f67fc640 1 -- 172.21.15.28:0/851347785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8005020 con 0x7f3a00156510 2024-09-10T20:43:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.263+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a000b60b0 con 0x7f3a00156510 2024-09-10T20:43:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.264+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a000d11c0 con 0x7f3a00156510 2024-09-10T20:43:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.265+0000 7f39f67fc640 1 -- 172.21.15.28:0/851347785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f39f800f040 con 0x7f3a00156510 2024-09-10T20:43:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.266+0000 7f39f67fc640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39e40783f0 0x7f39e407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.266+0000 7f39f67fc640 1 -- 172.21.15.28:0/851347785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7f39f809ac70 con 0x7f3a00156510 2024-09-10T20:43:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.266+0000 7f3a05b77640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39e40783f0 0x7f39e407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.266+0000 7f3a05b77640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39e40783f0 0x7f39e407a8b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39fc004f60 tx=0x7f39fc007e30 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:37.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.270+0000 7f39f67fc640 1 -- 172.21.15.28:0/851347785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3a000b1e20 con 0x7f3a00156510 2024-09-10T20:43:37.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.528+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7f3a000caa50 con 0x7f3a00156510 2024-09-10T20:43:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.804+0000 7f39f67fc640 1 -- 172.21.15.28:0/851347785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v298) ==== 149+0+0 (secure 0 0 0) 0x7f39f8016030 con 0x7f3a00156510 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39e40783f0 msgr2=0x7f39e407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39e40783f0 0x7f39e407a8b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39fc004f60 tx=0x7f39fc007e30 comp rx=0 tx=0).stop 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a00156510 msgr2=0x7f3a001f8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a00156510 0x7f3a001f8a10 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f39f800e990 tx=0x7f39f800ee50 comp rx=0 tx=0).stop 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 shutdown_connections 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f39e40783f0 0x7f39e407a8b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a00156510 0x7f3a001f8a10 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a00156130 0x7f3a0014f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 --2- 172.21.15.28:0/851347785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a0006d1f0 0x7f3a000b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 >> 172.21.15.28:0/851347785 conn(0x7f3a000d0420 msgr2=0x7f3a000689e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 shutdown_connections 2024-09-10T20:43:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.807+0000 7f3a07ddb640 1 -- 172.21.15.28:0/851347785 wait complete. 2024-09-10T20:43:37.823 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-10T20:43:37.823 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-10T20:43:37.823 INFO:tasks.cephfs_test_runner:test_subvolume_no_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:43:37.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:43:37.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 -- 172.21.15.28:0/1612169703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342806d1f0 msgr2=0x7f34280caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 --2- 172.21.15.28:0/1612169703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342806d1f0 0x7f34280caa70 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f341c005fd0 tx=0x7f341c02a5d0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 -- 172.21.15.28:0/1612169703 shutdown_connections 2024-09-10T20:43:37.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 --2- 172.21.15.28:0/1612169703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3428156510 0x7f342806ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 --2- 172.21.15.28:0/1612169703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342806d1f0 0x7f34280caa70 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 -- 172.21.15.28:0/1612169703 >> 172.21.15.28:0/1612169703 conn(0x7f34280d0420 msgr2=0x7f34280d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:37.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 -- 172.21.15.28:0/1612169703 shutdown_connections 2024-09-10T20:43:37.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.927+0000 7f342fd24640 1 -- 172.21.15.28:0/1612169703 wait complete. 2024-09-10T20:43:37.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.928+0000 7f342fd24640 1 Processor -- start 2024-09-10T20:43:37.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.928+0000 7f342fd24640 1 -- start start 2024-09-10T20:43:37.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342fd24640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f342806d1f0 0x7f34280ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342fd24640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3428156130 0x7f34281f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342fd24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3428156510 0x7f34280b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342fd24640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3428073580 con 0x7f3428156510 2024-09-10T20:43:37.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342fd24640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3428073400 con 0x7f342806d1f0 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342fd24640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3428073700 con 0x7f3428156130 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342e2c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3428156510 0x7f34280b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342e2c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3428156510 0x7f34280b33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53300/0 (socket says 172.21.15.28:53300) 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342e2c1640 1 -- 172.21.15.28:0/1291861880 learned_addr learned my addr 172.21.15.28:0/1291861880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342d2bf640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3428156130 0x7f34281f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.929+0000 7f342dac0640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f342806d1f0 0x7f34280ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342e2c1640 1 -- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3428156130 msgr2=0x7f34281f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342e2c1640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3428156130 0x7f34281f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342e2c1640 1 -- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f342806d1f0 msgr2=0x7f34280ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342e2c1640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f342806d1f0 0x7f34280ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342e2c1640 1 -- 172.21.15.28:0/1291861880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34280b53c0 con 0x7f3428156510 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342e2c1640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3428156510 0x7f34280b33f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f3420005d00 tx=0x7f342000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:37.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f34267fc640 1 -- 172.21.15.28:0/1291861880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420017070 con 0x7f3428156510 2024-09-10T20:43:37.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f34267fc640 1 -- 172.21.15.28:0/1291861880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3420005330 con 0x7f3428156510 2024-09-10T20:43:37.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34280b56a0 con 0x7f3428156510 2024-09-10T20:43:37.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f34267fc640 1 -- 172.21.15.28:0/1291861880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34200047a0 con 0x7f3428156510 2024-09-10T20:43:37.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.930+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34280b5c60 con 0x7f3428156510 2024-09-10T20:43:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.931+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3428073bb0 con 0x7f3428156510 2024-09-10T20:43:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.932+0000 7f34267fc640 1 -- 172.21.15.28:0/1291861880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f3420004940 con 0x7f3428156510 2024-09-10T20:43:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.932+0000 7f34267fc640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34140783f0 0x7f341407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.933+0000 7f34267fc640 1 -- 172.21.15.28:0/1291861880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7f342009a420 con 0x7f3428156510 2024-09-10T20:43:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.933+0000 7f342dac0640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34140783f0 0x7f341407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.933+0000 7f342dac0640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34140783f0 0x7f341407a8b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f34280b4ca0 tx=0x7f3418009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:37.937+0000 7f34267fc640 1 -- 172.21.15.28:0/1291861880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3420062410 con 0x7f3428156510 2024-09-10T20:43:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.120+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34280d11c0 con 0x7f3428156510 2024-09-10T20:43:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.121+0000 7f34267fc640 1 -- 172.21.15.28:0/1291861880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+13080 (secure 0 0 0) 0x7f342006b180 con 0x7f3428156510 2024-09-10T20:43:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:38.125 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:43:36.799357+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1426218269":"2024-09-11T20:43:33.788373+0000","172.21.15.28:6832/1426218269":"2024-09-11T20:43:33.788373+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:43:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34140783f0 msgr2=0x7f341407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34140783f0 0x7f341407a8b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f34280b4ca0 tx=0x7f3418009040 comp rx=0 tx=0).stop 2024-09-10T20:43:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3428156510 msgr2=0x7f34280b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3428156510 0x7f34280b33f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f3420005d00 tx=0x7f342000dae0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 shutdown_connections 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34140783f0 0x7f341407a8b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3428156510 0x7f34280b33f0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3428156130 0x7f34281f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 --2- 172.21.15.28:0/1291861880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f342806d1f0 0x7f34280ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 >> 172.21.15.28:0/1291861880 conn(0x7f34280d0420 msgr2=0x7f342815e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 shutdown_connections 2024-09-10T20:43:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.123+0000 7f342fd24640 1 -- 172.21.15.28:0/1291861880 wait complete. 2024-09-10T20:43:38.138 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:43:38.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 -- 172.21.15.28:0/927449927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 msgr2=0x7f2fc415b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/927449927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 0x7f2fc415b170 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4005fd0 tx=0x7f2fb402a820 comp rx=0 tx=0).stop 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 -- 172.21.15.28:0/927449927 shutdown_connections 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/927449927 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc406fc20 0x7f2fc415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/927449927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 0x7f2fc415b170 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 -- 172.21.15.28:0/927449927 >> 172.21.15.28:0/927449927 conn(0x7f2fc406d110 msgr2=0x7f2fc40b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 -- 172.21.15.28:0/927449927 shutdown_connections 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.243+0000 7f2fc9c33640 1 -- 172.21.15.28:0/927449927 wait complete. 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.245+0000 7f2fc9c33640 1 Processor -- start 2024-09-10T20:43:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.245+0000 7f2fc9c33640 1 -- start start 2024-09-10T20:43:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc9c33640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc406fc20 0x7f2fc4152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc9c33640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 0x7f2fc41557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc9c33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc415e7d0 0x7f2fc4155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc9c33640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc415d4e0 con 0x7f2fc415e7d0 2024-09-10T20:43:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc9c33640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc415d360 con 0x7f2fc41587a0 2024-09-10T20:43:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc9c33640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc415d660 con 0x7f2fc406fc20 2024-09-10T20:43:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc415e7d0 0x7f2fc4155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc415e7d0 0x7f2fc4155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53312/0 (socket says 172.21.15.28:53312) 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 -- 172.21.15.28:0/2682861188 learned_addr learned my addr 172.21.15.28:0/2682861188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc37fe640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc406fc20 0x7f2fc4152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc2ffd640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 0x7f2fc41557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 -- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc406fc20 msgr2=0x7f2fc4152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc406fc20 0x7f2fc4152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 -- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 msgr2=0x7f2fc41557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 0x7f2fc41557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 -- 172.21.15.28:0/2682861188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc41f27b0 con 0x7f2fc415e7d0 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc2ffd640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 0x7f2fc41557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fc3fff640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc415e7d0 0x7f2fc4155d30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00e940 tx=0x7f2fbc00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fabfff640 1 -- 172.21.15.28:0/2682861188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc00ccc0 con 0x7f2fc415e7d0 2024-09-10T20:43:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.246+0000 7f2fabfff640 1 -- 172.21.15.28:0/2682861188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fbc005020 con 0x7f2fc415e7d0 2024-09-10T20:43:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.247+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc41f2a90 con 0x7f2fc415e7d0 2024-09-10T20:43:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.247+0000 7f2fabfff640 1 -- 172.21.15.28:0/2682861188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc00ce60 con 0x7f2fc415e7d0 2024-09-10T20:43:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.247+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc41f3050 con 0x7f2fc415e7d0 2024-09-10T20:43:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.248+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc415cbd0 con 0x7f2fc415e7d0 2024-09-10T20:43:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.248+0000 7f2fabfff640 1 -- 172.21.15.28:0/2682861188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f2fbc005450 con 0x7f2fc415e7d0 2024-09-10T20:43:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.249+0000 7f2fabfff640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fb00783f0 0x7f2fb007a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.249+0000 7f2fc37fe640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fb00783f0 0x7f2fb007a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.249+0000 7f2fabfff640 1 -- 172.21.15.28:0/2682861188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7f2fbc09ad70 con 0x7f2fc415e7d0 2024-09-10T20:43:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.249+0000 7f2fc37fe640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fb00783f0 0x7f2fb007a8b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4034000 tx=0x7f2fb4004ab0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.253+0000 7f2fabfff640 1 -- 172.21.15.28:0/2682861188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2fbc063410 con 0x7f2fc415e7d0 2024-09-10T20:43:38.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.438+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2fc40b2540 con 0x7f2fc415e7d0 2024-09-10T20:43:38.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.438+0000 7f2fabfff640 1 -- 172.21.15.28:0/2682861188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f2fbc062d10 con 0x7f2fc415e7d0 2024-09-10T20:43:38.442 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:43:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fb00783f0 msgr2=0x7f2fb007a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fb00783f0 0x7f2fb007a8b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4034000 tx=0x7f2fb4004ab0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc415e7d0 msgr2=0x7f2fc4155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc415e7d0 0x7f2fc4155d30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00e940 tx=0x7f2fbc00ee00 comp rx=0 tx=0).stop 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 shutdown_connections 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2fb00783f0 0x7f2fb007a8b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc415e7d0 0x7f2fc4155d30 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2fc41587a0 0x7f2fc41557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 --2- 172.21.15.28:0/2682861188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2fc406fc20 0x7f2fc4152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 >> 172.21.15.28:0/2682861188 conn(0x7f2fc406d110 msgr2=0x7f2fc40ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 shutdown_connections 2024-09-10T20:43:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.440+0000 7f2fc9c33640 1 -- 172.21.15.28:0/2682861188 wait complete. 2024-09-10T20:43:38.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2' 2024-09-10T20:43:38.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.557+0000 7f381d34b640 1 -- 172.21.15.28:0/4229766086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3804004d40 msgr2=0x7f38040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.557+0000 7f381d34b640 1 --2- 172.21.15.28:0/4229766086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3804004d40 0x7f38040051c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f381000c610 tx=0x7f3804005830 comp rx=0 tx=0).stop 2024-09-10T20:43:38.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.557+0000 7f38177fe640 1 -- 172.21.15.28:0/4229766086 reap_dead start 2024-09-10T20:43:38.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.557+0000 7f381d34b640 1 -- 172.21.15.28:0/4229766086 shutdown_connections 2024-09-10T20:43:38.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.557+0000 7f381d34b640 1 -- 172.21.15.28:0/4229766086 >> 172.21.15.28:0/4229766086 conn(0x7f38180d0420 msgr2=0x7f38180d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:38.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.557+0000 7f381d34b640 1 -- 172.21.15.28:0/4229766086 shutdown_connections 2024-09-10T20:43:38.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.557+0000 7f381d34b640 1 -- 172.21.15.28:0/4229766086 wait complete. 2024-09-10T20:43:38.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.558+0000 7f381d34b640 1 Processor -- start 2024-09-10T20:43:38.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.558+0000 7f381d34b640 1 -- start start 2024-09-10T20:43:38.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f381d34b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3804004d40 0x7f381813cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f381d34b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3818158440 0x7f38180706c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f381d34b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38181588c0 0x7f38180b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f381d34b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3818073e70 con 0x7f38181588c0 2024-09-10T20:43:38.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f381d34b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3818073cf0 con 0x7f3818158440 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f381d34b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3818073ff0 con 0x7f3804004d40 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f38177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38181588c0 0x7f38180b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f38177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38181588c0 0x7f38180b0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53334/0 (socket says 172.21.15.28:53334) 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f38177fe640 1 -- 172.21.15.28:0/1954201707 learned_addr learned my addr 172.21.15.28:0/1954201707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f3816ffd640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3804004d40 0x7f381813cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.559+0000 7f380ffff640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3818158440 0x7f38180706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f38177fe640 1 -- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3804004d40 msgr2=0x7f381813cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f38177fe640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3804004d40 0x7f381813cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f38177fe640 1 -- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3818158440 msgr2=0x7f38180706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f38177fe640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3818158440 0x7f38180706c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f38177fe640 1 -- 172.21.15.28:0/1954201707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38181573a0 con 0x7f38181588c0 2024-09-10T20:43:38.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f3816ffd640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3804004d40 0x7f381813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f380ffff640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3818158440 0x7f38180706c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f38177fe640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38181588c0 0x7f38180b0bf0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f381002b4d0 tx=0x7f3810001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f380f7fe640 1 -- 172.21.15.28:0/1954201707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810039070 con 0x7f38181588c0 2024-09-10T20:43:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f380f7fe640 1 -- 172.21.15.28:0/1954201707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3810006680 con 0x7f38181588c0 2024-09-10T20:43:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3818157680 con 0x7f38181588c0 2024-09-10T20:43:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f380f7fe640 1 -- 172.21.15.28:0/1954201707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810002070 con 0x7f38181588c0 2024-09-10T20:43:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.560+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3818157c40 con 0x7f38181588c0 2024-09-10T20:43:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.561+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381815e7e0 con 0x7f38181588c0 2024-09-10T20:43:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.562+0000 7f380f7fe640 1 -- 172.21.15.28:0/1954201707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f3810046030 con 0x7f38181588c0 2024-09-10T20:43:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.562+0000 7f380f7fe640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37f80783f0 0x7f37f807a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.562+0000 7f380f7fe640 1 -- 172.21.15.28:0/1954201707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7f38100b9e60 con 0x7f38181588c0 2024-09-10T20:43:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.562+0000 7f3816ffd640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37f80783f0 0x7f37f807a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.563+0000 7f3816ffd640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37f80783f0 0x7f37f807a8b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3808009c50 tx=0x7f3808005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.566+0000 7f380f7fe640 1 -- 172.21.15.28:0/1954201707 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3810082740 con 0x7f38181588c0 2024-09-10T20:43:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:38.825+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]} v 0) -- 0x7f38180b5a80 con 0x7f38181588c0 2024-09-10T20:43:39.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.810+0000 7f380f7fe640 1 -- 172.21.15.28:0/1954201707 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]}]=0 v300) ==== 151+0+0 (secure 0 0 0) 0x7f3810082040 con 0x7f38181588c0 2024-09-10T20:43:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37f80783f0 msgr2=0x7f37f807a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37f80783f0 0x7f37f807a8b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3808009c50 tx=0x7f3808005f90 comp rx=0 tx=0).stop 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38181588c0 msgr2=0x7f38180b0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38181588c0 0x7f38180b0bf0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f381002b4d0 tx=0x7f3810001aa0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 shutdown_connections 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37f80783f0 0x7f37f807a8b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38181588c0 0x7f38180b0bf0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3818158440 0x7f38180706c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 --2- 172.21.15.28:0/1954201707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3804004d40 0x7f381813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.813+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 >> 172.21.15.28:0/1954201707 conn(0x7f38180d0420 msgr2=0x7f38180cbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.814+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 shutdown_connections 2024-09-10T20:43:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.814+0000 7f381d34b640 1 -- 172.21.15.28:0/1954201707 wait complete. 2024-09-10T20:43:39.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:43:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.938+0000 7f634cdf3640 1 -- 172.21.15.28:0/1852597636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 msgr2=0x7f63480d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.938+0000 7f634cdf3640 1 --2- 172.21.15.28:0/1852597636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f63480d0090 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f6330009900 tx=0x7f633002a610 comp rx=0 tx=0).stop 2024-09-10T20:43:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.939+0000 7f634cdf3640 1 -- 172.21.15.28:0/1852597636 shutdown_connections 2024-09-10T20:43:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.939+0000 7f634cdf3640 1 --2- 172.21.15.28:0/1852597636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f63480d0090 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.939+0000 7f634cdf3640 1 -- 172.21.15.28:0/1852597636 >> 172.21.15.28:0/1852597636 conn(0x7f634806cf00 msgr2=0x7f634806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.939+0000 7f634cdf3640 1 -- 172.21.15.28:0/1852597636 shutdown_connections 2024-09-10T20:43:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.939+0000 7f634cdf3640 1 -- 172.21.15.28:0/1852597636 wait complete. 2024-09-10T20:43:39.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.940+0000 7f634cdf3640 1 Processor -- start 2024-09-10T20:43:39.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.940+0000 7f634cdf3640 1 -- start start 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634cdf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f6348155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634cdf3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63480cd3d0 0x7f63480b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634cdf3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63480d05d0 0x7f63480b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634cdf3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63480b3590 con 0x7f63480cc570 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634cdf3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f63480b4b40 con 0x7f63480cd3d0 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634cdf3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f63480b3710 con 0x7f63480d05d0 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f6348155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f6348155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53344/0 (socket says 172.21.15.28:53344) 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 -- 172.21.15.28:0/721293261 learned_addr learned my addr 172.21.15.28:0/721293261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f6345d9b640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63480cd3d0 0x7f63480b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:39.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f6346d9d640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63480d05d0 0x7f63480b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 -- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63480d05d0 msgr2=0x7f63480b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63480d05d0 0x7f63480b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 -- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63480cd3d0 msgr2=0x7f63480b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63480cd3d0 0x7f63480b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 -- 172.21.15.28:0/721293261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63480b2630 con 0x7f63480cc570 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f6346d9d640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63480d05d0 0x7f63480b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.941+0000 7f634659c640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f6348155910 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f634000c7a0 tx=0x7f634000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.942+0000 7f633effd640 1 -- 172.21.15.28:0/721293261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340017070 con 0x7f63480cc570 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.942+0000 7f633effd640 1 -- 172.21.15.28:0/721293261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6340005070 con 0x7f63480cc570 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.942+0000 7f633effd640 1 -- 172.21.15.28:0/721293261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340005500 con 0x7f63480cc570 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.942+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6348153c30 con 0x7f63480cc570 2024-09-10T20:43:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.942+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63481541f0 con 0x7f63480cc570 2024-09-10T20:43:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.943+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63480b3b90 con 0x7f63480cc570 2024-09-10T20:43:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.943+0000 7f633effd640 1 -- 172.21.15.28:0/721293261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f634000a640 con 0x7f63480cc570 2024-09-10T20:43:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.944+0000 7f633effd640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63280783f0 0x7f632807a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.944+0000 7f633effd640 1 -- 172.21.15.28:0/721293261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7f6340099de0 con 0x7f63480cc570 2024-09-10T20:43:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.944+0000 7f6345d9b640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63280783f0 0x7f632807a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.944+0000 7f6345d9b640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63280783f0 0x7f632807a8b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6330033000 tx=0x7f6330004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:39.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:39.948+0000 7f633effd640 1 -- 172.21.15.28:0/721293261 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6340061da0 con 0x7f63480cc570 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.133+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f634806d6b0 con 0x7f63480cc570 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.134+0000 7f633effd640 1 -- 172.21.15.28:0/721293261 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v39) ==== 59+0+1998 (secure 0 0 0) 0x7f63400667d0 con 0x7f63480cc570 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:43:40.138 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:43:40.139 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:43:40.140 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:43:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63280783f0 msgr2=0x7f632807a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63280783f0 0x7f632807a8b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6330033000 tx=0x7f6330004730 comp rx=0 tx=0).stop 2024-09-10T20:43:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 msgr2=0x7f6348155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f6348155910 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f634000c7a0 tx=0x7f634000cc60 comp rx=0 tx=0).stop 2024-09-10T20:43:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 shutdown_connections 2024-09-10T20:43:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f63280783f0 0x7f632807a8b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f63480d05d0 0x7f63480b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f63480cd3d0 0x7f63480b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 --2- 172.21.15.28:0/721293261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63480cc570 0x7f6348155910 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 >> 172.21.15.28:0/721293261 conn(0x7f634806cf00 msgr2=0x7f6348075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 shutdown_connections 2024-09-10T20:43:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.136+0000 7f634cdf3640 1 -- 172.21.15.28:0/721293261 wait complete. 2024-09-10T20:43:40.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:43:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.256+0000 7fc76490e640 1 -- 172.21.15.28:0/1726968693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 msgr2=0x7fc74c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.256+0000 7fc76490e640 1 --2- 172.21.15.28:0/1726968693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 0x7fc74c00b9c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fc750006f90 tx=0x7fc74c005830 comp rx=0 tx=0).stop 2024-09-10T20:43:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.256+0000 7fc75ed9d640 1 -- 172.21.15.28:0/1726968693 reap_dead start 2024-09-10T20:43:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.256+0000 7fc76490e640 1 -- 172.21.15.28:0/1726968693 shutdown_connections 2024-09-10T20:43:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.256+0000 7fc76490e640 1 -- 172.21.15.28:0/1726968693 >> 172.21.15.28:0/1726968693 conn(0x7fc7600d1f30 msgr2=0x7fc7600d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.256+0000 7fc76490e640 1 -- 172.21.15.28:0/1726968693 shutdown_connections 2024-09-10T20:43:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.256+0000 7fc76490e640 1 -- 172.21.15.28:0/1726968693 wait complete. 2024-09-10T20:43:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.257+0000 7fc76490e640 1 Processor -- start 2024-09-10T20:43:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.257+0000 7fc76490e640 1 -- start start 2024-09-10T20:43:40.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc76490e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 0x7fc7601f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc76490e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7601f33d0 0x7fc7600c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc76490e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76009bc30 0x7fc760070da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc76490e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7600d1030 con 0x7fc74c0051c0 2024-09-10T20:43:40.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc76490e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc7600d0eb0 con 0x7fc7601f33d0 2024-09-10T20:43:40.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc76490e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc7600ccb20 con 0x7fc76009bc30 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 0x7fc7601f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 0x7fc7601f6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53366/0 (socket says 172.21.15.28:53366) 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 -- 172.21.15.28:0/823220418 learned_addr learned my addr 172.21.15.28:0/823220418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75ed9d640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76009bc30 0x7fc760070da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75dd9b640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7601f33d0 0x7fc7600c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 -- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76009bc30 msgr2=0x7fc760070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76009bc30 0x7fc760070da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 -- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7601f33d0 msgr2=0x7fc7600c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7601f33d0 0x7fc7600c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75e59c640 1 -- 172.21.15.28:0/823220418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7601f2670 con 0x7fc74c0051c0 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75ed9d640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76009bc30 0x7fc760070da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.258+0000 7fc75dd9b640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7601f33d0 0x7fc7600c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.259+0000 7fc75e59c640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 0x7fc7601f6670 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fc758006090 tx=0x7fc758006550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.259+0000 7fc756ffd640 1 -- 172.21.15.28:0/823220418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758015070 con 0x7fc74c0051c0 2024-09-10T20:43:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.259+0000 7fc756ffd640 1 -- 172.21.15.28:0/823220418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc75800fb50 con 0x7fc74c0051c0 2024-09-10T20:43:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.259+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7601f6bb0 con 0x7fc74c0051c0 2024-09-10T20:43:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.259+0000 7fc756ffd640 1 -- 172.21.15.28:0/823220418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758011430 con 0x7fc74c0051c0 2024-09-10T20:43:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.259+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7601f28f0 con 0x7fc74c0051c0 2024-09-10T20:43:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.260+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7600ce750 con 0x7fc74c0051c0 2024-09-10T20:43:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.260+0000 7fc756ffd640 1 -- 172.21.15.28:0/823220418 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fc7580115d0 con 0x7fc74c0051c0 2024-09-10T20:43:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.261+0000 7fc756ffd640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc7400783f0 0x7fc74007a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.261+0000 7fc756ffd640 1 -- 172.21.15.28:0/823220418 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7fc75809b330 con 0x7fc74c0051c0 2024-09-10T20:43:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.261+0000 7fc75dd9b640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc7400783f0 0x7fc74007a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.261+0000 7fc75dd9b640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc7400783f0 0x7fc74007a8b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc7601f5bc0 tx=0x7fc748005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.265+0000 7fc756ffd640 1 -- 172.21.15.28:0/823220418 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc7601f6bb0 con 0x7fc74c0051c0 2024-09-10T20:43:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.451+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc7600cc3d0 con 0x7fc74c0051c0 2024-09-10T20:43:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.455+0000 7fc756ffd640 1 -- 172.21.15.28:0/823220418 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7fc7601f2670 con 0x7fc74c0051c0 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc7400783f0 msgr2=0x7fc74007a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc7400783f0 0x7fc74007a8b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc7601f5bc0 tx=0x7fc748005f90 comp rx=0 tx=0).stop 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 msgr2=0x7fc7601f6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 0x7fc7601f6670 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fc758006090 tx=0x7fc758006550 comp rx=0 tx=0).stop 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 shutdown_connections 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc7400783f0 0x7fc74007a8b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76009bc30 0x7fc760070da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7601f33d0 0x7fc7600c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 --2- 172.21.15.28:0/823220418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc74c0051c0 0x7fc7601f6670 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 >> 172.21.15.28:0/823220418 conn(0x7fc7600d1f30 msgr2=0x7fc7600cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 shutdown_connections 2024-09-10T20:43:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.457+0000 7fc76490e640 1 -- 172.21.15.28:0/823220418 wait complete. 2024-09-10T20:43:40.472 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:43:40.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:43:40.518 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:43:40.559 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:43:40.559 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:40.559 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:40.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:43:40.575 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:43:40.575 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:40.617 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:43:40.617 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:40.617 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:40.633 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:43:40.633 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:43:40.633 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:40.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:43:40.674 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:43:40.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:40.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 -- 172.21.15.28:0/1698206798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35780d1060 msgr2=0x7f357815e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 --2- 172.21.15.28:0/1698206798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35780d1060 0x7f357815e6e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3568005fd0 tx=0x7f356802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 -- 172.21.15.28:0/1698206798 shutdown_connections 2024-09-10T20:43:40.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 --2- 172.21.15.28:0/1698206798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35780d1060 0x7f357815e6e0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 --2- 172.21.15.28:0/1698206798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35780d0c80 0x7f357815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 -- 172.21.15.28:0/1698206798 >> 172.21.15.28:0/1698206798 conn(0x7f357806dbf0 msgr2=0x7f357806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:40.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 -- 172.21.15.28:0/1698206798 shutdown_connections 2024-09-10T20:43:40.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.776+0000 7f3580008640 1 -- 172.21.15.28:0/1698206798 wait complete. 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.777+0000 7f3580008640 1 Processor -- start 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.777+0000 7f3580008640 1 -- start start 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f3580008640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35780d0c80 0x7f35781414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f3580008640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35780d1060 0x7f35780cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f3580008640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357815ec20 0x7f357809d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f3580008640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3578070990 con 0x7f357815ec20 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f3580008640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f35780b54a0 con 0x7f35780d1060 2024-09-10T20:43:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f3580008640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3578070b10 con 0x7f35780d0c80 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357815ec20 0x7f357809d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357815ec20 0x7f357809d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57996/0 (socket says 172.21.15.28:57996) 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 -- 172.21.15.28:0/1860331541 learned_addr learned my addr 172.21.15.28:0/1860331541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357dda4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35780d0c80 0x7f35781414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357d5a3640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35780d1060 0x7f35780cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 -- 172.21.15.28:0/1860331541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35780d0c80 msgr2=0x7f35781414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35780d0c80 0x7f35781414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 -- 172.21.15.28:0/1860331541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35780d1060 msgr2=0x7f35780cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35780d1060 0x7f35780cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.778+0000 7f357e5a5640 1 -- 172.21.15.28:0/1860331541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3578141a20 con 0x7f357815ec20 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.779+0000 7f357e5a5640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357815ec20 0x7f357809d550 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f357000c6b0 tx=0x7f357000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.779+0000 7f356e7fc640 1 -- 172.21.15.28:0/1860331541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570003b70 con 0x7f357815ec20 2024-09-10T20:43:40.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.779+0000 7f356e7fc640 1 -- 172.21.15.28:0/1860331541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3570003d30 con 0x7f357815ec20 2024-09-10T20:43:40.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.779+0000 7f356e7fc640 1 -- 172.21.15.28:0/1860331541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570005330 con 0x7f357815ec20 2024-09-10T20:43:40.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.779+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35780ce6e0 con 0x7f357815ec20 2024-09-10T20:43:40.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.779+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35780ceca0 con 0x7f357815ec20 2024-09-10T20:43:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.780+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f357815bc40 con 0x7f357815ec20 2024-09-10T20:43:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.781+0000 7f356e7fc640 1 -- 172.21.15.28:0/1860331541 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f35700054d0 con 0x7f357815ec20 2024-09-10T20:43:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.781+0000 7f356e7fc640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f355c078420 0x7f355c07a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.781+0000 7f357dda4640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f355c078420 0x7f355c07a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.782+0000 7f356e7fc640 1 -- 172.21.15.28:0/1860331541 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4971+0+0 (secure 0 0 0) 0x7f3570099df0 con 0x7f357815ec20 2024-09-10T20:43:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.782+0000 7f357dda4640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f355c078420 0x7f355c07a8e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3574005fd0 tx=0x7f357400b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:40.786+0000 7f356e7fc640 1 -- 172.21.15.28:0/1860331541 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35700057f0 con 0x7f357815ec20 2024-09-10T20:43:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.007+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35780d0a60 con 0x7f357815ec20 2024-09-10T20:43:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.007+0000 7f356e7fc640 1 -- 172.21.15.28:0/1860331541 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+3917 (secure 0 0 0) 0x7f3570061f70 con 0x7f357815ec20 2024-09-10T20:43:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:41.012 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":69,"btime":"2024-09-10T20:43:34:790115+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4277647396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4277647396},{"type":"v1","addr":"172.21.15.28:6834","nonce":4277647396}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/4032655811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":4032655811},{"type":"v1","addr":"172.21.15.159:6835","nonce":4032655811}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/3530092732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3530092732},{"type":"v1","addr":"172.21.15.159:6834","nonce":3530092732}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":7386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2586791093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2586791093},{"type":"v1","addr":"172.21.15.28:6835","nonce":2586791093}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[]} 2024-09-10T20:43:41.012 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 69 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.009+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f355c078420 msgr2=0x7f355c07a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.009+0000 7f3580008640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f355c078420 0x7f355c07a8e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3574005fd0 tx=0x7f357400b390 comp rx=0 tx=0).stop 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357815ec20 msgr2=0x7f357809d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357815ec20 0x7f357809d550 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f357000c6b0 tx=0x7f357000cb70 comp rx=0 tx=0).stop 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 shutdown_connections 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f355c078420 0x7f355c07a8e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357815ec20 0x7f357809d550 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35780d1060 0x7f35780cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 --2- 172.21.15.28:0/1860331541 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35780d0c80 0x7f35781414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 >> 172.21.15.28:0/1860331541 conn(0x7f357806dbf0 msgr2=0x7f3578152210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 shutdown_connections 2024-09-10T20:43:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:41.010+0000 7f3580008640 1 -- 172.21.15.28:0/1860331541 wait complete. 2024-09-10T20:43:41.024 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:43:41.024 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:43:41.025 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:43:41.025 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:43:41.025 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:43:41.025 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:43:41.025 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:43:41.025 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:43:41.026 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:43:41.026 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:43:41.026 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:43:41.026 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:43:41.026 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:43:41.026 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:43:41.026 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:43:41.026 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:43:41.035 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:43:41.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:43:41.041 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:43:41.041 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:43:41.042 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:43:41.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:43:41.078 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:43:41.078 DEBUG:teuthology.parallel:result is None 2024-09-10T20:43:41.083 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:43:41.084 DEBUG:teuthology.parallel:result is None 2024-09-10T20:43:41.084 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:43:41.085 DEBUG:teuthology.parallel:result is None 2024-09-10T20:43:41.154 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:43:41.167 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:43:41.167 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:43:45.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-09-10T20:43:47.036 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:43:47.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:43:47.039 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:43:47.039 DEBUG:teuthology.parallel:result is None 2024-09-10T20:43:47.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:43:47.128 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:43:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 -- 172.21.15.28:0/2667773773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 msgr2=0x7f9d00161fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 --2- 172.21.15.28:0/2667773773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00161fc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f9d0008c1a0 tx=0x7f9cf802aa30 comp rx=0 tx=0).stop 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 -- 172.21.15.28:0/2667773773 shutdown_connections 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 --2- 172.21.15.28:0/2667773773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00161fc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 --2- 172.21.15.28:0/2667773773 >> v2:172.21.15.159:3300/0 conn(0x7f9d000cf780 0x7f9d000cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 --2- 172.21.15.28:0/2667773773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0015de70 0x7f9d000d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 -- 172.21.15.28:0/2667773773 >> 172.21.15.28:0/2667773773 conn(0x7f9d000d3fb0 msgr2=0x7f9d000d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 -- 172.21.15.28:0/2667773773 shutdown_connections 2024-09-10T20:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.157+0000 7f9d05afe640 1 -- 172.21.15.28:0/2667773773 wait complete. 2024-09-10T20:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.158+0000 7f9d05afe640 1 Processor -- start 2024-09-10T20:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.159+0000 7f9d05afe640 1 -- start start 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.159+0000 7f9d05afe640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d000cf780 0x7f9d00153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.159+0000 7f9d05afe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0015de70 0x7f9d000cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.159+0000 7f9d05afe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.159+0000 7f9d05afe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d000b6e00 con 0x7f9d00161b50 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.159+0000 7f9d05afe640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0015e460 con 0x7f9d000cf780 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.159+0000 7f9d05afe640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9d000b6f80 con 0x7f9d0015de70 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58034/0 (socket says 172.21.15.28:58034) 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 -- 172.21.15.28:0/3726037065 learned_addr learned my addr 172.21.15.28:0/3726037065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfeffd640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0015de70 0x7f9d000cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d000cf780 0x7f9d00153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 -- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0015de70 msgr2=0x7f9d000cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0015de70 0x7f9d000cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 -- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d000cf780 msgr2=0x7f9d00153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d000cf780 0x7f9d00153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 -- 172.21.15.28:0/3726037065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d00154d10 con 0x7f9d00161b50 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d000cf780 0x7f9d00153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9cfffff640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00156d80 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009d60 tx=0x7f9cf8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3726037065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8033050 con 0x7f9d00161b50 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3726037065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cf8004450 con 0x7f9d00161b50 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3726037065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8038450 con 0x7f9d00161b50 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.160+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d00154ff0 con 0x7f9d00161b50 2024-09-10T20:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.161+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d000cb200 con 0x7f9d00161b50 2024-09-10T20:43:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.162+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d000cb400 con 0x7f9d00161b50 2024-09-10T20:43:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.162+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3726037065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f9cf8040070 con 0x7f9d00161b50 2024-09-10T20:43:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.163+0000 7f9ce7fff640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cec0783f0 0x7f9cec07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.163+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3726037065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 4971+0+0 (secure 0 0 0) 0x7f9cf80bb750 con 0x7f9d00161b50 2024-09-10T20:43:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.163+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cec0783f0 0x7f9cec07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.164+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cec0783f0 0x7f9cec07a8b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0005fd0 tx=0x7f9cf000b2c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.167+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3726037065 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9d000cb400 con 0x7f9d00161b50 2024-09-10T20:43:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:47.363+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f9d000b9f30 con 0x7f9d00161b50 2024-09-10T20:43:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.365+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3726037065 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v111) ==== 97+0+0 (secure 0 0 0) 0x7f9cf8083d20 con 0x7f9d00161b50 2024-09-10T20:43:48.369 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.368+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cec0783f0 msgr2=0x7f9cec07a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.368+0000 7f9d05afe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cec0783f0 0x7f9cec07a8b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0005fd0 tx=0x7f9cf000b2c0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.368+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 msgr2=0x7f9d00156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.368+0000 7f9d05afe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00156d80 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009d60 tx=0x7f9cf8003f10 comp rx=0 tx=0).stop 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 shutdown_connections 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cec0783f0 0x7f9cec07a8b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00161b50 0x7f9d00156d80 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9d0015de70 0x7f9d000cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 --2- 172.21.15.28:0/3726037065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9d000cf780 0x7f9d00153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 >> 172.21.15.28:0/3726037065 conn(0x7f9d000d3fb0 msgr2=0x7f9d000ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 shutdown_connections 2024-09-10T20:43:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.369+0000 7f9d05afe640 1 -- 172.21.15.28:0/3726037065 wait complete. 2024-09-10T20:43:48.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:43:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.496+0000 7fc239871640 1 -- 172.21.15.28:0/3434013099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c004d40 msgr2=0x7fc21c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.496+0000 7fc239871640 1 --2- 172.21.15.28:0/3434013099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c004d40 0x7fc21c0051c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009ed0 tx=0x7fc21c005830 comp rx=0 tx=0).stop 2024-09-10T20:43:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.496+0000 7fc2337fe640 1 -- 172.21.15.28:0/3434013099 reap_dead start 2024-09-10T20:43:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.496+0000 7fc239871640 1 -- 172.21.15.28:0/3434013099 shutdown_connections 2024-09-10T20:43:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.496+0000 7fc239871640 1 -- 172.21.15.28:0/3434013099 >> 172.21.15.28:0/3434013099 conn(0x7fc2340d0420 msgr2=0x7fc2340d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.496+0000 7fc239871640 1 -- 172.21.15.28:0/3434013099 shutdown_connections 2024-09-10T20:43:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.496+0000 7fc239871640 1 -- 172.21.15.28:0/3434013099 wait complete. 2024-09-10T20:43:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.497+0000 7fc239871640 1 Processor -- start 2024-09-10T20:43:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc239871640 1 -- start start 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc239871640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c004d40 0x7fc2341500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc239871640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc234150600 0x7fc2341f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc239871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2341f6910 0x7fc23406fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc239871640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc23415f5b0 con 0x7fc2341f6910 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc239871640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc23415f430 con 0x7fc234150600 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc239871640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc2340cb210 con 0x7fc21c004d40 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.498+0000 7fc2337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2341f6910 0x7fc23406fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2341f6910 0x7fc23406fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58052/0 (socket says 172.21.15.28:58052) 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 -- 172.21.15.28:0/1627319524 learned_addr learned my addr 172.21.15.28:0/1627319524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:48.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2327fc640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc234150600 0x7fc2341f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc232ffd640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c004d40 0x7fc2341500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 -- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c004d40 msgr2=0x7fc2341500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c004d40 0x7fc2341500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 -- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc234150600 msgr2=0x7fc2341f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc234150600 0x7fc2341f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 -- 172.21.15.28:0/1627319524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23414ec30 con 0x7fc2341f6910 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc2337fe640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2341f6910 0x7fc23406fe80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc22c02b860 tx=0x7fc22c033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc21bfff640 1 -- 172.21.15.28:0/1627319524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22c042070 con 0x7fc2341f6910 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc21bfff640 1 -- 172.21.15.28:0/1627319524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc22c003970 con 0x7fc2341f6910 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23414ef10 con 0x7fc2341f6910 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc21bfff640 1 -- 172.21.15.28:0/1627319524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22c038040 con 0x7fc2341f6910 2024-09-10T20:43:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.499+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc23414f4d0 con 0x7fc2341f6910 2024-09-10T20:43:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.500+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc23415e7e0 con 0x7fc2341f6910 2024-09-10T20:43:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.501+0000 7fc21bfff640 1 -- 172.21.15.28:0/1627319524 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fc22c03b070 con 0x7fc2341f6910 2024-09-10T20:43:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.502+0000 7fc21bfff640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc220078420 0x7fc22007a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.502+0000 7fc21bfff640 1 -- 172.21.15.28:0/1627319524 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4885+0+0 (secure 0 0 0) 0x7fc22c0ba280 con 0x7fc2341f6910 2024-09-10T20:43:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.502+0000 7fc232ffd640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc220078420 0x7fc22007a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.502+0000 7fc232ffd640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc220078420 0x7fc22007a8e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2240018a0 tx=0x7fc224006040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.506+0000 7fc21bfff640 1 -- 172.21.15.28:0/1627319524 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc22c082280 con 0x7fc2341f6910 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.761+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc234068d10 con 0x7fc2341f6910 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.761+0000 7fc21bfff640 1 -- 172.21.15.28:0/1627319524 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v32) ==== 80+0+5304 (secure 0 0 0) 0x7fc22c086cb0 con 0x7fc2341f6910 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:43:48.765 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.766 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:43:48.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:43:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.770 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.772 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:43:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.776 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:43:48.777 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:43:48.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:43:48.779 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:48.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc220078420 msgr2=0x7fc22007a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc220078420 0x7fc22007a8e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2240018a0 tx=0x7fc224006040 comp rx=0 tx=0).stop 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2341f6910 msgr2=0x7fc23406fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2341f6910 0x7fc23406fe80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc22c02b860 tx=0x7fc22c033c10 comp rx=0 tx=0).stop 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 shutdown_connections 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc220078420 0x7fc22007a8e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2341f6910 0x7fc23406fe80 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc234150600 0x7fc2341f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 --2- 172.21.15.28:0/1627319524 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc21c004d40 0x7fc2341500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 >> 172.21.15.28:0/1627319524 conn(0x7fc2340d0420 msgr2=0x7fc2340cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.763+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 shutdown_connections 2024-09-10T20:43:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.764+0000 7fc239871640 1 -- 172.21.15.28:0/1627319524 wait complete. 2024-09-10T20:43:48.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 -- 172.21.15.28:0/2313142617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff48015ed20 msgr2=0x7ff4800cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 --2- 172.21.15.28:0/2313142617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff48015ed20 0x7ff4800cb210 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7ff480074830 tx=0x7ff47802aa30 comp rx=0 tx=0).stop 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 -- 172.21.15.28:0/2313142617 shutdown_connections 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 --2- 172.21.15.28:0/2313142617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff48015ed20 0x7ff4800cb210 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 --2- 172.21.15.28:0/2313142617 >> v2:172.21.15.159:3300/0 conn(0x7ff4800cf0f0 0x7ff48015e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 -- 172.21.15.28:0/2313142617 >> 172.21.15.28:0/2313142617 conn(0x7ff4800d0420 msgr2=0x7ff4800d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 -- 172.21.15.28:0/2313142617 shutdown_connections 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.884+0000 7ff4874ed640 1 -- 172.21.15.28:0/2313142617 wait complete. 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.885+0000 7ff4874ed640 1 Processor -- start 2024-09-10T20:43:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.885+0000 7ff4874ed640 1 -- start start 2024-09-10T20:43:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff4874ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4800cf0f0 0x7ff4801f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff4874ed640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff480151cc0 0x7ff48006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff4874ed640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff48015ed20 0x7ff48006f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff4874ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff48015f430 con 0x7ff4800cf0f0 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff4874ed640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff480152120 con 0x7ff480151cc0 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff4874ed640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff48015f5b0 con 0x7ff48015ed20 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4800cf0f0 0x7ff4801f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4800cf0f0 0x7ff4801f1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58078/0 (socket says 172.21.15.28:58078) 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 -- 172.21.15.28:0/3914589606 learned_addr learned my addr 172.21.15.28:0/3914589606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485a8a640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff48015ed20 0x7ff48006f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff484a88640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff480151cc0 0x7ff48006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 -- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff48015ed20 msgr2=0x7ff48006f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff48015ed20 0x7ff48006f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 -- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff480151cc0 msgr2=0x7ff48006ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff480151cc0 0x7ff48006ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 -- 172.21.15.28:0/3914589606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4801588a0 con 0x7ff4800cf0f0 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.886+0000 7ff485289640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4800cf0f0 0x7ff4801f1f50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00c430 tx=0x7ff47c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.887+0000 7ff475ffb640 1 -- 172.21.15.28:0/3914589606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c011790 con 0x7ff4800cf0f0 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.887+0000 7ff475ffb640 1 -- 172.21.15.28:0/3914589606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff47c017070 con 0x7ff4800cf0f0 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.887+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff480158b80 con 0x7ff4800cf0f0 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.887+0000 7ff475ffb640 1 -- 172.21.15.28:0/3914589606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c011d00 con 0x7ff4800cf0f0 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.887+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4800b0d40 con 0x7ff4800cf0f0 2024-09-10T20:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.888+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48015e7e0 con 0x7ff4800cf0f0 2024-09-10T20:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.888+0000 7ff475ffb640 1 -- 172.21.15.28:0/3914589606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7ff47c01c460 con 0x7ff4800cf0f0 2024-09-10T20:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.889+0000 7ff475ffb640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff4640783f0 0x7ff46407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.889+0000 7ff475ffb640 1 -- 172.21.15.28:0/3914589606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4885+0+0 (secure 0 0 0) 0x7ff47c099f10 con 0x7ff4800cf0f0 2024-09-10T20:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.889+0000 7ff484a88640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff4640783f0 0x7ff46407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.889+0000 7ff484a88640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff4640783f0 0x7ff46407a8b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff4801f5a40 tx=0x7ff47000b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:48.893+0000 7ff475ffb640 1 -- 172.21.15.28:0/3914589606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff47c0620a0 con 0x7ff4800cf0f0 2024-09-10T20:43:49.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.081+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff48015e2b0 con 0x7ff4800cf0f0 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.082+0000 7ff475ffb640 1 -- 172.21.15.28:0/3914589606 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v111) ==== 84+0+65 (secure 0 0 0) 0x7ff47c09b090 con 0x7ff4800cf0f0 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:43:49.086 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:43:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff4640783f0 msgr2=0x7ff46407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff4640783f0 0x7ff46407a8b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff4801f5a40 tx=0x7ff47000b390 comp rx=0 tx=0).stop 2024-09-10T20:43:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4800cf0f0 msgr2=0x7ff4801f1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4800cf0f0 0x7ff4801f1f50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00c430 tx=0x7ff47c00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 shutdown_connections 2024-09-10T20:43:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff4640783f0 0x7ff46407a8b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff48015ed20 0x7ff48006f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff480151cc0 0x7ff48006ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 --2- 172.21.15.28:0/3914589606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4800cf0f0 0x7ff4801f1f50 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.084+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 >> 172.21.15.28:0/3914589606 conn(0x7ff4800d0420 msgr2=0x7ff4800cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.085+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 shutdown_connections 2024-09-10T20:43:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.085+0000 7ff4874ed640 1 -- 172.21.15.28:0/3914589606 wait complete. 2024-09-10T20:43:49.100 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:43:49.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:43:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.206+0000 7f704cda3640 1 -- 172.21.15.28:0/3287994292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 msgr2=0x7f70480cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.206+0000 7f704cda3640 1 --2- 172.21.15.28:0/3287994292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70480cde80 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f7040005fd0 tx=0x7f704002aa10 comp rx=0 tx=0).stop 2024-09-10T20:43:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.206+0000 7f704cda3640 1 -- 172.21.15.28:0/3287994292 shutdown_connections 2024-09-10T20:43:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.206+0000 7f704cda3640 1 --2- 172.21.15.28:0/3287994292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70480cde80 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.206+0000 7f704cda3640 1 -- 172.21.15.28:0/3287994292 >> 172.21.15.28:0/3287994292 conn(0x7f70480cfc80 msgr2=0x7f70480d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.206+0000 7f704cda3640 1 -- 172.21.15.28:0/3287994292 shutdown_connections 2024-09-10T20:43:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.206+0000 7f704cda3640 1 -- 172.21.15.28:0/3287994292 wait complete. 2024-09-10T20:43:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.207+0000 7f704cda3640 1 Processor -- start 2024-09-10T20:43:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.207+0000 7f704cda3640 1 -- start start 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f704cda3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70480cabc0 0x7f70480b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f704cda3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70481f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f704cda3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704815a350 0x7f7048152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f704cda3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70480729a0 con 0x7f70480cda70 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f704cda3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7048072820 con 0x7f70480cabc0 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f704cda3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7048072b20 con 0x7f704815a350 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f7045d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70481f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f7045d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70481f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58092/0 (socket says 172.21.15.28:58092) 2024-09-10T20:43:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f7045d9b640 1 -- 172.21.15.28:0/4030594435 learned_addr learned my addr 172.21.15.28:0/4030594435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f7046d9d640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704815a350 0x7f7048152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f704659c640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70480cabc0 0x7f70480b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.208+0000 7f7045d9b640 1 -- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704815a350 msgr2=0x7f7048152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7045d9b640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704815a350 0x7f7048152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7045d9b640 1 -- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70480cabc0 msgr2=0x7f70480b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7045d9b640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70480cabc0 0x7f70480b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7045d9b640 1 -- 172.21.15.28:0/4030594435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70480c94c0 con 0x7f70480cda70 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7045d9b640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70481f3750 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f703c00c430 tx=0x7f703c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7036ffd640 1 -- 172.21.15.28:0/4030594435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c011790 con 0x7f70480cda70 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7036ffd640 1 -- 172.21.15.28:0/4030594435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f703c017070 con 0x7f70480cda70 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70481f3c90 con 0x7f70480cda70 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f7036ffd640 1 -- 172.21.15.28:0/4030594435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c011d00 con 0x7f70480cda70 2024-09-10T20:43:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.209+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70480c9730 con 0x7f70480cda70 2024-09-10T20:43:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.210+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70481f41f0 con 0x7f70480cda70 2024-09-10T20:43:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.211+0000 7f7036ffd640 1 -- 172.21.15.28:0/4030594435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f703c01c460 con 0x7f70480cda70 2024-09-10T20:43:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.211+0000 7f7036ffd640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f70280783f0 0x7f702807a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.211+0000 7f704659c640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f70280783f0 0x7f702807a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.211+0000 7f7036ffd640 1 -- 172.21.15.28:0/4030594435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4885+0+0 (secure 0 0 0) 0x7f703c099fe0 con 0x7f70480cda70 2024-09-10T20:43:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.212+0000 7f704659c640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f70280783f0 0x7f702807a8b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f70480c93b0 tx=0x7f7038009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.216+0000 7f7036ffd640 1 -- 172.21.15.28:0/4030594435 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f70481f41f0 con 0x7f70480cda70 2024-09-10T20:43:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:49.406+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f70480cde80 con 0x7f70480cda70 2024-09-10T20:43:50.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.377+0000 7f7036ffd640 1 -- 172.21.15.28:0/4030594435 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v112) ==== 138+0+0 (secure 0 0 0) 0x7f70481f3c90 con 0x7f70480cda70 2024-09-10T20:43:50.381 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f70280783f0 msgr2=0x7f702807a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f70280783f0 0x7f702807a8b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f70480c93b0 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 msgr2=0x7f70481f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70481f3750 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f703c00c430 tx=0x7f703c00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 shutdown_connections 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f70280783f0 0x7f702807a8b0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f70480c93b0 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f704815a350 0x7f7048152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70480cda70 0x7f70481f3750 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 --2- 172.21.15.28:0/4030594435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f70480cabc0 0x7f70480b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 >> 172.21.15.28:0/4030594435 conn(0x7f70480cfc80 msgr2=0x7f70480c5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 shutdown_connections 2024-09-10T20:43:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.381+0000 7f704cda3640 1 -- 172.21.15.28:0/4030594435 wait complete. 2024-09-10T20:43:50.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3561691449 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae015e130 msgr2=0x7f9ae00d4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3561691449 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae015e130 0x7f9ae00d4b60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0005fd0 tx=0x7f9ad002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3561691449 shutdown_connections 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3561691449 >> v2:172.21.15.159:3300/0 conn(0x7f9ae00b0c90 0x7f9ae00cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3561691449 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae015e130 0x7f9ae00d4b60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3561691449 >> 172.21.15.28:0/3561691449 conn(0x7f9ae00d4040 msgr2=0x7f9ae00d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3561691449 shutdown_connections 2024-09-10T20:43:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.519+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3561691449 wait complete. 2024-09-10T20:43:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.520+0000 7f9ae5b6c640 1 Processor -- start 2024-09-10T20:43:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.520+0000 7f9ae5b6c640 1 -- start start 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9ae5b6c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae00b0c90 0x7f9ae00b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9ae5b6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae00b10d0 0x7f9ae00c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9ae5b6c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ae015e130 0x7f9ae00c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9ae5b6c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae00d1160 con 0x7f9ae00b10d0 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9ae5b6c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae00cfe00 con 0x7f9ae015e130 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9ae5b6c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae00d12e0 con 0x7f9ae00b0c90 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae00b10d0 0x7f9ae00c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adf7fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae00b0c90 0x7f9ae00b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae00b10d0 0x7f9ae00c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53044/0 (socket says 172.21.15.28:53044) 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 -- 172.21.15.28:0/3880003540 learned_addr learned my addr 172.21.15.28:0/3880003540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adffff640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ae015e130 0x7f9ae00c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 -- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae00b0c90 msgr2=0x7f9ae00b90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae00b0c90 0x7f9ae00b90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 -- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ae015e130 msgr2=0x7f9ae00c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ae015e130 0x7f9ae00c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adeffd640 1 -- 172.21.15.28:0/3880003540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae00c8c60 con 0x7f9ae00b10d0 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.521+0000 7f9adf7fe640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae00b0c90 0x7f9ae00b90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.522+0000 7f9adeffd640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae00b10d0 0x7f9ae00c8720 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400e9c0 tx=0x7f9ad400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.522+0000 7f9ac7fff640 1 -- 172.21.15.28:0/3880003540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad400cd80 con 0x7f9ae00b10d0 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.522+0000 7f9ac7fff640 1 -- 172.21.15.28:0/3880003540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ad4004030 con 0x7f9ae00b10d0 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.522+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae01f4ce0 con 0x7f9ae00b10d0 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.522+0000 7f9ac7fff640 1 -- 172.21.15.28:0/3880003540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4004480 con 0x7f9ae00b10d0 2024-09-10T20:43:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.522+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae01f52a0 con 0x7f9ae00b10d0 2024-09-10T20:43:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.523+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae00c9c20 con 0x7f9ae00b10d0 2024-09-10T20:43:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.524+0000 7f9ac7fff640 1 -- 172.21.15.28:0/3880003540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f9ad40035f0 con 0x7f9ae00b10d0 2024-09-10T20:43:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.524+0000 7f9ac7fff640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9acc0783f0 0x7f9acc07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.524+0000 7f9ac7fff640 1 -- 172.21.15.28:0/3880003540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5255+0+0 (secure 0 0 0) 0x7f9ad409aeb0 con 0x7f9ae00b10d0 2024-09-10T20:43:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.524+0000 7f9adf7fe640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9acc0783f0 0x7f9acc07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.525+0000 7f9adf7fe640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9acc0783f0 0x7f9acc07a8b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0033000 tx=0x7f9ad0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.529+0000 7f9ac7fff640 1 -- 172.21.15.28:0/3880003540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ad4062d60 con 0x7f9ae00b10d0 2024-09-10T20:43:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:50.714+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9ae00cf540 con 0x7f9ae00b10d0 2024-09-10T20:43:51.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.379+0000 7f9ac7fff640 1 -- 172.21.15.28:0/3880003540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v113) ==== 170+0+0 (secure 0 0 0) 0x7f9ad4067790 con 0x7f9ae00b10d0 2024-09-10T20:43:51.383 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:43:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9acc0783f0 msgr2=0x7f9acc07a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9acc0783f0 0x7f9acc07a8b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0033000 tx=0x7f9ad0004730 comp rx=0 tx=0).stop 2024-09-10T20:43:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae00b10d0 msgr2=0x7f9ae00c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae00b10d0 0x7f9ae00c8720 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400e9c0 tx=0x7f9ad400ee80 comp rx=0 tx=0).stop 2024-09-10T20:43:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 shutdown_connections 2024-09-10T20:43:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9acc0783f0 0x7f9acc07a8b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ae015e130 0x7f9ae00c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae00b10d0 0x7f9ae00c8720 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 --2- 172.21.15.28:0/3880003540 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9ae00b0c90 0x7f9ae00b90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 >> 172.21.15.28:0/3880003540 conn(0x7f9ae00d4040 msgr2=0x7f9ae00b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 shutdown_connections 2024-09-10T20:43:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.383+0000 7f9ae5b6c640 1 -- 172.21.15.28:0/3880003540 wait complete. 2024-09-10T20:43:51.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:43:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 -- 172.21.15.28:0/419644709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3900b3580 msgr2=0x7fd39006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 --2- 172.21.15.28:0/419644709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3900b3580 0x7fd39006e400 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd39015d260 tx=0x7fd37c02a560 comp rx=0 tx=0).stop 2024-09-10T20:43:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 -- 172.21.15.28:0/419644709 shutdown_connections 2024-09-10T20:43:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 --2- 172.21.15.28:0/419644709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3900b3580 0x7fd39006e400 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 --2- 172.21.15.28:0/419644709 >> v2:172.21.15.159:3300/0 conn(0x7fd39006e940 0x7fd3900ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 -- 172.21.15.28:0/419644709 >> 172.21.15.28:0/419644709 conn(0x7fd39006d110 msgr2=0x7fd3900b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 -- 172.21.15.28:0/419644709 shutdown_connections 2024-09-10T20:43:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.525+0000 7fd397047640 1 -- 172.21.15.28:0/419644709 wait complete. 2024-09-10T20:43:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.526+0000 7fd397047640 1 Processor -- start 2024-09-10T20:43:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.526+0000 7fd397047640 1 -- start start 2024-09-10T20:43:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd397047640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd39006e940 0x7fd390142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd397047640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd390152e00 0x7fd3901f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd397047640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3901f81a0 0x7fd39014f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd397047640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd39015f820 con 0x7fd390152e00 2024-09-10T20:43:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd397047640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd3900cf5b0 con 0x7fd3901f81a0 2024-09-10T20:43:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd397047640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd39015f9a0 con 0x7fd39006e940 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd38ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd390152e00 0x7fd3901f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd38ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd390152e00 0x7fd3901f7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53060/0 (socket says 172.21.15.28:53060) 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd38ffff640 1 -- 172.21.15.28:0/2185610136 learned_addr learned my addr 172.21.15.28:0/2185610136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd394de3640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd39006e940 0x7fd390142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.527+0000 7fd3955e4640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3901f81a0 0x7fd39014f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38ffff640 1 -- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd39006e940 msgr2=0x7fd390142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38ffff640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd39006e940 0x7fd390142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38ffff640 1 -- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3901f81a0 msgr2=0x7fd39014f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38ffff640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3901f81a0 0x7fd39014f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38ffff640 1 -- 172.21.15.28:0/2185610136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd390143180 con 0x7fd390152e00 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38ffff640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd390152e00 0x7fd3901f7c60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009850 tx=0x7fd37c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38d7fa640 1 -- 172.21.15.28:0/2185610136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c03f070 con 0x7fd390152e00 2024-09-10T20:43:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38d7fa640 1 -- 172.21.15.28:0/2185610136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd37c0047a0 con 0x7fd390152e00 2024-09-10T20:43:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd38d7fa640 1 -- 172.21.15.28:0/2185610136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c035030 con 0x7fd390152e00 2024-09-10T20:43:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3901f86b0 con 0x7fd390152e00 2024-09-10T20:43:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.528+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd390142440 con 0x7fd390152e00 2024-09-10T20:43:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.529+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd39013ff10 con 0x7fd390152e00 2024-09-10T20:43:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.530+0000 7fd38d7fa640 1 -- 172.21.15.28:0/2185610136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fd37c004bb0 con 0x7fd390152e00 2024-09-10T20:43:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.531+0000 7fd38d7fa640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd3740783f0 0x7fd37407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.531+0000 7fd38d7fa640 1 -- 172.21.15.28:0/2185610136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5637+0+0 (secure 0 0 0) 0x7fd37c0bbb80 con 0x7fd390152e00 2024-09-10T20:43:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.531+0000 7fd394de3640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd3740783f0 0x7fd37407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.531+0000 7fd394de3640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd3740783f0 0x7fd37407a8b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd384001720 tx=0x7fd384009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.535+0000 7fd38d7fa640 1 -- 172.21.15.28:0/2185610136 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd39013ff10 con 0x7fd390152e00 2024-09-10T20:43:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:51.762+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd39015fec0 con 0x7fd390152e00 2024-09-10T20:43:51.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-09-10T20:43:52.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.402+0000 7fd38d7fa640 1 -- 172.21.15.28:0/2185610136 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v73) ==== 174+0+0 (secure 0 0 0) 0x7fd37c0839a0 con 0x7fd390152e00 2024-09-10T20:43:52.410 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 18 and data pool 19 2024-09-10T20:43:52.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.421+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd3740783f0 msgr2=0x7fd37407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.421+0000 7fd397047640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd3740783f0 0x7fd37407a8b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd384001720 tx=0x7fd384009040 comp rx=0 tx=0).stop 2024-09-10T20:43:52.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.421+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd390152e00 msgr2=0x7fd3901f7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.421+0000 7fd397047640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd390152e00 0x7fd3901f7c60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009850 tx=0x7fd37c003910 comp rx=0 tx=0).stop 2024-09-10T20:43:52.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 shutdown_connections 2024-09-10T20:43:52.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd3740783f0 0x7fd37407a8b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3901f81a0 0x7fd39014f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd390152e00 0x7fd3901f7c60 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 --2- 172.21.15.28:0/2185610136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd39006e940 0x7fd390142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 >> 172.21.15.28:0/2185610136 conn(0x7fd39006d110 msgr2=0x7fd3900d05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:52.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 shutdown_connections 2024-09-10T20:43:52.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.422+0000 7fd397047640 1 -- 172.21.15.28:0/2185610136 wait complete. 2024-09-10T20:43:52.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:43:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.551+0000 7f701a4dd640 1 -- 172.21.15.28:0/2125559098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701415c050 msgr2=0x7f6ffc0039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.551+0000 7f701a4dd640 1 --2- 172.21.15.28:0/2125559098 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701415c050 0x7f6ffc0039e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7004009b20 tx=0x7f6ffc005830 comp rx=0 tx=0).stop 2024-09-10T20:43:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.551+0000 7f7018a7a640 1 -- 172.21.15.28:0/2125559098 reap_dead start 2024-09-10T20:43:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.551+0000 7f701a4dd640 1 -- 172.21.15.28:0/2125559098 shutdown_connections 2024-09-10T20:43:52.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.551+0000 7f701a4dd640 1 -- 172.21.15.28:0/2125559098 >> 172.21.15.28:0/2125559098 conn(0x7f70140d1e40 msgr2=0x7f70140d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:52.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.551+0000 7f701a4dd640 1 -- 172.21.15.28:0/2125559098 shutdown_connections 2024-09-10T20:43:52.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.551+0000 7f701a4dd640 1 -- 172.21.15.28:0/2125559098 wait complete. 2024-09-10T20:43:52.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.552+0000 7f701a4dd640 1 Processor -- start 2024-09-10T20:43:52.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.552+0000 7f701a4dd640 1 -- start start 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f701a4dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701415c050 0x7f70141f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f701a4dd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701414f1f0 0x7f70140c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f701a4dd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f701406ef10 0x7f701414c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f701a4dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70140cfda0 con 0x7f701415c050 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f701a4dd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f70140cfc20 con 0x7f701406ef10 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f701a4dd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f70140cff20 con 0x7f701414f1f0 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f7013fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701415c050 0x7f70141f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f7013fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701415c050 0x7f70141f7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53078/0 (socket says 172.21.15.28:53078) 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.553+0000 7f7013fff640 1 -- 172.21.15.28:0/1240523055 learned_addr learned my addr 172.21.15.28:0/1240523055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7013fff640 1 -- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701414f1f0 msgr2=0x7f70140c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7013fff640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701414f1f0 0x7f70140c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7013fff640 1 -- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f701406ef10 msgr2=0x7f701414c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7018a7a640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f701406ef10 0x7f701414c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7013fff640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f701406ef10 0x7f701414c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7013fff640 1 -- 172.21.15.28:0/1240523055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70140cc1c0 con 0x7f701415c050 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7018a7a640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f701406ef10 0x7f701414c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7013fff640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701415c050 0x7f70141f7c10 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7004009a70 tx=0x7f7004033650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7010ff9640 1 -- 172.21.15.28:0/1240523055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7004038040 con 0x7f701415c050 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7010ff9640 1 -- 172.21.15.28:0/1240523055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7004041070 con 0x7f701415c050 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f7010ff9640 1 -- 172.21.15.28:0/1240523055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7004033d10 con 0x7f701415c050 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70140cbc40 con 0x7f701415c050 2024-09-10T20:43:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.554+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f701406e710 con 0x7f701415c050 2024-09-10T20:43:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.555+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70140b1de0 con 0x7f701415c050 2024-09-10T20:43:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.556+0000 7f7010ff9640 1 -- 172.21.15.28:0/1240523055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f700403b070 con 0x7f701415c050 2024-09-10T20:43:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.556+0000 7f7010ff9640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7000078420 0x7f700007a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.556+0000 7f7010ff9640 1 -- 172.21.15.28:0/1240523055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5737+0+0 (secure 0 0 0) 0x7f70040baa90 con 0x7f701415c050 2024-09-10T20:43:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.556+0000 7f70137fe640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7000078420 0x7f700007a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.557+0000 7f70137fe640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7000078420 0x7f700007a8e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7008006850 tx=0x7f70080067e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:52.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.561+0000 7f7010ff9640 1 -- 172.21.15.28:0/1240523055 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7004082e10 con 0x7f701415c050 2024-09-10T20:43:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.747+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70140b3300 con 0x7f701415c050 2024-09-10T20:43:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.748+0000 7f7010ff9640 1 -- 172.21.15.28:0/1240523055 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+16744 (secure 0 0 0) 0x7f7004082710 con 0x7f701415c050 2024-09-10T20:43:52.752 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:52.752 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:43:51.763858+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:43:49.411437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-10T20:43:50.719111+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:43:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7000078420 msgr2=0x7f700007a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7000078420 0x7f700007a8e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7008006850 tx=0x7f70080067e0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701415c050 msgr2=0x7f70141f7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701415c050 0x7f70141f7c10 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7004009a70 tx=0x7f7004033650 comp rx=0 tx=0).stop 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 shutdown_connections 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7000078420 0x7f700007a8e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f701406ef10 0x7f701414c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f701414f1f0 0x7f70140c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 --2- 172.21.15.28:0/1240523055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701415c050 0x7f70141f7c10 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 >> 172.21.15.28:0/1240523055 conn(0x7f70140d1e40 msgr2=0x7f70140b53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 shutdown_connections 2024-09-10T20:43:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.750+0000 7f701a4dd640 1 -- 172.21.15.28:0/1240523055 wait complete. 2024-09-10T20:43:52.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:43:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.867+0000 7fb4f2e47640 1 -- 172.21.15.28:0/922654262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 msgr2=0x7fb4dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.867+0000 7fb4f2e47640 1 --2- 172.21.15.28:0/922654262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 0x7fb4dc004c80 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800c310 tx=0x7fb4dc005830 comp rx=0 tx=0).stop 2024-09-10T20:43:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.867+0000 7fb4f13e4640 1 -- 172.21.15.28:0/922654262 reap_dead start 2024-09-10T20:43:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.867+0000 7fb4f2e47640 1 -- 172.21.15.28:0/922654262 shutdown_connections 2024-09-10T20:43:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.867+0000 7fb4f2e47640 1 -- 172.21.15.28:0/922654262 >> 172.21.15.28:0/922654262 conn(0x7fb4ec06dbf0 msgr2=0x7fb4ec06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.867+0000 7fb4f2e47640 1 -- 172.21.15.28:0/922654262 shutdown_connections 2024-09-10T20:43:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.867+0000 7fb4f2e47640 1 -- 172.21.15.28:0/922654262 wait complete. 2024-09-10T20:43:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.868+0000 7fb4f2e47640 1 Processor -- start 2024-09-10T20:43:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.868+0000 7fb4f2e47640 1 -- start start 2024-09-10T20:43:52.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f2e47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 0x7fb4ec09e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f2e47640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4ec14eff0 0x7fb4ec154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f2e47640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4ec154f50 0x7fb4ec157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f2e47640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ec164990 con 0x7fb4dc004800 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f2e47640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ec164810 con 0x7fb4ec154f50 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f2e47640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb4ec164b10 con 0x7fb4ec14eff0 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f0be3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 0x7fb4ec09e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f0be3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 0x7fb4ec09e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53088/0 (socket says 172.21.15.28:53088) 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.869+0000 7fb4f0be3640 1 -- 172.21.15.28:0/241984263 learned_addr learned my addr 172.21.15.28:0/241984263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4ebfff640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4ec14eff0 0x7fb4ec154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f0be3640 1 -- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4ec14eff0 msgr2=0x7fb4ec154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f0be3640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4ec14eff0 0x7fb4ec154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f13e4640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4ec154f50 0x7fb4ec157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f0be3640 1 -- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4ec154f50 msgr2=0x7fb4ec157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f0be3640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4ec154f50 0x7fb4ec157400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f0be3640 1 -- 172.21.15.28:0/241984263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ec151230 con 0x7fb4dc004800 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4ebfff640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4ec14eff0 0x7fb4ec154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f0be3640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 0x7fb4ec09e120 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0004df0 tx=0x7fb4e0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4e97fa640 1 -- 172.21.15.28:0/241984263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0001d40 con 0x7fb4dc004800 2024-09-10T20:43:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4e97fa640 1 -- 172.21.15.28:0/241984263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4e0016070 con 0x7fb4dc004800 2024-09-10T20:43:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4ec14f5b0 con 0x7fb4dc004800 2024-09-10T20:43:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4e97fa640 1 -- 172.21.15.28:0/241984263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0010ac0 con 0x7fb4dc004800 2024-09-10T20:43:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.870+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4ec156920 con 0x7fb4dc004800 2024-09-10T20:43:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.871+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4ec0af660 con 0x7fb4dc004800 2024-09-10T20:43:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.872+0000 7fb4e97fa640 1 -- 172.21.15.28:0/241984263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fb4e0010c60 con 0x7fb4dc004800 2024-09-10T20:43:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.872+0000 7fb4e97fa640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4d0078420 0x7fb4d007a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.872+0000 7fb4e97fa640 1 -- 172.21.15.28:0/241984263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5737+0+0 (secure 0 0 0) 0x7fb4e009adf0 con 0x7fb4dc004800 2024-09-10T20:43:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.872+0000 7fb4ebfff640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4d0078420 0x7fb4d007a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.873+0000 7fb4ebfff640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4d0078420 0x7fb4d007a8e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8033000 tx=0x7fb4d80019f0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:52.876+0000 7fb4e97fa640 1 -- 172.21.15.28:0/241984263 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb4e0063230 con 0x7fb4dc004800 2024-09-10T20:43:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.064+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4ec158830 con 0x7fb4dc004800 2024-09-10T20:43:53.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.065+0000 7fb4e97fa640 1 -- 172.21.15.28:0/241984263 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+16744 (secure 0 0 0) 0x7fb4e0062b30 con 0x7fb4dc004800 2024-09-10T20:43:53.069 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:53.069 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:43:51.763858+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:43:49.411437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-10T20:43:50.719111+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:43:53.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.067+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4d0078420 msgr2=0x7fb4d007a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.067+0000 7fb4f2e47640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4d0078420 0x7fb4d007a8e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8033000 tx=0x7fb4d80019f0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.067+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 msgr2=0x7fb4ec09e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.067+0000 7fb4f2e47640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 0x7fb4ec09e120 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0004df0 tx=0x7fb4e0001630 comp rx=0 tx=0).stop 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 shutdown_connections 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4d0078420 0x7fb4d007a8e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4ec154f50 0x7fb4ec157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4ec14eff0 0x7fb4ec154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 --2- 172.21.15.28:0/241984263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc004800 0x7fb4ec09e120 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 >> 172.21.15.28:0/241984263 conn(0x7fb4ec06dbf0 msgr2=0x7fb4ec153ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 shutdown_connections 2024-09-10T20:43:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.068+0000 7fb4f2e47640 1 -- 172.21.15.28:0/241984263 wait complete. 2024-09-10T20:43:53.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:43:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.188+0000 7fa977e87640 1 -- 172.21.15.28:0/133540032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 msgr2=0x7fa970169030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:53.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.188+0000 7fa977e87640 1 --2- 172.21.15.28:0/133540032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970169030 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fa968009830 tx=0x7fa96802a640 comp rx=0 tx=0).stop 2024-09-10T20:43:53.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.188+0000 7fa977e87640 1 -- 172.21.15.28:0/133540032 shutdown_connections 2024-09-10T20:43:53.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.188+0000 7fa977e87640 1 --2- 172.21.15.28:0/133540032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970169030 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.188+0000 7fa977e87640 1 -- 172.21.15.28:0/133540032 >> 172.21.15.28:0/133540032 conn(0x7fa9700d4130 msgr2=0x7fa9700d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:53.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.188+0000 7fa977e87640 1 -- 172.21.15.28:0/133540032 shutdown_connections 2024-09-10T20:43:53.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.188+0000 7fa977e87640 1 -- 172.21.15.28:0/133540032 wait complete. 2024-09-10T20:43:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.189+0000 7fa977e87640 1 Processor -- start 2024-09-10T20:43:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.189+0000 7fa977e87640 1 -- start start 2024-09-10T20:43:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa977e87640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9700d1f00 0x7fa9701583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa977e87640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa977e87640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa97015e220 0x7fa970159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa977e87640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9701696f0 con 0x7fa9700d2d70 2024-09-10T20:43:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa977e87640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa970169570 con 0x7fa97015e220 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa977e87640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa970169870 con 0x7fa9700d1f00 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa975422640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa975422640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53100/0 (socket says 172.21.15.28:53100) 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa975422640 1 -- 172.21.15.28:0/701841763 learned_addr learned my addr 172.21.15.28:0/701841763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa976424640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa97015e220 0x7fa970159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.190+0000 7fa975c23640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9700d1f00 0x7fa9701583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa975422640 1 -- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9700d1f00 msgr2=0x7fa9701583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa975422640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9700d1f00 0x7fa9701583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa975422640 1 -- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa97015e220 msgr2=0x7fa970159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa975422640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa97015e220 0x7fa970159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa975422640 1 -- 172.21.15.28:0/701841763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9701588f0 con 0x7fa9700d2d70 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa975c23640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9700d1f00 0x7fa9701583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa976424640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa97015e220 0x7fa970159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa975422640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970158ae0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c500 tx=0x7fa96400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa96e7fc640 1 -- 172.21.15.28:0/701841763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9640052d0 con 0x7fa9700d2d70 2024-09-10T20:43:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa96e7fc640 1 -- 172.21.15.28:0/701841763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa964005470 con 0x7fa9700d2d70 2024-09-10T20:43:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa96e7fc640 1 -- 172.21.15.28:0/701841763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964012870 con 0x7fa9700d2d70 2024-09-10T20:43:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9700cb600 con 0x7fa9700d2d70 2024-09-10T20:43:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.191+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9700cb8a0 con 0x7fa9700d2d70 2024-09-10T20:43:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.192+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9700d2660 con 0x7fa9700d2d70 2024-09-10T20:43:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.193+0000 7fa96e7fc640 1 -- 172.21.15.28:0/701841763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fa9640035f0 con 0x7fa9700d2d70 2024-09-10T20:43:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.193+0000 7fa96e7fc640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa95c0783f0 0x7fa95c07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.193+0000 7fa96e7fc640 1 -- 172.21.15.28:0/701841763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5737+0+0 (secure 0 0 0) 0x7fa96409a220 con 0x7fa9700d2d70 2024-09-10T20:43:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.193+0000 7fa975c23640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa95c0783f0 0x7fa95c07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.194+0000 7fa975c23640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa95c0783f0 0x7fa95c07a8b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa960004a30 tx=0x7fa960009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.198+0000 7fa96e7fc640 1 -- 172.21.15.28:0/701841763 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa964062630 con 0x7fa9700d2d70 2024-09-10T20:43:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:53.423+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa970074c40 con 0x7fa9700d2d70 2024-09-10T20:43:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.414+0000 7fa96e7fc640 1 -- 172.21.15.28:0/701841763 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v75) ==== 118+0+0 (secure 0 0 0) 0x7fa964061f30 con 0x7fa9700d2d70 2024-09-10T20:43:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa95c0783f0 msgr2=0x7fa95c07a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa95c0783f0 0x7fa95c07a8b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa960004a30 tx=0x7fa960009040 comp rx=0 tx=0).stop 2024-09-10T20:43:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 msgr2=0x7fa970158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970158ae0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c500 tx=0x7fa96400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 shutdown_connections 2024-09-10T20:43:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa95c0783f0 0x7fa95c07a8b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa97015e220 0x7fa970159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9700d2d70 0x7fa970158ae0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 --2- 172.21.15.28:0/701841763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa9700d1f00 0x7fa9701583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 >> 172.21.15.28:0/701841763 conn(0x7fa9700d4130 msgr2=0x7fa9700b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 shutdown_connections 2024-09-10T20:43:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.417+0000 7fa977e87640 1 -- 172.21.15.28:0/701841763 wait complete. 2024-09-10T20:43:54.434 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:43:54.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 -- 172.21.15.28:0/2409219672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe474161b50 msgr2=0x7fe4740d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2409219672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe474161b50 0x7fe4740d2620 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009830 tx=0x7fe45c02a480 comp rx=0 tx=0).stop 2024-09-10T20:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 -- 172.21.15.28:0/2409219672 shutdown_connections 2024-09-10T20:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2409219672 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe474161b50 0x7fe4740d2620 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2409219672 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe474161770 0x7fe4740d20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 -- 172.21.15.28:0/2409219672 >> 172.21.15.28:0/2409219672 conn(0x7fe4740d4070 msgr2=0x7fe4740d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 -- 172.21.15.28:0/2409219672 shutdown_connections 2024-09-10T20:43:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.547+0000 7fe4797a3640 1 -- 172.21.15.28:0/2409219672 wait complete. 2024-09-10T20:43:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.548+0000 7fe4797a3640 1 Processor -- start 2024-09-10T20:43:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.548+0000 7fe4797a3640 1 -- start start 2024-09-10T20:43:54.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4797a3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4740d4b90 0x7fe4741f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:54.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4797a3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe474161770 0x7fe47415c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:54.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4797a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe474161b50 0x7fe47415d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:54.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4797a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4740b2f50 con 0x7fe474161b50 2024-09-10T20:43:54.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4797a3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe4740b2dd0 con 0x7fe4740d4b90 2024-09-10T20:43:54.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4797a3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe4740b30d0 con 0x7fe474161770 2024-09-10T20:43:54.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe474161b50 0x7fe47415d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe474161b50 0x7fe47415d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53114/0 (socket says 172.21.15.28:53114) 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4737fe640 1 -- 172.21.15.28:0/2946757195 learned_addr learned my addr 172.21.15.28:0/2946757195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe472ffd640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4740d4b90 0x7fe4741f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.549+0000 7fe4727fc640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe474161770 0x7fe47415c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4737fe640 1 -- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe474161770 msgr2=0x7fe47415c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4737fe640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe474161770 0x7fe47415c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4737fe640 1 -- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4740d4b90 msgr2=0x7fe4741f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4737fe640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4740d4b90 0x7fe4741f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4737fe640 1 -- 172.21.15.28:0/2946757195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47409e910 con 0x7fe474161b50 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe472ffd640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4740d4b90 0x7fe4741f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4737fe640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe474161b50 0x7fe47415d980 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c3f0 tx=0x7fe46400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe46b7fe640 1 -- 172.21.15.28:0/2946757195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe464017070 con 0x7fe474161b50 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe46b7fe640 1 -- 172.21.15.28:0/2946757195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4640053c0 con 0x7fe474161b50 2024-09-10T20:43:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe46b7fe640 1 -- 172.21.15.28:0/2946757195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe464003850 con 0x7fe474161b50 2024-09-10T20:43:54.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47409eb80 con 0x7fe474161b50 2024-09-10T20:43:54.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.550+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47409f140 con 0x7fe474161b50 2024-09-10T20:43:54.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.551+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4740b3870 con 0x7fe474161b50 2024-09-10T20:43:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.552+0000 7fe46b7fe640 1 -- 172.21.15.28:0/2946757195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fe4640039f0 con 0x7fe474161b50 2024-09-10T20:43:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.552+0000 7fe46b7fe640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4540783f0 0x7fe45407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.552+0000 7fe46b7fe640 1 -- 172.21.15.28:0/2946757195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fe464099d60 con 0x7fe474161b50 2024-09-10T20:43:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.552+0000 7fe472ffd640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4540783f0 0x7fe45407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.553+0000 7fe472ffd640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4540783f0 0x7fe45407a8b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe46c005fd0 tx=0x7fe46c0095c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.557+0000 7fe46b7fe640 1 -- 172.21.15.28:0/2946757195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe464062070 con 0x7fe474161b50 2024-09-10T20:43:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:54.782+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe4740b3490 con 0x7fe474161b50 2024-09-10T20:43:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.415+0000 7fe46b7fe640 1 -- 172.21.15.28:0/2946757195 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v76) ==== 122+0+0 (secure 0 0 0) 0x7fe464061970 con 0x7fe474161b50 2024-09-10T20:43:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4540783f0 msgr2=0x7fe45407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4540783f0 0x7fe45407a8b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe46c005fd0 tx=0x7fe46c0095c0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe474161b50 msgr2=0x7fe47415d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe474161b50 0x7fe47415d980 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c3f0 tx=0x7fe46400c8b0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 shutdown_connections 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4540783f0 0x7fe45407a8b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe474161b50 0x7fe47415d980 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe474161770 0x7fe47415c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 --2- 172.21.15.28:0/2946757195 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4740d4b90 0x7fe4741f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.418+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 >> 172.21.15.28:0/2946757195 conn(0x7fe4740d4070 msgr2=0x7fe4740cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.419+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 shutdown_connections 2024-09-10T20:43:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.419+0000 7fe4797a3640 1 -- 172.21.15.28:0/2946757195 wait complete. 2024-09-10T20:43:55.436 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:43:55.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.544+0000 7f09cbec6640 1 -- 172.21.15.28:0/889977384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40b4d10 msgr2=0x7f09c40ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.544+0000 7f09cbec6640 1 --2- 172.21.15.28:0/889977384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40b4d10 0x7f09c40ce030 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f09bc005fd0 tx=0x7f09bc02aa90 comp rx=0 tx=0).stop 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.545+0000 7f09cbec6640 1 -- 172.21.15.28:0/889977384 shutdown_connections 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.545+0000 7f09cbec6640 1 --2- 172.21.15.28:0/889977384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40b4d10 0x7f09c40ce030 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.545+0000 7f09cbec6640 1 --2- 172.21.15.28:0/889977384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09c415df70 0x7f09c40b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.545+0000 7f09cbec6640 1 -- 172.21.15.28:0/889977384 >> 172.21.15.28:0/889977384 conn(0x7f09c406cf00 msgr2=0x7f09c406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.545+0000 7f09cbec6640 1 -- 172.21.15.28:0/889977384 shutdown_connections 2024-09-10T20:43:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.545+0000 7f09cbec6640 1 -- 172.21.15.28:0/889977384 wait complete. 2024-09-10T20:43:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.546+0000 7f09cbec6640 1 Processor -- start 2024-09-10T20:43:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.546+0000 7f09cbec6640 1 -- start start 2024-09-10T20:43:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09cbec6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09c40b4d10 0x7f09c41f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09cbec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c415a190 0x7f09c4157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09cbec6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09c415df70 0x7f09c4151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09cbec6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40d0270 con 0x7f09c415a190 2024-09-10T20:43:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09cbec6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40d00f0 con 0x7f09c415df70 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09cbec6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f09c40d03f0 con 0x7f09c40b4d10 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c415a190 0x7f09c4157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c415a190 0x7f09c4157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53124/0 (socket says 172.21.15.28:53124) 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 -- 172.21.15.28:0/2540827310 learned_addr learned my addr 172.21.15.28:0/2540827310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9c62640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09c40b4d10 0x7f09c41f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09ca463640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09c415df70 0x7f09c4151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 -- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09c40b4d10 msgr2=0x7f09c41f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09c40b4d10 0x7f09c41f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 -- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09c415df70 msgr2=0x7f09c4151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09c415df70 0x7f09c4151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 -- 172.21.15.28:0/2540827310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c40af8c0 con 0x7f09c415a190 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09c9461640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c415a190 0x7f09c4157ea0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f09b400dae0 tx=0x7f09b400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.547+0000 7f09ba7fc640 1 -- 172.21.15.28:0/2540827310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b400ae30 con 0x7f09c415a190 2024-09-10T20:43:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.548+0000 7f09ba7fc640 1 -- 172.21.15.28:0/2540827310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09b4005110 con 0x7f09c415a190 2024-09-10T20:43:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.548+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c40afba0 con 0x7f09c415a190 2024-09-10T20:43:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.548+0000 7f09ba7fc640 1 -- 172.21.15.28:0/2540827310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4005540 con 0x7f09c415a190 2024-09-10T20:43:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.548+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c40b0680 con 0x7f09c415a190 2024-09-10T20:43:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.549+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c40b3f00 con 0x7f09c415a190 2024-09-10T20:43:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.549+0000 7f09ba7fc640 1 -- 172.21.15.28:0/2540827310 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f09b400b040 con 0x7f09c415a190 2024-09-10T20:43:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.550+0000 7f09ba7fc640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f09a80783f0 0x7f09a807a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.550+0000 7f09ba7fc640 1 -- 172.21.15.28:0/2540827310 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f09b409a890 con 0x7f09c415a190 2024-09-10T20:43:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.550+0000 7f09c9c62640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f09a80783f0 0x7f09a807a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.550+0000 7f09c9c62640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f09a80783f0 0x7f09a807a8b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09c00017c0 tx=0x7f09c0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:55.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.554+0000 7f09ba7fc640 1 -- 172.21.15.28:0/2540827310 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f09c40b3f00 con 0x7f09c415a190 2024-09-10T20:43:55.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.776+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09c40ce030 con 0x7f09c415a190 2024-09-10T20:43:55.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.776+0000 7f09ba7fc640 1 -- 172.21.15.28:0/2540827310 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5231 (secure 0 0 0) 0x7f09b406b2c0 con 0x7f09c415a190 2024-09-10T20:43:55.780 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:55.781 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":76,"btime":"2024-09-10T20:43:55:414872+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3157854991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3157854991},{"type":"v1","addr":"172.21.15.159:6835","nonce":3157854991}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/3291396887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3291396887},{"type":"v1","addr":"172.21.15.159:6801","nonce":3291396887}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7461,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3500302836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3500302836},{"type":"v1","addr":"172.21.15.28:6833","nonce":3500302836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:43:51.763823+0000","modified":"2024-09-10T20:43:54.783424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7482":{"gid":7482,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1765205934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1765205934},{"type":"v1","addr":"172.21.15.28:6835","nonce":1765205934}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7482,"qdb_cluster":[7482]},"id":9}]} 2024-09-10T20:43:55.781 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 76 2024-09-10T20:43:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f09a80783f0 msgr2=0x7f09a807a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f09a80783f0 0x7f09a807a8b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09c00017c0 tx=0x7f09c0009040 comp rx=0 tx=0).stop 2024-09-10T20:43:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c415a190 msgr2=0x7f09c4157ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c415a190 0x7f09c4157ea0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f09b400dae0 tx=0x7f09b400a250 comp rx=0 tx=0).stop 2024-09-10T20:43:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 shutdown_connections 2024-09-10T20:43:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f09a80783f0 0x7f09a807a8b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09c415df70 0x7f09c4151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c415a190 0x7f09c4157ea0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 --2- 172.21.15.28:0/2540827310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09c40b4d10 0x7f09c41f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 >> 172.21.15.28:0/2540827310 conn(0x7f09c406cf00 msgr2=0x7f09c40acd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:55.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 shutdown_connections 2024-09-10T20:43:55.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.779+0000 7f09cbec6640 1 -- 172.21.15.28:0/2540827310 wait complete. 2024-09-10T20:43:55.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:43:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 -- 172.21.15.28:0/974059026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c1554e0 msgr2=0x7ff34c15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 --2- 172.21.15.28:0/974059026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c1554e0 0x7ff34c15e930 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ff33c005fd0 tx=0x7ff33c02a530 comp rx=0 tx=0).stop 2024-09-10T20:43:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 -- 172.21.15.28:0/974059026 shutdown_connections 2024-09-10T20:43:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 --2- 172.21.15.28:0/974059026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c1554e0 0x7ff34c15e930 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 --2- 172.21.15.28:0/974059026 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff34c155100 0x7ff34c15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 -- 172.21.15.28:0/974059026 >> 172.21.15.28:0/974059026 conn(0x7ff34c06cf00 msgr2=0x7ff34c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 -- 172.21.15.28:0/974059026 shutdown_connections 2024-09-10T20:43:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.895+0000 7ff354194640 1 -- 172.21.15.28:0/974059026 wait complete. 2024-09-10T20:43:55.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.896+0000 7ff354194640 1 Processor -- start 2024-09-10T20:43:55.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.896+0000 7ff354194640 1 -- start start 2024-09-10T20:43:55.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff354194640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff34c155100 0x7ff34c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff354194640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff34c1554e0 0x7ff34c1466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff354194640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c15ee70 0x7ff34c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff354194640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c072f60 con 0x7ff34c15ee70 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff354194640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c072de0 con 0x7ff34c1554e0 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff354194640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff34c0730e0 con 0x7ff34c155100 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c15ee70 0x7ff34c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c15ee70 0x7ff34c0b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53140/0 (socket says 172.21.15.28:53140) 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 -- 172.21.15.28:0/2909499025 learned_addr learned my addr 172.21.15.28:0/2909499025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff351f30640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff34c155100 0x7ff34c14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff35172f640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff34c1554e0 0x7ff34c1466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 -- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff34c155100 msgr2=0x7ff34c14e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff34c155100 0x7ff34c14e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 -- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff34c1554e0 msgr2=0x7ff34c1466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff34c1554e0 0x7ff34c1466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.897+0000 7ff352731640 1 -- 172.21.15.28:0/2909499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34c146c10 con 0x7ff34c15ee70 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.898+0000 7ff352731640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c15ee70 0x7ff34c0b02b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7ff34400c730 tx=0x7ff34400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.898+0000 7ff3427fc640 1 -- 172.21.15.28:0/2909499025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344016020 con 0x7ff34c15ee70 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.898+0000 7ff3427fc640 1 -- 172.21.15.28:0/2909499025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff34400ab40 con 0x7ff34c15ee70 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.898+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34c0b1b60 con 0x7ff34c15ee70 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.898+0000 7ff3427fc640 1 -- 172.21.15.28:0/2909499025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34400ce40 con 0x7ff34c15ee70 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.898+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff34c0b1080 con 0x7ff34c15ee70 2024-09-10T20:43:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.899+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34c0735b0 con 0x7ff34c15ee70 2024-09-10T20:43:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.899+0000 7ff3427fc640 1 -- 172.21.15.28:0/2909499025 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7ff34401a3b0 con 0x7ff34c15ee70 2024-09-10T20:43:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.900+0000 7ff3427fc640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3300783f0 0x7ff33007a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.900+0000 7ff3427fc640 1 -- 172.21.15.28:0/2909499025 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7ff34409a720 con 0x7ff34c15ee70 2024-09-10T20:43:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.900+0000 7ff351f30640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3300783f0 0x7ff33007a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.901+0000 7ff351f30640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3300783f0 0x7ff33007a8b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff348005fd0 tx=0x7ff34800a250 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:55.904+0000 7ff3427fc640 1 -- 172.21.15.28:0/2909499025 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff344020030 con 0x7ff34c15ee70 2024-09-10T20:43:56.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.090+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff34c0b2330 con 0x7ff34c15ee70 2024-09-10T20:43:56.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.091+0000 7ff3427fc640 1 -- 172.21.15.28:0/2909499025 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+16731 (secure 0 0 0) 0x7ff34c146c10 con 0x7ff34c15ee70 2024-09-10T20:43:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:56.095 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":115,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:43:53.402754+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:43:49.411437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-10T20:43:50.719111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:43:56.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3300783f0 msgr2=0x7ff33007a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3300783f0 0x7ff33007a8b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff348005fd0 tx=0x7ff34800a250 comp rx=0 tx=0).stop 2024-09-10T20:43:56.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c15ee70 msgr2=0x7ff34c0b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c15ee70 0x7ff34c0b02b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7ff34400c730 tx=0x7ff34400cbf0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 shutdown_connections 2024-09-10T20:43:56.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3300783f0 0x7ff33007a8b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff34c15ee70 0x7ff34c0b02b0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff34c1554e0 0x7ff34c1466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 --2- 172.21.15.28:0/2909499025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff34c155100 0x7ff34c14e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 >> 172.21.15.28:0/2909499025 conn(0x7ff34c06cf00 msgr2=0x7ff34c15c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 shutdown_connections 2024-09-10T20:43:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.093+0000 7ff354194640 1 -- 172.21.15.28:0/2909499025 wait complete. 2024-09-10T20:43:56.108 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:43:56.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:43:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.210+0000 7f928cd9c640 1 -- 172.21.15.28:0/1819962180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9274003a00 msgr2=0x7f9274003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.210+0000 7f928cd9c640 1 --2- 172.21.15.28:0/1819962180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9274003a00 0x7f9274003e20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f9278009b80 tx=0x7f9274005830 comp rx=0 tx=0).stop 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.210+0000 7f9286d9d640 1 -- 172.21.15.28:0/1819962180 reap_dead start 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.210+0000 7f928cd9c640 1 -- 172.21.15.28:0/1819962180 shutdown_connections 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.210+0000 7f928cd9c640 1 -- 172.21.15.28:0/1819962180 >> 172.21.15.28:0/1819962180 conn(0x7f92880d0420 msgr2=0x7f92880d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.210+0000 7f928cd9c640 1 -- 172.21.15.28:0/1819962180 shutdown_connections 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.210+0000 7f928cd9c640 1 -- 172.21.15.28:0/1819962180 wait complete. 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 Processor -- start 2024-09-10T20:43:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 -- start start 2024-09-10T20:43:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9274003a00 0x7f92881f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92881f5380 0x7f9288151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92880c9770 0x7f92880c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f928806f2c0 con 0x7f92881f5380 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f928806f140 con 0x7f9274003a00 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f928cd9c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92880731f0 con 0x7f92880c9770 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f927ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92881f5380 0x7f9288151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f927ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92881f5380 0x7f9288151ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53166/0 (socket says 172.21.15.28:53166) 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.212+0000 7f927ffff640 1 -- 172.21.15.28:0/1815959078 learned_addr learned my addr 172.21.15.28:0/1815959078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f928659c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9274003a00 0x7f92881f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f9286d9d640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92880c9770 0x7f92880c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927ffff640 1 -- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92880c9770 msgr2=0x7f92880c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927ffff640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92880c9770 0x7f92880c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927ffff640 1 -- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9274003a00 msgr2=0x7f92881f5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927ffff640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9274003a00 0x7f92881f5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927ffff640 1 -- 172.21.15.28:0/1815959078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92880b5d10 con 0x7f92881f5380 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f9286d9d640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92880c9770 0x7f92880c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927ffff640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92881f5380 0x7f9288151ff0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f927000e600 tx=0x7f927000eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927effd640 1 -- 172.21.15.28:0/1815959078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9270015070 con 0x7f92881f5380 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927effd640 1 -- 172.21.15.28:0/1815959078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f927000ca30 con 0x7f92881f5380 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f927effd640 1 -- 172.21.15.28:0/1815959078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927000ce60 con 0x7f92881f5380 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9288152530 con 0x7f92881f5380 2024-09-10T20:43:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.213+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92880b5750 con 0x7f92881f5380 2024-09-10T20:43:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.214+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92880ca130 con 0x7f92881f5380 2024-09-10T20:43:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.215+0000 7f927effd640 1 -- 172.21.15.28:0/1815959078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f9270011460 con 0x7f92881f5380 2024-09-10T20:43:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.215+0000 7f927effd640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078420 0x7f926807a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.216+0000 7f928659c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078420 0x7f926807a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.216+0000 7f927effd640 1 -- 172.21.15.28:0/1815959078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f927009ad60 con 0x7f92881f5380 2024-09-10T20:43:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.216+0000 7f928659c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078420 0x7f926807a8e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9278035000 tx=0x7f927802d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.220+0000 7f927effd640 1 -- 172.21.15.28:0/1815959078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9270063100 con 0x7f92881f5380 2024-09-10T20:43:56.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.401+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92880d0000 con 0x7f9268078420 2024-09-10T20:43:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.405+0000 7f927effd640 1 -- 172.21.15.28:0/1815959078 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f927802b850 con 0x7f9268078420 2024-09-10T20:43:56.411 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:43:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078420 msgr2=0x7f926807a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078420 0x7f926807a8e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9278035000 tx=0x7f927802d040 comp rx=0 tx=0).stop 2024-09-10T20:43:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92881f5380 msgr2=0x7f9288151ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92881f5380 0x7f9288151ff0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f927000e600 tx=0x7f927000eac0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 shutdown_connections 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078420 0x7f926807a8e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92880c9770 0x7f92880c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92881f5380 0x7f9288151ff0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 --2- 172.21.15.28:0/1815959078 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9274003a00 0x7f92881f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 >> 172.21.15.28:0/1815959078 conn(0x7f92880d0420 msgr2=0x7f92880b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 shutdown_connections 2024-09-10T20:43:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.408+0000 7f928cd9c640 1 -- 172.21.15.28:0/1815959078 wait complete. 2024-09-10T20:43:56.424 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-10T20:43:56.313435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278548,"kb_used_data":12368,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754696172,"statfs":{"total":773094113280,"available":772808880128,"internally_reserved":0,"allocated":12664832,"data_stored":6213004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.935137"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408628+0000","last_change":"2024-09-10T20:43:52.408397+0000","last_active":"2024-09-10T20:43:53.408628+0000","last_peered":"2024-09-10T20:43:53.408628+0000","last_clean":"2024-09-10T20:43:53.408628+0000","last_became_active":"2024-09-10T20:43:52.408104+0000","last_became_peered":"2024-09-10T20:43:52.408104+0000","last_unstale":"2024-09-10T20:43:53.408628+0000","last_undegraded":"2024-09-10T20:43:53.408628+0000","last_fullsized":"2024-09-10T20:43:53.408628+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:12.359581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408850+0000","last_change":"2024-09-10T20:43:52.412350+0000","last_active":"2024-09-10T20:43:53.408850+0000","last_peered":"2024-09-10T20:43:53.408850+0000","last_clean":"2024-09-10T20:43:53.408850+0000","last_became_active":"2024-09-10T20:43:52.411891+0000","last_became_peered":"2024-09-10T20:43:52.411891+0000","last_unstale":"2024-09-10T20:43:53.408850+0000","last_undegraded":"2024-09-10T20:43:53.408850+0000","last_fullsized":"2024-09-10T20:43:53.408850+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:45.026682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477745+0000","last_change":"2024-09-10T20:43:52.412391+0000","last_active":"2024-09-10T20:43:53.477745+0000","last_peered":"2024-09-10T20:43:53.477745+0000","last_clean":"2024-09-10T20:43:53.477745+0000","last_became_active":"2024-09-10T20:43:52.412107+0000","last_became_peered":"2024-09-10T20:43:52.412107+0000","last_unstale":"2024-09-10T20:43:53.477745+0000","last_undegraded":"2024-09-10T20:43:53.477745+0000","last_fullsized":"2024-09-10T20:43:53.477745+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:56:48.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407565+0000","last_change":"2024-09-10T20:43:52.408407+0000","last_active":"2024-09-10T20:43:53.407565+0000","last_peered":"2024-09-10T20:43:53.407565+0000","last_clean":"2024-09-10T20:43:53.407565+0000","last_became_active":"2024-09-10T20:43:52.408149+0000","last_became_peered":"2024-09-10T20:43:52.408149+0000","last_unstale":"2024-09-10T20:43:53.407565+0000","last_undegraded":"2024-09-10T20:43:53.407565+0000","last_fullsized":"2024-09-10T20:43:53.407565+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:52.035814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406177+0000","last_change":"2024-09-10T20:43:52.408695+0000","last_active":"2024-09-10T20:43:53.406177+0000","last_peered":"2024-09-10T20:43:53.406177+0000","last_clean":"2024-09-10T20:43:53.406177+0000","last_became_active":"2024-09-10T20:43:52.408221+0000","last_became_peered":"2024-09-10T20:43:52.408221+0000","last_unstale":"2024-09-10T20:43:53.406177+0000","last_undegraded":"2024-09-10T20:43:53.406177+0000","last_fullsized":"2024-09-10T20:43:53.406177+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:48:23.047553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412904+0000","last_change":"2024-09-10T20:43:52.410087+0000","last_active":"2024-09-10T20:43:53.412904+0000","last_peered":"2024-09-10T20:43:53.412904+0000","last_clean":"2024-09-10T20:43:53.412904+0000","last_became_active":"2024-09-10T20:43:52.409699+0000","last_became_peered":"2024-09-10T20:43:52.409699+0000","last_unstale":"2024-09-10T20:43:53.412904+0000","last_undegraded":"2024-09-10T20:43:53.412904+0000","last_fullsized":"2024-09-10T20:43:53.412904+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:45.052374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.405818+0000","last_change":"2024-09-10T20:43:52.408416+0000","last_active":"2024-09-10T20:43:53.405818+0000","last_peered":"2024-09-10T20:43:53.405818+0000","last_clean":"2024-09-10T20:43:53.405818+0000","last_became_active":"2024-09-10T20:43:52.408101+0000","last_became_peered":"2024-09-10T20:43:52.408101+0000","last_unstale":"2024-09-10T20:43:53.405818+0000","last_undegraded":"2024-09-10T20:43:53.405818+0000","last_fullsized":"2024-09-10T20:43:53.405818+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:48.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.405720+0000","last_change":"2024-09-10T20:43:52.404371+0000","last_active":"2024-09-10T20:43:53.405720+0000","last_peered":"2024-09-10T20:43:53.405720+0000","last_clean":"2024-09-10T20:43:53.405720+0000","last_became_active":"2024-09-10T20:43:52.404073+0000","last_became_peered":"2024-09-10T20:43:52.404073+0000","last_unstale":"2024-09-10T20:43:53.405720+0000","last_undegraded":"2024-09-10T20:43:53.405720+0000","last_fullsized":"2024-09-10T20:43:53.405720+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:51.683106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477876+0000","last_change":"2024-09-10T20:43:52.414392+0000","last_active":"2024-09-10T20:43:53.477876+0000","last_peered":"2024-09-10T20:43:53.477876+0000","last_clean":"2024-09-10T20:43:53.477876+0000","last_became_active":"2024-09-10T20:43:52.414089+0000","last_became_peered":"2024-09-10T20:43:52.414089+0000","last_unstale":"2024-09-10T20:43:53.477876+0000","last_undegraded":"2024-09-10T20:43:53.477876+0000","last_fullsized":"2024-09-10T20:43:53.477876+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:49.567937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408821+0000","last_change":"2024-09-10T20:43:52.408994+0000","last_active":"2024-09-10T20:43:53.408821+0000","last_peered":"2024-09-10T20:43:53.408821+0000","last_clean":"2024-09-10T20:43:53.408821+0000","last_became_active":"2024-09-10T20:43:52.408725+0000","last_became_peered":"2024-09-10T20:43:52.408725+0000","last_unstale":"2024-09-10T20:43:53.408821+0000","last_undegraded":"2024-09-10T20:43:53.408821+0000","last_fullsized":"2024-09-10T20:43:53.408821+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:27:00.590021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407919+0000","last_change":"2024-09-10T20:43:52.410655+0000","last_active":"2024-09-10T20:43:53.407919+0000","last_peered":"2024-09-10T20:43:53.407919+0000","last_clean":"2024-09-10T20:43:53.407919+0000","last_became_active":"2024-09-10T20:43:52.410389+0000","last_became_peered":"2024-09-10T20:43:52.410389+0000","last_unstale":"2024-09-10T20:43:53.407919+0000","last_undegraded":"2024-09-10T20:43:53.407919+0000","last_fullsized":"2024-09-10T20:43:53.407919+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:18:23.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413308+0000","last_change":"2024-09-10T20:43:52.421943+0000","last_active":"2024-09-10T20:43:53.413308+0000","last_peered":"2024-09-10T20:43:53.413308+0000","last_clean":"2024-09-10T20:43:53.413308+0000","last_became_active":"2024-09-10T20:43:52.421632+0000","last_became_peered":"2024-09-10T20:43:52.421632+0000","last_unstale":"2024-09-10T20:43:53.413308+0000","last_undegraded":"2024-09-10T20:43:53.413308+0000","last_fullsized":"2024-09-10T20:43:53.413308+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:52:11.137255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407572+0000","last_change":"2024-09-10T20:43:52.407464+0000","last_active":"2024-09-10T20:43:53.407572+0000","last_peered":"2024-09-10T20:43:53.407572+0000","last_clean":"2024-09-10T20:43:53.407572+0000","last_became_active":"2024-09-10T20:43:52.407124+0000","last_became_peered":"2024-09-10T20:43:52.407124+0000","last_unstale":"2024-09-10T20:43:53.407572+0000","last_undegraded":"2024-09-10T20:43:53.407572+0000","last_fullsized":"2024-09-10T20:43:53.407572+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:54.752584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477763+0000","last_change":"2024-09-10T20:43:52.414427+0000","last_active":"2024-09-10T20:43:53.477763+0000","last_peered":"2024-09-10T20:43:53.477763+0000","last_clean":"2024-09-10T20:43:53.477763+0000","last_became_active":"2024-09-10T20:43:52.414075+0000","last_became_peered":"2024-09-10T20:43:52.414075+0000","last_unstale":"2024-09-10T20:43:53.477763+0000","last_undegraded":"2024-09-10T20:43:53.477763+0000","last_fullsized":"2024-09-10T20:43:53.477763+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:20.178183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477836+0000","last_change":"2024-09-10T20:43:52.421929+0000","last_active":"2024-09-10T20:43:53.477836+0000","last_peered":"2024-09-10T20:43:53.477836+0000","last_clean":"2024-09-10T20:43:53.477836+0000","last_became_active":"2024-09-10T20:43:52.421522+0000","last_became_peered":"2024-09-10T20:43:52.421522+0000","last_unstale":"2024-09-10T20:43:53.477836+0000","last_undegraded":"2024-09-10T20:43:53.477836+0000","last_fullsized":"2024-09-10T20:43:53.477836+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:47.243959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477577+0000","last_change":"2024-09-10T20:43:52.418281+0000","last_active":"2024-09-10T20:43:53.477577+0000","last_peered":"2024-09-10T20:43:53.477577+0000","last_clean":"2024-09-10T20:43:53.477577+0000","last_became_active":"2024-09-10T20:43:52.417719+0000","last_became_peered":"2024-09-10T20:43:52.417719+0000","last_unstale":"2024-09-10T20:43:53.477577+0000","last_undegraded":"2024-09-10T20:43:53.477577+0000","last_fullsized":"2024-09-10T20:43:53.477577+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:45.052374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477947+0000","last_change":"2024-09-10T20:43:52.411778+0000","last_active":"2024-09-10T20:43:53.477947+0000","last_peered":"2024-09-10T20:43:53.477947+0000","last_clean":"2024-09-10T20:43:53.477947+0000","last_became_active":"2024-09-10T20:43:52.411484+0000","last_became_peered":"2024-09-10T20:43:52.411484+0000","last_unstale":"2024-09-10T20:43:53.477947+0000","last_undegraded":"2024-09-10T20:43:53.477947+0000","last_fullsized":"2024-09-10T20:43:53.477947+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:51.933448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478707+0000","last_change":"2024-09-10T20:43:52.418892+0000","last_active":"2024-09-10T20:43:53.478707+0000","last_peered":"2024-09-10T20:43:53.478707+0000","last_clean":"2024-09-10T20:43:53.478707+0000","last_became_active":"2024-09-10T20:43:52.418614+0000","last_became_peered":"2024-09-10T20:43:52.418614+0000","last_unstale":"2024-09-10T20:43:53.478707+0000","last_undegraded":"2024-09-10T20:43:53.478707+0000","last_fullsized":"2024-09-10T20:43:53.478707+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:52.284247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408321+0000","last_change":"2024-09-10T20:43:52.411549+0000","last_active":"2024-09-10T20:43:53.408321+0000","last_peered":"2024-09-10T20:43:53.408321+0000","last_clean":"2024-09-10T20:43:53.408321+0000","last_became_active":"2024-09-10T20:43:52.411279+0000","last_became_peered":"2024-09-10T20:43:52.411279+0000","last_unstale":"2024-09-10T20:43:53.408321+0000","last_undegraded":"2024-09-10T20:43:53.408321+0000","last_fullsized":"2024-09-10T20:43:53.408321+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:50.252590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478128+0000","last_change":"2024-09-10T20:43:52.415022+0000","last_active":"2024-09-10T20:43:53.478128+0000","last_peered":"2024-09-10T20:43:53.478128+0000","last_clean":"2024-09-10T20:43:53.478128+0000","last_became_active":"2024-09-10T20:43:52.414637+0000","last_became_peered":"2024-09-10T20:43:52.414637+0000","last_unstale":"2024-09-10T20:43:53.478128+0000","last_undegraded":"2024-09-10T20:43:53.478128+0000","last_fullsized":"2024-09-10T20:43:53.478128+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:46:21.335954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413098+0000","last_change":"2024-09-10T20:43:52.409483+0000","last_active":"2024-09-10T20:43:53.413098+0000","last_peered":"2024-09-10T20:43:53.413098+0000","last_clean":"2024-09-10T20:43:53.413098+0000","last_became_active":"2024-09-10T20:43:52.409135+0000","last_became_peered":"2024-09-10T20:43:52.409135+0000","last_unstale":"2024-09-10T20:43:53.413098+0000","last_undegraded":"2024-09-10T20:43:53.413098+0000","last_fullsized":"2024-09-10T20:43:53.413098+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:36:38.170888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407903+0000","last_change":"2024-09-10T20:43:52.413059+0000","last_active":"2024-09-10T20:43:53.407903+0000","last_peered":"2024-09-10T20:43:53.407903+0000","last_clean":"2024-09-10T20:43:53.407903+0000","last_became_active":"2024-09-10T20:43:52.412407+0000","last_became_peered":"2024-09-10T20:43:52.412407+0000","last_unstale":"2024-09-10T20:43:53.407903+0000","last_undegraded":"2024-09-10T20:43:53.407903+0000","last_fullsized":"2024-09-10T20:43:53.407903+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:30.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477137+0000","last_change":"2024-09-10T20:43:52.407647+0000","last_active":"2024-09-10T20:43:53.477137+0000","last_peered":"2024-09-10T20:43:53.477137+0000","last_clean":"2024-09-10T20:43:53.477137+0000","last_became_active":"2024-09-10T20:43:52.407353+0000","last_became_peered":"2024-09-10T20:43:52.407353+0000","last_unstale":"2024-09-10T20:43:53.477137+0000","last_undegraded":"2024-09-10T20:43:53.477137+0000","last_fullsized":"2024-09-10T20:43:53.477137+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:14:35.107362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477596+0000","last_change":"2024-09-10T20:43:52.417718+0000","last_active":"2024-09-10T20:43:53.477596+0000","last_peered":"2024-09-10T20:43:53.477596+0000","last_clean":"2024-09-10T20:43:53.477596+0000","last_became_active":"2024-09-10T20:43:52.417311+0000","last_became_peered":"2024-09-10T20:43:52.417311+0000","last_unstale":"2024-09-10T20:43:53.477596+0000","last_undegraded":"2024-09-10T20:43:53.477596+0000","last_fullsized":"2024-09-10T20:43:53.477596+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:10.430197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408192+0000","last_change":"2024-09-10T20:43:52.408268+0000","last_active":"2024-09-10T20:43:53.408192+0000","last_peered":"2024-09-10T20:43:53.408192+0000","last_clean":"2024-09-10T20:43:53.408192+0000","last_became_active":"2024-09-10T20:43:52.407995+0000","last_became_peered":"2024-09-10T20:43:52.407995+0000","last_unstale":"2024-09-10T20:43:53.408192+0000","last_undegraded":"2024-09-10T20:43:53.408192+0000","last_fullsized":"2024-09-10T20:43:53.408192+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:31.680937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.409000+0000","last_change":"2024-09-10T20:43:52.413099+0000","last_active":"2024-09-10T20:43:53.409000+0000","last_peered":"2024-09-10T20:43:53.409000+0000","last_clean":"2024-09-10T20:43:53.409000+0000","last_became_active":"2024-09-10T20:43:52.412720+0000","last_became_peered":"2024-09-10T20:43:52.412720+0000","last_unstale":"2024-09-10T20:43:53.409000+0000","last_undegraded":"2024-09-10T20:43:53.409000+0000","last_fullsized":"2024-09-10T20:43:53.409000+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:37.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478266+0000","last_change":"2024-09-10T20:43:52.428537+0000","last_active":"2024-09-10T20:43:53.478266+0000","last_peered":"2024-09-10T20:43:53.478266+0000","last_clean":"2024-09-10T20:43:53.478266+0000","last_became_active":"2024-09-10T20:43:52.428221+0000","last_became_peered":"2024-09-10T20:43:52.428221+0000","last_unstale":"2024-09-10T20:43:53.478266+0000","last_undegraded":"2024-09-10T20:43:53.478266+0000","last_fullsized":"2024-09-10T20:43:53.478266+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:27.850900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477341+0000","last_change":"2024-09-10T20:43:52.407894+0000","last_active":"2024-09-10T20:43:53.477341+0000","last_peered":"2024-09-10T20:43:53.477341+0000","last_clean":"2024-09-10T20:43:53.477341+0000","last_became_active":"2024-09-10T20:43:52.407595+0000","last_became_peered":"2024-09-10T20:43:52.407595+0000","last_unstale":"2024-09-10T20:43:53.477341+0000","last_undegraded":"2024-09-10T20:43:53.477341+0000","last_fullsized":"2024-09-10T20:43:53.477341+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:00.098258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407746+0000","last_change":"2024-09-10T20:43:52.415674+0000","last_active":"2024-09-10T20:43:53.407746+0000","last_peered":"2024-09-10T20:43:53.407746+0000","last_clean":"2024-09-10T20:43:53.407746+0000","last_became_active":"2024-09-10T20:43:52.415340+0000","last_became_peered":"2024-09-10T20:43:52.415340+0000","last_unstale":"2024-09-10T20:43:53.407746+0000","last_undegraded":"2024-09-10T20:43:53.407746+0000","last_fullsized":"2024-09-10T20:43:53.407746+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:00.817420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407960+0000","last_change":"2024-09-10T20:43:52.413462+0000","last_active":"2024-09-10T20:43:53.407960+0000","last_peered":"2024-09-10T20:43:53.407960+0000","last_clean":"2024-09-10T20:43:53.407960+0000","last_became_active":"2024-09-10T20:43:52.412062+0000","last_became_peered":"2024-09-10T20:43:52.412062+0000","last_unstale":"2024-09-10T20:43:53.407960+0000","last_undegraded":"2024-09-10T20:43:53.407960+0000","last_fullsized":"2024-09-10T20:43:53.407960+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:57.069937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477142+0000","last_change":"2024-09-10T20:43:52.412937+0000","last_active":"2024-09-10T20:43:53.477142+0000","last_peered":"2024-09-10T20:43:53.477142+0000","last_clean":"2024-09-10T20:43:53.477142+0000","last_became_active":"2024-09-10T20:43:52.412650+0000","last_became_peered":"2024-09-10T20:43:52.412650+0000","last_unstale":"2024-09-10T20:43:53.477142+0000","last_undegraded":"2024-09-10T20:43:53.477142+0000","last_fullsized":"2024-09-10T20:43:53.477142+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:48:58.854630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477262+0000","last_change":"2024-09-10T20:43:52.415534+0000","last_active":"2024-09-10T20:43:53.477262+0000","last_peered":"2024-09-10T20:43:53.477262+0000","last_clean":"2024-09-10T20:43:53.477262+0000","last_became_active":"2024-09-10T20:43:52.415260+0000","last_became_peered":"2024-09-10T20:43:52.415260+0000","last_unstale":"2024-09-10T20:43:53.477262+0000","last_undegraded":"2024-09-10T20:43:53.477262+0000","last_fullsized":"2024-09-10T20:43:53.477262+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:05.159002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478342+0000","last_change":"2024-09-10T20:43:52.418047+0000","last_active":"2024-09-10T20:43:53.478342+0000","last_peered":"2024-09-10T20:43:53.478342+0000","last_clean":"2024-09-10T20:43:53.478342+0000","last_became_active":"2024-09-10T20:43:52.417729+0000","last_became_peered":"2024-09-10T20:43:52.417729+0000","last_unstale":"2024-09-10T20:43:53.478342+0000","last_undegraded":"2024-09-10T20:43:53.478342+0000","last_fullsized":"2024-09-10T20:43:53.478342+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:41:43.768841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1f","version":"114'4","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406153+0000","last_change":"2024-09-10T20:43:51.401947+0000","last_active":"2024-09-10T20:43:53.406153+0000","last_peered":"2024-09-10T20:43:53.406153+0000","last_clean":"2024-09-10T20:43:53.406153+0000","last_became_active":"2024-09-10T20:43:51.401661+0000","last_became_peered":"2024-09-10T20:43:51.401661+0000","last_unstale":"2024-09-10T20:43:53.406153+0000","last_undegraded":"2024-09-10T20:43:53.406153+0000","last_fullsized":"2024-09-10T20:43:53.406153+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:54.753371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477983+0000","last_change":"2024-09-10T20:43:52.406537+0000","last_active":"2024-09-10T20:43:53.477983+0000","last_peered":"2024-09-10T20:43:53.477983+0000","last_clean":"2024-09-10T20:43:53.477983+0000","last_became_active":"2024-09-10T20:43:52.406184+0000","last_became_peered":"2024-09-10T20:43:52.406184+0000","last_unstale":"2024-09-10T20:43:53.477983+0000","last_undegraded":"2024-09-10T20:43:53.477983+0000","last_fullsized":"2024-09-10T20:43:53.477983+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:56:44.720530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1e","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477677+0000","last_change":"2024-09-10T20:43:51.399291+0000","last_active":"2024-09-10T20:43:53.477677+0000","last_peered":"2024-09-10T20:43:53.477677+0000","last_clean":"2024-09-10T20:43:53.477677+0000","last_became_active":"2024-09-10T20:43:51.398975+0000","last_became_peered":"2024-09-10T20:43:51.398975+0000","last_unstale":"2024-09-10T20:43:53.477677+0000","last_undegraded":"2024-09-10T20:43:53.477677+0000","last_fullsized":"2024-09-10T20:43:53.477677+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:10.016773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477694+0000","last_change":"2024-09-10T20:43:52.416656+0000","last_active":"2024-09-10T20:43:53.477694+0000","last_peered":"2024-09-10T20:43:53.477694+0000","last_clean":"2024-09-10T20:43:53.477694+0000","last_became_active":"2024-09-10T20:43:52.416312+0000","last_became_peered":"2024-09-10T20:43:52.416312+0000","last_unstale":"2024-09-10T20:43:53.477694+0000","last_undegraded":"2024-09-10T20:43:53.477694+0000","last_fullsized":"2024-09-10T20:43:53.477694+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:33.324330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1d","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408059+0000","last_change":"2024-09-10T20:43:51.400653+0000","last_active":"2024-09-10T20:43:53.408059+0000","last_peered":"2024-09-10T20:43:53.408059+0000","last_clean":"2024-09-10T20:43:53.408059+0000","last_became_active":"2024-09-10T20:43:51.400345+0000","last_became_peered":"2024-09-10T20:43:51.400345+0000","last_unstale":"2024-09-10T20:43:53.408059+0000","last_undegraded":"2024-09-10T20:43:53.408059+0000","last_fullsized":"2024-09-10T20:43:53.408059+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:55.549813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413265+0000","last_change":"2024-09-10T20:43:52.413607+0000","last_active":"2024-09-10T20:43:53.413265+0000","last_peered":"2024-09-10T20:43:53.413265+0000","last_clean":"2024-09-10T20:43:53.413265+0000","last_became_active":"2024-09-10T20:43:52.413342+0000","last_became_peered":"2024-09-10T20:43:52.413342+0000","last_unstale":"2024-09-10T20:43:53.413265+0000","last_undegraded":"2024-09-10T20:43:53.413265+0000","last_fullsized":"2024-09-10T20:43:53.413265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:41:43.768841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1c","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408442+0000","last_change":"2024-09-10T20:43:51.400665+0000","last_active":"2024-09-10T20:43:53.408442+0000","last_peered":"2024-09-10T20:43:53.408442+0000","last_clean":"2024-09-10T20:43:53.408442+0000","last_became_active":"2024-09-10T20:43:51.400375+0000","last_became_peered":"2024-09-10T20:43:51.400375+0000","last_unstale":"2024-09-10T20:43:53.408442+0000","last_undegraded":"2024-09-10T20:43:53.408442+0000","last_fullsized":"2024-09-10T20:43:53.408442+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:28.493731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478337+0000","last_change":"2024-09-10T20:43:52.421796+0000","last_active":"2024-09-10T20:43:53.478337+0000","last_peered":"2024-09-10T20:43:53.478337+0000","last_clean":"2024-09-10T20:43:53.478337+0000","last_became_active":"2024-09-10T20:43:52.421502+0000","last_became_peered":"2024-09-10T20:43:52.421502+0000","last_unstale":"2024-09-10T20:43:53.478337+0000","last_undegraded":"2024-09-10T20:43:53.478337+0000","last_fullsized":"2024-09-10T20:43:53.478337+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:45.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1b","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407452+0000","last_change":"2024-09-10T20:43:51.391089+0000","last_active":"2024-09-10T20:43:53.407452+0000","last_peered":"2024-09-10T20:43:53.407452+0000","last_clean":"2024-09-10T20:43:53.407452+0000","last_became_active":"2024-09-10T20:43:51.390763+0000","last_became_peered":"2024-09-10T20:43:51.390763+0000","last_unstale":"2024-09-10T20:43:53.407452+0000","last_undegraded":"2024-09-10T20:43:53.407452+0000","last_fullsized":"2024-09-10T20:43:53.407452+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:08.578252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413156+0000","last_change":"2024-09-10T20:43:52.415078+0000","last_active":"2024-09-10T20:43:53.413156+0000","last_peered":"2024-09-10T20:43:53.413156+0000","last_clean":"2024-09-10T20:43:53.413156+0000","last_became_active":"2024-09-10T20:43:52.414436+0000","last_became_peered":"2024-09-10T20:43:52.414436+0000","last_unstale":"2024-09-10T20:43:53.413156+0000","last_undegraded":"2024-09-10T20:43:53.413156+0000","last_fullsized":"2024-09-10T20:43:53.413156+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:27.850900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478488+0000","last_change":"2024-09-10T20:43:51.400219+0000","last_active":"2024-09-10T20:43:53.478488+0000","last_peered":"2024-09-10T20:43:53.478488+0000","last_clean":"2024-09-10T20:43:53.478488+0000","last_became_active":"2024-09-10T20:43:51.399942+0000","last_became_peered":"2024-09-10T20:43:51.399942+0000","last_unstale":"2024-09-10T20:43:53.478488+0000","last_undegraded":"2024-09-10T20:43:53.478488+0000","last_fullsized":"2024-09-10T20:43:53.478488+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:20.699035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412927+0000","last_change":"2024-09-10T20:43:52.406959+0000","last_active":"2024-09-10T20:43:53.412927+0000","last_peered":"2024-09-10T20:43:53.412927+0000","last_clean":"2024-09-10T20:43:53.412927+0000","last_became_active":"2024-09-10T20:43:52.406668+0000","last_became_peered":"2024-09-10T20:43:52.406668+0000","last_unstale":"2024-09-10T20:43:53.412927+0000","last_undegraded":"2024-09-10T20:43:53.412927+0000","last_fullsized":"2024-09-10T20:43:53.412927+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:20.178183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477188+0000","last_change":"2024-09-10T20:43:51.390469+0000","last_active":"2024-09-10T20:43:53.477188+0000","last_peered":"2024-09-10T20:43:53.477188+0000","last_clean":"2024-09-10T20:43:53.477188+0000","last_became_active":"2024-09-10T20:43:51.390165+0000","last_became_peered":"2024-09-10T20:43:51.390165+0000","last_unstale":"2024-09-10T20:43:53.477188+0000","last_undegraded":"2024-09-10T20:43:53.477188+0000","last_fullsized":"2024-09-10T20:43:53.477188+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:09.640298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407638+0000","last_change":"2024-09-10T20:43:52.403234+0000","last_active":"2024-09-10T20:43:53.407638+0000","last_peered":"2024-09-10T20:43:53.407638+0000","last_clean":"2024-09-10T20:43:53.407638+0000","last_became_active":"2024-09-10T20:43:52.402948+0000","last_became_peered":"2024-09-10T20:43:52.402948+0000","last_unstale":"2024-09-10T20:43:53.407638+0000","last_undegraded":"2024-09-10T20:43:53.407638+0000","last_fullsized":"2024-09-10T20:43:53.407638+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:08:56.610321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407347+0000","last_change":"2024-09-10T20:43:51.397159+0000","last_active":"2024-09-10T20:43:53.407347+0000","last_peered":"2024-09-10T20:43:53.407347+0000","last_clean":"2024-09-10T20:43:53.407347+0000","last_became_active":"2024-09-10T20:43:51.396880+0000","last_became_peered":"2024-09-10T20:43:51.396880+0000","last_unstale":"2024-09-10T20:43:53.407347+0000","last_undegraded":"2024-09-10T20:43:53.407347+0000","last_fullsized":"2024-09-10T20:43:53.407347+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:09:25.981861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.409016+0000","last_change":"2024-09-10T20:43:52.404136+0000","last_active":"2024-09-10T20:43:53.409016+0000","last_peered":"2024-09-10T20:43:53.409016+0000","last_clean":"2024-09-10T20:43:53.409016+0000","last_became_active":"2024-09-10T20:43:52.403458+0000","last_became_peered":"2024-09-10T20:43:52.403458+0000","last_unstale":"2024-09-10T20:43:53.409016+0000","last_undegraded":"2024-09-10T20:43:53.409016+0000","last_fullsized":"2024-09-10T20:43:53.409016+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:38.068750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413358+0000","last_change":"2024-09-10T20:43:51.400577+0000","last_active":"2024-09-10T20:43:53.413358+0000","last_peered":"2024-09-10T20:43:53.413358+0000","last_clean":"2024-09-10T20:43:53.413358+0000","last_became_active":"2024-09-10T20:43:51.400279+0000","last_became_peered":"2024-09-10T20:43:51.400279+0000","last_unstale":"2024-09-10T20:43:53.413358+0000","last_undegraded":"2024-09-10T20:43:53.413358+0000","last_fullsized":"2024-09-10T20:43:53.413358+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:20.816146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406054+0000","last_change":"2024-09-10T20:43:52.415847+0000","last_active":"2024-09-10T20:43:53.406054+0000","last_peered":"2024-09-10T20:43:53.406054+0000","last_clean":"2024-09-10T20:43:53.406054+0000","last_became_active":"2024-09-10T20:43:52.415554+0000","last_became_peered":"2024-09-10T20:43:52.415554+0000","last_unstale":"2024-09-10T20:43:53.406054+0000","last_undegraded":"2024-09-10T20:43:53.406054+0000","last_fullsized":"2024-09-10T20:43:53.406054+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:25.749401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478556+0000","last_change":"2024-09-10T20:43:51.403485+0000","last_active":"2024-09-10T20:43:53.478556+0000","last_peered":"2024-09-10T20:43:53.478556+0000","last_clean":"2024-09-10T20:43:53.478556+0000","last_became_active":"2024-09-10T20:43:51.403151+0000","last_became_peered":"2024-09-10T20:43:51.403151+0000","last_unstale":"2024-09-10T20:43:53.478556+0000","last_undegraded":"2024-09-10T20:43:53.478556+0000","last_fullsized":"2024-09-10T20:43:53.478556+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:00.680902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413428+0000","last_change":"2024-09-10T20:43:52.424582+0000","last_active":"2024-09-10T20:43:53.413428+0000","last_peered":"2024-09-10T20:43:53.413428+0000","last_clean":"2024-09-10T20:43:53.413428+0000","last_became_active":"2024-09-10T20:43:52.424298+0000","last_became_peered":"2024-09-10T20:43:52.424298+0000","last_unstale":"2024-09-10T20:43:53.413428+0000","last_undegraded":"2024-09-10T20:43:53.413428+0000","last_fullsized":"2024-09-10T20:43:53.413428+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:05:45.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.15","version":"114'3","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406064+0000","last_change":"2024-09-10T20:43:51.400582+0000","last_active":"2024-09-10T20:43:53.406064+0000","last_peered":"2024-09-10T20:43:53.406064+0000","last_clean":"2024-09-10T20:43:53.406064+0000","last_became_active":"2024-09-10T20:43:51.400291+0000","last_became_peered":"2024-09-10T20:43:51.400291+0000","last_unstale":"2024-09-10T20:43:53.406064+0000","last_undegraded":"2024-09-10T20:43:53.406064+0000","last_fullsized":"2024-09-10T20:43:53.406064+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:15.784132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477506+0000","last_change":"2024-09-10T20:43:52.419497+0000","last_active":"2024-09-10T20:43:53.477506+0000","last_peered":"2024-09-10T20:43:53.477506+0000","last_clean":"2024-09-10T20:43:53.477506+0000","last_became_active":"2024-09-10T20:43:52.419221+0000","last_became_peered":"2024-09-10T20:43:52.419221+0000","last_unstale":"2024-09-10T20:43:53.477506+0000","last_undegraded":"2024-09-10T20:43:53.477506+0000","last_fullsized":"2024-09-10T20:43:53.477506+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:41.304730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.14","version":"114'6","reported_seq":21,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477288+0000","last_change":"2024-09-10T20:43:51.402444+0000","last_active":"2024-09-10T20:43:53.477288+0000","last_peered":"2024-09-10T20:43:53.477288+0000","last_clean":"2024-09-10T20:43:53.477288+0000","last_became_active":"2024-09-10T20:43:51.402176+0000","last_became_peered":"2024-09-10T20:43:51.402176+0000","last_unstale":"2024-09-10T20:43:53.477288+0000","last_undegraded":"2024-09-10T20:43:53.477288+0000","last_fullsized":"2024-09-10T20:43:53.477288+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:56:36.223340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478398+0000","last_change":"2024-09-10T20:43:52.415634+0000","last_active":"2024-09-10T20:43:53.478398+0000","last_peered":"2024-09-10T20:43:53.478398+0000","last_clean":"2024-09-10T20:43:53.478398+0000","last_became_active":"2024-09-10T20:43:52.414726+0000","last_became_peered":"2024-09-10T20:43:52.414726+0000","last_unstale":"2024-09-10T20:43:53.478398+0000","last_undegraded":"2024-09-10T20:43:53.478398+0000","last_fullsized":"2024-09-10T20:43:53.478398+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:52:11.137255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477845+0000","last_change":"2024-09-10T20:43:51.403836+0000","last_active":"2024-09-10T20:43:53.477845+0000","last_peered":"2024-09-10T20:43:53.477845+0000","last_clean":"2024-09-10T20:43:53.477845+0000","last_became_active":"2024-09-10T20:43:51.403505+0000","last_became_peered":"2024-09-10T20:43:51.403505+0000","last_unstale":"2024-09-10T20:43:53.477845+0000","last_undegraded":"2024-09-10T20:43:53.477845+0000","last_fullsized":"2024-09-10T20:43:53.477845+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:50:23.673158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477587+0000","last_change":"2024-09-10T20:43:52.417801+0000","last_active":"2024-09-10T20:43:53.477587+0000","last_peered":"2024-09-10T20:43:53.477587+0000","last_clean":"2024-09-10T20:43:53.477587+0000","last_became_active":"2024-09-10T20:43:52.417492+0000","last_became_peered":"2024-09-10T20:43:52.417492+0000","last_unstale":"2024-09-10T20:43:53.477587+0000","last_undegraded":"2024-09-10T20:43:53.477587+0000","last_fullsized":"2024-09-10T20:43:53.477587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:01.745662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407153+0000","last_change":"2024-09-10T20:43:51.398171+0000","last_active":"2024-09-10T20:43:53.407153+0000","last_peered":"2024-09-10T20:43:53.407153+0000","last_clean":"2024-09-10T20:43:53.407153+0000","last_became_active":"2024-09-10T20:43:51.397803+0000","last_became_peered":"2024-09-10T20:43:51.397803+0000","last_unstale":"2024-09-10T20:43:53.407153+0000","last_undegraded":"2024-09-10T20:43:53.407153+0000","last_fullsized":"2024-09-10T20:43:53.407153+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:21:16.651238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412857+0000","last_change":"2024-09-10T20:43:52.409293+0000","last_active":"2024-09-10T20:43:53.412857+0000","last_peered":"2024-09-10T20:43:53.412857+0000","last_clean":"2024-09-10T20:43:53.412857+0000","last_became_active":"2024-09-10T20:43:52.408940+0000","last_became_peered":"2024-09-10T20:43:52.408940+0000","last_unstale":"2024-09-10T20:43:53.412857+0000","last_undegraded":"2024-09-10T20:43:53.412857+0000","last_fullsized":"2024-09-10T20:43:53.412857+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:39:39.173777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.10","version":"114'8","reported_seq":23,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412788+0000","last_change":"2024-09-10T20:43:51.399871+0000","last_active":"2024-09-10T20:43:53.412788+0000","last_peered":"2024-09-10T20:43:53.412788+0000","last_clean":"2024-09-10T20:43:53.412788+0000","last_became_active":"2024-09-10T20:43:51.399549+0000","last_became_peered":"2024-09-10T20:43:51.399549+0000","last_unstale":"2024-09-10T20:43:53.412788+0000","last_undegraded":"2024-09-10T20:43:53.412788+0000","last_fullsized":"2024-09-10T20:43:53.412788+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:30:20.003721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406006+0000","last_change":"2024-09-10T20:43:52.409212+0000","last_active":"2024-09-10T20:43:53.406006+0000","last_peered":"2024-09-10T20:43:53.406006+0000","last_clean":"2024-09-10T20:43:53.406006+0000","last_became_active":"2024-09-10T20:43:52.408926+0000","last_became_peered":"2024-09-10T20:43:52.408926+0000","last_unstale":"2024-09-10T20:43:53.406006+0000","last_undegraded":"2024-09-10T20:43:53.406006+0000","last_fullsized":"2024-09-10T20:43:53.406006+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:21.545302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.f","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413414+0000","last_change":"2024-09-10T20:43:51.406171+0000","last_active":"2024-09-10T20:43:53.413414+0000","last_peered":"2024-09-10T20:43:53.413414+0000","last_clean":"2024-09-10T20:43:53.413414+0000","last_became_active":"2024-09-10T20:43:51.405833+0000","last_became_peered":"2024-09-10T20:43:51.405833+0000","last_unstale":"2024-09-10T20:43:53.413414+0000","last_undegraded":"2024-09-10T20:43:53.413414+0000","last_fullsized":"2024-09-10T20:43:53.413414+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:44.079778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406118+0000","last_change":"2024-09-10T20:43:52.402032+0000","last_active":"2024-09-10T20:43:53.406118+0000","last_peered":"2024-09-10T20:43:53.406118+0000","last_clean":"2024-09-10T20:43:53.406118+0000","last_became_active":"2024-09-10T20:43:52.401721+0000","last_became_peered":"2024-09-10T20:43:52.401721+0000","last_unstale":"2024-09-10T20:43:53.406118+0000","last_undegraded":"2024-09-10T20:43:53.406118+0000","last_fullsized":"2024-09-10T20:43:53.406118+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:59.645506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407807+0000","last_change":"2024-09-10T20:43:51.392667+0000","last_active":"2024-09-10T20:43:53.407807+0000","last_peered":"2024-09-10T20:43:53.407807+0000","last_clean":"2024-09-10T20:43:53.407807+0000","last_became_active":"2024-09-10T20:43:51.392369+0000","last_became_peered":"2024-09-10T20:43:51.392369+0000","last_unstale":"2024-09-10T20:43:53.407807+0000","last_undegraded":"2024-09-10T20:43:53.407807+0000","last_fullsized":"2024-09-10T20:43:53.407807+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:52.657407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408339+0000","last_change":"2024-09-10T20:43:52.404514+0000","last_active":"2024-09-10T20:43:53.408339+0000","last_peered":"2024-09-10T20:43:53.408339+0000","last_clean":"2024-09-10T20:43:53.408339+0000","last_became_active":"2024-09-10T20:43:52.404192+0000","last_became_peered":"2024-09-10T20:43:52.404192+0000","last_unstale":"2024-09-10T20:43:53.408339+0000","last_undegraded":"2024-09-10T20:43:53.408339+0000","last_fullsized":"2024-09-10T20:43:53.408339+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:04.186296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.d","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477853+0000","last_change":"2024-09-10T20:43:51.393006+0000","last_active":"2024-09-10T20:43:53.477853+0000","last_peered":"2024-09-10T20:43:53.477853+0000","last_clean":"2024-09-10T20:43:53.477853+0000","last_became_active":"2024-09-10T20:43:51.392652+0000","last_became_peered":"2024-09-10T20:43:51.392652+0000","last_unstale":"2024-09-10T20:43:53.477853+0000","last_undegraded":"2024-09-10T20:43:53.477853+0000","last_fullsized":"2024-09-10T20:43:53.477853+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:28.100705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408524+0000","last_change":"2024-09-10T20:43:52.409343+0000","last_active":"2024-09-10T20:43:53.408524+0000","last_peered":"2024-09-10T20:43:53.408524+0000","last_clean":"2024-09-10T20:43:53.408524+0000","last_became_active":"2024-09-10T20:43:52.409059+0000","last_became_peered":"2024-09-10T20:43:52.409059+0000","last_unstale":"2024-09-10T20:43:53.408524+0000","last_undegraded":"2024-09-10T20:43:53.408524+0000","last_fullsized":"2024-09-10T20:43:53.408524+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:28.542728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.405953+0000","last_change":"2024-09-10T20:43:51.401556+0000","last_active":"2024-09-10T20:43:53.405953+0000","last_peered":"2024-09-10T20:43:53.405953+0000","last_clean":"2024-09-10T20:43:53.405953+0000","last_became_active":"2024-09-10T20:43:51.401271+0000","last_became_peered":"2024-09-10T20:43:51.401271+0000","last_unstale":"2024-09-10T20:43:53.405953+0000","last_undegraded":"2024-09-10T20:43:53.405953+0000","last_fullsized":"2024-09-10T20:43:53.405953+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:41:51.957414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408119+0000","last_change":"2024-09-10T20:43:52.413366+0000","last_active":"2024-09-10T20:43:53.408119+0000","last_peered":"2024-09-10T20:43:53.408119+0000","last_clean":"2024-09-10T20:43:53.408119+0000","last_became_active":"2024-09-10T20:43:52.412744+0000","last_became_peered":"2024-09-10T20:43:52.412744+0000","last_unstale":"2024-09-10T20:43:53.408119+0000","last_undegraded":"2024-09-10T20:43:53.408119+0000","last_fullsized":"2024-09-10T20:43:53.408119+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:45.643260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.b","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477493+0000","last_change":"2024-09-10T20:43:51.392907+0000","last_active":"2024-09-10T20:43:53.477493+0000","last_peered":"2024-09-10T20:43:53.477493+0000","last_clean":"2024-09-10T20:43:53.477493+0000","last_became_active":"2024-09-10T20:43:51.392435+0000","last_became_peered":"2024-09-10T20:43:51.392435+0000","last_unstale":"2024-09-10T20:43:53.477493+0000","last_undegraded":"2024-09-10T20:43:53.477493+0000","last_fullsized":"2024-09-10T20:43:53.477493+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:50:51.186640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477170+0000","last_change":"2024-09-10T20:43:52.409667+0000","last_active":"2024-09-10T20:43:53.477170+0000","last_peered":"2024-09-10T20:43:53.477170+0000","last_clean":"2024-09-10T20:43:53.477170+0000","last_became_active":"2024-09-10T20:43:52.407879+0000","last_became_peered":"2024-09-10T20:43:52.407879+0000","last_unstale":"2024-09-10T20:43:53.477170+0000","last_undegraded":"2024-09-10T20:43:53.477170+0000","last_fullsized":"2024-09-10T20:43:53.477170+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:41.570822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478473+0000","last_change":"2024-09-10T20:43:51.399348+0000","last_active":"2024-09-10T20:43:53.478473+0000","last_peered":"2024-09-10T20:43:53.478473+0000","last_clean":"2024-09-10T20:43:53.478473+0000","last_became_active":"2024-09-10T20:43:51.399092+0000","last_became_peered":"2024-09-10T20:43:51.399092+0000","last_unstale":"2024-09-10T20:43:53.478473+0000","last_undegraded":"2024-09-10T20:43:53.478473+0000","last_fullsized":"2024-09-10T20:43:53.478473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:44:32.315572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477847+0000","last_change":"2024-09-10T20:43:52.415252+0000","last_active":"2024-09-10T20:43:53.477847+0000","last_peered":"2024-09-10T20:43:53.477847+0000","last_clean":"2024-09-10T20:43:53.477847+0000","last_became_active":"2024-09-10T20:43:52.414961+0000","last_became_peered":"2024-09-10T20:43:52.414961+0000","last_unstale":"2024-09-10T20:43:53.477847+0000","last_undegraded":"2024-09-10T20:43:53.477847+0000","last_fullsized":"2024-09-10T20:43:53.477847+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:43.049678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477914+0000","last_change":"2024-09-10T20:43:51.402453+0000","last_active":"2024-09-10T20:43:53.477914+0000","last_peered":"2024-09-10T20:43:53.477914+0000","last_clean":"2024-09-10T20:43:53.477914+0000","last_became_active":"2024-09-10T20:43:51.402098+0000","last_became_peered":"2024-09-10T20:43:51.402098+0000","last_unstale":"2024-09-10T20:43:53.477914+0000","last_undegraded":"2024-09-10T20:43:53.477914+0000","last_fullsized":"2024-09-10T20:43:53.477914+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:45:56.629766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412917+0000","last_change":"2024-09-10T20:43:52.418107+0000","last_active":"2024-09-10T20:43:53.412917+0000","last_peered":"2024-09-10T20:43:53.412917+0000","last_clean":"2024-09-10T20:43:53.412917+0000","last_became_active":"2024-09-10T20:43:52.417761+0000","last_became_peered":"2024-09-10T20:43:52.417761+0000","last_unstale":"2024-09-10T20:43:53.412917+0000","last_undegraded":"2024-09-10T20:43:53.412917+0000","last_fullsized":"2024-09-10T20:43:53.412917+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:39.479110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407097+0000","last_change":"2024-09-10T20:43:51.386361+0000","last_active":"2024-09-10T20:43:53.407097+0000","last_peered":"2024-09-10T20:43:53.407097+0000","last_clean":"2024-09-10T20:43:53.407097+0000","last_became_active":"2024-09-10T20:43:51.386073+0000","last_became_peered":"2024-09-10T20:43:51.386073+0000","last_unstale":"2024-09-10T20:43:53.407097+0000","last_undegraded":"2024-09-10T20:43:53.407097+0000","last_fullsized":"2024-09-10T20:43:53.407097+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:09:15.809400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477613+0000","last_change":"2024-09-10T20:43:52.417449+0000","last_active":"2024-09-10T20:43:53.477613+0000","last_peered":"2024-09-10T20:43:53.477613+0000","last_clean":"2024-09-10T20:43:53.477613+0000","last_became_active":"2024-09-10T20:43:52.417159+0000","last_became_peered":"2024-09-10T20:43:52.417159+0000","last_unstale":"2024-09-10T20:43:53.477613+0000","last_undegraded":"2024-09-10T20:43:53.477613+0000","last_fullsized":"2024-09-10T20:43:53.477613+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:15.346163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.7","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478684+0000","last_change":"2024-09-10T20:43:51.400298+0000","last_active":"2024-09-10T20:43:53.478684+0000","last_peered":"2024-09-10T20:43:53.478684+0000","last_clean":"2024-09-10T20:43:53.478684+0000","last_became_active":"2024-09-10T20:43:51.400009+0000","last_became_peered":"2024-09-10T20:43:51.400009+0000","last_unstale":"2024-09-10T20:43:53.478684+0000","last_undegraded":"2024-09-10T20:43:53.478684+0000","last_fullsized":"2024-09-10T20:43:53.478684+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:46:29.035453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408852+0000","last_change":"2024-09-10T20:43:52.411523+0000","last_active":"2024-09-10T20:43:53.408852+0000","last_peered":"2024-09-10T20:43:53.408852+0000","last_clean":"2024-09-10T20:43:53.408852+0000","last_became_active":"2024-09-10T20:43:52.411245+0000","last_became_peered":"2024-09-10T20:43:52.411245+0000","last_unstale":"2024-09-10T20:43:53.408852+0000","last_undegraded":"2024-09-10T20:43:53.408852+0000","last_fullsized":"2024-09-10T20:43:53.408852+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:55.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.6","version":"115'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412242+0000","last_change":"2024-09-10T20:43:51.402779+0000","last_active":"2024-09-10T20:43:53.412242+0000","last_peered":"2024-09-10T20:43:53.412242+0000","last_clean":"2024-09-10T20:43:53.412242+0000","last_became_active":"2024-09-10T20:43:51.402448+0000","last_became_peered":"2024-09-10T20:43:51.402448+0000","last_unstale":"2024-09-10T20:43:53.412242+0000","last_undegraded":"2024-09-10T20:43:53.412242+0000","last_fullsized":"2024-09-10T20:43:53.412242+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:14:34.101637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478404+0000","last_change":"2024-09-10T20:43:52.414285+0000","last_active":"2024-09-10T20:43:53.478404+0000","last_peered":"2024-09-10T20:43:53.478404+0000","last_clean":"2024-09-10T20:43:53.478404+0000","last_became_active":"2024-09-10T20:43:52.413729+0000","last_became_peered":"2024-09-10T20:43:52.413729+0000","last_unstale":"2024-09-10T20:43:53.478404+0000","last_undegraded":"2024-09-10T20:43:53.478404+0000","last_fullsized":"2024-09-10T20:43:53.478404+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:19.949840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478335+0000","last_change":"2024-09-10T20:43:51.406447+0000","last_active":"2024-09-10T20:43:53.478335+0000","last_peered":"2024-09-10T20:43:53.478335+0000","last_clean":"2024-09-10T20:43:53.478335+0000","last_became_active":"2024-09-10T20:43:51.406221+0000","last_became_peered":"2024-09-10T20:43:51.406221+0000","last_unstale":"2024-09-10T20:43:53.478335+0000","last_undegraded":"2024-09-10T20:43:53.478335+0000","last_fullsized":"2024-09-10T20:43:53.478335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:30.540249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477789+0000","last_change":"2024-09-10T20:43:52.412318+0000","last_active":"2024-09-10T20:43:53.477789+0000","last_peered":"2024-09-10T20:43:53.477789+0000","last_clean":"2024-09-10T20:43:53.477789+0000","last_became_active":"2024-09-10T20:43:52.412054+0000","last_became_peered":"2024-09-10T20:43:52.412054+0000","last_unstale":"2024-09-10T20:43:53.477789+0000","last_undegraded":"2024-09-10T20:43:53.477789+0000","last_fullsized":"2024-09-10T20:43:53.477789+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:13.044452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478210+0000","last_change":"2024-09-10T20:43:51.403560+0000","last_active":"2024-09-10T20:43:53.478210+0000","last_peered":"2024-09-10T20:43:53.478210+0000","last_clean":"2024-09-10T20:43:53.478210+0000","last_became_active":"2024-09-10T20:43:51.403283+0000","last_became_peered":"2024-09-10T20:43:51.403283+0000","last_unstale":"2024-09-10T20:43:53.478210+0000","last_undegraded":"2024-09-10T20:43:53.478210+0000","last_fullsized":"2024-09-10T20:43:53.478210+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:13:50.843696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412933+0000","last_change":"2024-09-10T20:43:52.410397+0000","last_active":"2024-09-10T20:43:53.412933+0000","last_peered":"2024-09-10T20:43:53.412933+0000","last_clean":"2024-09-10T20:43:53.412933+0000","last_became_active":"2024-09-10T20:43:52.410096+0000","last_became_peered":"2024-09-10T20:43:52.410096+0000","last_unstale":"2024-09-10T20:43:53.412933+0000","last_undegraded":"2024-09-10T20:43:53.412933+0000","last_fullsized":"2024-09-10T20:43:53.412933+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:34:31.913191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407538+0000","last_change":"2024-09-10T20:43:51.402457+0000","last_active":"2024-09-10T20:43:53.407538+0000","last_peered":"2024-09-10T20:43:53.407538+0000","last_clean":"2024-09-10T20:43:53.407538+0000","last_became_active":"2024-09-10T20:43:51.402198+0000","last_became_peered":"2024-09-10T20:43:51.402198+0000","last_unstale":"2024-09-10T20:43:53.407538+0000","last_undegraded":"2024-09-10T20:43:53.407538+0000","last_fullsized":"2024-09-10T20:43:53.407538+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:24.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413490+0000","last_change":"2024-09-10T20:43:52.414717+0000","last_active":"2024-09-10T20:43:53.413490+0000","last_peered":"2024-09-10T20:43:53.413490+0000","last_clean":"2024-09-10T20:43:53.413490+0000","last_became_active":"2024-09-10T20:43:52.414105+0000","last_became_peered":"2024-09-10T20:43:52.414105+0000","last_unstale":"2024-09-10T20:43:53.413490+0000","last_undegraded":"2024-09-10T20:43:53.413490+0000","last_fullsized":"2024-09-10T20:43:53.413490+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:02.446253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413027+0000","last_change":"2024-09-10T20:43:51.401690+0000","last_active":"2024-09-10T20:43:53.413027+0000","last_peered":"2024-09-10T20:43:53.413027+0000","last_clean":"2024-09-10T20:43:53.413027+0000","last_became_active":"2024-09-10T20:43:51.401420+0000","last_became_peered":"2024-09-10T20:43:51.401420+0000","last_unstale":"2024-09-10T20:43:53.413027+0000","last_undegraded":"2024-09-10T20:43:53.413027+0000","last_fullsized":"2024-09-10T20:43:53.413027+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:29:04.826111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477701+0000","last_change":"2024-09-10T20:43:52.428756+0000","last_active":"2024-09-10T20:43:53.477701+0000","last_peered":"2024-09-10T20:43:53.477701+0000","last_clean":"2024-09-10T20:43:53.477701+0000","last_became_active":"2024-09-10T20:43:52.428176+0000","last_became_peered":"2024-09-10T20:43:52.428176+0000","last_unstale":"2024-09-10T20:43:53.477701+0000","last_undegraded":"2024-09-10T20:43:53.477701+0000","last_fullsized":"2024-09-10T20:43:53.477701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:59:37.007457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407263+0000","last_change":"2024-09-10T20:43:51.398079+0000","last_active":"2024-09-10T20:43:53.407263+0000","last_peered":"2024-09-10T20:43:53.407263+0000","last_clean":"2024-09-10T20:43:53.407263+0000","last_became_active":"2024-09-10T20:43:51.397757+0000","last_became_peered":"2024-09-10T20:43:51.397757+0000","last_unstale":"2024-09-10T20:43:53.407263+0000","last_undegraded":"2024-09-10T20:43:53.407263+0000","last_fullsized":"2024-09-10T20:43:53.407263+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:12.990418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477977+0000","last_change":"2024-09-10T20:43:52.415122+0000","last_active":"2024-09-10T20:43:53.477977+0000","last_peered":"2024-09-10T20:43:53.477977+0000","last_clean":"2024-09-10T20:43:53.477977+0000","last_became_active":"2024-09-10T20:43:52.414804+0000","last_became_peered":"2024-09-10T20:43:52.414804+0000","last_unstale":"2024-09-10T20:43:53.477977+0000","last_undegraded":"2024-09-10T20:43:53.477977+0000","last_fullsized":"2024-09-10T20:43:53.477977+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:44:54.549791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477725+0000","last_change":"2024-09-10T20:43:51.403657+0000","last_active":"2024-09-10T20:43:53.477725+0000","last_peered":"2024-09-10T20:43:53.477725+0000","last_clean":"2024-09-10T20:43:53.477725+0000","last_became_active":"2024-09-10T20:43:51.403362+0000","last_became_peered":"2024-09-10T20:43:51.403362+0000","last_unstale":"2024-09-10T20:43:53.477725+0000","last_undegraded":"2024-09-10T20:43:53.477725+0000","last_fullsized":"2024-09-10T20:43:53.477725+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:31.071123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477582+0000","last_change":"2024-09-10T20:43:52.409599+0000","last_active":"2024-09-10T20:43:53.477582+0000","last_peered":"2024-09-10T20:43:53.477582+0000","last_clean":"2024-09-10T20:43:53.477582+0000","last_became_active":"2024-09-10T20:43:52.409312+0000","last_became_peered":"2024-09-10T20:43:52.409312+0000","last_unstale":"2024-09-10T20:43:53.477582+0000","last_undegraded":"2024-09-10T20:43:53.477582+0000","last_fullsized":"2024-09-10T20:43:53.477582+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:07.867978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.13","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413322+0000","last_change":"2024-09-10T20:43:51.404408+0000","last_active":"2024-09-10T20:43:53.413322+0000","last_peered":"2024-09-10T20:43:53.413322+0000","last_clean":"2024-09-10T20:43:53.413322+0000","last_became_active":"2024-09-10T20:43:51.403983+0000","last_became_peered":"2024-09-10T20:43:51.403983+0000","last_unstale":"2024-09-10T20:43:53.413322+0000","last_undegraded":"2024-09-10T20:43:53.413322+0000","last_fullsized":"2024-09-10T20:43:53.413322+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:31.071123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":312,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.476848+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:43:53.476848+0000","last_peered":"2024-09-10T20:43:53.476848+0000","last_clean":"2024-09-10T20:43:53.476848+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:43:53.476848+0000","last_undegraded":"2024-09-10T20:43:53.476848+0000","last_fullsized":"2024-09-10T20:43:53.476848+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542218,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1421312,"data_stored":623719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542219,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":630013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542217,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33164,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338676,"statfs":{"total":96636764160,"available":96602804224,"internally_reserved":0,"allocated":2023424,"data_stored":1214795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542217,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1429504,"data_stored":630013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509515,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1433600,"data_stored":630083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509514,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1437696,"data_stored":630685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509514,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37348,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334492,"statfs":{"total":96636764160,"available":96598519808,"internally_reserved":0,"allocated":2039808,"data_stored":1222015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509514,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1449984,"data_stored":631681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:43:56.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:43:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 -- 172.21.15.28:0/2127044470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 msgr2=0x7f311815e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 --2- 172.21.15.28:0/2127044470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311815e6e0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f31180d1d10 tx=0x7f310802a560 comp rx=0 tx=0).stop 2024-09-10T20:43:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 -- 172.21.15.28:0/2127044470 shutdown_connections 2024-09-10T20:43:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 --2- 172.21.15.28:0/2127044470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311815e6e0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 --2- 172.21.15.28:0/2127044470 >> v2:172.21.15.159:3300/0 conn(0x7f311815ec20 0x7f3118070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 -- 172.21.15.28:0/2127044470 >> 172.21.15.28:0/2127044470 conn(0x7f311806dbf0 msgr2=0x7f311806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 -- 172.21.15.28:0/2127044470 shutdown_connections 2024-09-10T20:43:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.565+0000 7f311fe8c640 1 -- 172.21.15.28:0/2127044470 wait complete. 2024-09-10T20:43:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.566+0000 7f311fe8c640 1 Processor -- start 2024-09-10T20:43:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.566+0000 7f311fe8c640 1 -- start start 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311fe8c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31180d0c80 0x7f311814c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311fe8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311fe8c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f311815ec20 0x7f311806d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311fe8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3118070940 con 0x7f311815bc40 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311fe8c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f31180707c0 con 0x7f31180d0c80 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311fe8c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3118070ac0 con 0x7f311815ec20 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311806cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53192/0 (socket says 172.21.15.28:53192) 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 -- 172.21.15.28:0/1740960108 learned_addr learned my addr 172.21.15.28:0/1740960108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311e429640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f311815ec20 0x7f311806d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311dc28640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31180d0c80 0x7f311814c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 -- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f311815ec20 msgr2=0x7f311806d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f311815ec20 0x7f311806d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 -- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31180d0c80 msgr2=0x7f311814c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31180d0c80 0x7f311814c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.567+0000 7f311d427640 1 -- 172.21.15.28:0/1740960108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311814d3d0 con 0x7f311815bc40 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.568+0000 7f311d427640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311806cae0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3108000c00 tx=0x7f3108003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.568+0000 7f310e7fc640 1 -- 172.21.15.28:0/1740960108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310803f070 con 0x7f311815bc40 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.568+0000 7f310e7fc640 1 -- 172.21.15.28:0/1740960108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31080047a0 con 0x7f311815bc40 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.568+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31181f5ec0 con 0x7f311815bc40 2024-09-10T20:43:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.568+0000 7f310e7fc640 1 -- 172.21.15.28:0/1740960108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108035030 con 0x7f311815bc40 2024-09-10T20:43:56.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.568+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31181f6480 con 0x7f311815bc40 2024-09-10T20:43:56.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.569+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311815e6e0 con 0x7f311815bc40 2024-09-10T20:43:56.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.569+0000 7f310e7fc640 1 -- 172.21.15.28:0/1740960108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f3108004bb0 con 0x7f311815bc40 2024-09-10T20:43:56.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.570+0000 7f310e7fc640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f30fc078420 0x7f30fc07a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.570+0000 7f310e7fc640 1 -- 172.21.15.28:0/1740960108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f31080bb5f0 con 0x7f311815bc40 2024-09-10T20:43:56.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.570+0000 7f311dc28640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f30fc078420 0x7f30fc07a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.570+0000 7f311dc28640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f30fc078420 0x7f30fc07a8e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f31140017c0 tx=0x7f3114009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.574+0000 7f310e7fc640 1 -- 172.21.15.28:0/1740960108 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3108083440 con 0x7f311815bc40 2024-09-10T20:43:56.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.756+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f31180d0a60 con 0x7f30fc078420 2024-09-10T20:43:56.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.760+0000 7f310e7fc640 1 -- 172.21.15.28:0/1740960108 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f31180d0a60 con 0x7f30fc078420 2024-09-10T20:43:56.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:56.769 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:43:56.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f30fc078420 msgr2=0x7f30fc07a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f30fc078420 0x7f30fc07a8e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f31140017c0 tx=0x7f3114009040 comp rx=0 tx=0).stop 2024-09-10T20:43:56.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 msgr2=0x7f311806cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311806cae0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3108000c00 tx=0x7f3108003910 comp rx=0 tx=0).stop 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 shutdown_connections 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f30fc078420 0x7f30fc07a8e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f311815ec20 0x7f311806d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311815bc40 0x7f311806cae0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 --2- 172.21.15.28:0/1740960108 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f31180d0c80 0x7f311814c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 >> 172.21.15.28:0/1740960108 conn(0x7f311806dbf0 msgr2=0x7f311806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.763+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 shutdown_connections 2024-09-10T20:43:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.764+0000 7f311fe8c640 1 -- 172.21.15.28:0/1740960108 wait complete. 2024-09-10T20:43:56.779 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-10T20:43:56.313435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278548,"kb_used_data":12368,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754696172,"statfs":{"total":773094113280,"available":772808880128,"internally_reserved":0,"allocated":12664832,"data_stored":6213004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.935137"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408628+0000","last_change":"2024-09-10T20:43:52.408397+0000","last_active":"2024-09-10T20:43:53.408628+0000","last_peered":"2024-09-10T20:43:53.408628+0000","last_clean":"2024-09-10T20:43:53.408628+0000","last_became_active":"2024-09-10T20:43:52.408104+0000","last_became_peered":"2024-09-10T20:43:52.408104+0000","last_unstale":"2024-09-10T20:43:53.408628+0000","last_undegraded":"2024-09-10T20:43:53.408628+0000","last_fullsized":"2024-09-10T20:43:53.408628+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:12.359581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408850+0000","last_change":"2024-09-10T20:43:52.412350+0000","last_active":"2024-09-10T20:43:53.408850+0000","last_peered":"2024-09-10T20:43:53.408850+0000","last_clean":"2024-09-10T20:43:53.408850+0000","last_became_active":"2024-09-10T20:43:52.411891+0000","last_became_peered":"2024-09-10T20:43:52.411891+0000","last_unstale":"2024-09-10T20:43:53.408850+0000","last_undegraded":"2024-09-10T20:43:53.408850+0000","last_fullsized":"2024-09-10T20:43:53.408850+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:45.026682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477745+0000","last_change":"2024-09-10T20:43:52.412391+0000","last_active":"2024-09-10T20:43:53.477745+0000","last_peered":"2024-09-10T20:43:53.477745+0000","last_clean":"2024-09-10T20:43:53.477745+0000","last_became_active":"2024-09-10T20:43:52.412107+0000","last_became_peered":"2024-09-10T20:43:52.412107+0000","last_unstale":"2024-09-10T20:43:53.477745+0000","last_undegraded":"2024-09-10T20:43:53.477745+0000","last_fullsized":"2024-09-10T20:43:53.477745+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:56:48.742953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407565+0000","last_change":"2024-09-10T20:43:52.408407+0000","last_active":"2024-09-10T20:43:53.407565+0000","last_peered":"2024-09-10T20:43:53.407565+0000","last_clean":"2024-09-10T20:43:53.407565+0000","last_became_active":"2024-09-10T20:43:52.408149+0000","last_became_peered":"2024-09-10T20:43:52.408149+0000","last_unstale":"2024-09-10T20:43:53.407565+0000","last_undegraded":"2024-09-10T20:43:53.407565+0000","last_fullsized":"2024-09-10T20:43:53.407565+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:52.035814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406177+0000","last_change":"2024-09-10T20:43:52.408695+0000","last_active":"2024-09-10T20:43:53.406177+0000","last_peered":"2024-09-10T20:43:53.406177+0000","last_clean":"2024-09-10T20:43:53.406177+0000","last_became_active":"2024-09-10T20:43:52.408221+0000","last_became_peered":"2024-09-10T20:43:52.408221+0000","last_unstale":"2024-09-10T20:43:53.406177+0000","last_undegraded":"2024-09-10T20:43:53.406177+0000","last_fullsized":"2024-09-10T20:43:53.406177+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:48:23.047553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412904+0000","last_change":"2024-09-10T20:43:52.410087+0000","last_active":"2024-09-10T20:43:53.412904+0000","last_peered":"2024-09-10T20:43:53.412904+0000","last_clean":"2024-09-10T20:43:53.412904+0000","last_became_active":"2024-09-10T20:43:52.409699+0000","last_became_peered":"2024-09-10T20:43:52.409699+0000","last_unstale":"2024-09-10T20:43:53.412904+0000","last_undegraded":"2024-09-10T20:43:53.412904+0000","last_fullsized":"2024-09-10T20:43:53.412904+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:45.052374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.405818+0000","last_change":"2024-09-10T20:43:52.408416+0000","last_active":"2024-09-10T20:43:53.405818+0000","last_peered":"2024-09-10T20:43:53.405818+0000","last_clean":"2024-09-10T20:43:53.405818+0000","last_became_active":"2024-09-10T20:43:52.408101+0000","last_became_peered":"2024-09-10T20:43:52.408101+0000","last_unstale":"2024-09-10T20:43:53.405818+0000","last_undegraded":"2024-09-10T20:43:53.405818+0000","last_fullsized":"2024-09-10T20:43:53.405818+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:48.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.405720+0000","last_change":"2024-09-10T20:43:52.404371+0000","last_active":"2024-09-10T20:43:53.405720+0000","last_peered":"2024-09-10T20:43:53.405720+0000","last_clean":"2024-09-10T20:43:53.405720+0000","last_became_active":"2024-09-10T20:43:52.404073+0000","last_became_peered":"2024-09-10T20:43:52.404073+0000","last_unstale":"2024-09-10T20:43:53.405720+0000","last_undegraded":"2024-09-10T20:43:53.405720+0000","last_fullsized":"2024-09-10T20:43:53.405720+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:41:51.683106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477876+0000","last_change":"2024-09-10T20:43:52.414392+0000","last_active":"2024-09-10T20:43:53.477876+0000","last_peered":"2024-09-10T20:43:53.477876+0000","last_clean":"2024-09-10T20:43:53.477876+0000","last_became_active":"2024-09-10T20:43:52.414089+0000","last_became_peered":"2024-09-10T20:43:52.414089+0000","last_unstale":"2024-09-10T20:43:53.477876+0000","last_undegraded":"2024-09-10T20:43:53.477876+0000","last_fullsized":"2024-09-10T20:43:53.477876+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:49.567937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408821+0000","last_change":"2024-09-10T20:43:52.408994+0000","last_active":"2024-09-10T20:43:53.408821+0000","last_peered":"2024-09-10T20:43:53.408821+0000","last_clean":"2024-09-10T20:43:53.408821+0000","last_became_active":"2024-09-10T20:43:52.408725+0000","last_became_peered":"2024-09-10T20:43:52.408725+0000","last_unstale":"2024-09-10T20:43:53.408821+0000","last_undegraded":"2024-09-10T20:43:53.408821+0000","last_fullsized":"2024-09-10T20:43:53.408821+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:27:00.590021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407919+0000","last_change":"2024-09-10T20:43:52.410655+0000","last_active":"2024-09-10T20:43:53.407919+0000","last_peered":"2024-09-10T20:43:53.407919+0000","last_clean":"2024-09-10T20:43:53.407919+0000","last_became_active":"2024-09-10T20:43:52.410389+0000","last_became_peered":"2024-09-10T20:43:52.410389+0000","last_unstale":"2024-09-10T20:43:53.407919+0000","last_undegraded":"2024-09-10T20:43:53.407919+0000","last_fullsized":"2024-09-10T20:43:53.407919+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:18:23.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413308+0000","last_change":"2024-09-10T20:43:52.421943+0000","last_active":"2024-09-10T20:43:53.413308+0000","last_peered":"2024-09-10T20:43:53.413308+0000","last_clean":"2024-09-10T20:43:53.413308+0000","last_became_active":"2024-09-10T20:43:52.421632+0000","last_became_peered":"2024-09-10T20:43:52.421632+0000","last_unstale":"2024-09-10T20:43:53.413308+0000","last_undegraded":"2024-09-10T20:43:53.413308+0000","last_fullsized":"2024-09-10T20:43:53.413308+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:52:11.137255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407572+0000","last_change":"2024-09-10T20:43:52.407464+0000","last_active":"2024-09-10T20:43:53.407572+0000","last_peered":"2024-09-10T20:43:53.407572+0000","last_clean":"2024-09-10T20:43:53.407572+0000","last_became_active":"2024-09-10T20:43:52.407124+0000","last_became_peered":"2024-09-10T20:43:52.407124+0000","last_unstale":"2024-09-10T20:43:53.407572+0000","last_undegraded":"2024-09-10T20:43:53.407572+0000","last_fullsized":"2024-09-10T20:43:53.407572+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:54.752584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477763+0000","last_change":"2024-09-10T20:43:52.414427+0000","last_active":"2024-09-10T20:43:53.477763+0000","last_peered":"2024-09-10T20:43:53.477763+0000","last_clean":"2024-09-10T20:43:53.477763+0000","last_became_active":"2024-09-10T20:43:52.414075+0000","last_became_peered":"2024-09-10T20:43:52.414075+0000","last_unstale":"2024-09-10T20:43:53.477763+0000","last_undegraded":"2024-09-10T20:43:53.477763+0000","last_fullsized":"2024-09-10T20:43:53.477763+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:20.178183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477836+0000","last_change":"2024-09-10T20:43:52.421929+0000","last_active":"2024-09-10T20:43:53.477836+0000","last_peered":"2024-09-10T20:43:53.477836+0000","last_clean":"2024-09-10T20:43:53.477836+0000","last_became_active":"2024-09-10T20:43:52.421522+0000","last_became_peered":"2024-09-10T20:43:52.421522+0000","last_unstale":"2024-09-10T20:43:53.477836+0000","last_undegraded":"2024-09-10T20:43:53.477836+0000","last_fullsized":"2024-09-10T20:43:53.477836+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:47.243959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477577+0000","last_change":"2024-09-10T20:43:52.418281+0000","last_active":"2024-09-10T20:43:53.477577+0000","last_peered":"2024-09-10T20:43:53.477577+0000","last_clean":"2024-09-10T20:43:53.477577+0000","last_became_active":"2024-09-10T20:43:52.417719+0000","last_became_peered":"2024-09-10T20:43:52.417719+0000","last_unstale":"2024-09-10T20:43:53.477577+0000","last_undegraded":"2024-09-10T20:43:53.477577+0000","last_fullsized":"2024-09-10T20:43:53.477577+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:30:45.052374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477947+0000","last_change":"2024-09-10T20:43:52.411778+0000","last_active":"2024-09-10T20:43:53.477947+0000","last_peered":"2024-09-10T20:43:53.477947+0000","last_clean":"2024-09-10T20:43:53.477947+0000","last_became_active":"2024-09-10T20:43:52.411484+0000","last_became_peered":"2024-09-10T20:43:52.411484+0000","last_unstale":"2024-09-10T20:43:53.477947+0000","last_undegraded":"2024-09-10T20:43:53.477947+0000","last_fullsized":"2024-09-10T20:43:53.477947+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:51.933448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478707+0000","last_change":"2024-09-10T20:43:52.418892+0000","last_active":"2024-09-10T20:43:53.478707+0000","last_peered":"2024-09-10T20:43:53.478707+0000","last_clean":"2024-09-10T20:43:53.478707+0000","last_became_active":"2024-09-10T20:43:52.418614+0000","last_became_peered":"2024-09-10T20:43:52.418614+0000","last_unstale":"2024-09-10T20:43:53.478707+0000","last_undegraded":"2024-09-10T20:43:53.478707+0000","last_fullsized":"2024-09-10T20:43:53.478707+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:52.284247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408321+0000","last_change":"2024-09-10T20:43:52.411549+0000","last_active":"2024-09-10T20:43:53.408321+0000","last_peered":"2024-09-10T20:43:53.408321+0000","last_clean":"2024-09-10T20:43:53.408321+0000","last_became_active":"2024-09-10T20:43:52.411279+0000","last_became_peered":"2024-09-10T20:43:52.411279+0000","last_unstale":"2024-09-10T20:43:53.408321+0000","last_undegraded":"2024-09-10T20:43:53.408321+0000","last_fullsized":"2024-09-10T20:43:53.408321+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:50.252590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478128+0000","last_change":"2024-09-10T20:43:52.415022+0000","last_active":"2024-09-10T20:43:53.478128+0000","last_peered":"2024-09-10T20:43:53.478128+0000","last_clean":"2024-09-10T20:43:53.478128+0000","last_became_active":"2024-09-10T20:43:52.414637+0000","last_became_peered":"2024-09-10T20:43:52.414637+0000","last_unstale":"2024-09-10T20:43:53.478128+0000","last_undegraded":"2024-09-10T20:43:53.478128+0000","last_fullsized":"2024-09-10T20:43:53.478128+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:46:21.335954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413098+0000","last_change":"2024-09-10T20:43:52.409483+0000","last_active":"2024-09-10T20:43:53.413098+0000","last_peered":"2024-09-10T20:43:53.413098+0000","last_clean":"2024-09-10T20:43:53.413098+0000","last_became_active":"2024-09-10T20:43:52.409135+0000","last_became_peered":"2024-09-10T20:43:52.409135+0000","last_unstale":"2024-09-10T20:43:53.413098+0000","last_undegraded":"2024-09-10T20:43:53.413098+0000","last_fullsized":"2024-09-10T20:43:53.413098+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:36:38.170888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407903+0000","last_change":"2024-09-10T20:43:52.413059+0000","last_active":"2024-09-10T20:43:53.407903+0000","last_peered":"2024-09-10T20:43:53.407903+0000","last_clean":"2024-09-10T20:43:53.407903+0000","last_became_active":"2024-09-10T20:43:52.412407+0000","last_became_peered":"2024-09-10T20:43:52.412407+0000","last_unstale":"2024-09-10T20:43:53.407903+0000","last_undegraded":"2024-09-10T20:43:53.407903+0000","last_fullsized":"2024-09-10T20:43:53.407903+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:30.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477137+0000","last_change":"2024-09-10T20:43:52.407647+0000","last_active":"2024-09-10T20:43:53.477137+0000","last_peered":"2024-09-10T20:43:53.477137+0000","last_clean":"2024-09-10T20:43:53.477137+0000","last_became_active":"2024-09-10T20:43:52.407353+0000","last_became_peered":"2024-09-10T20:43:52.407353+0000","last_unstale":"2024-09-10T20:43:53.477137+0000","last_undegraded":"2024-09-10T20:43:53.477137+0000","last_fullsized":"2024-09-10T20:43:53.477137+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:14:35.107362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477596+0000","last_change":"2024-09-10T20:43:52.417718+0000","last_active":"2024-09-10T20:43:53.477596+0000","last_peered":"2024-09-10T20:43:53.477596+0000","last_clean":"2024-09-10T20:43:53.477596+0000","last_became_active":"2024-09-10T20:43:52.417311+0000","last_became_peered":"2024-09-10T20:43:52.417311+0000","last_unstale":"2024-09-10T20:43:53.477596+0000","last_undegraded":"2024-09-10T20:43:53.477596+0000","last_fullsized":"2024-09-10T20:43:53.477596+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:10.430197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408192+0000","last_change":"2024-09-10T20:43:52.408268+0000","last_active":"2024-09-10T20:43:53.408192+0000","last_peered":"2024-09-10T20:43:53.408192+0000","last_clean":"2024-09-10T20:43:53.408192+0000","last_became_active":"2024-09-10T20:43:52.407995+0000","last_became_peered":"2024-09-10T20:43:52.407995+0000","last_unstale":"2024-09-10T20:43:53.408192+0000","last_undegraded":"2024-09-10T20:43:53.408192+0000","last_fullsized":"2024-09-10T20:43:53.408192+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:31.680937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.409000+0000","last_change":"2024-09-10T20:43:52.413099+0000","last_active":"2024-09-10T20:43:53.409000+0000","last_peered":"2024-09-10T20:43:53.409000+0000","last_clean":"2024-09-10T20:43:53.409000+0000","last_became_active":"2024-09-10T20:43:52.412720+0000","last_became_peered":"2024-09-10T20:43:52.412720+0000","last_unstale":"2024-09-10T20:43:53.409000+0000","last_undegraded":"2024-09-10T20:43:53.409000+0000","last_fullsized":"2024-09-10T20:43:53.409000+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:37.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478266+0000","last_change":"2024-09-10T20:43:52.428537+0000","last_active":"2024-09-10T20:43:53.478266+0000","last_peered":"2024-09-10T20:43:53.478266+0000","last_clean":"2024-09-10T20:43:53.478266+0000","last_became_active":"2024-09-10T20:43:52.428221+0000","last_became_peered":"2024-09-10T20:43:52.428221+0000","last_unstale":"2024-09-10T20:43:53.478266+0000","last_undegraded":"2024-09-10T20:43:53.478266+0000","last_fullsized":"2024-09-10T20:43:53.478266+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:27.850900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477341+0000","last_change":"2024-09-10T20:43:52.407894+0000","last_active":"2024-09-10T20:43:53.477341+0000","last_peered":"2024-09-10T20:43:53.477341+0000","last_clean":"2024-09-10T20:43:53.477341+0000","last_became_active":"2024-09-10T20:43:52.407595+0000","last_became_peered":"2024-09-10T20:43:52.407595+0000","last_unstale":"2024-09-10T20:43:53.477341+0000","last_undegraded":"2024-09-10T20:43:53.477341+0000","last_fullsized":"2024-09-10T20:43:53.477341+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:00.098258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407746+0000","last_change":"2024-09-10T20:43:52.415674+0000","last_active":"2024-09-10T20:43:53.407746+0000","last_peered":"2024-09-10T20:43:53.407746+0000","last_clean":"2024-09-10T20:43:53.407746+0000","last_became_active":"2024-09-10T20:43:52.415340+0000","last_became_peered":"2024-09-10T20:43:52.415340+0000","last_unstale":"2024-09-10T20:43:53.407746+0000","last_undegraded":"2024-09-10T20:43:53.407746+0000","last_fullsized":"2024-09-10T20:43:53.407746+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:00.817420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407960+0000","last_change":"2024-09-10T20:43:52.413462+0000","last_active":"2024-09-10T20:43:53.407960+0000","last_peered":"2024-09-10T20:43:53.407960+0000","last_clean":"2024-09-10T20:43:53.407960+0000","last_became_active":"2024-09-10T20:43:52.412062+0000","last_became_peered":"2024-09-10T20:43:52.412062+0000","last_unstale":"2024-09-10T20:43:53.407960+0000","last_undegraded":"2024-09-10T20:43:53.407960+0000","last_fullsized":"2024-09-10T20:43:53.407960+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:57.069937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477142+0000","last_change":"2024-09-10T20:43:52.412937+0000","last_active":"2024-09-10T20:43:53.477142+0000","last_peered":"2024-09-10T20:43:53.477142+0000","last_clean":"2024-09-10T20:43:53.477142+0000","last_became_active":"2024-09-10T20:43:52.412650+0000","last_became_peered":"2024-09-10T20:43:52.412650+0000","last_unstale":"2024-09-10T20:43:53.477142+0000","last_undegraded":"2024-09-10T20:43:53.477142+0000","last_fullsized":"2024-09-10T20:43:53.477142+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:48:58.854630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477262+0000","last_change":"2024-09-10T20:43:52.415534+0000","last_active":"2024-09-10T20:43:53.477262+0000","last_peered":"2024-09-10T20:43:53.477262+0000","last_clean":"2024-09-10T20:43:53.477262+0000","last_became_active":"2024-09-10T20:43:52.415260+0000","last_became_peered":"2024-09-10T20:43:52.415260+0000","last_unstale":"2024-09-10T20:43:53.477262+0000","last_undegraded":"2024-09-10T20:43:53.477262+0000","last_fullsized":"2024-09-10T20:43:53.477262+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:05.159002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478342+0000","last_change":"2024-09-10T20:43:52.418047+0000","last_active":"2024-09-10T20:43:53.478342+0000","last_peered":"2024-09-10T20:43:53.478342+0000","last_clean":"2024-09-10T20:43:53.478342+0000","last_became_active":"2024-09-10T20:43:52.417729+0000","last_became_peered":"2024-09-10T20:43:52.417729+0000","last_unstale":"2024-09-10T20:43:53.478342+0000","last_undegraded":"2024-09-10T20:43:53.478342+0000","last_fullsized":"2024-09-10T20:43:53.478342+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:41:43.768841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1f","version":"114'4","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406153+0000","last_change":"2024-09-10T20:43:51.401947+0000","last_active":"2024-09-10T20:43:53.406153+0000","last_peered":"2024-09-10T20:43:53.406153+0000","last_clean":"2024-09-10T20:43:53.406153+0000","last_became_active":"2024-09-10T20:43:51.401661+0000","last_became_peered":"2024-09-10T20:43:51.401661+0000","last_unstale":"2024-09-10T20:43:53.406153+0000","last_undegraded":"2024-09-10T20:43:53.406153+0000","last_fullsized":"2024-09-10T20:43:53.406153+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:54.753371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477983+0000","last_change":"2024-09-10T20:43:52.406537+0000","last_active":"2024-09-10T20:43:53.477983+0000","last_peered":"2024-09-10T20:43:53.477983+0000","last_clean":"2024-09-10T20:43:53.477983+0000","last_became_active":"2024-09-10T20:43:52.406184+0000","last_became_peered":"2024-09-10T20:43:52.406184+0000","last_unstale":"2024-09-10T20:43:53.477983+0000","last_undegraded":"2024-09-10T20:43:53.477983+0000","last_fullsized":"2024-09-10T20:43:53.477983+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:56:44.720530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1e","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477677+0000","last_change":"2024-09-10T20:43:51.399291+0000","last_active":"2024-09-10T20:43:53.477677+0000","last_peered":"2024-09-10T20:43:53.477677+0000","last_clean":"2024-09-10T20:43:53.477677+0000","last_became_active":"2024-09-10T20:43:51.398975+0000","last_became_peered":"2024-09-10T20:43:51.398975+0000","last_unstale":"2024-09-10T20:43:53.477677+0000","last_undegraded":"2024-09-10T20:43:53.477677+0000","last_fullsized":"2024-09-10T20:43:53.477677+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:10.016773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477694+0000","last_change":"2024-09-10T20:43:52.416656+0000","last_active":"2024-09-10T20:43:53.477694+0000","last_peered":"2024-09-10T20:43:53.477694+0000","last_clean":"2024-09-10T20:43:53.477694+0000","last_became_active":"2024-09-10T20:43:52.416312+0000","last_became_peered":"2024-09-10T20:43:52.416312+0000","last_unstale":"2024-09-10T20:43:53.477694+0000","last_undegraded":"2024-09-10T20:43:53.477694+0000","last_fullsized":"2024-09-10T20:43:53.477694+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:33.324330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1d","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408059+0000","last_change":"2024-09-10T20:43:51.400653+0000","last_active":"2024-09-10T20:43:53.408059+0000","last_peered":"2024-09-10T20:43:53.408059+0000","last_clean":"2024-09-10T20:43:53.408059+0000","last_became_active":"2024-09-10T20:43:51.400345+0000","last_became_peered":"2024-09-10T20:43:51.400345+0000","last_unstale":"2024-09-10T20:43:53.408059+0000","last_undegraded":"2024-09-10T20:43:53.408059+0000","last_fullsized":"2024-09-10T20:43:53.408059+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:55.549813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413265+0000","last_change":"2024-09-10T20:43:52.413607+0000","last_active":"2024-09-10T20:43:53.413265+0000","last_peered":"2024-09-10T20:43:53.413265+0000","last_clean":"2024-09-10T20:43:53.413265+0000","last_became_active":"2024-09-10T20:43:52.413342+0000","last_became_peered":"2024-09-10T20:43:52.413342+0000","last_unstale":"2024-09-10T20:43:53.413265+0000","last_undegraded":"2024-09-10T20:43:53.413265+0000","last_fullsized":"2024-09-10T20:43:53.413265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:41:43.768841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1c","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408442+0000","last_change":"2024-09-10T20:43:51.400665+0000","last_active":"2024-09-10T20:43:53.408442+0000","last_peered":"2024-09-10T20:43:53.408442+0000","last_clean":"2024-09-10T20:43:53.408442+0000","last_became_active":"2024-09-10T20:43:51.400375+0000","last_became_peered":"2024-09-10T20:43:51.400375+0000","last_unstale":"2024-09-10T20:43:53.408442+0000","last_undegraded":"2024-09-10T20:43:53.408442+0000","last_fullsized":"2024-09-10T20:43:53.408442+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:28.493731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478337+0000","last_change":"2024-09-10T20:43:52.421796+0000","last_active":"2024-09-10T20:43:53.478337+0000","last_peered":"2024-09-10T20:43:53.478337+0000","last_clean":"2024-09-10T20:43:53.478337+0000","last_became_active":"2024-09-10T20:43:52.421502+0000","last_became_peered":"2024-09-10T20:43:52.421502+0000","last_unstale":"2024-09-10T20:43:53.478337+0000","last_undegraded":"2024-09-10T20:43:53.478337+0000","last_fullsized":"2024-09-10T20:43:53.478337+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:45.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1b","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407452+0000","last_change":"2024-09-10T20:43:51.391089+0000","last_active":"2024-09-10T20:43:53.407452+0000","last_peered":"2024-09-10T20:43:53.407452+0000","last_clean":"2024-09-10T20:43:53.407452+0000","last_became_active":"2024-09-10T20:43:51.390763+0000","last_became_peered":"2024-09-10T20:43:51.390763+0000","last_unstale":"2024-09-10T20:43:53.407452+0000","last_undegraded":"2024-09-10T20:43:53.407452+0000","last_fullsized":"2024-09-10T20:43:53.407452+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:08.578252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413156+0000","last_change":"2024-09-10T20:43:52.415078+0000","last_active":"2024-09-10T20:43:53.413156+0000","last_peered":"2024-09-10T20:43:53.413156+0000","last_clean":"2024-09-10T20:43:53.413156+0000","last_became_active":"2024-09-10T20:43:52.414436+0000","last_became_peered":"2024-09-10T20:43:52.414436+0000","last_unstale":"2024-09-10T20:43:53.413156+0000","last_undegraded":"2024-09-10T20:43:53.413156+0000","last_fullsized":"2024-09-10T20:43:53.413156+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:27.850900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478488+0000","last_change":"2024-09-10T20:43:51.400219+0000","last_active":"2024-09-10T20:43:53.478488+0000","last_peered":"2024-09-10T20:43:53.478488+0000","last_clean":"2024-09-10T20:43:53.478488+0000","last_became_active":"2024-09-10T20:43:51.399942+0000","last_became_peered":"2024-09-10T20:43:51.399942+0000","last_unstale":"2024-09-10T20:43:53.478488+0000","last_undegraded":"2024-09-10T20:43:53.478488+0000","last_fullsized":"2024-09-10T20:43:53.478488+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:20.699035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412927+0000","last_change":"2024-09-10T20:43:52.406959+0000","last_active":"2024-09-10T20:43:53.412927+0000","last_peered":"2024-09-10T20:43:53.412927+0000","last_clean":"2024-09-10T20:43:53.412927+0000","last_became_active":"2024-09-10T20:43:52.406668+0000","last_became_peered":"2024-09-10T20:43:52.406668+0000","last_unstale":"2024-09-10T20:43:53.412927+0000","last_undegraded":"2024-09-10T20:43:53.412927+0000","last_fullsized":"2024-09-10T20:43:53.412927+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:20.178183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477188+0000","last_change":"2024-09-10T20:43:51.390469+0000","last_active":"2024-09-10T20:43:53.477188+0000","last_peered":"2024-09-10T20:43:53.477188+0000","last_clean":"2024-09-10T20:43:53.477188+0000","last_became_active":"2024-09-10T20:43:51.390165+0000","last_became_peered":"2024-09-10T20:43:51.390165+0000","last_unstale":"2024-09-10T20:43:53.477188+0000","last_undegraded":"2024-09-10T20:43:53.477188+0000","last_fullsized":"2024-09-10T20:43:53.477188+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:09.640298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407638+0000","last_change":"2024-09-10T20:43:52.403234+0000","last_active":"2024-09-10T20:43:53.407638+0000","last_peered":"2024-09-10T20:43:53.407638+0000","last_clean":"2024-09-10T20:43:53.407638+0000","last_became_active":"2024-09-10T20:43:52.402948+0000","last_became_peered":"2024-09-10T20:43:52.402948+0000","last_unstale":"2024-09-10T20:43:53.407638+0000","last_undegraded":"2024-09-10T20:43:53.407638+0000","last_fullsized":"2024-09-10T20:43:53.407638+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:08:56.610321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407347+0000","last_change":"2024-09-10T20:43:51.397159+0000","last_active":"2024-09-10T20:43:53.407347+0000","last_peered":"2024-09-10T20:43:53.407347+0000","last_clean":"2024-09-10T20:43:53.407347+0000","last_became_active":"2024-09-10T20:43:51.396880+0000","last_became_peered":"2024-09-10T20:43:51.396880+0000","last_unstale":"2024-09-10T20:43:53.407347+0000","last_undegraded":"2024-09-10T20:43:53.407347+0000","last_fullsized":"2024-09-10T20:43:53.407347+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:09:25.981861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.409016+0000","last_change":"2024-09-10T20:43:52.404136+0000","last_active":"2024-09-10T20:43:53.409016+0000","last_peered":"2024-09-10T20:43:53.409016+0000","last_clean":"2024-09-10T20:43:53.409016+0000","last_became_active":"2024-09-10T20:43:52.403458+0000","last_became_peered":"2024-09-10T20:43:52.403458+0000","last_unstale":"2024-09-10T20:43:53.409016+0000","last_undegraded":"2024-09-10T20:43:53.409016+0000","last_fullsized":"2024-09-10T20:43:53.409016+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:43:38.068750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413358+0000","last_change":"2024-09-10T20:43:51.400577+0000","last_active":"2024-09-10T20:43:53.413358+0000","last_peered":"2024-09-10T20:43:53.413358+0000","last_clean":"2024-09-10T20:43:53.413358+0000","last_became_active":"2024-09-10T20:43:51.400279+0000","last_became_peered":"2024-09-10T20:43:51.400279+0000","last_unstale":"2024-09-10T20:43:53.413358+0000","last_undegraded":"2024-09-10T20:43:53.413358+0000","last_fullsized":"2024-09-10T20:43:53.413358+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:09:20.816146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406054+0000","last_change":"2024-09-10T20:43:52.415847+0000","last_active":"2024-09-10T20:43:53.406054+0000","last_peered":"2024-09-10T20:43:53.406054+0000","last_clean":"2024-09-10T20:43:53.406054+0000","last_became_active":"2024-09-10T20:43:52.415554+0000","last_became_peered":"2024-09-10T20:43:52.415554+0000","last_unstale":"2024-09-10T20:43:53.406054+0000","last_undegraded":"2024-09-10T20:43:53.406054+0000","last_fullsized":"2024-09-10T20:43:53.406054+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:25.749401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478556+0000","last_change":"2024-09-10T20:43:51.403485+0000","last_active":"2024-09-10T20:43:53.478556+0000","last_peered":"2024-09-10T20:43:53.478556+0000","last_clean":"2024-09-10T20:43:53.478556+0000","last_became_active":"2024-09-10T20:43:51.403151+0000","last_became_peered":"2024-09-10T20:43:51.403151+0000","last_unstale":"2024-09-10T20:43:53.478556+0000","last_undegraded":"2024-09-10T20:43:53.478556+0000","last_fullsized":"2024-09-10T20:43:53.478556+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:00.680902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413428+0000","last_change":"2024-09-10T20:43:52.424582+0000","last_active":"2024-09-10T20:43:53.413428+0000","last_peered":"2024-09-10T20:43:53.413428+0000","last_clean":"2024-09-10T20:43:53.413428+0000","last_became_active":"2024-09-10T20:43:52.424298+0000","last_became_peered":"2024-09-10T20:43:52.424298+0000","last_unstale":"2024-09-10T20:43:53.413428+0000","last_undegraded":"2024-09-10T20:43:53.413428+0000","last_fullsized":"2024-09-10T20:43:53.413428+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:05:45.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.15","version":"114'3","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406064+0000","last_change":"2024-09-10T20:43:51.400582+0000","last_active":"2024-09-10T20:43:53.406064+0000","last_peered":"2024-09-10T20:43:53.406064+0000","last_clean":"2024-09-10T20:43:53.406064+0000","last_became_active":"2024-09-10T20:43:51.400291+0000","last_became_peered":"2024-09-10T20:43:51.400291+0000","last_unstale":"2024-09-10T20:43:53.406064+0000","last_undegraded":"2024-09-10T20:43:53.406064+0000","last_fullsized":"2024-09-10T20:43:53.406064+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:15.784132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477506+0000","last_change":"2024-09-10T20:43:52.419497+0000","last_active":"2024-09-10T20:43:53.477506+0000","last_peered":"2024-09-10T20:43:53.477506+0000","last_clean":"2024-09-10T20:43:53.477506+0000","last_became_active":"2024-09-10T20:43:52.419221+0000","last_became_peered":"2024-09-10T20:43:52.419221+0000","last_unstale":"2024-09-10T20:43:53.477506+0000","last_undegraded":"2024-09-10T20:43:53.477506+0000","last_fullsized":"2024-09-10T20:43:53.477506+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:37:41.304730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.14","version":"114'6","reported_seq":21,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477288+0000","last_change":"2024-09-10T20:43:51.402444+0000","last_active":"2024-09-10T20:43:53.477288+0000","last_peered":"2024-09-10T20:43:53.477288+0000","last_clean":"2024-09-10T20:43:53.477288+0000","last_became_active":"2024-09-10T20:43:51.402176+0000","last_became_peered":"2024-09-10T20:43:51.402176+0000","last_unstale":"2024-09-10T20:43:53.477288+0000","last_undegraded":"2024-09-10T20:43:53.477288+0000","last_fullsized":"2024-09-10T20:43:53.477288+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:56:36.223340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478398+0000","last_change":"2024-09-10T20:43:52.415634+0000","last_active":"2024-09-10T20:43:53.478398+0000","last_peered":"2024-09-10T20:43:53.478398+0000","last_clean":"2024-09-10T20:43:53.478398+0000","last_became_active":"2024-09-10T20:43:52.414726+0000","last_became_peered":"2024-09-10T20:43:52.414726+0000","last_unstale":"2024-09-10T20:43:53.478398+0000","last_undegraded":"2024-09-10T20:43:53.478398+0000","last_fullsized":"2024-09-10T20:43:53.478398+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:52:11.137255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477845+0000","last_change":"2024-09-10T20:43:51.403836+0000","last_active":"2024-09-10T20:43:53.477845+0000","last_peered":"2024-09-10T20:43:53.477845+0000","last_clean":"2024-09-10T20:43:53.477845+0000","last_became_active":"2024-09-10T20:43:51.403505+0000","last_became_peered":"2024-09-10T20:43:51.403505+0000","last_unstale":"2024-09-10T20:43:53.477845+0000","last_undegraded":"2024-09-10T20:43:53.477845+0000","last_fullsized":"2024-09-10T20:43:53.477845+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:50:23.673158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477587+0000","last_change":"2024-09-10T20:43:52.417801+0000","last_active":"2024-09-10T20:43:53.477587+0000","last_peered":"2024-09-10T20:43:53.477587+0000","last_clean":"2024-09-10T20:43:53.477587+0000","last_became_active":"2024-09-10T20:43:52.417492+0000","last_became_peered":"2024-09-10T20:43:52.417492+0000","last_unstale":"2024-09-10T20:43:53.477587+0000","last_undegraded":"2024-09-10T20:43:53.477587+0000","last_fullsized":"2024-09-10T20:43:53.477587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:49:01.745662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407153+0000","last_change":"2024-09-10T20:43:51.398171+0000","last_active":"2024-09-10T20:43:53.407153+0000","last_peered":"2024-09-10T20:43:53.407153+0000","last_clean":"2024-09-10T20:43:53.407153+0000","last_became_active":"2024-09-10T20:43:51.397803+0000","last_became_peered":"2024-09-10T20:43:51.397803+0000","last_unstale":"2024-09-10T20:43:53.407153+0000","last_undegraded":"2024-09-10T20:43:53.407153+0000","last_fullsized":"2024-09-10T20:43:53.407153+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:21:16.651238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412857+0000","last_change":"2024-09-10T20:43:52.409293+0000","last_active":"2024-09-10T20:43:53.412857+0000","last_peered":"2024-09-10T20:43:53.412857+0000","last_clean":"2024-09-10T20:43:53.412857+0000","last_became_active":"2024-09-10T20:43:52.408940+0000","last_became_peered":"2024-09-10T20:43:52.408940+0000","last_unstale":"2024-09-10T20:43:53.412857+0000","last_undegraded":"2024-09-10T20:43:53.412857+0000","last_fullsized":"2024-09-10T20:43:53.412857+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:39:39.173777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.10","version":"114'8","reported_seq":23,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412788+0000","last_change":"2024-09-10T20:43:51.399871+0000","last_active":"2024-09-10T20:43:53.412788+0000","last_peered":"2024-09-10T20:43:53.412788+0000","last_clean":"2024-09-10T20:43:53.412788+0000","last_became_active":"2024-09-10T20:43:51.399549+0000","last_became_peered":"2024-09-10T20:43:51.399549+0000","last_unstale":"2024-09-10T20:43:53.412788+0000","last_undegraded":"2024-09-10T20:43:53.412788+0000","last_fullsized":"2024-09-10T20:43:53.412788+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:30:20.003721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406006+0000","last_change":"2024-09-10T20:43:52.409212+0000","last_active":"2024-09-10T20:43:53.406006+0000","last_peered":"2024-09-10T20:43:53.406006+0000","last_clean":"2024-09-10T20:43:53.406006+0000","last_became_active":"2024-09-10T20:43:52.408926+0000","last_became_peered":"2024-09-10T20:43:52.408926+0000","last_unstale":"2024-09-10T20:43:53.406006+0000","last_undegraded":"2024-09-10T20:43:53.406006+0000","last_fullsized":"2024-09-10T20:43:53.406006+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:21.545302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.f","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413414+0000","last_change":"2024-09-10T20:43:51.406171+0000","last_active":"2024-09-10T20:43:53.413414+0000","last_peered":"2024-09-10T20:43:53.413414+0000","last_clean":"2024-09-10T20:43:53.413414+0000","last_became_active":"2024-09-10T20:43:51.405833+0000","last_became_peered":"2024-09-10T20:43:51.405833+0000","last_unstale":"2024-09-10T20:43:53.413414+0000","last_undegraded":"2024-09-10T20:43:53.413414+0000","last_fullsized":"2024-09-10T20:43:53.413414+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:44.079778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.406118+0000","last_change":"2024-09-10T20:43:52.402032+0000","last_active":"2024-09-10T20:43:53.406118+0000","last_peered":"2024-09-10T20:43:53.406118+0000","last_clean":"2024-09-10T20:43:53.406118+0000","last_became_active":"2024-09-10T20:43:52.401721+0000","last_became_peered":"2024-09-10T20:43:52.401721+0000","last_unstale":"2024-09-10T20:43:53.406118+0000","last_undegraded":"2024-09-10T20:43:53.406118+0000","last_fullsized":"2024-09-10T20:43:53.406118+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:59.645506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407807+0000","last_change":"2024-09-10T20:43:51.392667+0000","last_active":"2024-09-10T20:43:53.407807+0000","last_peered":"2024-09-10T20:43:53.407807+0000","last_clean":"2024-09-10T20:43:53.407807+0000","last_became_active":"2024-09-10T20:43:51.392369+0000","last_became_peered":"2024-09-10T20:43:51.392369+0000","last_unstale":"2024-09-10T20:43:53.407807+0000","last_undegraded":"2024-09-10T20:43:53.407807+0000","last_fullsized":"2024-09-10T20:43:53.407807+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:52.657407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408339+0000","last_change":"2024-09-10T20:43:52.404514+0000","last_active":"2024-09-10T20:43:53.408339+0000","last_peered":"2024-09-10T20:43:53.408339+0000","last_clean":"2024-09-10T20:43:53.408339+0000","last_became_active":"2024-09-10T20:43:52.404192+0000","last_became_peered":"2024-09-10T20:43:52.404192+0000","last_unstale":"2024-09-10T20:43:53.408339+0000","last_undegraded":"2024-09-10T20:43:53.408339+0000","last_fullsized":"2024-09-10T20:43:53.408339+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:04.186296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.d","version":"114'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477853+0000","last_change":"2024-09-10T20:43:51.393006+0000","last_active":"2024-09-10T20:43:53.477853+0000","last_peered":"2024-09-10T20:43:53.477853+0000","last_clean":"2024-09-10T20:43:53.477853+0000","last_became_active":"2024-09-10T20:43:51.392652+0000","last_became_peered":"2024-09-10T20:43:51.392652+0000","last_unstale":"2024-09-10T20:43:53.477853+0000","last_undegraded":"2024-09-10T20:43:53.477853+0000","last_fullsized":"2024-09-10T20:43:53.477853+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:28.100705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408524+0000","last_change":"2024-09-10T20:43:52.409343+0000","last_active":"2024-09-10T20:43:53.408524+0000","last_peered":"2024-09-10T20:43:53.408524+0000","last_clean":"2024-09-10T20:43:53.408524+0000","last_became_active":"2024-09-10T20:43:52.409059+0000","last_became_peered":"2024-09-10T20:43:52.409059+0000","last_unstale":"2024-09-10T20:43:53.408524+0000","last_undegraded":"2024-09-10T20:43:53.408524+0000","last_fullsized":"2024-09-10T20:43:53.408524+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:28.542728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.405953+0000","last_change":"2024-09-10T20:43:51.401556+0000","last_active":"2024-09-10T20:43:53.405953+0000","last_peered":"2024-09-10T20:43:53.405953+0000","last_clean":"2024-09-10T20:43:53.405953+0000","last_became_active":"2024-09-10T20:43:51.401271+0000","last_became_peered":"2024-09-10T20:43:51.401271+0000","last_unstale":"2024-09-10T20:43:53.405953+0000","last_undegraded":"2024-09-10T20:43:53.405953+0000","last_fullsized":"2024-09-10T20:43:53.405953+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:41:51.957414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408119+0000","last_change":"2024-09-10T20:43:52.413366+0000","last_active":"2024-09-10T20:43:53.408119+0000","last_peered":"2024-09-10T20:43:53.408119+0000","last_clean":"2024-09-10T20:43:53.408119+0000","last_became_active":"2024-09-10T20:43:52.412744+0000","last_became_peered":"2024-09-10T20:43:52.412744+0000","last_unstale":"2024-09-10T20:43:53.408119+0000","last_undegraded":"2024-09-10T20:43:53.408119+0000","last_fullsized":"2024-09-10T20:43:53.408119+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:45.643260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.b","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477493+0000","last_change":"2024-09-10T20:43:51.392907+0000","last_active":"2024-09-10T20:43:53.477493+0000","last_peered":"2024-09-10T20:43:53.477493+0000","last_clean":"2024-09-10T20:43:53.477493+0000","last_became_active":"2024-09-10T20:43:51.392435+0000","last_became_peered":"2024-09-10T20:43:51.392435+0000","last_unstale":"2024-09-10T20:43:53.477493+0000","last_undegraded":"2024-09-10T20:43:53.477493+0000","last_fullsized":"2024-09-10T20:43:53.477493+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:50:51.186640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477170+0000","last_change":"2024-09-10T20:43:52.409667+0000","last_active":"2024-09-10T20:43:53.477170+0000","last_peered":"2024-09-10T20:43:53.477170+0000","last_clean":"2024-09-10T20:43:53.477170+0000","last_became_active":"2024-09-10T20:43:52.407879+0000","last_became_peered":"2024-09-10T20:43:52.407879+0000","last_unstale":"2024-09-10T20:43:53.477170+0000","last_undegraded":"2024-09-10T20:43:53.477170+0000","last_fullsized":"2024-09-10T20:43:53.477170+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:41.570822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478473+0000","last_change":"2024-09-10T20:43:51.399348+0000","last_active":"2024-09-10T20:43:53.478473+0000","last_peered":"2024-09-10T20:43:53.478473+0000","last_clean":"2024-09-10T20:43:53.478473+0000","last_became_active":"2024-09-10T20:43:51.399092+0000","last_became_peered":"2024-09-10T20:43:51.399092+0000","last_unstale":"2024-09-10T20:43:53.478473+0000","last_undegraded":"2024-09-10T20:43:53.478473+0000","last_fullsized":"2024-09-10T20:43:53.478473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:44:32.315572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477847+0000","last_change":"2024-09-10T20:43:52.415252+0000","last_active":"2024-09-10T20:43:53.477847+0000","last_peered":"2024-09-10T20:43:53.477847+0000","last_clean":"2024-09-10T20:43:53.477847+0000","last_became_active":"2024-09-10T20:43:52.414961+0000","last_became_peered":"2024-09-10T20:43:52.414961+0000","last_unstale":"2024-09-10T20:43:53.477847+0000","last_undegraded":"2024-09-10T20:43:53.477847+0000","last_fullsized":"2024-09-10T20:43:53.477847+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:37:43.049678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477914+0000","last_change":"2024-09-10T20:43:51.402453+0000","last_active":"2024-09-10T20:43:53.477914+0000","last_peered":"2024-09-10T20:43:53.477914+0000","last_clean":"2024-09-10T20:43:53.477914+0000","last_became_active":"2024-09-10T20:43:51.402098+0000","last_became_peered":"2024-09-10T20:43:51.402098+0000","last_unstale":"2024-09-10T20:43:53.477914+0000","last_undegraded":"2024-09-10T20:43:53.477914+0000","last_fullsized":"2024-09-10T20:43:53.477914+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:45:56.629766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412917+0000","last_change":"2024-09-10T20:43:52.418107+0000","last_active":"2024-09-10T20:43:53.412917+0000","last_peered":"2024-09-10T20:43:53.412917+0000","last_clean":"2024-09-10T20:43:53.412917+0000","last_became_active":"2024-09-10T20:43:52.417761+0000","last_became_peered":"2024-09-10T20:43:52.417761+0000","last_unstale":"2024-09-10T20:43:53.412917+0000","last_undegraded":"2024-09-10T20:43:53.412917+0000","last_fullsized":"2024-09-10T20:43:53.412917+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:39.479110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407097+0000","last_change":"2024-09-10T20:43:51.386361+0000","last_active":"2024-09-10T20:43:53.407097+0000","last_peered":"2024-09-10T20:43:53.407097+0000","last_clean":"2024-09-10T20:43:53.407097+0000","last_became_active":"2024-09-10T20:43:51.386073+0000","last_became_peered":"2024-09-10T20:43:51.386073+0000","last_unstale":"2024-09-10T20:43:53.407097+0000","last_undegraded":"2024-09-10T20:43:53.407097+0000","last_fullsized":"2024-09-10T20:43:53.407097+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:09:15.809400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477613+0000","last_change":"2024-09-10T20:43:52.417449+0000","last_active":"2024-09-10T20:43:53.477613+0000","last_peered":"2024-09-10T20:43:53.477613+0000","last_clean":"2024-09-10T20:43:53.477613+0000","last_became_active":"2024-09-10T20:43:52.417159+0000","last_became_peered":"2024-09-10T20:43:52.417159+0000","last_unstale":"2024-09-10T20:43:53.477613+0000","last_undegraded":"2024-09-10T20:43:53.477613+0000","last_fullsized":"2024-09-10T20:43:53.477613+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:15.346163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.7","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478684+0000","last_change":"2024-09-10T20:43:51.400298+0000","last_active":"2024-09-10T20:43:53.478684+0000","last_peered":"2024-09-10T20:43:53.478684+0000","last_clean":"2024-09-10T20:43:53.478684+0000","last_became_active":"2024-09-10T20:43:51.400009+0000","last_became_peered":"2024-09-10T20:43:51.400009+0000","last_unstale":"2024-09-10T20:43:53.478684+0000","last_undegraded":"2024-09-10T20:43:53.478684+0000","last_fullsized":"2024-09-10T20:43:53.478684+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:46:29.035453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.408852+0000","last_change":"2024-09-10T20:43:52.411523+0000","last_active":"2024-09-10T20:43:53.408852+0000","last_peered":"2024-09-10T20:43:53.408852+0000","last_clean":"2024-09-10T20:43:53.408852+0000","last_became_active":"2024-09-10T20:43:52.411245+0000","last_became_peered":"2024-09-10T20:43:52.411245+0000","last_unstale":"2024-09-10T20:43:53.408852+0000","last_undegraded":"2024-09-10T20:43:53.408852+0000","last_fullsized":"2024-09-10T20:43:53.408852+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:55.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.6","version":"115'1","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412242+0000","last_change":"2024-09-10T20:43:51.402779+0000","last_active":"2024-09-10T20:43:53.412242+0000","last_peered":"2024-09-10T20:43:53.412242+0000","last_clean":"2024-09-10T20:43:53.412242+0000","last_became_active":"2024-09-10T20:43:51.402448+0000","last_became_peered":"2024-09-10T20:43:51.402448+0000","last_unstale":"2024-09-10T20:43:53.412242+0000","last_undegraded":"2024-09-10T20:43:53.412242+0000","last_fullsized":"2024-09-10T20:43:53.412242+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:14:34.101637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478404+0000","last_change":"2024-09-10T20:43:52.414285+0000","last_active":"2024-09-10T20:43:53.478404+0000","last_peered":"2024-09-10T20:43:53.478404+0000","last_clean":"2024-09-10T20:43:53.478404+0000","last_became_active":"2024-09-10T20:43:52.413729+0000","last_became_peered":"2024-09-10T20:43:52.413729+0000","last_unstale":"2024-09-10T20:43:53.478404+0000","last_undegraded":"2024-09-10T20:43:53.478404+0000","last_fullsized":"2024-09-10T20:43:53.478404+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:19.949840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478335+0000","last_change":"2024-09-10T20:43:51.406447+0000","last_active":"2024-09-10T20:43:53.478335+0000","last_peered":"2024-09-10T20:43:53.478335+0000","last_clean":"2024-09-10T20:43:53.478335+0000","last_became_active":"2024-09-10T20:43:51.406221+0000","last_became_peered":"2024-09-10T20:43:51.406221+0000","last_unstale":"2024-09-10T20:43:53.478335+0000","last_undegraded":"2024-09-10T20:43:53.478335+0000","last_fullsized":"2024-09-10T20:43:53.478335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:30.540249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477789+0000","last_change":"2024-09-10T20:43:52.412318+0000","last_active":"2024-09-10T20:43:53.477789+0000","last_peered":"2024-09-10T20:43:53.477789+0000","last_clean":"2024-09-10T20:43:53.477789+0000","last_became_active":"2024-09-10T20:43:52.412054+0000","last_became_peered":"2024-09-10T20:43:52.412054+0000","last_unstale":"2024-09-10T20:43:53.477789+0000","last_undegraded":"2024-09-10T20:43:53.477789+0000","last_fullsized":"2024-09-10T20:43:53.477789+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:13.044452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.478210+0000","last_change":"2024-09-10T20:43:51.403560+0000","last_active":"2024-09-10T20:43:53.478210+0000","last_peered":"2024-09-10T20:43:53.478210+0000","last_clean":"2024-09-10T20:43:53.478210+0000","last_became_active":"2024-09-10T20:43:51.403283+0000","last_became_peered":"2024-09-10T20:43:51.403283+0000","last_unstale":"2024-09-10T20:43:53.478210+0000","last_undegraded":"2024-09-10T20:43:53.478210+0000","last_fullsized":"2024-09-10T20:43:53.478210+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:13:50.843696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.412933+0000","last_change":"2024-09-10T20:43:52.410397+0000","last_active":"2024-09-10T20:43:53.412933+0000","last_peered":"2024-09-10T20:43:53.412933+0000","last_clean":"2024-09-10T20:43:53.412933+0000","last_became_active":"2024-09-10T20:43:52.410096+0000","last_became_peered":"2024-09-10T20:43:52.410096+0000","last_unstale":"2024-09-10T20:43:53.412933+0000","last_undegraded":"2024-09-10T20:43:53.412933+0000","last_fullsized":"2024-09-10T20:43:53.412933+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:34:31.913191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407538+0000","last_change":"2024-09-10T20:43:51.402457+0000","last_active":"2024-09-10T20:43:53.407538+0000","last_peered":"2024-09-10T20:43:53.407538+0000","last_clean":"2024-09-10T20:43:53.407538+0000","last_became_active":"2024-09-10T20:43:51.402198+0000","last_became_peered":"2024-09-10T20:43:51.402198+0000","last_unstale":"2024-09-10T20:43:53.407538+0000","last_undegraded":"2024-09-10T20:43:53.407538+0000","last_fullsized":"2024-09-10T20:43:53.407538+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:24.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413490+0000","last_change":"2024-09-10T20:43:52.414717+0000","last_active":"2024-09-10T20:43:53.413490+0000","last_peered":"2024-09-10T20:43:53.413490+0000","last_clean":"2024-09-10T20:43:53.413490+0000","last_became_active":"2024-09-10T20:43:52.414105+0000","last_became_peered":"2024-09-10T20:43:52.414105+0000","last_unstale":"2024-09-10T20:43:53.413490+0000","last_undegraded":"2024-09-10T20:43:53.413490+0000","last_fullsized":"2024-09-10T20:43:53.413490+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:02.446253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413027+0000","last_change":"2024-09-10T20:43:51.401690+0000","last_active":"2024-09-10T20:43:53.413027+0000","last_peered":"2024-09-10T20:43:53.413027+0000","last_clean":"2024-09-10T20:43:53.413027+0000","last_became_active":"2024-09-10T20:43:51.401420+0000","last_became_peered":"2024-09-10T20:43:51.401420+0000","last_unstale":"2024-09-10T20:43:53.413027+0000","last_undegraded":"2024-09-10T20:43:53.413027+0000","last_fullsized":"2024-09-10T20:43:53.413027+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:29:04.826111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477701+0000","last_change":"2024-09-10T20:43:52.428756+0000","last_active":"2024-09-10T20:43:53.477701+0000","last_peered":"2024-09-10T20:43:53.477701+0000","last_clean":"2024-09-10T20:43:53.477701+0000","last_became_active":"2024-09-10T20:43:52.428176+0000","last_became_peered":"2024-09-10T20:43:52.428176+0000","last_unstale":"2024-09-10T20:43:53.477701+0000","last_undegraded":"2024-09-10T20:43:53.477701+0000","last_fullsized":"2024-09-10T20:43:53.477701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:59:37.007457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.407263+0000","last_change":"2024-09-10T20:43:51.398079+0000","last_active":"2024-09-10T20:43:53.407263+0000","last_peered":"2024-09-10T20:43:53.407263+0000","last_clean":"2024-09-10T20:43:53.407263+0000","last_became_active":"2024-09-10T20:43:51.397757+0000","last_became_peered":"2024-09-10T20:43:51.397757+0000","last_unstale":"2024-09-10T20:43:53.407263+0000","last_undegraded":"2024-09-10T20:43:53.407263+0000","last_fullsized":"2024-09-10T20:43:53.407263+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:12.990418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477977+0000","last_change":"2024-09-10T20:43:52.415122+0000","last_active":"2024-09-10T20:43:53.477977+0000","last_peered":"2024-09-10T20:43:53.477977+0000","last_clean":"2024-09-10T20:43:53.477977+0000","last_became_active":"2024-09-10T20:43:52.414804+0000","last_became_peered":"2024-09-10T20:43:52.414804+0000","last_unstale":"2024-09-10T20:43:53.477977+0000","last_undegraded":"2024-09-10T20:43:53.477977+0000","last_fullsized":"2024-09-10T20:43:53.477977+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:44:54.549791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477725+0000","last_change":"2024-09-10T20:43:51.403657+0000","last_active":"2024-09-10T20:43:53.477725+0000","last_peered":"2024-09-10T20:43:53.477725+0000","last_clean":"2024-09-10T20:43:53.477725+0000","last_became_active":"2024-09-10T20:43:51.403362+0000","last_became_peered":"2024-09-10T20:43:51.403362+0000","last_unstale":"2024-09-10T20:43:53.477725+0000","last_undegraded":"2024-09-10T20:43:53.477725+0000","last_fullsized":"2024-09-10T20:43:53.477725+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:31.071123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.477582+0000","last_change":"2024-09-10T20:43:52.409599+0000","last_active":"2024-09-10T20:43:53.477582+0000","last_peered":"2024-09-10T20:43:53.477582+0000","last_clean":"2024-09-10T20:43:53.477582+0000","last_became_active":"2024-09-10T20:43:52.409312+0000","last_became_peered":"2024-09-10T20:43:52.409312+0000","last_unstale":"2024-09-10T20:43:53.477582+0000","last_undegraded":"2024-09-10T20:43:53.477582+0000","last_fullsized":"2024-09-10T20:43:53.477582+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:51.377773+0000","last_clean_scrub_stamp":"2024-09-10T20:43:51.377773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:07.867978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.13","version":"114'2","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.413322+0000","last_change":"2024-09-10T20:43:51.404408+0000","last_active":"2024-09-10T20:43:53.413322+0000","last_peered":"2024-09-10T20:43:53.413322+0000","last_clean":"2024-09-10T20:43:53.413322+0000","last_became_active":"2024-09-10T20:43:51.403983+0000","last_became_peered":"2024-09-10T20:43:51.403983+0000","last_unstale":"2024-09-10T20:43:53.413322+0000","last_undegraded":"2024-09-10T20:43:53.413322+0000","last_fullsized":"2024-09-10T20:43:53.413322+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:43:50.372049+0000","last_clean_scrub_stamp":"2024-09-10T20:43:50.372049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:31.071123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":312,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-10T20:43:53.476848+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:43:53.476848+0000","last_peered":"2024-09-10T20:43:53.476848+0000","last_clean":"2024-09-10T20:43:53.476848+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:43:53.476848+0000","last_undegraded":"2024-09-10T20:43:53.476848+0000","last_fullsized":"2024-09-10T20:43:53.476848+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542218,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1421312,"data_stored":623719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542219,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":630013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542217,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33164,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338676,"statfs":{"total":96636764160,"available":96602804224,"internally_reserved":0,"allocated":2023424,"data_stored":1214795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542217,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1429504,"data_stored":630013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509515,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1433600,"data_stored":630083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509514,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1437696,"data_stored":630685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509514,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37348,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334492,"statfs":{"total":96636764160,"available":96598519808,"internally_reserved":0,"allocated":2039808,"data_stored":1222015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509514,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1449984,"data_stored":631681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:43:56.783 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:43:56.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 -- 172.21.15.28:0/3355383347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00d00b0 msgr2=0x7f4cc00d0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/3355383347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00d00b0 0x7f4cc00d0490 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00b2dc0 tx=0x7f4cbc02a560 comp rx=0 tx=0).stop 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 -- 172.21.15.28:0/3355383347 shutdown_connections 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/3355383347 >> v2:172.21.15.159:3300/0 conn(0x7f4cc00ad8c0 0x7f4cc00b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/3355383347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00d00b0 0x7f4cc00d0490 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 -- 172.21.15.28:0/3355383347 >> 172.21.15.28:0/3355383347 conn(0x7f4cc006cf00 msgr2=0x7f4cc006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 -- 172.21.15.28:0/3355383347 shutdown_connections 2024-09-10T20:43:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.928+0000 7f4cc7d91640 1 -- 172.21.15.28:0/3355383347 wait complete. 2024-09-10T20:43:56.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.929+0000 7f4cc7d91640 1 Processor -- start 2024-09-10T20:43:56.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.929+0000 7f4cc7d91640 1 -- start start 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc7d91640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cc00ad460 0x7f4cc00b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc7d91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00ad8c0 0x7f4cc01efa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc7d91640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cc00d00b0 0x7f4cc01f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc7d91640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc00b4c20 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc7d91640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc00b3870 con 0x7f4cc00d00b0 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc7d91640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc00b4da0 con 0x7f4cc00ad460 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00ad8c0 0x7f4cc01efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00ad8c0 0x7f4cc01efa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53218/0 (socket says 172.21.15.28:53218) 2024-09-10T20:43:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 -- 172.21.15.28:0/4097611019 learned_addr learned my addr 172.21.15.28:0/4097611019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc5b2d640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cc00ad460 0x7f4cc00b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc632e640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cc00d00b0 0x7f4cc01f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 -- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cc00ad460 msgr2=0x7f4cc00b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cc00ad460 0x7f4cc00b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 -- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cc00d00b0 msgr2=0x7f4cc01f5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cc00d00b0 0x7f4cc01f5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc532c640 1 -- 172.21.15.28:0/4097611019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc00b2740 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.930+0000 7f4cc5b2d640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cc00ad460 0x7f4cc00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.931+0000 7f4cc532c640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00ad8c0 0x7f4cc01efa70 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000c410 tx=0x7f4cb000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.931+0000 7f4cb67fc640 1 -- 172.21.15.28:0/4097611019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0017070 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.931+0000 7f4cb67fc640 1 -- 172.21.15.28:0/4097611019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cb00035f0 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.931+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc00b0470 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.931+0000 7f4cb67fc640 1 -- 172.21.15.28:0/4097611019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0003a80 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.931+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc00b0a30 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.932+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc00b46e0 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.932+0000 7f4cb67fc640 1 -- 172.21.15.28:0/4097611019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f4cb0003c20 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.933+0000 7f4cb67fc640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ca40783f0 0x7f4ca407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:56.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.933+0000 7f4cb67fc640 1 -- 172.21.15.28:0/4097611019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f4cb0099d20 con 0x7f4cc00ad8c0 2024-09-10T20:43:56.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.933+0000 7f4cc5b2d640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ca40783f0 0x7f4ca407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:56.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.933+0000 7f4cc5b2d640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ca40783f0 0x7f4ca407a8b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc033000 tx=0x7f4cbc004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:56.937+0000 7f4cb67fc640 1 -- 172.21.15.28:0/4097611019 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4cb009f020 con 0x7f4cc00ad8c0 2024-09-10T20:43:57.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.192+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f4cc00aea30 con 0x7f4cc00ad8c0 2024-09-10T20:43:57.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.194+0000 7f4cb67fc640 1 -- 172.21.15.28:0/4097611019 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v33) ==== 190+0+147 (secure 0 0 0) 0x7f4cb00620c0 con 0x7f4cc00ad8c0 2024-09-10T20:43:57.199 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:43:57.200 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:43:57.200 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:43:57.200 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:43:57.200 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:43:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ca40783f0 msgr2=0x7f4ca407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ca40783f0 0x7f4ca407a8b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc033000 tx=0x7f4cbc004730 comp rx=0 tx=0).stop 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00ad8c0 msgr2=0x7f4cc01efa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00ad8c0 0x7f4cc01efa70 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000c410 tx=0x7f4cb000c8d0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 shutdown_connections 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ca40783f0 0x7f4ca407a8b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cc00d00b0 0x7f4cc01f5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc00ad8c0 0x7f4cc01efa70 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 --2- 172.21.15.28:0/4097611019 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4cc00ad460 0x7f4cc00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 >> 172.21.15.28:0/4097611019 conn(0x7f4cc006cf00 msgr2=0x7f4cc01542d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 shutdown_connections 2024-09-10T20:43:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.197+0000 7f4cc7d91640 1 -- 172.21.15.28:0/4097611019 wait complete. 2024-09-10T20:43:57.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:43:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.313+0000 7fef671e2640 1 -- 172.21.15.28:0/1476608488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef4c004000 msgr2=0x7fef4c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.313+0000 7fef671e2640 1 --2- 172.21.15.28:0/1476608488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef4c004000 0x7fef4c004800 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fef50001dc0 tx=0x7fef4c005830 comp rx=0 tx=0).stop 2024-09-10T20:43:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.313+0000 7fef6577f640 1 -- 172.21.15.28:0/1476608488 reap_dead start 2024-09-10T20:43:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.313+0000 7fef671e2640 1 -- 172.21.15.28:0/1476608488 shutdown_connections 2024-09-10T20:43:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.313+0000 7fef671e2640 1 -- 172.21.15.28:0/1476608488 >> 172.21.15.28:0/1476608488 conn(0x7fef600d1e40 msgr2=0x7fef600d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.313+0000 7fef671e2640 1 -- 172.21.15.28:0/1476608488 shutdown_connections 2024-09-10T20:43:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.313+0000 7fef671e2640 1 -- 172.21.15.28:0/1476608488 wait complete. 2024-09-10T20:43:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.314+0000 7fef671e2640 1 Processor -- start 2024-09-10T20:43:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.314+0000 7fef671e2640 1 -- start start 2024-09-10T20:43:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef671e2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef4c004000 0x7fef601f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef671e2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef600cb7b0 0x7fef600cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef671e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef60157c40 0x7fef601580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef671e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef600cd5c0 con 0x7fef60157c40 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef671e2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fef600d0d50 con 0x7fef600cb7b0 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef671e2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fef600b5d20 con 0x7fef4c004000 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef6577f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef60157c40 0x7fef601580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef6577f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef60157c40 0x7fef601580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53234/0 (socket says 172.21.15.28:53234) 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef6577f640 1 -- 172.21.15.28:0/3357478704 learned_addr learned my addr 172.21.15.28:0/3357478704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef64f7e640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef4c004000 0x7fef601f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.315+0000 7fef5ffff640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef600cb7b0 0x7fef600cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef6577f640 1 -- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef4c004000 msgr2=0x7fef601f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef6577f640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef4c004000 0x7fef601f1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef6577f640 1 -- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef600cb7b0 msgr2=0x7fef600cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef6577f640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef600cb7b0 0x7fef600cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef6577f640 1 -- 172.21.15.28:0/3357478704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef600cc170 con 0x7fef60157c40 2024-09-10T20:43:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef6577f640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef60157c40 0x7fef601580f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fef600d1a20 tx=0x7fef48011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef5d7fa640 1 -- 172.21.15.28:0/3357478704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4800f920 con 0x7fef60157c40 2024-09-10T20:43:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef5d7fa640 1 -- 172.21.15.28:0/3357478704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef4800fae0 con 0x7fef60157c40 2024-09-10T20:43:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef60158630 con 0x7fef60157c40 2024-09-10T20:43:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef5d7fa640 1 -- 172.21.15.28:0/3357478704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48005020 con 0x7fef60157c40 2024-09-10T20:43:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.316+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef60158bc0 con 0x7fef60157c40 2024-09-10T20:43:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.317+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef600b5610 con 0x7fef60157c40 2024-09-10T20:43:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.318+0000 7fef5d7fa640 1 -- 172.21.15.28:0/3357478704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fef48021030 con 0x7fef60157c40 2024-09-10T20:43:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.318+0000 7fef5d7fa640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef40078420 0x7fef4007a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.318+0000 7fef5d7fa640 1 -- 172.21.15.28:0/3357478704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fef4809ab20 con 0x7fef60157c40 2024-09-10T20:43:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.318+0000 7fef64f7e640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef40078420 0x7fef4007a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.319+0000 7fef64f7e640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef40078420 0x7fef4007a8e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fef600c8590 tx=0x7fef580067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.322+0000 7fef5d7fa640 1 -- 172.21.15.28:0/3357478704 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fef48011e20 con 0x7fef60157c40 2024-09-10T20:43:57.377 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-10T20:43:57.373+0000 7fd010142640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:43:57.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.545+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef6015b5a0 con 0x7fef60157c40 2024-09-10T20:43:57.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.546+0000 7fef5d7fa640 1 -- 172.21.15.28:0/3357478704 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5231 (secure 0 0 0) 0x7fef48062870 con 0x7fef60157c40 2024-09-10T20:43:57.551 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:43:57.551 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":76,"btime":"2024-09-10T20:43:55:414872+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3157854991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3157854991},{"type":"v1","addr":"172.21.15.159:6835","nonce":3157854991}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/3291396887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3291396887},{"type":"v1","addr":"172.21.15.159:6801","nonce":3291396887}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7461,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3500302836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3500302836},{"type":"v1","addr":"172.21.15.28:6833","nonce":3500302836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:43:51.763823+0000","modified":"2024-09-10T20:43:54.783424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7482":{"gid":7482,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1765205934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1765205934},{"type":"v1","addr":"172.21.15.28:6835","nonce":1765205934}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7482,"qdb_cluster":[7482]},"id":9}]} 2024-09-10T20:43:57.551 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 76 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef40078420 msgr2=0x7fef4007a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef40078420 0x7fef4007a8e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fef600c8590 tx=0x7fef580067a0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef60157c40 msgr2=0x7fef601580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef60157c40 0x7fef601580f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fef600d1a20 tx=0x7fef48011bd0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 shutdown_connections 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef40078420 0x7fef4007a8e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef60157c40 0x7fef601580f0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef600cb7b0 0x7fef600cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 --2- 172.21.15.28:0/3357478704 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef4c004000 0x7fef601f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 >> 172.21.15.28:0/3357478704 conn(0x7fef600d1e40 msgr2=0x7fef600ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:57.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 shutdown_connections 2024-09-10T20:43:57.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:57.548+0000 7fef671e2640 1 -- 172.21.15.28:0/3357478704 wait complete. 2024-09-10T20:43:57.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:43:51.763823+0000', 'modified': '2024-09-10T20:43:54.783424+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7482}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7482': {'gid': 7482, 'name': 'a', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1765205934', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 1765205934}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1765205934}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7482, 'qdb_cluster': [7482]} 2024-09-10T20:43:57.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:43:57.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-10T20:43:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.079+0000 7fc311bf8640 1 -- 172.21.15.28:0/1940039628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c15de70 msgr2=0x7fc30c0d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.079+0000 7fc311bf8640 1 --2- 172.21.15.28:0/1940039628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c15de70 0x7fc30c0d2b90 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc005fd0 tx=0x7fc2fc02aa90 comp rx=0 tx=0).stop 2024-09-10T20:43:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.080+0000 7fc311bf8640 1 -- 172.21.15.28:0/1940039628 shutdown_connections 2024-09-10T20:43:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.080+0000 7fc311bf8640 1 --2- 172.21.15.28:0/1940039628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c0cf780 0x7fc30c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.080+0000 7fc311bf8640 1 --2- 172.21.15.28:0/1940039628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c15de70 0x7fc30c0d2b90 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.080+0000 7fc311bf8640 1 -- 172.21.15.28:0/1940039628 >> 172.21.15.28:0/1940039628 conn(0x7fc30c0d3fb0 msgr2=0x7fc30c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.080+0000 7fc311bf8640 1 -- 172.21.15.28:0/1940039628 shutdown_connections 2024-09-10T20:43:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.080+0000 7fc311bf8640 1 -- 172.21.15.28:0/1940039628 wait complete. 2024-09-10T20:43:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.081+0000 7fc311bf8640 1 Processor -- start 2024-09-10T20:43:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.081+0000 7fc311bf8640 1 -- start start 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc311bf8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c0cf780 0x7fc30c15a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc311bf8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15de70 0x7fc30c14d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc311bf8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c161b50 0x7fc30c15bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc311bf8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c162350 con 0x7fc30c0cf780 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc311bf8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c1621d0 con 0x7fc30c15de70 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc311bf8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c1624d0 con 0x7fc30c161b50 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c0cf780 0x7fc30c15a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c0cf780 0x7fc30c15a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53258/0 (socket says 172.21.15.28:53258) 2024-09-10T20:43:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 -- 172.21.15.28:0/2899793989 learned_addr learned my addr 172.21.15.28:0/2899793989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30bfff640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c161b50 0x7fc30c15bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30affd640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15de70 0x7fc30c14d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 -- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c161b50 msgr2=0x7fc30c15bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c161b50 0x7fc30c15bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 -- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15de70 msgr2=0x7fc30c14d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15de70 0x7fc30c14d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 -- 172.21.15.28:0/2899793989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c14d920 con 0x7fc30c0cf780 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.082+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c0cf780 0x7fc30c15a600 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc000f80 tx=0x7fc2fc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.083+0000 7fc303fff640 1 -- 172.21.15.28:0/2899793989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc046070 con 0x7fc30c0cf780 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.083+0000 7fc303fff640 1 -- 172.21.15.28:0/2899793989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2fc004250 con 0x7fc30c0cf780 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.083+0000 7fc303fff640 1 -- 172.21.15.28:0/2899793989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc004680 con 0x7fc30c0cf780 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.083+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c159560 con 0x7fc30c0cf780 2024-09-10T20:43:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.083+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c159b20 con 0x7fc30c0cf780 2024-09-10T20:43:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.084+0000 7fc303fff640 1 -- 172.21.15.28:0/2899793989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fc2fc04b440 con 0x7fc30c0cf780 2024-09-10T20:43:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.085+0000 7fc303fff640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078420 0x7fc2ec07a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.085+0000 7fc303fff640 1 -- 172.21.15.28:0/2899793989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fc2fc0bb830 con 0x7fc30c0cf780 2024-09-10T20:43:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.085+0000 7fc30affd640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078420 0x7fc2ec07a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.086+0000 7fc30affd640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078420 0x7fc2ec07a8e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0ca660 tx=0x7fc2f400b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.086+0000 7fc3149cd740 1 Processor -- start 2024-09-10T20:43:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.086+0000 7fc3149cd740 1 -- start start 2024-09-10T20:43:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b725b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b76e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b7235f0 0x56467b723a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc3149cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56467b69a820 con 0x56467b76ddf0 2024-09-10T20:43:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56467b689c50 con 0x56467b7235f0 2024-09-10T20:43:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56467b68da60 con 0x56467b74e1b0 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b76e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b76e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53268/0 (socket says 172.21.15.28:53268) 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 -- 172.21.15.28:0/976708137 learned_addr learned my addr 172.21.15.28:0/976708137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b725b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30bfff640 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b7235f0 0x56467b723a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 -- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 msgr2=0x56467b725b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b725b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 -- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b7235f0 msgr2=0x56467b723a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b7235f0 0x56467b723a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 -- 172.21.15.28:0/976708137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56467b3eb850 con 0x56467b76ddf0 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30bfff640 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b7235f0 0x56467b723a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc30affd640 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b76e1f0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0d0100 tx=0x7fc2f4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9270cbf80cfd3f71 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.087+0000 7fc3017fa640 1 -- 172.21.15.28:0/976708137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f401e620 con 0x56467b76ddf0 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3017fa640 1 -- 172.21.15.28:0/976708137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f4011e50 con 0x56467b76ddf0 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3017fa640 1 -- 172.21.15.28:0/976708137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f401eab0 con 0x56467b76ddf0 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 -- 172.21.15.28:0/976708137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 msgr2=0x56467b76e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b76e1f0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0d0100 tx=0x7fc2f4013000 comp rx=0 tx=0).stop 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 -- 172.21.15.28:0/976708137 shutdown_connections 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b7235f0 0x56467b723a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b76e1f0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 --2- 172.21.15.28:0/976708137 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b725b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 -- 172.21.15.28:0/976708137 >> 172.21.15.28:0/976708137 conn(0x56467b7246a0 msgr2=0x56467b7280c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 -- 172.21.15.28:0/976708137 shutdown_connections 2024-09-10T20:43:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.088+0000 7fc3149cd740 1 -- 172.21.15.28:0/976708137 wait complete. 2024-09-10T20:43:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.089+0000 7fc3149cd740 1 -- start start 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.089+0000 7fc3149cd740 1 Processor -- start 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b77cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b737390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc3149cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56467b3fa620 con 0x56467b76ddf0 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56467b62f5f0 con 0x56467b74e1b0 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56467b722b30 con 0x56467b74d360 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc3149cd740 10 client.0 _ll_register_callbacks cb 0x56467b6330a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53276/0 (socket says 172.21.15.28:53276) 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 -- 172.21.15.28:0/2607723370 learned_addr learned my addr 172.21.15.28:0/2607723370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b77cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30bfff640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b737390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 -- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 msgr2=0x56467b737390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b737390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 -- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 msgr2=0x56467b77cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b77cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30affd640 1 -- 172.21.15.28:0/2607723370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56467b4626b0 con 0x56467b76ddf0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.090+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b77cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc30affd640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ce20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4005940 tx=0x7fc2f401e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b798d512473ebe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc2ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/2607723370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4034060 con 0x56467b76ddf0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/2607723370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fc2f401eac0 con 0x56467b76ddf0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f4011e50 con 0x56467b76ddf0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/2607723370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fc2f403a050 con 0x56467b76ddf0 2024-09-10T20:43:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 10 client.7590 fetch_fsmap learned FSMap version 76 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x56467b737da0 con 0x56467b76ddf0 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/2607723370 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f4011e50 con 0x56467b76ddf0 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/2607723370 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 76) ==== 3502+0+0 (secure 0 0 0) 0x7fc2f403d020 con 0x56467b76ddf0 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 10 client.7590 fetch_fsmap finished waiting for FSMap version 76 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 10 client.7590 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 20 client.7590 populate_metadata read hostname 'smithi028' 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b748080 0x56467b7803c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 4 client.7590 mds_command: new command op to 7482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 --> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56467b69c770 con 0x56467b748080 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.091+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b748080 0x56467b7803c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.092+0000 7fc2ea7fc640 10 client.7590 ms_handle_connect on v2:172.21.15.28:6834/1765205934 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.092+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b748080 0x56467b7803c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fc64067c2448a5e server_cookie=8984dc3a3d1baf5e in_seq=0 out_seq=0 2024-09-10T20:43:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.092+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/2607723370 <== mds.0 v2:172.21.15.28:6834/1765205934 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56467b69c770 con 0x56467b748080 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.092+0000 7fc2ea7fc640 10 client.7590 handle_command_reply: tid=0 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.092+0000 7fc3149cd740 1 client.7590 shutdown 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 msgr2=0x56467b74ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ce20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4005940 tx=0x7fc2f401e650 comp rx=0 tx=0).stop 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 shutdown_connections 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b748080 0x56467b7803c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b737390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ce20 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc2ea7fc640 0 client.7590 ms_handle_reset on v2:172.21.15.28:6834/1765205934 2024-09-10T20:43:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 --2- 172.21.15.28:0/2607723370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b77cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 >> 172.21.15.28:0/2607723370 conn(0x56467b7235f0 msgr2=0x56467b727ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 shutdown_connections 2024-09-10T20:43:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 1 -- 172.21.15.28:0/2607723370 wait complete. 2024-09-10T20:43:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.093+0000 7fc3149cd740 20 client.7590 trim_cache size 0 max 16384 2024-09-10T20:43:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.111+0000 7fc3149cd740 1 Processor -- start 2024-09-10T20:43:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.111+0000 7fc3149cd740 1 -- start start 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b74e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b74ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b7863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc3149cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0cbd20 con 0x56467b76ddf0 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0cb080 con 0x56467b74e1b0 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c0cb6d0 con 0x56467b74d360 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b7863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b7863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53282/0 (socket says 172.21.15.28:53282) 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 -- 172.21.15.28:0/3593077741 learned_addr learned my addr 172.21.15.28:0/3593077741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b74e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30affd640 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b74ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 -- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 msgr2=0x56467b74e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b74e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 -- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 msgr2=0x56467b74ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b74ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.112+0000 7fc30bfff640 1 -- 172.21.15.28:0/3593077741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56467b746f60 con 0x56467b76ddf0 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc30bfff640 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b7863e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc3040055d0 tx=0x7fc304007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50f646596d6b2ef server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc300ff9640 1 -- 172.21.15.28:0/3593077741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304014850 con 0x56467b76ddf0 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc300ff9640 1 -- 172.21.15.28:0/3593077741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc304014ae0 con 0x56467b76ddf0 2024-09-10T20:43:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc300ff9640 1 -- 172.21.15.28:0/3593077741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304016420 con 0x56467b76ddf0 2024-09-10T20:43:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc3149cd740 1 -- 172.21.15.28:0/3593077741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 msgr2=0x56467b7863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc3149cd740 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b7863e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc3040055d0 tx=0x7fc304007db0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc3149cd740 1 -- 172.21.15.28:0/3593077741 shutdown_connections 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc3149cd740 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b7863e0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc3149cd740 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74e1b0 0x56467b74ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc3149cd740 1 --2- 172.21.15.28:0/3593077741 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74d360 0x56467b74e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.113+0000 7fc3149cd740 1 -- 172.21.15.28:0/3593077741 >> 172.21.15.28:0/3593077741 conn(0x56467b7235f0 msgr2=0x56467b76efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.114+0000 7fc3149cd740 1 -- 172.21.15.28:0/3593077741 shutdown_connections 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.114+0000 7fc3149cd740 1 -- 172.21.15.28:0/3593077741 wait complete. 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.114+0000 7fc3149cd740 1 -- start start 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.114+0000 7fc3149cd740 1 Processor -- start 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74d360 0x56467b7e6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b74e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc3149cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc3149cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56467b747f00 con 0x56467b76ddf0 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56467b738030 con 0x56467b74d360 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc3149cd740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x56467b748550 con 0x56467b74e1b0 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc3149cd740 10 client.0 _ll_register_callbacks cb 0x7fc304007c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53286/0 (socket says 172.21.15.28:53286) 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 -- 172.21.15.28:0/4054919659 learned_addr learned my addr 172.21.15.28:0/4054919659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30affd640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b74e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74d360 0x56467b7e6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 -- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 msgr2=0x56467b74e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b74e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 -- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74d360 msgr2=0x56467b7e6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74d360 0x56467b7e6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30bfff640 1 -- 172.21.15.28:0/4054919659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56467b735c20 con 0x56467b76ddf0 2024-09-10T20:43:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.115+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74d360 0x56467b7e6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc30bfff640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ecc0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x56467b7484f0 tx=0x7fc304014f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03333c5e24eaf49 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc2ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/4054919659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304016b80 con 0x56467b76ddf0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/4054919659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fc304016d20 con 0x56467b76ddf0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc304014ae0 con 0x56467b76ddf0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/4054919659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fc304030050 con 0x56467b76ddf0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 10 client.7596 fetch_fsmap learned FSMap version 76 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x56467b77c1d0 con 0x56467b76ddf0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/4054919659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc304014ae0 con 0x56467b76ddf0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/4054919659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 76) ==== 3502+0+0 (secure 0 0 0) 0x7fc304033020 con 0x56467b76ddf0 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 10 client.7596 fetch_fsmap finished waiting for FSMap version 76 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 10 client.7596 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 20 client.7596 populate_metadata read hostname 'smithi028' 2024-09-10T20:43:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b7edbe0 0x56467b725b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 4 client.7596 mds_command: new command op to 7482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 --> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] -- command(tid 0: {"prefix": "status"}) -- 0x56467b698220 con 0x56467b7edbe0 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.116+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b7edbe0 0x56467b725b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc2ea7fc640 10 client.7596 ms_handle_connect on v2:172.21.15.28:6834/1765205934 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc30b7fe640 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b7edbe0 0x56467b725b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=417b634111763d8e server_cookie=7d777cf3697c81d4 in_seq=0 out_seq=0 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc2ea7fc640 1 -- 172.21.15.28:0/4054919659 <== mds.0 v2:172.21.15.28:6834/1765205934 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56467b698220 con 0x56467b7edbe0 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc2ea7fc640 10 client.7596 handle_command_reply: tid=0 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 client.7596 shutdown 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 msgr2=0x56467b74ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ecc0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x56467b7484f0 tx=0x7fc304014f90 comp rx=0 tx=0).stop 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 shutdown_connections 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:6834/1765205934,v1:172.21.15.28:6835/1765205934] conn(0x56467b7edbe0 0x56467b725b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56467b76ddf0 0x56467b74ecc0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56467b74e1b0 0x56467b74e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc2ea7fc640 0 client.7596 ms_handle_reset on v2:172.21.15.28:6834/1765205934 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 --2- 172.21.15.28:0/4054919659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56467b74d360 0x56467b7e6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 >> 172.21.15.28:0/4054919659 conn(0x56467b7235f0 msgr2=0x56467b76efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 shutdown_connections 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 1 -- 172.21.15.28:0/4054919659 wait complete. 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.117+0000 7fc3149cd740 20 client.7596 trim_cache size 0 max 16384 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:43:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7482, 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7144217470000003, 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 76, 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 115, 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 115, 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.991561011 2024-09-10T20:43:58.129 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:43:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078420 msgr2=0x7fc2ec07a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078420 0x7fc2ec07a8e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0ca660 tx=0x7fc2f400b390 comp rx=0 tx=0).stop 2024-09-10T20:43:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c0cf780 msgr2=0x7fc30c15a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c0cf780 0x7fc30c15a600 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc000f80 tx=0x7fc2fc003f10 comp rx=0 tx=0).stop 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 shutdown_connections 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2ec078420 0x7fc2ec07a8e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c161b50 0x7fc30c15bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15de70 0x7fc30c14d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 --2- 172.21.15.28:0/2899793989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc30c0cf780 0x7fc30c15a600 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 >> 172.21.15.28:0/2899793989 conn(0x7fc30c0d3fb0 msgr2=0x7fc30c0b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 shutdown_connections 2024-09-10T20:43:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.118+0000 7fc311bf8640 1 -- 172.21.15.28:0/2899793989 wait complete. 2024-09-10T20:43:58.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:43:58.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 -- 172.21.15.28:0/1548439033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 msgr2=0x7fd33815bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 --2- 172.21.15.28:0/1548439033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 0x7fd33815bc40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd334009890 tx=0x7fd33402e940 comp rx=0 tx=0).stop 2024-09-10T20:43:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 -- 172.21.15.28:0/1548439033 shutdown_connections 2024-09-10T20:43:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 --2- 172.21.15.28:0/1548439033 >> v2:172.21.15.159:3300/0 conn(0x7fd3380d1060 0x7fd33815e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 --2- 172.21.15.28:0/1548439033 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 0x7fd33815bc40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 -- 172.21.15.28:0/1548439033 >> 172.21.15.28:0/1548439033 conn(0x7fd33806dbf0 msgr2=0x7fd33806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 -- 172.21.15.28:0/1548439033 shutdown_connections 2024-09-10T20:43:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.234+0000 7fd33fc47640 1 -- 172.21.15.28:0/1548439033 wait complete. 2024-09-10T20:43:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 Processor -- start 2024-09-10T20:43:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 -- start start 2024-09-10T20:43:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 0x7fd3381468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3380d1060 0x7fd3380b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd33815ec20 0x7fd33815c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd338070990 con 0x7fd33815ec20 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd3380b54a0 con 0x7fd3380d1060 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33fc47640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd338070b10 con 0x7fd3380d0c80 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33e1e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd33815ec20 0x7fd33815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33e1e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd33815ec20 0x7fd33815c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53290/0 (socket says 172.21.15.28:53290) 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.236+0000 7fd33e1e4640 1 -- 172.21.15.28:0/3546148508 learned_addr learned my addr 172.21.15.28:0/3546148508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33d9e3640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 0x7fd3381468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33d1e2640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3380d1060 0x7fd3380b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33e1e4640 1 -- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 msgr2=0x7fd3381468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33e1e4640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 0x7fd3381468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33e1e4640 1 -- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3380d1060 msgr2=0x7fd3380b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33e1e4640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3380d1060 0x7fd3380b2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33e1e4640 1 -- 172.21.15.28:0/3546148508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3380cfb10 con 0x7fd33815ec20 2024-09-10T20:43:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33e1e4640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd33815ec20 0x7fd33815c220 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fd33000cb40 tx=0x7fd33000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd32e7fc640 1 -- 172.21.15.28:0/3546148508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd330016020 con 0x7fd33815ec20 2024-09-10T20:43:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3380cfdf0 con 0x7fd33815ec20 2024-09-10T20:43:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd32e7fc640 1 -- 172.21.15.28:0/3546148508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd33000adb0 con 0x7fd33815ec20 2024-09-10T20:43:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd32e7fc640 1 -- 172.21.15.28:0/3546148508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd330012680 con 0x7fd33815ec20 2024-09-10T20:43:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.237+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3380cdc90 con 0x7fd33815ec20 2024-09-10T20:43:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.238+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3380d0030 con 0x7fd33815ec20 2024-09-10T20:43:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.239+0000 7fd32e7fc640 1 -- 172.21.15.28:0/3546148508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fd330012820 con 0x7fd33815ec20 2024-09-10T20:43:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.240+0000 7fd32e7fc640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd31c078420 0x7fd31c07a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.240+0000 7fd32e7fc640 1 -- 172.21.15.28:0/3546148508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fd33009aeb0 con 0x7fd33815ec20 2024-09-10T20:43:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.240+0000 7fd33d9e3640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd31c078420 0x7fd31c07a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.240+0000 7fd33d9e3640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd31c078420 0x7fd31c07a8e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd334039000 tx=0x7fd334004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:58.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.244+0000 7fd32e7fc640 1 -- 172.21.15.28:0/3546148508 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd3380d0030 con 0x7fd33815ec20 2024-09-10T20:43:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.470+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd33815bc40 con 0x7fd33815ec20 2024-09-10T20:43:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.470+0000 7fd32e7fc640 1 -- 172.21.15.28:0/3546148508 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) ==== 53+0+73 (secure 0 0 0) 0x7fd330020020 con 0x7fd33815ec20 2024-09-10T20:43:58.475 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:43:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd31c078420 msgr2=0x7fd31c07a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd31c078420 0x7fd31c07a8e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd334039000 tx=0x7fd334004730 comp rx=0 tx=0).stop 2024-09-10T20:43:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd33815ec20 msgr2=0x7fd33815c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd33815ec20 0x7fd33815c220 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fd33000cb40 tx=0x7fd33000a060 comp rx=0 tx=0).stop 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 shutdown_connections 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd31c078420 0x7fd31c07a8e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd33815ec20 0x7fd33815c220 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3380d1060 0x7fd3380b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 --2- 172.21.15.28:0/3546148508 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3380d0c80 0x7fd3381468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 >> 172.21.15.28:0/3546148508 conn(0x7fd33806dbf0 msgr2=0x7fd33806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 shutdown_connections 2024-09-10T20:43:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58.472+0000 7fd33fc47640 1 -- 172.21.15.28:0/3546148508 wait complete. 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:43:58.488 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:43:58.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:58.489 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:43:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:43:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:43:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:58.506 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42392sec preferred_lft 42392sec 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:58.507 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:58.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:58.508 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:43:58.569 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:43:58.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:58.571 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:43:58.628 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:43:58.629 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:43:58.629 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:43:58.629 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:43:58.629 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:43:58.629 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:43:58.629 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:43:58.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:43:58.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:43:58.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:43:58.629 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:43:58.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:43:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:58.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:58.757 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:43:58.773 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:43:58.773 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:43:58.774 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:43:58.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:58.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:43:58.839 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:43:58.840 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:58.895 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:43:58.896 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:43:58.896 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:58.951 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:43:58.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:58.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:59.035 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:59.035 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:43:59.035 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:43:59.044 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:59.044 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:43:59.044 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:43:59.044 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:43:59.044 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:43:59.050 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:43:59.051 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:43:59.051 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:43:59.051 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:43:59.051 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:43:59.051 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:43:59.053 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:43:59.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:59.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:43:59.078 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:43:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:client6344 2024-09-10T20:43:59.182 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:43:59.182 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:43:59.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:43:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 -- 172.21.15.28:0/2708439216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18501554e0 msgr2=0x7f185015e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 --2- 172.21.15.28:0/2708439216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18501554e0 0x7f185015e930 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f18500ced80 tx=0x7f184402a560 comp rx=0 tx=0).stop 2024-09-10T20:43:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 -- 172.21.15.28:0/2708439216 shutdown_connections 2024-09-10T20:43:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 --2- 172.21.15.28:0/2708439216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18501554e0 0x7f185015e930 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 --2- 172.21.15.28:0/2708439216 >> v2:172.21.15.159:3300/0 conn(0x7f1850155100 0x7f185015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 -- 172.21.15.28:0/2708439216 >> 172.21.15.28:0/2708439216 conn(0x7f185006cf00 msgr2=0x7f185006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 -- 172.21.15.28:0/2708439216 shutdown_connections 2024-09-10T20:43:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.339+0000 7f1854cc6640 1 -- 172.21.15.28:0/2708439216 wait complete. 2024-09-10T20:43:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.340+0000 7f1854cc6640 1 Processor -- start 2024-09-10T20:43:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f1854cc6640 1 -- start start 2024-09-10T20:43:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f1854cc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1850155100 0x7f1850147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f1854cc6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18501554e0 0x7f1850159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f1854cc6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f185015ee70 0x7f185015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f1854cc6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1850072d50 con 0x7f1850155100 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f1854cc6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1850072bd0 con 0x7f18501554e0 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f1854cc6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1850072ed0 con 0x7f185015ee70 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f184e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1850155100 0x7f1850147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f184e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1850155100 0x7f1850147060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53314/0 (socket says 172.21.15.28:53314) 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.341+0000 7f184e59c640 1 -- 172.21.15.28:0/921102782 learned_addr learned my addr 172.21.15.28:0/921102782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184dd9b640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18501554e0 0x7f1850159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184ed9d640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f185015ee70 0x7f185015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184e59c640 1 -- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f185015ee70 msgr2=0x7f185015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184e59c640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f185015ee70 0x7f185015a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184e59c640 1 -- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18501554e0 msgr2=0x7f1850159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184e59c640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18501554e0 0x7f1850159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184e59c640 1 -- 172.21.15.28:0/921102782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18501f55e0 con 0x7f1850155100 2024-09-10T20:43:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184ed9d640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f185015ee70 0x7f185015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184dd9b640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18501554e0 0x7f1850159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:43:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f184e59c640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1850155100 0x7f1850147060 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f184000d740 tx=0x7f184000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f183effd640 1 -- 172.21.15.28:0/921102782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840017070 con 0x7f1850155100 2024-09-10T20:43:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f183effd640 1 -- 172.21.15.28:0/921102782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1840009d20 con 0x7f1850155100 2024-09-10T20:43:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f183effd640 1 -- 172.21.15.28:0/921102782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18400106c0 con 0x7f1850155100 2024-09-10T20:43:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18501f5300 con 0x7f1850155100 2024-09-10T20:43:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.342+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1850147b40 con 0x7f1850155100 2024-09-10T20:43:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.343+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185015bd50 con 0x7f1850155100 2024-09-10T20:43:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.344+0000 7f183effd640 1 -- 172.21.15.28:0/921102782 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f1840010860 con 0x7f1850155100 2024-09-10T20:43:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.344+0000 7f183effd640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18300783f0 0x7f183007a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:43:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.345+0000 7f184dd9b640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18300783f0 0x7f183007a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:43:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.345+0000 7f183effd640 1 -- 172.21.15.28:0/921102782 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f184009aab0 con 0x7f1850155100 2024-09-10T20:43:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.345+0000 7f184dd9b640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18300783f0 0x7f183007a8b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1844033000 tx=0x7f1844004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:43:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.349+0000 7f183effd640 1 -- 172.21.15.28:0/921102782 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1840062e50 con 0x7f1850155100 2024-09-10T20:43:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.572+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f185009d750 con 0x7f1850155100 2024-09-10T20:43:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.573+0000 7f183effd640 1 -- 172.21.15.28:0/921102782 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) ==== 53+0+73 (secure 0 0 0) 0x7f1840062750 con 0x7f1850155100 2024-09-10T20:43:59.578 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18300783f0 msgr2=0x7f183007a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18300783f0 0x7f183007a8b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1844033000 tx=0x7f1844004730 comp rx=0 tx=0).stop 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1850155100 msgr2=0x7f1850147060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1850155100 0x7f1850147060 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f184000d740 tx=0x7f184000dc00 comp rx=0 tx=0).stop 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 shutdown_connections 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18300783f0 0x7f183007a8b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f185015ee70 0x7f185015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18501554e0 0x7f1850159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 --2- 172.21.15.28:0/921102782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1850155100 0x7f1850147060 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:43:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 >> 172.21.15.28:0/921102782 conn(0x7f185006cf00 msgr2=0x7f18500ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:43:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 shutdown_connections 2024-09-10T20:43:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:43:59.575+0000 7f1854cc6640 1 -- 172.21.15.28:0/921102782 wait complete. 2024-09-10T20:43:59.590 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:43:59.591 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:43:59.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:59.591 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:43:59.611 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:43:59.611 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:43:59.611 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:43:59.611 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:43:59.612 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42409sec preferred_lft 42409sec 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:59.613 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:43:59.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:59.615 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:43:59.680 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:43:59.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:59.682 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:43:59.741 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:43:59.741 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:43:59.741 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:43:59.741 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:43:59.741 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:43:59.742 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:43:59.742 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:43:59.742 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:43:59.742 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:43:59.742 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:43:59.742 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:43:59.835 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:43:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:43:59.869 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:43:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:43:59.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:59.876 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:43:59.891 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:43:59.891 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:43:59.891 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:43:59.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:43:59.892 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:43:59.957 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:43:59.957 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:00.013 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:44:00.013 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:44:00.013 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:00.068 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:44:00.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:00.068 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:00.151 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:00.151 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:44:00.151 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:44:00.160 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:00.160 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:44:00.160 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:44:00.160 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:44:00.160 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:44:00.165 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:44:00.166 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:44:00.166 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:44:00.166 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:44:00.166 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:44:00.166 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:44:00.167 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:44:00.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:00.168 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:00.233 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:44:00.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:44:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:44:00.323 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:44:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:44:00.329 INFO:teuthology.orchestra.run.smithi159.stdout:client6350 2024-09-10T20:44:00.330 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:00.330 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:00.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:44:00.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2104751261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680cce90 msgr2=0x7fbb680cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2104751261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680cce90 0x7fbb680cd2d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009860 tx=0x7fbb5c02a540 comp rx=0 tx=0).stop 2024-09-10T20:44:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2104751261 shutdown_connections 2024-09-10T20:44:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2104751261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680cce90 0x7fbb680cd2d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2104751261 >> v2:172.21.15.159:3300/0 conn(0x7fbb6815f540 0x7fbb680cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2104751261 >> 172.21.15.28:0/2104751261 conn(0x7fbb680d1e40 msgr2=0x7fbb680d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2104751261 shutdown_connections 2024-09-10T20:44:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.469+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2104751261 wait complete. 2024-09-10T20:44:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 Processor -- start 2024-09-10T20:44:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 -- start start 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb680cce90 0x7fbb681e7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680ce580 0x7fbb680c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb6815f540 0x7fbb680c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb680b6750 con 0x7fbb680cce90 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbb680b65d0 con 0x7fbb6815f540 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb6d52b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbb680b68d0 con 0x7fbb680ce580 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb680cce90 0x7fbb681e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb680cce90 0x7fbb681e7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52742/0 (socket says 172.21.15.28:52742) 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.471+0000 7fbb66ffd640 1 -- 172.21.15.28:0/2426297821 learned_addr learned my addr 172.21.15.28:0/2426297821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb667fc640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680ce580 0x7fbb680c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb677fe640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb6815f540 0x7fbb680c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb66ffd640 1 -- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680ce580 msgr2=0x7fbb680c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb66ffd640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680ce580 0x7fbb680c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb66ffd640 1 -- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb6815f540 msgr2=0x7fbb680c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb66ffd640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb6815f540 0x7fbb680c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb66ffd640 1 -- 172.21.15.28:0/2426297821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb681f6d80 con 0x7fbb680cce90 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb66ffd640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb680cce90 0x7fbb681e7170 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800ea20 tx=0x7fbb5800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/2426297821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800ce40 con 0x7fbb680cce90 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/2426297821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb58004020 con 0x7fbb680cce90 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb681f5fc0 con 0x7fbb680cce90 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/2426297821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58004470 con 0x7fbb680cce90 2024-09-10T20:44:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.472+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb681f6580 con 0x7fbb680cce90 2024-09-10T20:44:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.473+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb680d1010 con 0x7fbb680cce90 2024-09-10T20:44:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.474+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/2426297821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fbb580035f0 con 0x7fbb680cce90 2024-09-10T20:44:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.474+0000 7fbb4f7fe640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbb540783f0 0x7fbb5407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.475+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/2426297821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fbb5809aff0 con 0x7fbb680cce90 2024-09-10T20:44:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.475+0000 7fbb667fc640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbb540783f0 0x7fbb5407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.475+0000 7fbb667fc640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbb540783f0 0x7fbb5407a8b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c034000 tx=0x7fbb5c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.479+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/2426297821 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbb680d1010 con 0x7fbb680cce90 2024-09-10T20:44:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:00.703+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fbb681e8400 con 0x7fbb680cce90 2024-09-10T20:44:01.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.436+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/2426297821 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v77) ==== 136+0+0 (secure 0 0 0) 0x7fbb58062c80 con 0x7fbb680cce90 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbb540783f0 msgr2=0x7fbb5407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbb540783f0 0x7fbb5407a8b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c034000 tx=0x7fbb5c004730 comp rx=0 tx=0).stop 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb680cce90 msgr2=0x7fbb681e7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb680cce90 0x7fbb681e7170 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800ea20 tx=0x7fbb5800eee0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 shutdown_connections 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbb540783f0 0x7fbb5407a8b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbb6815f540 0x7fbb680c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbb680ce580 0x7fbb680c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 --2- 172.21.15.28:0/2426297821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb680cce90 0x7fbb681e7170 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 >> 172.21.15.28:0/2426297821 conn(0x7fbb680d1e40 msgr2=0x7fbb6815d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 shutdown_connections 2024-09-10T20:44:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.441+0000 7fbb6d52b640 1 -- 172.21.15.28:0/2426297821 wait complete. 2024-09-10T20:44:01.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:44:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.567+0000 7f52db448640 1 -- 172.21.15.28:0/1975065925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 msgr2=0x7f52c0004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.567+0000 7f52db448640 1 --2- 172.21.15.28:0/1975065925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52c0004f80 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009b50 tx=0x7f52c0006940 comp rx=0 tx=0).stop 2024-09-10T20:44:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.567+0000 7f52db448640 1 -- 172.21.15.28:0/1975065925 shutdown_connections 2024-09-10T20:44:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.567+0000 7f52db448640 1 --2- 172.21.15.28:0/1975065925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52c0004f80 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.567+0000 7f52db448640 1 -- 172.21.15.28:0/1975065925 >> 172.21.15.28:0/1975065925 conn(0x7f52d406cf00 msgr2=0x7f52d406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.568+0000 7f52db448640 1 -- 172.21.15.28:0/1975065925 shutdown_connections 2024-09-10T20:44:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.568+0000 7f52db448640 1 -- 172.21.15.28:0/1975065925 wait complete. 2024-09-10T20:44:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.569+0000 7f52db448640 1 Processor -- start 2024-09-10T20:44:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.569+0000 7f52db448640 1 -- start start 2024-09-10T20:44:01.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52db448640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52d4148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52db448640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52d40c4b10 0x7f52d40c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52db448640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52d414b370 0x7f52d414b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52db448640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52d40b4780 con 0x7f52c0004b60 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52db448640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f52d40b4600 con 0x7f52d414b370 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52db448640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f52d40b4900 con 0x7f52d40c4b10 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52d4148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52d4148b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52768/0 (socket says 172.21.15.28:52768) 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 -- 172.21.15.28:0/2372638060 learned_addr learned my addr 172.21.15.28:0/2372638060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d99e5640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52d414b370 0x7f52d414b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d89e3640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52d40c4b10 0x7f52d40c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 -- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52d40c4b10 msgr2=0x7f52d40c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52d40c4b10 0x7f52d40c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 -- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52d414b370 msgr2=0x7f52d414b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52d414b370 0x7f52d414b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 -- 172.21.15.28:0/2372638060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d414bf80 con 0x7f52c0004b60 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d99e5640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52d414b370 0x7f52d414b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d89e3640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52d40c4b10 0x7f52d40c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52d91e4640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52d4148b10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f52d0036040 tx=0x7f52d00336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52c9ffb640 1 -- 172.21.15.28:0/2372638060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0038040 con 0x7f52c0004b60 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52c9ffb640 1 -- 172.21.15.28:0/2372638060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52d0004230 con 0x7f52c0004b60 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52c9ffb640 1 -- 172.21.15.28:0/2372638060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d00338e0 con 0x7f52c0004b60 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.570+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d414c260 con 0x7f52c0004b60 2024-09-10T20:44:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.571+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d409be00 con 0x7f52c0004b60 2024-09-10T20:44:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.572+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d40d0260 con 0x7f52c0004b60 2024-09-10T20:44:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.572+0000 7f52c9ffb640 1 -- 172.21.15.28:0/2372638060 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f52d003b070 con 0x7f52c0004b60 2024-09-10T20:44:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.573+0000 7f52c9ffb640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f52b8078420 0x7f52b807a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.573+0000 7f52c9ffb640 1 -- 172.21.15.28:0/2372638060 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f52d00ba650 con 0x7f52c0004b60 2024-09-10T20:44:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.573+0000 7f52d89e3640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f52b8078420 0x7f52b807a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.573+0000 7f52d89e3640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f52b8078420 0x7f52b807a8e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f52c4006850 tx=0x7f52c40067e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.577+0000 7f52c9ffb640 1 -- 172.21.15.28:0/2372638060 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f52d0082c20 con 0x7f52c0004b60 2024-09-10T20:44:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.784+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f52d41576e0 con 0x7f52b8078420 2024-09-10T20:44:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.785+0000 7f52c9ffb640 1 -- 172.21.15.28:0/2372638060 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f52c4006dc0 con 0x7f52b8078420 2024-09-10T20:44:01.790 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:44:01.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:01.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:44:01.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:01.791 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:44:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f52b8078420 msgr2=0x7f52b807a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f52b8078420 0x7f52b807a8e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f52c4006850 tx=0x7f52c40067e0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 msgr2=0x7f52d4148b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52d4148b10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f52d0036040 tx=0x7f52d00336f0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 shutdown_connections 2024-09-10T20:44:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f52b8078420 0x7f52b807a8e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f52d414b370 0x7f52d414b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f52d40c4b10 0x7f52d40c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 --2- 172.21.15.28:0/2372638060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52c0004b60 0x7f52d4148b10 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 >> 172.21.15.28:0/2372638060 conn(0x7f52d406cf00 msgr2=0x7f52d4159080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 shutdown_connections 2024-09-10T20:44:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.788+0000 7f52db448640 1 -- 172.21.15.28:0/2372638060 wait complete. 2024-09-10T20:44:01.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:44:01.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.908+0000 7faa55b56640 1 -- 172.21.15.28:0/1743775501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 msgr2=0x7faa38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.908+0000 7faa55b56640 1 --2- 172.21.15.28:0/1743775501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 0x7faa38004800 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7faa44009b50 tx=0x7faa38005830 comp rx=0 tx=0).stop 2024-09-10T20:44:01.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.908+0000 7faa4ffff640 1 -- 172.21.15.28:0/1743775501 reap_dead start 2024-09-10T20:44:01.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.908+0000 7faa55b56640 1 -- 172.21.15.28:0/1743775501 shutdown_connections 2024-09-10T20:44:01.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.908+0000 7faa55b56640 1 -- 172.21.15.28:0/1743775501 >> 172.21.15.28:0/1743775501 conn(0x7faa500d0d50 msgr2=0x7faa500d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:01.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.908+0000 7faa55b56640 1 -- 172.21.15.28:0/1743775501 shutdown_connections 2024-09-10T20:44:01.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.908+0000 7faa55b56640 1 -- 172.21.15.28:0/1743775501 wait complete. 2024-09-10T20:44:01.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.909+0000 7faa55b56640 1 Processor -- start 2024-09-10T20:44:01.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.909+0000 7faa55b56640 1 -- start start 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa55b56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 0x7faa500c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa55b56640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa500affb0 0x7faa50150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa55b56640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa500ca500 0x7faa500ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa55b56640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa50070860 con 0x7faa38003f70 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa55b56640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faa500706e0 con 0x7faa500affb0 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa55b56640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faa500709e0 con 0x7faa500ca500 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 0x7faa500c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 0x7faa500c7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52788/0 (socket says 172.21.15.28:52788) 2024-09-10T20:44:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 -- 172.21.15.28:0/2162771491 learned_addr learned my addr 172.21.15.28:0/2162771491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4effd640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa500affb0 0x7faa50150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4ffff640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa500ca500 0x7faa500ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 -- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa500ca500 msgr2=0x7faa500ca980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa500ca500 0x7faa500ca980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 -- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa500affb0 msgr2=0x7faa50150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa500affb0 0x7faa50150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4f7fe640 1 -- 172.21.15.28:0/2162771491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa500caec0 con 0x7faa38003f70 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.910+0000 7faa4ffff640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa500ca500 0x7faa500ca980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.911+0000 7faa4f7fe640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 0x7faa500c7460 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7faa40006660 tx=0x7faa400039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.911+0000 7faa37fff640 1 -- 172.21.15.28:0/2162771491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40015070 con 0x7faa38003f70 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.911+0000 7faa37fff640 1 -- 172.21.15.28:0/2162771491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa40011900 con 0x7faa38003f70 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.911+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa501f5c70 con 0x7faa38003f70 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.911+0000 7faa37fff640 1 -- 172.21.15.28:0/2162771491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40011d90 con 0x7faa38003f70 2024-09-10T20:44:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.911+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa501f6230 con 0x7faa38003f70 2024-09-10T20:44:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.912+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa50073790 con 0x7faa38003f70 2024-09-10T20:44:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.913+0000 7faa37fff640 1 -- 172.21.15.28:0/2162771491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7faa40010420 con 0x7faa38003f70 2024-09-10T20:44:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.913+0000 7faa37fff640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa3c078420 0x7faa3c07a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.913+0000 7faa37fff640 1 -- 172.21.15.28:0/2162771491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7faa4009b370 con 0x7faa38003f70 2024-09-10T20:44:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.913+0000 7faa4effd640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa3c078420 0x7faa3c07a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:01.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.914+0000 7faa4effd640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa3c078420 0x7faa3c07a8e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa44035000 tx=0x7faa440046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:01.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:01.918+0000 7faa37fff640 1 -- 172.21.15.28:0/2162771491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faa40063040 con 0x7faa38003f70 2024-09-10T20:44:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.104+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7faa500749a0 con 0x7faa38003f70 2024-09-10T20:44:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.104+0000 7faa37fff640 1 -- 172.21.15.28:0/2162771491 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7faa40067a70 con 0x7faa38003f70 2024-09-10T20:44:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa3c078420 msgr2=0x7faa3c07a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa3c078420 0x7faa3c07a8e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa44035000 tx=0x7faa440046c0 comp rx=0 tx=0).stop 2024-09-10T20:44:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 msgr2=0x7faa500c7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 0x7faa500c7460 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7faa40006660 tx=0x7faa400039c0 comp rx=0 tx=0).stop 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 shutdown_connections 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa3c078420 0x7faa3c07a8e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa500ca500 0x7faa500ca980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa500affb0 0x7faa50150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 --2- 172.21.15.28:0/2162771491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa38003f70 0x7faa500c7460 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 >> 172.21.15.28:0/2162771491 conn(0x7faa500d0d50 msgr2=0x7faa500cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 shutdown_connections 2024-09-10T20:44:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:02.107+0000 7faa55b56640 1 -- 172.21.15.28:0/2162771491 wait complete. 2024-09-10T20:44:02.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.123 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol31370/9e18844c-eabb-456e-9bab-26162b769ef0) 2024-09-10T20:44:02.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:44:02.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.154 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol31370/9e18844c-eabb-456e-9bab-26162b769ef0/.snap/fake) 2024-09-10T20:44:02.197 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:44:02.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.228 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:44:02.312 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:44:02.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.313 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol31370/9e18844c-eabb-456e-9bab-26162b769ef0) 2024-09-10T20:44:02.382 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:02.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol31370/.meta 2024-09-10T20:44:02.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.452 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp74240/subvol20671/19fa2599-6812-4f48-a1e8-4a95426e497d) 2024-09-10T20:44:02.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.522 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp74240/subvol20671/19fa2599-6812-4f48-a1e8-4a95426e497d/.snap/fake) 2024-09-10T20:44:02.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.594 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:44:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:44:02.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.652 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/subvol_grp74240/subvol20671/19fa2599-6812-4f48-a1e8-4a95426e497d) 2024-09-10T20:44:02.721 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:02.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/subvol_grp74240/subvol20671/.meta 2024-09-10T20:44:02.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.788 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol62402/cd1f6c8b-861e-459b-85d8-0eb7828ec75e) 2024-09-10T20:44:02.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.857 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:44:02.913 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:44:02.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:02.913 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol62402/cd1f6c8b-861e-459b-85d8-0eb7828ec75e) 2024-09-10T20:44:02.982 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:02.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol62402/.meta 2024-09-10T20:44:03.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol31370 2024-09-10T20:44:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.193+0000 7f76d420f640 1 -- 172.21.15.28:0/4100665073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 msgr2=0x7f76b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.193+0000 7f76d420f640 1 --2- 172.21.15.28:0/4100665073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 0x7f76b80051c0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c610 tx=0x7f76b8005830 comp rx=0 tx=0).stop 2024-09-10T20:44:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.193+0000 7f76d27ac640 1 -- 172.21.15.28:0/4100665073 reap_dead start 2024-09-10T20:44:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.193+0000 7f76d420f640 1 -- 172.21.15.28:0/4100665073 shutdown_connections 2024-09-10T20:44:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.193+0000 7f76d420f640 1 -- 172.21.15.28:0/4100665073 >> 172.21.15.28:0/4100665073 conn(0x7f76cc06cf00 msgr2=0x7f76cc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.194+0000 7f76d420f640 1 -- 172.21.15.28:0/4100665073 shutdown_connections 2024-09-10T20:44:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.194+0000 7f76d420f640 1 -- 172.21.15.28:0/4100665073 wait complete. 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.195+0000 7f76d420f640 1 Processor -- start 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.195+0000 7f76d420f640 1 -- start start 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 0x7f76cc15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f57b0 0x7f76cc1f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0b4fe0 0x7f76cc0b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc15cd80 con 0x7f76b8004d40 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc15cc00 con 0x7f76cc0b4fe0 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f76cc15cf00 con 0x7f76cc1f57b0 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 0x7f76cc15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 0x7f76cc15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52806/0 (socket says 172.21.15.28:52806) 2024-09-10T20:44:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 -- 172.21.15.28:0/1925175110 learned_addr learned my addr 172.21.15.28:0/1925175110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d27ac640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0b4fe0 0x7f76cc0b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d17aa640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f57b0 0x7f76cc1f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 -- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f57b0 msgr2=0x7f76cc1f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f57b0 0x7f76cc1f5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 -- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0b4fe0 msgr2=0x7f76cc0b5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0b4fe0 0x7f76cc0b5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 -- 172.21.15.28:0/1925175110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc0b5d60 con 0x7f76b8004d40 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d1fab640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 0x7f76cc15b290 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f76c4004650 tx=0x7f76c400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76c27fc640 1 -- 172.21.15.28:0/1925175110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4010b10 con 0x7f76b8004d40 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76c27fc640 1 -- 172.21.15.28:0/1925175110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76c4015070 con 0x7f76b8004d40 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76c27fc640 1 -- 172.21.15.28:0/1925175110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c401a470 con 0x7f76b8004d40 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76cc1f3390 con 0x7f76b8004d40 2024-09-10T20:44:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.196+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76cc1f3950 con 0x7f76b8004d40 2024-09-10T20:44:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.197+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76cc14f4e0 con 0x7f76b8004d40 2024-09-10T20:44:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.198+0000 7f76c27fc640 1 -- 172.21.15.28:0/1925175110 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f76c401a610 con 0x7f76b8004d40 2024-09-10T20:44:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.198+0000 7f76c27fc640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76b00783f0 0x7f76b007a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.199+0000 7f76c27fc640 1 -- 172.21.15.28:0/1925175110 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f76c409b3c0 con 0x7f76b8004d40 2024-09-10T20:44:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.199+0000 7f76d17aa640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76b00783f0 0x7f76b007a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.199+0000 7f76d17aa640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76b00783f0 0x7f76b007a8b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76bc005940 tx=0x7f76bc005890 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:03.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.203+0000 7f76c27fc640 1 -- 172.21.15.28:0/1925175110 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f76c4063690 con 0x7f76b8004d40 2024-09-10T20:44:03.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.415+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol31370", "target": ["mon-mgr", ""]}) -- 0x7f76cc073bc0 con 0x7f76b00783f0 2024-09-10T20:44:03.426 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:03.421+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:03.426 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:03.421+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:03.426 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:03.421+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:03.426 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:03.421+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:03.426 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:03.421+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.455+0000 7f76c27fc640 1 -- 172.21.15.28:0/1925175110 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f76cc073bc0 con 0x7f76b00783f0 2024-09-10T20:44:03.460 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol31370/9e18844c-eabb-456e-9bab-26162b769ef0 2024-09-10T20:44:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76b00783f0 msgr2=0x7f76b007a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76b00783f0 0x7f76b007a8b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76bc005940 tx=0x7f76bc005890 comp rx=0 tx=0).stop 2024-09-10T20:44:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 msgr2=0x7f76cc15b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 0x7f76cc15b290 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f76c4004650 tx=0x7f76c400bac0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 shutdown_connections 2024-09-10T20:44:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f76b00783f0 0x7f76b007a8b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76cc0b4fe0 0x7f76cc0b5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76cc1f57b0 0x7f76cc1f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 --2- 172.21.15.28:0/1925175110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b8004d40 0x7f76cc15b290 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 >> 172.21.15.28:0/1925175110 conn(0x7f76cc06cf00 msgr2=0x7f76cc15d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 shutdown_connections 2024-09-10T20:44:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.457+0000 7f76d420f640 1 -- 172.21.15.28:0/1925175110 wait complete. 2024-09-10T20:44:03.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol20671 subvol_grp74240 2024-09-10T20:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.578+0000 7fa1399a0640 1 -- 172.21.15.28:0/773148029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa120004800 msgr2=0x7fa120004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.578+0000 7fa1399a0640 1 --2- 172.21.15.28:0/773148029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa120004800 0x7fa120004c80 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009b50 tx=0x7fa120005830 comp rx=0 tx=0).stop 2024-09-10T20:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.578+0000 7fa1337fe640 1 -- 172.21.15.28:0/773148029 reap_dead start 2024-09-10T20:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.578+0000 7fa1399a0640 1 -- 172.21.15.28:0/773148029 shutdown_connections 2024-09-10T20:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.578+0000 7fa1399a0640 1 -- 172.21.15.28:0/773148029 >> 172.21.15.28:0/773148029 conn(0x7fa1340ccb60 msgr2=0x7fa1340ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.578+0000 7fa1399a0640 1 -- 172.21.15.28:0/773148029 shutdown_connections 2024-09-10T20:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.579+0000 7fa1399a0640 1 -- 172.21.15.28:0/773148029 wait complete. 2024-09-10T20:44:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 Processor -- start 2024-09-10T20:44:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 -- start start 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa120004800 0x7fa1341f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1340c6240 0x7fa1340b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1340b0d50 0x7fa13415d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa13415f3b0 con 0x7fa1340c6240 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa13415f230 con 0x7fa120004800 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.580+0000 7fa1399a0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa13415f530 con 0x7fa1340b0d50 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1340c6240 0x7fa1340b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1340c6240 0x7fa1340b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52842/0 (socket says 172.21.15.28:52842) 2024-09-10T20:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 -- 172.21.15.28:0/3820316042 learned_addr learned my addr 172.21.15.28:0/3820316042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa132ffd640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa120004800 0x7fa1341f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa1337fe640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1340b0d50 0x7fa13415d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 -- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1340b0d50 msgr2=0x7fa13415d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1340b0d50 0x7fa13415d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 -- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa120004800 msgr2=0x7fa1341f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa120004800 0x7fa1341f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 -- 172.21.15.28:0/3820316042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1340c8a00 con 0x7fa1340c6240 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa132ffd640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa120004800 0x7fa1341f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa1337fe640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1340b0d50 0x7fa13415d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa12bfff640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1340c6240 0x7fa1340b0810 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009a70 tx=0x7fa11c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa130ff9640 1 -- 172.21.15.28:0/3820316042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c043070 con 0x7fa1340c6240 2024-09-10T20:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa130ff9640 1 -- 172.21.15.28:0/3820316042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa11c004320 con 0x7fa1340c6240 2024-09-10T20:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1340c8ce0 con 0x7fa1340c6240 2024-09-10T20:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa130ff9640 1 -- 172.21.15.28:0/3820316042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c038040 con 0x7fa1340c6240 2024-09-10T20:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.581+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1340c86f0 con 0x7fa1340c6240 2024-09-10T20:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.582+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13415b9b0 con 0x7fa1340c6240 2024-09-10T20:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.583+0000 7fa130ff9640 1 -- 172.21.15.28:0/3820316042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fa11c03b030 con 0x7fa1340c6240 2024-09-10T20:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.584+0000 7fa130ff9640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1140783f0 0x7fa11407a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.584+0000 7fa132ffd640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1140783f0 0x7fa11407a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.584+0000 7fa130ff9640 1 -- 172.21.15.28:0/3820316042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fa11c0bab00 con 0x7fa1340c6240 2024-09-10T20:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.584+0000 7fa132ffd640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1140783f0 0x7fa11407a8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa1240018a0 tx=0x7fa124001e10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.588+0000 7fa130ff9640 1 -- 172.21.15.28:0/3820316042 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa11c0827c0 con 0x7fa1340c6240 2024-09-10T20:44:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.801+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol20671", "group_name": "subvol_grp74240", "target": ["mon-mgr", ""]}) -- 0x7fa1340b4db0 con 0x7fa1140783f0 2024-09-10T20:44:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.835+0000 7fa130ff9640 1 -- 172.21.15.28:0/3820316042 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+74 (secure 0 0 0) 0x7fa1340b4db0 con 0x7fa1140783f0 2024-09-10T20:44:03.840 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp74240/subvol20671/19fa2599-6812-4f48-a1e8-4a95426e497d 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1140783f0 msgr2=0x7fa11407a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1140783f0 0x7fa11407a8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa1240018a0 tx=0x7fa124001e10 comp rx=0 tx=0).stop 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1340c6240 msgr2=0x7fa1340b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1340c6240 0x7fa1340b0810 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009a70 tx=0x7fa11c033670 comp rx=0 tx=0).stop 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 shutdown_connections 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1140783f0 0x7fa11407a8b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1340b0d50 0x7fa13415d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1340c6240 0x7fa1340b0810 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 --2- 172.21.15.28:0/3820316042 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa120004800 0x7fa1341f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 >> 172.21.15.28:0/3820316042 conn(0x7fa1340ccb60 msgr2=0x7fa13406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 shutdown_connections 2024-09-10T20:44:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.837+0000 7fa1399a0640 1 -- 172.21.15.28:0/3820316042 wait complete. 2024-09-10T20:44:03.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol62402 2024-09-10T20:44:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.957+0000 7fe7d35d3640 1 -- 172.21.15.28:0/2513605541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 msgr2=0x7fe7b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.957+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/2513605541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7b4004c80 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009b20 tx=0x7fe7b4005830 comp rx=0 tx=0).stop 2024-09-10T20:44:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.957+0000 7fe7d35d3640 1 -- 172.21.15.28:0/2513605541 shutdown_connections 2024-09-10T20:44:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.957+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/2513605541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7b4004c80 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.957+0000 7fe7d35d3640 1 -- 172.21.15.28:0/2513605541 >> 172.21.15.28:0/2513605541 conn(0x7fe7cc06d110 msgr2=0x7fe7cc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:03.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.957+0000 7fe7d35d3640 1 -- 172.21.15.28:0/2513605541 shutdown_connections 2024-09-10T20:44:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.957+0000 7fe7d35d3640 1 -- 172.21.15.28:0/2513605541 wait complete. 2024-09-10T20:44:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.958+0000 7fe7d35d3640 1 Processor -- start 2024-09-10T20:44:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.958+0000 7fe7d35d3640 1 -- start start 2024-09-10T20:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d35d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7cc1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d35d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7cc1f5310 0x7fe7cc0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d35d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7cc15d650 0x7fe7cc15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d35d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc15b500 con 0x7fe7b4004800 2024-09-10T20:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d35d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc15b380 con 0x7fe7cc1f5310 2024-09-10T20:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d35d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc15b680 con 0x7fe7cc15d650 2024-09-10T20:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7cc1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7cc1538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52862/0 (socket says 172.21.15.28:52862) 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 -- 172.21.15.28:0/1312675027 learned_addr learned my addr 172.21.15.28:0/1312675027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d1b70640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7cc15d650 0x7fe7cc15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d0b6e640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7cc1f5310 0x7fe7cc0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 -- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7cc15d650 msgr2=0x7fe7cc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7cc15d650 0x7fe7cc15dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 -- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7cc1f5310 msgr2=0x7fe7cc0af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7cc1f5310 0x7fe7cc0af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.959+0000 7fe7d136f640 1 -- 172.21.15.28:0/1312675027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc0cdff0 con 0x7fe7b4004800 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.960+0000 7fe7d136f640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7cc1538c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006680 tx=0x7fe7c40066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.960+0000 7fe7c9ffb640 1 -- 172.21.15.28:0/1312675027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4015070 con 0x7fe7b4004800 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.960+0000 7fe7c9ffb640 1 -- 172.21.15.28:0/1312675027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7c4011920 con 0x7fe7b4004800 2024-09-10T20:44:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.960+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc0cd390 con 0x7fe7b4004800 2024-09-10T20:44:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.960+0000 7fe7c9ffb640 1 -- 172.21.15.28:0/1312675027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4011d70 con 0x7fe7b4004800 2024-09-10T20:44:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.960+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc1f5770 con 0x7fe7b4004800 2024-09-10T20:44:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.961+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc0b4d20 con 0x7fe7b4004800 2024-09-10T20:44:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.962+0000 7fe7c9ffb640 1 -- 172.21.15.28:0/1312675027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7fe7c4010420 con 0x7fe7b4004800 2024-09-10T20:44:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.962+0000 7fe7c9ffb640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe7b8078420 0x7fe7b807a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.962+0000 7fe7c9ffb640 1 -- 172.21.15.28:0/1312675027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fe7c409b5d0 con 0x7fe7b4004800 2024-09-10T20:44:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.962+0000 7fe7d0b6e640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe7b8078420 0x7fe7b807a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.963+0000 7fe7d0b6e640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe7b8078420 0x7fe7b807a8e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0035000 tx=0x7fe7c00046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:03.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:03.967+0000 7fe7c9ffb640 1 -- 172.21.15.28:0/1312675027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe7cc0b4d20 con 0x7fe7b4004800 2024-09-10T20:44:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.191+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol62402", "target": ["mon-mgr", ""]}) -- 0x7fe7cc0ce2d0 con 0x7fe7b8078420 2024-09-10T20:44:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.216+0000 7fe7c9ffb640 1 -- 172.21.15.28:0/1312675027 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+144 (secure 0 0 0) 0x7fe7cc0ce2d0 con 0x7fe7b8078420 2024-09-10T20:44:04.221 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:44:04.221 INFO:teuthology.orchestra.run.smithi028.stdout: "status": { 2024-09-10T20:44:04.221 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "pending", 2024-09-10T20:44:04.222 INFO:teuthology.orchestra.run.smithi028.stdout: "source": { 2024-09-10T20:44:04.222 INFO:teuthology.orchestra.run.smithi028.stdout: "volume": "fake", 2024-09-10T20:44:04.222 INFO:teuthology.orchestra.run.smithi028.stdout: "subvolume": "fake", 2024-09-10T20:44:04.222 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot": "fake" 2024-09-10T20:44:04.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:04.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:04.222 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.218+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe7b8078420 msgr2=0x7fe7b807a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.218+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe7b8078420 0x7fe7b807a8e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0035000 tx=0x7fe7c00046c0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.218+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 msgr2=0x7fe7cc1538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7cc1538c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006680 tx=0x7fe7c40066b0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 shutdown_connections 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe7b8078420 0x7fe7b807a8e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe7cc15d650 0x7fe7cc15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe7cc1f5310 0x7fe7cc0af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 --2- 172.21.15.28:0/1312675027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b4004800 0x7fe7cc1538c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 >> 172.21.15.28:0/1312675027 conn(0x7fe7cc06d110 msgr2=0x7fe7cc15a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 shutdown_connections 2024-09-10T20:44:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.219+0000 7fe7d35d3640 1 -- 172.21.15.28:0/1312675027 wait complete. 2024-09-10T20:44:04.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol31370 fake 2024-09-10T20:44:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.341+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/1173449073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cda70 msgr2=0x7f5d280cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.341+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/1173449073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cda70 0x7f5d280cde80 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009830 tx=0x7f5d2002a480 comp rx=0 tx=0).stop 2024-09-10T20:44:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.341+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/1173449073 shutdown_connections 2024-09-10T20:44:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.341+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/1173449073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cda70 0x7f5d280cde80 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.341+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/1173449073 >> 172.21.15.28:0/1173449073 conn(0x7f5d280cfc80 msgr2=0x7f5d280d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.341+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/1173449073 shutdown_connections 2024-09-10T20:44:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.341+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/1173449073 wait complete. 2024-09-10T20:44:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.342+0000 7f5d2fcc5640 1 Processor -- start 2024-09-10T20:44:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.342+0000 7f5d2fcc5640 1 -- start start 2024-09-10T20:44:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2fcc5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cabc0 0x7f5d2814c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2fcc5640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d280cda70 0x7f5d280c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2fcc5640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d2815a350 0x7f5d280c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2fcc5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2806e720 con 0x7f5d280cabc0 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2fcc5640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2806e5a0 con 0x7f5d280cda70 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2fcc5640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2806e8a0 con 0x7f5d2815a350 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2da61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cabc0 0x7f5d2814c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2da61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cabc0 0x7f5d2814c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52882/0 (socket says 172.21.15.28:52882) 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2da61640 1 -- 172.21.15.28:0/4025613426 learned_addr learned my addr 172.21.15.28:0/4025613426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2e262640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d2815a350 0x7f5d280c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.343+0000 7f5d2d260640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d280cda70 0x7f5d280c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2da61640 1 -- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d2815a350 msgr2=0x7f5d280c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2da61640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d2815a350 0x7f5d280c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2da61640 1 -- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d280cda70 msgr2=0x7f5d280c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2da61640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d280cda70 0x7f5d280c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2da61640 1 -- 172.21.15.28:0/4025613426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2814cf30 con 0x7f5d280cabc0 2024-09-10T20:44:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2da61640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cabc0 0x7f5d2814c1c0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400c7a0 tx=0x7f5d2400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24017070 con 0x7f5d280cabc0 2024-09-10T20:44:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d24005070 con 0x7f5d280cabc0 2024-09-10T20:44:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2814ca10 con 0x7f5d280cabc0 2024-09-10T20:44:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d240054a0 con 0x7f5d280cabc0 2024-09-10T20:44:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.344+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d280c94e0 con 0x7f5d280cabc0 2024-09-10T20:44:04.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.345+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d280cde80 con 0x7f5d280cabc0 2024-09-10T20:44:04.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.346+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 101009+0+0 (secure 0 0 0) 0x7f5d24005640 con 0x7f5d280cabc0 2024-09-10T20:44:04.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.346+0000 7f5d1e7fc640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5d0c078420 0x7f5d0c07a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.346+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f5d24099f60 con 0x7f5d280cabc0 2024-09-10T20:44:04.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.346+0000 7f5d2d260640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5d0c078420 0x7f5d0c07a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:04.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.347+0000 7f5d2d260640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5d0c078420 0x7f5d0c07a8e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5d281481e0 tx=0x7f5d18009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:04.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.350+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5d280ce3c0 con 0x7f5d280cabc0 2024-09-10T20:44:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.459+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 21) ==== 101060+0+0 (secure 0 0 0) 0x7f5d2409d030 con 0x7f5d280cabc0 2024-09-10T20:44:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.571+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol31370", "snap_name": "fake", "target": ["mon-mgr", ""]}) -- 0x7f5d280c9880 con 0x7f5d0c078420 2024-09-10T20:44:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.605+0000 7f5d1e7fc640 1 -- 172.21.15.28:0/4025613426 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5d280c9880 con 0x7f5d0c078420 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.607+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5d0c078420 msgr2=0x7f5d0c07a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.607+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5d0c078420 0x7f5d0c07a8e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5d281481e0 tx=0x7f5d18009040 comp rx=0 tx=0).stop 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.607+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cabc0 msgr2=0x7f5d2814c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.607+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cabc0 0x7f5d2814c1c0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400c7a0 tx=0x7f5d2400cc60 comp rx=0 tx=0).stop 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 shutdown_connections 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5d0c078420 0x7f5d0c07a8e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5d2815a350 0x7f5d280c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5d280cda70 0x7f5d280c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 --2- 172.21.15.28:0/4025613426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d280cabc0 0x7f5d2814c1c0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 >> 172.21.15.28:0/4025613426 conn(0x7f5d280cfc80 msgr2=0x7f5d280d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 shutdown_connections 2024-09-10T20:44:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.608+0000 7f5d2fcc5640 1 -- 172.21.15.28:0/4025613426 wait complete. 2024-09-10T20:44:04.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol20671 fake subvol_grp74240 2024-09-10T20:44:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.730+0000 7f53fe3ad640 1 -- 172.21.15.28:0/330999502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 msgr2=0x7f53e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.730+0000 7f53fe3ad640 1 --2- 172.21.15.28:0/330999502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 0x7f53e0004c80 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f53f80b5b30 tx=0x7f53e0005830 comp rx=0 tx=0).stop 2024-09-10T20:44:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.730+0000 7f53fc94a640 1 -- 172.21.15.28:0/330999502 reap_dead start 2024-09-10T20:44:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.730+0000 7f53fe3ad640 1 -- 172.21.15.28:0/330999502 shutdown_connections 2024-09-10T20:44:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.730+0000 7f53fe3ad640 1 -- 172.21.15.28:0/330999502 >> 172.21.15.28:0/330999502 conn(0x7f53f80d4070 msgr2=0x7f53f80d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.730+0000 7f53fe3ad640 1 -- 172.21.15.28:0/330999502 shutdown_connections 2024-09-10T20:44:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.730+0000 7f53fe3ad640 1 -- 172.21.15.28:0/330999502 wait complete. 2024-09-10T20:44:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.731+0000 7f53fe3ad640 1 Processor -- start 2024-09-10T20:44:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.731+0000 7f53fe3ad640 1 -- start start 2024-09-10T20:44:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53fe3ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 0x7f53f81f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53fe3ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53f80a1900 0x7f53f80b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53fe3ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53f80b1fb0 0x7f53f8152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53fe3ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53f80b5770 con 0x7f53e0004800 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53fe3ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f53f80b55f0 con 0x7f53f80a1900 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53fe3ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f53f80b58f0 con 0x7f53f80b1fb0 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 0x7f53f81f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 0x7f53f81f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52898/0 (socket says 172.21.15.28:52898) 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.732+0000 7f53f7fff640 1 -- 172.21.15.28:0/1764101154 learned_addr learned my addr 172.21.15.28:0/1764101154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f77fe640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53f80a1900 0x7f53f80b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f7fff640 1 -- 172.21.15.28:0/1764101154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53f80b1fb0 msgr2=0x7f53f8152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f7fff640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53f80b1fb0 0x7f53f8152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f7fff640 1 -- 172.21.15.28:0/1764101154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53f80a1900 msgr2=0x7f53f80b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f7fff640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53f80a1900 0x7f53f80b1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f7fff640 1 -- 172.21.15.28:0/1764101154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f80a1d80 con 0x7f53e0004800 2024-09-10T20:44:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f7fff640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 0x7f53f81f8930 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f53e8011a80 tx=0x7f53e8011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f4ff9640 1 -- 172.21.15.28:0/1764101154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e800fd50 con 0x7f53e0004800 2024-09-10T20:44:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f4ff9640 1 -- 172.21.15.28:0/1764101154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53e8016070 con 0x7f53e0004800 2024-09-10T20:44:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f8153b70 con 0x7f53e0004800 2024-09-10T20:44:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53f4ff9640 1 -- 172.21.15.28:0/1764101154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e80126b0 con 0x7f53e0004800 2024-09-10T20:44:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.733+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f81f7e50 con 0x7f53e0004800 2024-09-10T20:44:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.734+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f80d1fe0 con 0x7f53e0004800 2024-09-10T20:44:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.735+0000 7f53f4ff9640 1 -- 172.21.15.28:0/1764101154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 101060+0+0 (secure 0 0 0) 0x7f53e8012850 con 0x7f53e0004800 2024-09-10T20:44:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.735+0000 7f53f4ff9640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f53e40784b0 0x7f53e407a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.736+0000 7f53f77fe640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f53e40784b0 0x7f53e407a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.736+0000 7f53f4ff9640 1 -- 172.21.15.28:0/1764101154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f53e809b290 con 0x7f53e0004800 2024-09-10T20:44:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.736+0000 7f53f77fe640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f53e40784b0 0x7f53e407a970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f53ec033000 tx=0x7f53f80b3610 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.740+0000 7f53f4ff9640 1 -- 172.21.15.28:0/1764101154 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f53e80634d0 con 0x7f53e0004800 2024-09-10T20:44:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.959+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol20671", "snap_name": "fake", "group_name": "subvol_grp74240", "target": ["mon-mgr", ""]}) -- 0x7f53f81f4b90 con 0x7f53e40784b0 2024-09-10T20:44:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.997+0000 7f53f4ff9640 1 -- 172.21.15.28:0/1764101154 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f53ec0014f0 con 0x7f53e40784b0 2024-09-10T20:44:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f53e40784b0 msgr2=0x7f53e407a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f53e40784b0 0x7f53e407a970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f53ec033000 tx=0x7f53f80b3610 comp rx=0 tx=0).stop 2024-09-10T20:44:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 msgr2=0x7f53f81f8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 0x7f53f81f8930 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f53e8011a80 tx=0x7f53e8011f40 comp rx=0 tx=0).stop 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 shutdown_connections 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f53e40784b0 0x7f53e407a970 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53f80b1fb0 0x7f53f8152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53f80a1900 0x7f53f80b1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 --2- 172.21.15.28:0/1764101154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0004800 0x7f53f81f8930 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 >> 172.21.15.28:0/1764101154 conn(0x7f53f80d4070 msgr2=0x7f53f80d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 shutdown_connections 2024-09-10T20:44:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:04.999+0000 7f53fe3ad640 1 -- 172.21.15.28:0/1764101154 wait complete. 2024-09-10T20:44:05.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:05.015 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol31370/.meta) 2024-09-10T20:44:05.041 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:44:05.041 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:44:05.041 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:44:05.041 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol31370/9e18844c-eabb-456e-9bab-26162b769ef0 2024-09-10T20:44:05.041 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:44:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:05.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:05.043 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/subvol_grp74240/subvol20671/.meta) 2024-09-10T20:44:05.112 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:44:05.112 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:44:05.112 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:44:05.112 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/subvol_grp74240/subvol20671/19fa2599-6812-4f48-a1e8-4a95426e497d 2024-09-10T20:44:05.112 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:44:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:05.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol31370 2024-09-10T20:44:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 -- 172.21.15.28:0/3619061683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8664073ea0 msgr2=0x7f86640cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 --2- 172.21.15.28:0/3619061683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8664073ea0 0x7f86640cc3b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f865c009be0 tx=0x7f865c02a890 comp rx=0 tx=0).stop 2024-09-10T20:44:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 -- 172.21.15.28:0/3619061683 shutdown_connections 2024-09-10T20:44:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 --2- 172.21.15.28:0/3619061683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8664073ea0 0x7f86640cc3b0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 --2- 172.21.15.28:0/3619061683 >> v2:172.21.15.159:3300/0 conn(0x7f8664151db0 0x7f86640706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 -- 172.21.15.28:0/3619061683 >> 172.21.15.28:0/3619061683 conn(0x7f86640d0d50 msgr2=0x7f86640d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 -- 172.21.15.28:0/3619061683 shutdown_connections 2024-09-10T20:44:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.257+0000 7f866bc60640 1 -- 172.21.15.28:0/3619061683 wait complete. 2024-09-10T20:44:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.258+0000 7f866bc60640 1 Processor -- start 2024-09-10T20:44:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.258+0000 7f866bc60640 1 -- start start 2024-09-10T20:44:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866bc60640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8664073ea0 0x7f86641511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866bc60640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8664151db0 0x7f86640cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866bc60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866415bd80 0x7f86640cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866bc60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86640cce70 con 0x7f866415bd80 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866bc60640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f86640cccf0 con 0x7f8664151db0 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866bc60640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f86640ccff0 con 0x7f8664073ea0 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866415bd80 0x7f86640cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866415bd80 0x7f86640cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52910/0 (socket says 172.21.15.28:52910) 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 -- 172.21.15.28:0/1535999842 learned_addr learned my addr 172.21.15.28:0/1535999842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f86699fc640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8664073ea0 0x7f86641511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f86691fb640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8664151db0 0x7f86640cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 -- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8664073ea0 msgr2=0x7f86641511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8664073ea0 0x7f86641511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 -- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8664151db0 msgr2=0x7f86640cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8664151db0 0x7f86640cf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.259+0000 7f866a1fd640 1 -- 172.21.15.28:0/1535999842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8664150730 con 0x7f866415bd80 2024-09-10T20:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.260+0000 7f866a1fd640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866415bd80 0x7f86640cfe20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f865c035040 tx=0x7f865c033d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.260+0000 7f865a7fc640 1 -- 172.21.15.28:0/1535999842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c042020 con 0x7f866415bd80 2024-09-10T20:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.260+0000 7f865a7fc640 1 -- 172.21.15.28:0/1535999842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f865c004040 con 0x7f866415bd80 2024-09-10T20:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.260+0000 7f865a7fc640 1 -- 172.21.15.28:0/1535999842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c037040 con 0x7f866415bd80 2024-09-10T20:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.260+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866415a0b0 con 0x7f866415bd80 2024-09-10T20:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.260+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866415a390 con 0x7f866415bd80 2024-09-10T20:44:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.261+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f866414d7b0 con 0x7f866415bd80 2024-09-10T20:44:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.261+0000 7f865a7fc640 1 -- 172.21.15.28:0/1535999842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 101060+0+0 (secure 0 0 0) 0x7f865c0044b0 con 0x7f866415bd80 2024-09-10T20:44:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.262+0000 7f865a7fc640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8648078480 0x7f864807a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.262+0000 7f865a7fc640 1 -- 172.21.15.28:0/1535999842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f865c0bbaf0 con 0x7f866415bd80 2024-09-10T20:44:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.262+0000 7f86699fc640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8648078480 0x7f864807a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.262+0000 7f86699fc640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8648078480 0x7f864807a940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8660005fd0 tx=0x7f8660005850 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:05.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.266+0000 7f865a7fc640 1 -- 172.21.15.28:0/1535999842 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f865c04c030 con 0x7f866415bd80 2024-09-10T20:44:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.479+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol31370", "target": ["mon-mgr", ""]}) -- 0x7f86640749a0 con 0x7f8648078480 2024-09-10T20:44:05.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.510+0000 7f865a7fc640 1 -- 172.21.15.28:0/1535999842 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f86640749a0 con 0x7f8648078480 2024-09-10T20:44:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8648078480 msgr2=0x7f864807a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8648078480 0x7f864807a940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8660005fd0 tx=0x7f8660005850 comp rx=0 tx=0).stop 2024-09-10T20:44:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866415bd80 msgr2=0x7f86640cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866415bd80 0x7f86640cfe20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f865c035040 tx=0x7f865c033d20 comp rx=0 tx=0).stop 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 shutdown_connections 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8648078480 0x7f864807a940 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866415bd80 0x7f86640cfe20 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8664151db0 0x7f86640cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 --2- 172.21.15.28:0/1535999842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8664073ea0 0x7f86641511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 >> 172.21.15.28:0/1535999842 conn(0x7f86640d0d50 msgr2=0x7f86640b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 shutdown_connections 2024-09-10T20:44:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.512+0000 7f866bc60640 1 -- 172.21.15.28:0/1535999842 wait complete. 2024-09-10T20:44:05.519 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.515+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.520 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.515+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.520 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.515+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.520 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.515+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.520 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.515+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol20671 subvol_grp74240 2024-09-10T20:44:05.533 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.529+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.533 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.529+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.533 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.529+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.533 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.529+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.534 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:05.529+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:05.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.637+0000 7f9a10191640 1 -- 172.21.15.28:0/215769313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a08162000 msgr2=0x7f9a080d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.637+0000 7f9a10191640 1 --2- 172.21.15.28:0/215769313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a08162000 0x7f9a080d4a80 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f9a04005fd0 tx=0x7f9a0402a670 comp rx=0 tx=0).stop 2024-09-10T20:44:05.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.637+0000 7f9a10191640 1 -- 172.21.15.28:0/215769313 shutdown_connections 2024-09-10T20:44:05.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.637+0000 7f9a10191640 1 --2- 172.21.15.28:0/215769313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a08162000 0x7f9a080d4a80 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.637+0000 7f9a10191640 1 -- 172.21.15.28:0/215769313 >> 172.21.15.28:0/215769313 conn(0x7f9a080d3f90 msgr2=0x7f9a080d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:05.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.637+0000 7f9a10191640 1 -- 172.21.15.28:0/215769313 shutdown_connections 2024-09-10T20:44:05.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.637+0000 7f9a10191640 1 -- 172.21.15.28:0/215769313 wait complete. 2024-09-10T20:44:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.638+0000 7f9a10191640 1 Processor -- start 2024-09-10T20:44:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.638+0000 7f9a10191640 1 -- start start 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a10191640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a080b4c20 0x7f9a080cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a10191640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a08161850 0x7f9a081f8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a10191640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a08162000 0x7f9a080b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a10191640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a080d2370 con 0x7f9a080b4c20 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a10191640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9a080d24f0 con 0x7f9a08162000 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a10191640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9a080d2670 con 0x7f9a08161850 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a080b4c20 0x7f9a080cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a080b4c20 0x7f9a080cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52954/0 (socket says 172.21.15.28:52954) 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 -- 172.21.15.28:0/958415048 learned_addr learned my addr 172.21.15.28:0/958415048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0e72e640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a08162000 0x7f9a080b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0d72c640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a08161850 0x7f9a081f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 -- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a08161850 msgr2=0x7f9a081f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a08161850 0x7f9a081f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 -- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a08162000 msgr2=0x7f9a080b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a08162000 0x7f9a080b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 -- 172.21.15.28:0/958415048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a081f8aa0 con 0x7f9a080b4c20 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.639+0000 7f9a0df2d640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a080b4c20 0x7f9a080cd4a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f9a04034040 tx=0x7f9a040039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.640+0000 7f99fe7fc640 1 -- 172.21.15.28:0/958415048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0403e030 con 0x7f9a080b4c20 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.640+0000 7f99fe7fc640 1 -- 172.21.15.28:0/958415048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a040047a0 con 0x7f9a080b4c20 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.640+0000 7f99fe7fc640 1 -- 172.21.15.28:0/958415048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04035040 con 0x7f9a080b4c20 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.640+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a081f8d80 con 0x7f9a080b4c20 2024-09-10T20:44:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.640+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a080cd9e0 con 0x7f9a080b4c20 2024-09-10T20:44:05.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.641+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a080b9f10 con 0x7f9a080b4c20 2024-09-10T20:44:05.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.641+0000 7f99fe7fc640 1 -- 172.21.15.28:0/958415048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 101060+0+0 (secure 0 0 0) 0x7f9a04004b90 con 0x7f9a080b4c20 2024-09-10T20:44:05.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.642+0000 7f99fe7fc640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99ec078480 0x7f99ec07a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:05.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.642+0000 7f9a0d72c640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99ec078480 0x7f99ec07a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:05.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.642+0000 7f99fe7fc640 1 -- 172.21.15.28:0/958415048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f9a040bb1d0 con 0x7f9a080b4c20 2024-09-10T20:44:05.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.642+0000 7f9a0d72c640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99ec078480 0x7f99ec07a940 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9a080b12f0 tx=0x7f99f8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.647+0000 7f99fe7fc640 1 -- 172.21.15.28:0/958415048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9a040834a0 con 0x7f9a080b4c20 2024-09-10T20:44:05.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.860+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol20671", "group_name": "subvol_grp74240", "target": ["mon-mgr", ""]}) -- 0x7f9a081f8f50 con 0x7f99ec078480 2024-09-10T20:44:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.892+0000 7f99fe7fc640 1 -- 172.21.15.28:0/958415048 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f99f8007db0 con 0x7f99ec078480 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99ec078480 msgr2=0x7f99ec07a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99ec078480 0x7f99ec07a940 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9a080b12f0 tx=0x7f99f8009040 comp rx=0 tx=0).stop 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a080b4c20 msgr2=0x7f9a080cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a080b4c20 0x7f9a080cd4a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f9a04034040 tx=0x7f9a040039e0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 shutdown_connections 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99ec078480 0x7f99ec07a940 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a08162000 0x7f9a080b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a08161850 0x7f9a081f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 --2- 172.21.15.28:0/958415048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a080b4c20 0x7f9a080cd4a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 >> 172.21.15.28:0/958415048 conn(0x7f9a080d3f90 msgr2=0x7f9a080b86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 shutdown_connections 2024-09-10T20:44:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:05.895+0000 7f9a10191640 1 -- 172.21.15.28:0/958415048 wait complete. 2024-09-10T20:44:05.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol62402 2024-09-10T20:44:06.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.055+0000 7f2d7569b640 1 -- 172.21.15.28:0/1488014386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d700d27f0 msgr2=0x7f2d700b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.055+0000 7f2d7569b640 1 --2- 172.21.15.28:0/1488014386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d700d27f0 0x7f2d700b1830 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009830 tx=0x7f2d5802a480 comp rx=0 tx=0).stop 2024-09-10T20:44:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.055+0000 7f2d7569b640 1 -- 172.21.15.28:0/1488014386 shutdown_connections 2024-09-10T20:44:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.055+0000 7f2d7569b640 1 --2- 172.21.15.28:0/1488014386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d700d27f0 0x7f2d700b1830 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.055+0000 7f2d7569b640 1 -- 172.21.15.28:0/1488014386 >> 172.21.15.28:0/1488014386 conn(0x7f2d700d4070 msgr2=0x7f2d700d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.055+0000 7f2d7569b640 1 -- 172.21.15.28:0/1488014386 shutdown_connections 2024-09-10T20:44:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.055+0000 7f2d7569b640 1 -- 172.21.15.28:0/1488014386 wait complete. 2024-09-10T20:44:06.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.056+0000 7f2d7569b640 1 Processor -- start 2024-09-10T20:44:06.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d7569b640 1 -- start start 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d7569b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d700d27f0 0x7f2d700cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d7569b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d70161770 0x7f2d7015b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d7569b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d70161b50 0x7f2d700ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d7569b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d700b2f50 con 0x7f2d70161770 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d7569b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2d700b2dd0 con 0x7f2d70161b50 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d7569b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2d700b30d0 con 0x7f2d700d27f0 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d6e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d70161770 0x7f2d7015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.057+0000 7f2d6e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d70161770 0x7f2d7015b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52972/0 (socket says 172.21.15.28:52972) 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6e7fc640 1 -- 172.21.15.28:0/17869907 learned_addr learned my addr 172.21.15.28:0/17869907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6effd640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d700d27f0 0x7f2d700cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6f7fe640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d70161b50 0x7f2d700ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6e7fc640 1 -- 172.21.15.28:0/17869907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d700d27f0 msgr2=0x7f2d700cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6e7fc640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2d700d27f0 0x7f2d700cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6e7fc640 1 -- 172.21.15.28:0/17869907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d70161b50 msgr2=0x7f2d700ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6e7fc640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2d70161b50 0x7f2d700ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6e7fc640 1 -- 172.21.15.28:0/17869907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d700cb140 con 0x7f2d70161770 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d6e7fc640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d70161770 0x7f2d7015b220 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009950 tx=0x7f2d58032cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d677fe640 1 -- 172.21.15.28:0/17869907 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58040070 con 0x7f2d70161770 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d677fe640 1 -- 172.21.15.28:0/17869907 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d58005310 con 0x7f2d70161770 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d677fe640 1 -- 172.21.15.28:0/17869907 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58003800 con 0x7f2d70161770 2024-09-10T20:44:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d7569b640 1 -- 172.21.15.28:0/17869907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d701f5330 con 0x7f2d70161770 2024-09-10T20:44:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.058+0000 7f2d7569b640 1 -- 172.21.15.28:0/17869907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d701f58f0 con 0x7f2d70161770 2024-09-10T20:44:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.059+0000 7f2d7569b640 1 -- 172.21.15.28:0/17869907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d700b2690 con 0x7f2d70161770 2024-09-10T20:44:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.060+0000 7f2d677fe640 1 -- 172.21.15.28:0/17869907 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 101060+0+0 (secure 0 0 0) 0x7f2d5804c030 con 0x7f2d70161770 2024-09-10T20:44:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.060+0000 7f2d677fe640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2d50078480 0x7f2d5007a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.060+0000 7f2d677fe640 1 -- 172.21.15.28:0/17869907 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f2d580bb710 con 0x7f2d70161770 2024-09-10T20:44:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.061+0000 7f2d6effd640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2d50078480 0x7f2d5007a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.061+0000 7f2d6effd640 1 --2- 172.21.15.28:0/17869907 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2d50078480 0x7f2d5007a940 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2d60005660 tx=0x7f2d60003d90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:06.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.066+0000 7f2d677fe640 1 -- 172.21.15.28:0/17869907 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2d58083410 con 0x7f2d70161770 2024-09-10T20:44:06.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.277+0000 7f2d7569b640 1 -- 172.21.15.28:0/17869907 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol62402", "target": ["mon-mgr", ""]}) -- 0x7f2d7015fb70 con 0x7f2d50078480 2024-09-10T20:44:06.298 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:06.295+0000 7f93ce3fd640 -1 mgr.server reply reply (11) Resource temporarily unavailable subvolume 'subvol62402' is not ready for operation rm (use --force to override) 2024-09-10T20:44:06.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.295+0000 7f2d677fe640 1 -- 172.21.15.28:0/17869907 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -11 subvolume 'subvol62402' is not ready for operation rm (use --force to override)) ==== 87+0+0 (secure 0 0 0) 0x7f2d7015fb70 con 0x7f2d50078480 2024-09-10T20:44:06.299 INFO:teuthology.orchestra.run.smithi028.stderr:Error EAGAIN: subvolume 'subvol62402' is not ready for operation rm (use --force to override) 2024-09-10T20:44:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-09-10T20:44:06.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:06.305 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol62402/.meta) 2024-09-10T20:44:06.331 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:44:06.331 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:44:06.331 INFO:teuthology.orchestra.run.smithi028.stdout:type = clone 2024-09-10T20:44:06.331 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol62402/cd1f6c8b-861e-459b-85d8-0eb7828ec75e 2024-09-10T20:44:06.331 INFO:teuthology.orchestra.run.smithi028.stdout:state = pending 2024-09-10T20:44:06.331 INFO:teuthology.orchestra.run.smithi028.stdout:[source] 2024-09-10T20:44:06.331 INFO:teuthology.orchestra.run.smithi028.stdout:volume = fake 2024-09-10T20:44:06.332 INFO:teuthology.orchestra.run.smithi028.stdout:subvolume = fake 2024-09-10T20:44:06.332 INFO:teuthology.orchestra.run.smithi028.stdout:snapshot = fake 2024-09-10T20:44:06.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol62402 --force 2024-09-10T20:44:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 -- 172.21.15.28:0/2406383248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 msgr2=0x7fc5580741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/2406383248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5580741b0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc55815ff60 tx=0x7fc55002aac0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 -- 172.21.15.28:0/2406383248 shutdown_connections 2024-09-10T20:44:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/2406383248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5580741b0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/2406383248 >> v2:172.21.15.159:3300/0 conn(0x7fc55806e390 0x7fc55815cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 -- 172.21.15.28:0/2406383248 >> 172.21.15.28:0/2406383248 conn(0x7fc5580cfc80 msgr2=0x7fc5580d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:06.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 -- 172.21.15.28:0/2406383248 shutdown_connections 2024-09-10T20:44:06.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.481+0000 7fc55d5f3640 1 -- 172.21.15.28:0/2406383248 wait complete. 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.482+0000 7fc55d5f3640 1 Processor -- start 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.482+0000 7fc55d5f3640 1 -- start start 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc55d5f3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc55806e390 0x7fc5581565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc55d5f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5581e64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc55d5f3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b5030 0x7fc558158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc55d5f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5580ca450 con 0x7fc558073d40 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc55d5f3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc5580ca2d0 con 0x7fc55806e390 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc55d5f3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc5580ca5d0 con 0x7fc5580b5030 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5581e64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5581e64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52982/0 (socket says 172.21.15.28:52982) 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 -- 172.21.15.28:0/4284830026 learned_addr learned my addr 172.21.15.28:0/4284830026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc556ffd640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc55806e390 0x7fc5581565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc5577fe640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b5030 0x7fc558158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 -- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b5030 msgr2=0x7fc558158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b5030 0x7fc558158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 -- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc55806e390 msgr2=0x7fc5581565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc55806e390 0x7fc5581565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 -- 172.21.15.28:0/4284830026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5580c9de0 con 0x7fc558073d40 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc556ffd640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc55806e390 0x7fc5581565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.483+0000 7fc54ffff640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5581e64e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc54000d950 tx=0x7fc54000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.484+0000 7fc54f7fe640 1 -- 172.21.15.28:0/4284830026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc540017070 con 0x7fc558073d40 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.484+0000 7fc54f7fe640 1 -- 172.21.15.28:0/4284830026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc540009cc0 con 0x7fc558073d40 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.484+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5580c9260 con 0x7fc558073d40 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.484+0000 7fc54f7fe640 1 -- 172.21.15.28:0/4284830026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5400106c0 con 0x7fc558073d40 2024-09-10T20:44:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.484+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5580c9820 con 0x7fc558073d40 2024-09-10T20:44:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.485+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5580cbc40 con 0x7fc558073d40 2024-09-10T20:44:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.485+0000 7fc54f7fe640 1 -- 172.21.15.28:0/4284830026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fc540010860 con 0x7fc558073d40 2024-09-10T20:44:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.486+0000 7fc54f7fe640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5380785d0 0x7fc53807aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.486+0000 7fc54f7fe640 1 -- 172.21.15.28:0/4284830026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fc54009acf0 con 0x7fc558073d40 2024-09-10T20:44:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.486+0000 7fc556ffd640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5380785d0 0x7fc53807aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.486+0000 7fc556ffd640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5380785d0 0x7fc53807aa90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc558070630 tx=0x7fc548009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.490+0000 7fc54f7fe640 1 -- 172.21.15.28:0/4284830026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc5580c9de0 con 0x7fc558073d40 2024-09-10T20:44:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.703+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol62402", "force": true, "target": ["mon-mgr", ""]}) -- 0x7fc55815fd70 con 0x7fc5380785d0 2024-09-10T20:44:06.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.730+0000 7fc54f7fe640 1 -- 172.21.15.28:0/4284830026 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc55815fd70 con 0x7fc5380785d0 2024-09-10T20:44:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5380785d0 msgr2=0x7fc53807aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5380785d0 0x7fc53807aa90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc558070630 tx=0x7fc548009040 comp rx=0 tx=0).stop 2024-09-10T20:44:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 msgr2=0x7fc5581e64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5581e64e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc54000d950 tx=0x7fc54000de10 comp rx=0 tx=0).stop 2024-09-10T20:44:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 shutdown_connections 2024-09-10T20:44:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5380785d0 0x7fc53807aa90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5580b5030 0x7fc558158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc558073d40 0x7fc5581e64e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 --2- 172.21.15.28:0/4284830026 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc55806e390 0x7fc5581565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 >> 172.21.15.28:0/4284830026 conn(0x7fc5580cfc80 msgr2=0x7fc5580d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 shutdown_connections 2024-09-10T20:44:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.733+0000 7fc55d5f3640 1 -- 172.21.15.28:0/4284830026 wait complete. 2024-09-10T20:44:06.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2024-09-10T20:44:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.854+0000 7fc585079640 1 -- 172.21.15.28:0/3932088908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc56c004d40 msgr2=0x7fc56c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.854+0000 7fc585079640 1 --2- 172.21.15.28:0/3932088908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc56c004d40 0x7fc56c0051c0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc578009e20 tx=0x7fc56c005830 comp rx=0 tx=0).stop 2024-09-10T20:44:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.854+0000 7fc57f59e640 1 -- 172.21.15.28:0/3932088908 reap_dead start 2024-09-10T20:44:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.854+0000 7fc585079640 1 -- 172.21.15.28:0/3932088908 shutdown_connections 2024-09-10T20:44:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.854+0000 7fc585079640 1 -- 172.21.15.28:0/3932088908 >> 172.21.15.28:0/3932088908 conn(0x7fc58006dbf0 msgr2=0x7fc58006dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.854+0000 7fc585079640 1 -- 172.21.15.28:0/3932088908 shutdown_connections 2024-09-10T20:44:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.854+0000 7fc585079640 1 -- 172.21.15.28:0/3932088908 wait complete. 2024-09-10T20:44:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 Processor -- start 2024-09-10T20:44:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 -- start start 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc56c004d40 0x7fc5801f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58014c5b0 0x7fc5801568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58014bcf0 0x7fc5800b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5800b3fb0 con 0x7fc58014c5b0 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc5800b3e30 con 0x7fc56c004d40 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc585079640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc5800b4130 con 0x7fc58014bcf0 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.856+0000 7fc577fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58014c5b0 0x7fc5801568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58014c5b0 0x7fc5801568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52996/0 (socket says 172.21.15.28:52996) 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 -- 172.21.15.28:0/4069001622 learned_addr learned my addr 172.21.15.28:0/4069001622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc57f59e640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58014bcf0 0x7fc5800b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc57ed9d640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc56c004d40 0x7fc5801f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 -- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58014bcf0 msgr2=0x7fc5800b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58014bcf0 0x7fc5800b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 -- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc56c004d40 msgr2=0x7fc5801f6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc56c004d40 0x7fc5801f6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 -- 172.21.15.28:0/4069001622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc580157080 con 0x7fc58014c5b0 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc57f59e640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58014bcf0 0x7fc5800b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc577fff640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58014c5b0 0x7fc5801568e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc56800e630 tx=0x7fc56800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc5777fe640 1 -- 172.21.15.28:0/4069001622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568015070 con 0x7fc58014c5b0 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc5777fe640 1 -- 172.21.15.28:0/4069001622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc56800caa0 con 0x7fc58014c5b0 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5801f5fd0 con 0x7fc58014c5b0 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc5777fe640 1 -- 172.21.15.28:0/4069001622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568011460 con 0x7fc58014c5b0 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.857+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc580157360 con 0x7fc58014c5b0 2024-09-10T20:44:06.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.858+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5800b46c0 con 0x7fc58014c5b0 2024-09-10T20:44:06.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.859+0000 7fc5777fe640 1 -- 172.21.15.28:0/4069001622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fc568011600 con 0x7fc58014c5b0 2024-09-10T20:44:06.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.859+0000 7fc5777fe640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5600785a0 0x7fc56007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:06.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.860+0000 7fc57ed9d640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5600785a0 0x7fc56007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:06.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.860+0000 7fc5777fe640 1 -- 172.21.15.28:0/4069001622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7fc56809b190 con 0x7fc58014c5b0 2024-09-10T20:44:06.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.860+0000 7fc57ed9d640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5600785a0 0x7fc56007aa60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc570003940 tx=0x7fc570006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:06.864+0000 7fc5777fe640 1 -- 172.21.15.28:0/4069001622 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc568063220 con 0x7fc58014c5b0 2024-09-10T20:44:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.079+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) -- 0x7fc5800b56d0 con 0x7fc5600785a0 2024-09-10T20:44:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.082+0000 7fc5777fe640 1 -- 172.21.15.28:0/4069001622 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fc5800b56d0 con 0x7fc5600785a0 2024-09-10T20:44:07.086 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-10T20:44:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5600785a0 msgr2=0x7fc56007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5600785a0 0x7fc56007aa60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc570003940 tx=0x7fc570006b70 comp rx=0 tx=0).stop 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58014c5b0 msgr2=0x7fc5801568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58014c5b0 0x7fc5801568e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc56800e630 tx=0x7fc56800eaf0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 shutdown_connections 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5600785a0 0x7fc56007aa60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58014bcf0 0x7fc5800b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58014c5b0 0x7fc5801568e0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 --2- 172.21.15.28:0/4069001622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc56c004d40 0x7fc5801f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 >> 172.21.15.28:0/4069001622 conn(0x7fc58006dbf0 msgr2=0x7fc58015d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 shutdown_connections 2024-09-10T20:44:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.084+0000 7fc585079640 1 -- 172.21.15.28:0/4069001622 wait complete. 2024-09-10T20:44:07.099 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-10T20:44:07.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:44:07.151 INFO:teuthology.orchestra.run.smithi028.stdout:2 2024-09-10T20:44:07.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:44:07.157 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:44:07.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:07.193 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:07.245 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:44:07.245 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:07.245 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:07.245 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:07.261 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:07.261 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:07.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 -- 172.21.15.28:0/2138144053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81680b2e30 msgr2=0x7f81680aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 --2- 172.21.15.28:0/2138144053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81680b2e30 0x7f81680aeab0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8168073960 tx=0x7f816002aac0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 -- 172.21.15.28:0/2138144053 shutdown_connections 2024-09-10T20:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 --2- 172.21.15.28:0/2138144053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81680b2e30 0x7f81680aeab0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 --2- 172.21.15.28:0/2138144053 >> v2:172.21.15.159:3300/0 conn(0x7f816815f780 0x7f8168160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 -- 172.21.15.28:0/2138144053 >> 172.21.15.28:0/2138144053 conn(0x7f81680d1de0 msgr2=0x7f81680d21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 -- 172.21.15.28:0/2138144053 shutdown_connections 2024-09-10T20:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.464+0000 7f816f184640 1 -- 172.21.15.28:0/2138144053 wait complete. 2024-09-10T20:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.466+0000 7f816f184640 1 Processor -- start 2024-09-10T20:44:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.466+0000 7f816f184640 1 -- start start 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.466+0000 7f816f184640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81680b2e30 0x7f81681f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.466+0000 7f816f184640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816815f3a0 0x7f81680ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.466+0000 7f816f184640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816815f780 0x7f81680c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816f184640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81680aeff0 con 0x7f816815f3a0 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816f184640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8168160ab0 con 0x7f81680b2e30 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816f184640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f81680af170 con 0x7f816815f780 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816815f3a0 0x7f81680ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816815f3a0 0x7f81680ca610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53012/0 (socket says 172.21.15.28:53012) 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 -- 172.21.15.28:0/3195422800 learned_addr learned my addr 172.21.15.28:0/3195422800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816d721640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816815f780 0x7f81680c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816cf20640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81680b2e30 0x7f81681f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 -- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816815f780 msgr2=0x7f81680c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816815f780 0x7f81680c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 -- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81680b2e30 msgr2=0x7f81681f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81680b2e30 0x7f81681f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 -- 172.21.15.28:0/3195422800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81681f3790 con 0x7f816815f3a0 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816cf20640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81680b2e30 0x7f81681f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f8167fff640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816815f3a0 0x7f81680ca610 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f815400d740 tx=0x7f815400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f81657fa640 1 -- 172.21.15.28:0/3195422800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154017070 con 0x7f816815f3a0 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f81657fa640 1 -- 172.21.15.28:0/3195422800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8154009d20 con 0x7f816815f3a0 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81680c8600 con 0x7f816815f3a0 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f81657fa640 1 -- 172.21.15.28:0/3195422800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81540106c0 con 0x7f816815f3a0 2024-09-10T20:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.467+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81680c6a70 con 0x7f816815f3a0 2024-09-10T20:44:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.468+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f816806d6e0 con 0x7f816815f3a0 2024-09-10T20:44:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.469+0000 7f81657fa640 1 -- 172.21.15.28:0/3195422800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f8154010860 con 0x7f816815f3a0 2024-09-10T20:44:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.470+0000 7f81657fa640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f814c0785a0 0x7f814c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.470+0000 7f81657fa640 1 -- 172.21.15.28:0/3195422800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5737+0+0 (secure 0 0 0) 0x7f815409adb0 con 0x7f816815f3a0 2024-09-10T20:44:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.470+0000 7f816cf20640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f814c0785a0 0x7f814c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.471+0000 7f816cf20640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f814c0785a0 0x7f814c07aa60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f815c007fd0 tx=0x7f815c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:07.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.478+0000 7f81657fa640 1 -- 172.21.15.28:0/3195422800 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f81681f3790 con 0x7f816815f3a0 2024-09-10T20:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.683+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f81680aeab0 con 0x7f816815f3a0 2024-09-10T20:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.684+0000 7f81657fa640 1 -- 172.21.15.28:0/3195422800 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v116) ==== 80+0+0 (secure 0 0 0) 0x7f8154062880 con 0x7f816815f3a0 2024-09-10T20:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:44:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f814c0785a0 msgr2=0x7f814c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f814c0785a0 0x7f814c07aa60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f815c007fd0 tx=0x7f815c009040 comp rx=0 tx=0).stop 2024-09-10T20:44:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816815f3a0 msgr2=0x7f81680ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816815f3a0 0x7f81680ca610 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f815400d740 tx=0x7f815400dc00 comp rx=0 tx=0).stop 2024-09-10T20:44:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 shutdown_connections 2024-09-10T20:44:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f814c0785a0 0x7f814c07aa60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f816815f780 0x7f81680c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816815f3a0 0x7f81680ca610 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 --2- 172.21.15.28:0/3195422800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f81680b2e30 0x7f81681f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 >> 172.21.15.28:0/3195422800 conn(0x7f81680d1de0 msgr2=0x7f81680cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 shutdown_connections 2024-09-10T20:44:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.686+0000 7f816f184640 1 -- 172.21.15.28:0/3195422800 wait complete. 2024-09-10T20:44:07.701 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:44:07.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:07.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:44:07.753 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:44:07.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:07.754 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:44:07.782 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:07.782 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:07.782 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:07.798 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:07.798 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:07.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:44:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 -- 172.21.15.28:0/2871064038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec1554e0 msgr2=0x7f5eec15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2871064038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec1554e0 0x7f5eec15e930 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0cbd10 tx=0x7f5edc02a560 comp rx=0 tx=0).stop 2024-09-10T20:44:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 -- 172.21.15.28:0/2871064038 shutdown_connections 2024-09-10T20:44:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2871064038 >> v2:172.21.15.159:3300/0 conn(0x7f5eec15ee70 0x7f5eec0d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2871064038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec1554e0 0x7f5eec15e930 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2871064038 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eec155100 0x7f5eec15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 -- 172.21.15.28:0/2871064038 >> 172.21.15.28:0/2871064038 conn(0x7f5eec06cf00 msgr2=0x7f5eec06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 -- 172.21.15.28:0/2871064038 shutdown_connections 2024-09-10T20:44:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.956+0000 7f5ef3994640 1 -- 172.21.15.28:0/2871064038 wait complete. 2024-09-10T20:44:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.957+0000 7f5ef3994640 1 Processor -- start 2024-09-10T20:44:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.957+0000 7f5ef3994640 1 -- start start 2024-09-10T20:44:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef3994640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eec155100 0x7f5eec09bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef3994640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eec1554e0 0x7f5eec148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef3994640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec15ee70 0x7f5eec06f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef3994640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec072bd0 con 0x7f5eec15ee70 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef3994640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec0d0ab0 con 0x7f5eec1554e0 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef3994640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5eec072d50 con 0x7f5eec155100 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef1f31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec15ee70 0x7f5eec06f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef1f31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec15ee70 0x7f5eec06f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53040/0 (socket says 172.21.15.28:53040) 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef1f31640 1 -- 172.21.15.28:0/2022781837 learned_addr learned my addr 172.21.15.28:0/2022781837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef0f2f640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eec1554e0 0x7f5eec148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.958+0000 7f5ef1730640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eec155100 0x7f5eec09bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef1f31640 1 -- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eec155100 msgr2=0x7f5eec09bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef1f31640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eec155100 0x7f5eec09bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef1f31640 1 -- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eec1554e0 msgr2=0x7f5eec148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef1f31640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eec1554e0 0x7f5eec148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef1f31640 1 -- 172.21.15.28:0/2022781837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eec148a70 con 0x7f5eec15ee70 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef1f31640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec15ee70 0x7f5eec06f400 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400d720 tx=0x7f5ee400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ee1ffb640 1 -- 172.21.15.28:0/2022781837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4017070 con 0x7f5eec15ee70 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ee1ffb640 1 -- 172.21.15.28:0/2022781837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ee4009ce0 con 0x7f5eec15ee70 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eec09c360 con 0x7f5eec15ee70 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ee1ffb640 1 -- 172.21.15.28:0/2022781837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4005280 con 0x7f5eec15ee70 2024-09-10T20:44:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.959+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eec09c920 con 0x7f5eec15ee70 2024-09-10T20:44:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.960+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eec15e930 con 0x7f5eec15ee70 2024-09-10T20:44:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.961+0000 7f5ee1ffb640 1 -- 172.21.15.28:0/2022781837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f5ee4005420 con 0x7f5eec15ee70 2024-09-10T20:44:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.961+0000 7f5ee1ffb640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ed00785a0 0x7f5ed007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.962+0000 7f5ee1ffb640 1 -- 172.21.15.28:0/2022781837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7f5ee409aa90 con 0x7f5eec15ee70 2024-09-10T20:44:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.962+0000 7f5ef1730640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ed00785a0 0x7f5ed007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.962+0000 7f5ef1730640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ed00785a0 0x7f5ed007aa60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005fd0 tx=0x7f5ee8009540 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:07.966+0000 7f5ee1ffb640 1 -- 172.21.15.28:0/2022781837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ee40624e0 con 0x7f5eec15ee70 2024-09-10T20:44:08.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.163+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5eec0cd5a0 con 0x7f5eec15ee70 2024-09-10T20:44:08.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.163+0000 7f5ee1ffb640 1 -- 172.21.15.28:0/2022781837 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v116) ==== 80+0+0 (secure 0 0 0) 0x7f5ee406b250 con 0x7f5eec15ee70 2024-09-10T20:44:08.167 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.165+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ed00785a0 msgr2=0x7f5ed007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.165+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ed00785a0 0x7f5ed007aa60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005fd0 tx=0x7f5ee8009540 comp rx=0 tx=0).stop 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.165+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec15ee70 msgr2=0x7f5eec06f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.165+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec15ee70 0x7f5eec06f400 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400d720 tx=0x7f5ee400dbe0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.165+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 shutdown_connections 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.165+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5ed00785a0 0x7f5ed007aa60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.166+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eec15ee70 0x7f5eec06f400 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.166+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5eec1554e0 0x7f5eec148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.166+0000 7f5ef3994640 1 --2- 172.21.15.28:0/2022781837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5eec155100 0x7f5eec09bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.166+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 >> 172.21.15.28:0/2022781837 conn(0x7f5eec06cf00 msgr2=0x7f5eec06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.166+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 shutdown_connections 2024-09-10T20:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.166+0000 7f5ef3994640 1 -- 172.21.15.28:0/2022781837 wait complete. 2024-09-10T20:44:08.180 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:44:08.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:08.180 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:08.209 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:44:08.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:08.209 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:44:08.266 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:08.266 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:08.266 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:08.282 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:44:08.282 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:08.282 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:08.283 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:08.322 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:44:08.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 -- 172.21.15.28:0/3332287975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c162260 msgr2=0x7fb87c0cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 --2- 172.21.15.28:0/3332287975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c162260 0x7fb87c0cfbf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb874005fd0 tx=0x7fb87402a840 comp rx=0 tx=0).stop 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 -- 172.21.15.28:0/3332287975 shutdown_connections 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 --2- 172.21.15.28:0/3332287975 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c162260 0x7fb87c0cfbf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 --2- 172.21.15.28:0/3332287975 >> v2:172.21.15.159:3300/0 conn(0x7fb87c15de70 0x7fb87c0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 -- 172.21.15.28:0/3332287975 >> 172.21.15.28:0/3332287975 conn(0x7fb87c0d3fb0 msgr2=0x7fb87c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 -- 172.21.15.28:0/3332287975 shutdown_connections 2024-09-10T20:44:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.428+0000 7fb882dbb640 1 -- 172.21.15.28:0/3332287975 wait complete. 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.429+0000 7fb882dbb640 1 Processor -- start 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.429+0000 7fb882dbb640 1 -- start start 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb882dbb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c0cf780 0x7fb87c0b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb882dbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb87c15de70 0x7fb87c159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb882dbb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb87c162260 0x7fb87c160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb882dbb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c0b7190 con 0x7fb87c15de70 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb882dbb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c0b7010 con 0x7fb87c162260 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb882dbb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb87c0b7310 con 0x7fb87c0cf780 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb87c15de70 0x7fb87c159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb87c15de70 0x7fb87c159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53048/0 (socket says 172.21.15.28:53048) 2024-09-10T20:44:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 -- 172.21.15.28:0/4143024163 learned_addr learned my addr 172.21.15.28:0/4143024163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb880b57640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c0cf780 0x7fb87c0b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb881358640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb87c162260 0x7fb87c160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 -- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c0cf780 msgr2=0x7fb87c0b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c0cf780 0x7fb87c0b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 -- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb87c162260 msgr2=0x7fb87c160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb87c162260 0x7fb87c160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.430+0000 7fb87bfff640 1 -- 172.21.15.28:0/4143024163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c1f4b30 con 0x7fb87c15de70 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.431+0000 7fb87bfff640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb87c15de70 0x7fb87c159bc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb86800c6e0 tx=0x7fb86800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.431+0000 7fb8797fa640 1 -- 172.21.15.28:0/4143024163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868016020 con 0x7fb87c15de70 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.431+0000 7fb8797fa640 1 -- 172.21.15.28:0/4143024163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb86800a930 con 0x7fb87c15de70 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.431+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c1f4e10 con 0x7fb87c15de70 2024-09-10T20:44:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.431+0000 7fb8797fa640 1 -- 172.21.15.28:0/4143024163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86800ad60 con 0x7fb87c15de70 2024-09-10T20:44:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.431+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c1f53d0 con 0x7fb87c15de70 2024-09-10T20:44:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.432+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87c0b7870 con 0x7fb87c15de70 2024-09-10T20:44:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.433+0000 7fb8797fa640 1 -- 172.21.15.28:0/4143024163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fb86801b400 con 0x7fb87c15de70 2024-09-10T20:44:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.433+0000 7fb8797fa640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8600785a0 0x7fb86007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.433+0000 7fb8797fa640 1 -- 172.21.15.28:0/4143024163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7fb86809ae60 con 0x7fb87c15de70 2024-09-10T20:44:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.433+0000 7fb880b57640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8600785a0 0x7fb86007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.434+0000 7fb880b57640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8600785a0 0x7fb86007aa60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb8700095f0 tx=0x7fb870005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.437+0000 7fb8797fa640 1 -- 172.21.15.28:0/4143024163 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb868020020 con 0x7fb87c15de70 2024-09-10T20:44:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.657+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb87c0b9f30 con 0x7fb87c15de70 2024-09-10T20:44:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.658+0000 7fb8797fa640 1 -- 172.21.15.28:0/4143024163 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7fb868062e30 con 0x7fb87c15de70 2024-09-10T20:44:08.662 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:08.662 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"btime":"2024-09-10T20:44:01:436697+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3157854991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3157854991},{"type":"v1","addr":"172.21.15.159:6835","nonce":3157854991}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/3291396887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3291396887},{"type":"v1","addr":"172.21.15.159:6801","nonce":3291396887}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7461,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3500302836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3500302836},{"type":"v1","addr":"172.21.15.28:6833","nonce":3500302836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:43:51.763823+0000","modified":"2024-09-10T20:43:54.783424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7482":{"gid":7482,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1765205934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1765205934},{"type":"v1","addr":"172.21.15.28:6835","nonce":1765205934}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7482,"qdb_cluster":[7482]},"id":9}]} 2024-09-10T20:44:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 77 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8600785a0 msgr2=0x7fb86007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8600785a0 0x7fb86007aa60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb8700095f0 tx=0x7fb870005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb87c15de70 msgr2=0x7fb87c159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb87c15de70 0x7fb87c159bc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb86800c6e0 tx=0x7fb86800cba0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 shutdown_connections 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8600785a0 0x7fb86007aa60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb87c162260 0x7fb87c160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb87c15de70 0x7fb87c159bc0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 --2- 172.21.15.28:0/4143024163 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb87c0cf780 0x7fb87c0b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 >> 172.21.15.28:0/4143024163 conn(0x7fb87c0d3fb0 msgr2=0x7fb87c0ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 shutdown_connections 2024-09-10T20:44:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.660+0000 7fb882dbb640 1 -- 172.21.15.28:0/4143024163 wait complete. 2024-09-10T20:44:08.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 -- 172.21.15.28:0/2910074189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33001554e0 msgr2=0x7f330015e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 --2- 172.21.15.28:0/2910074189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33001554e0 0x7f330015e930 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f32f0009830 tx=0x7f32f002a510 comp rx=0 tx=0).stop 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 -- 172.21.15.28:0/2910074189 shutdown_connections 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 --2- 172.21.15.28:0/2910074189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f330015ee70 0x7f33000d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 --2- 172.21.15.28:0/2910074189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33001554e0 0x7f330015e930 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 -- 172.21.15.28:0/2910074189 >> 172.21.15.28:0/2910074189 conn(0x7f330006cf00 msgr2=0x7f330006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 -- 172.21.15.28:0/2910074189 shutdown_connections 2024-09-10T20:44:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.776+0000 7f330739b640 1 -- 172.21.15.28:0/2910074189 wait complete. 2024-09-10T20:44:08.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.777+0000 7f330739b640 1 Processor -- start 2024-09-10T20:44:08.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.777+0000 7f330739b640 1 -- start start 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f330739b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3300155100 0x7f3300144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f330739b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33001554e0 0x7f330009ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f330739b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330015ee70 0x7f330009cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f330739b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3300072de0 con 0x7f330015ee70 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f330739b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f33000d0af0 con 0x7f33001554e0 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f330739b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3300072f60 con 0x7f3300155100 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330015ee70 0x7f330009cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330015ee70 0x7f330009cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53070/0 (socket says 172.21.15.28:53070) 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 -- 172.21.15.28:0/2891544304 learned_addr learned my addr 172.21.15.28:0/2891544304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305137640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3300155100 0x7f3300144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3304936640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33001554e0 0x7f330009ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 -- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3300155100 msgr2=0x7f3300144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3300155100 0x7f3300144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 -- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33001554e0 msgr2=0x7f330009ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33001554e0 0x7f330009ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.778+0000 7f3305938640 1 -- 172.21.15.28:0/2891544304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33001559b0 con 0x7f330015ee70 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.779+0000 7f3305938640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330015ee70 0x7f330009cf80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f32f800f960 tx=0x7f32f800fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.779+0000 7f32f5ffb640 1 -- 172.21.15.28:0/2891544304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8009a80 con 0x7f330015ee70 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.779+0000 7f32f5ffb640 1 -- 172.21.15.28:0/2891544304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32f8009d10 con 0x7f330015ee70 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.779+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3300155c90 con 0x7f330015ee70 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.779+0000 7f32f5ffb640 1 -- 172.21.15.28:0/2891544304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8005280 con 0x7f330015ee70 2024-09-10T20:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.779+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3300156250 con 0x7f330015ee70 2024-09-10T20:44:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.780+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f330015e930 con 0x7f330015ee70 2024-09-10T20:44:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.781+0000 7f32f5ffb640 1 -- 172.21.15.28:0/2891544304 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f32f8005420 con 0x7f330015ee70 2024-09-10T20:44:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.781+0000 7f32f5ffb640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32e40785a0 0x7f32e407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.781+0000 7f32f5ffb640 1 -- 172.21.15.28:0/2891544304 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7f32f809c890 con 0x7f330015ee70 2024-09-10T20:44:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.781+0000 7f3305137640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32e40785a0 0x7f32e407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.782+0000 7f3305137640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32e40785a0 0x7f32e407aa60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0012e0 tx=0x7f32fc009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:08.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:08.785+0000 7f32f5ffb640 1 -- 172.21.15.28:0/2891544304 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f32f8014070 con 0x7f330015ee70 2024-09-10T20:44:09.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.005+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33000701b0 con 0x7f330015ee70 2024-09-10T20:44:09.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.006+0000 7f32f5ffb640 1 -- 172.21.15.28:0/2891544304 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7f32f806d090 con 0x7f330015ee70 2024-09-10T20:44:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:09.010 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"btime":"2024-09-10T20:44:01:436697+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3157854991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3157854991},{"type":"v1","addr":"172.21.15.159:6835","nonce":3157854991}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/3291396887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3291396887},{"type":"v1","addr":"172.21.15.159:6801","nonce":3291396887}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7461,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3500302836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3500302836},{"type":"v1","addr":"172.21.15.28:6833","nonce":3500302836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:43:51.763823+0000","modified":"2024-09-10T20:43:54.783424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7482":{"gid":7482,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1765205934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1765205934},{"type":"v1","addr":"172.21.15.28:6835","nonce":1765205934}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7482,"qdb_cluster":[7482]},"id":9}]} 2024-09-10T20:44:09.010 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 77 2024-09-10T20:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32e40785a0 msgr2=0x7f32e407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32e40785a0 0x7f32e407aa60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0012e0 tx=0x7f32fc009040 comp rx=0 tx=0).stop 2024-09-10T20:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330015ee70 msgr2=0x7f330009cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330015ee70 0x7f330009cf80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f32f800f960 tx=0x7f32f800fe20 comp rx=0 tx=0).stop 2024-09-10T20:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 shutdown_connections 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32e40785a0 0x7f32e407aa60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330015ee70 0x7f330009cf80 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33001554e0 0x7f330009ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 --2- 172.21.15.28:0/2891544304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3300155100 0x7f3300144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 >> 172.21.15.28:0/2891544304 conn(0x7f330006cf00 msgr2=0x7f33001549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 shutdown_connections 2024-09-10T20:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.008+0000 7f330739b640 1 -- 172.21.15.28:0/2891544304 wait complete. 2024-09-10T20:44:09.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 -- 172.21.15.28:0/1106529423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 msgr2=0x7f331c0d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 --2- 172.21.15.28:0/1106529423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 0x7f331c0d2b90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3314005fd0 tx=0x7f331402aa90 comp rx=0 tx=0).stop 2024-09-10T20:44:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 -- 172.21.15.28:0/1106529423 shutdown_connections 2024-09-10T20:44:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 --2- 172.21.15.28:0/1106529423 >> v2:172.21.15.159:3300/0 conn(0x7f331c162260 0x7f331c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 --2- 172.21.15.28:0/1106529423 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 0x7f331c0d2b90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 -- 172.21.15.28:0/1106529423 >> 172.21.15.28:0/1106529423 conn(0x7f331c0d3fb0 msgr2=0x7f331c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 -- 172.21.15.28:0/1106529423 shutdown_connections 2024-09-10T20:44:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.126+0000 7f332182a640 1 -- 172.21.15.28:0/1106529423 wait complete. 2024-09-10T20:44:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.127+0000 7f332182a640 1 Processor -- start 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.127+0000 7f332182a640 1 -- start start 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f332182a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f331c0cf780 0x7f331c0a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f332182a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 0x7f331c1f56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f332182a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c162260 0x7f331c0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f332182a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f331c0b7010 con 0x7f331c162260 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f332182a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f331c15e490 con 0x7f331c0cf780 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f332182a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f331c0b7190 con 0x7f331c15de70 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c162260 0x7f331c0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c162260 0x7f331c0ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53082/0 (socket says 172.21.15.28:53082) 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 -- 172.21.15.28:0/3496416896 learned_addr learned my addr 172.21.15.28:0/3496416896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331affd640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f331c0cf780 0x7f331c0a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331a7fc640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 0x7f331c1f56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 -- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 msgr2=0x7f331c1f56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 0x7f331c1f56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 -- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f331c0cf780 msgr2=0x7f331c0a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f331c0cf780 0x7f331c0a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 -- 172.21.15.28:0/3496416896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331c1f5c20 con 0x7f331c162260 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f331b7fe640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c162260 0x7f331c0ca900 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f330c00e700 tx=0x7f330c00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.128+0000 7f3313fff640 1 -- 172.21.15.28:0/3496416896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330c00cab0 con 0x7f331c162260 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.129+0000 7f3313fff640 1 -- 172.21.15.28:0/3496416896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f330c00cc50 con 0x7f331c162260 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.129+0000 7f3313fff640 1 -- 172.21.15.28:0/3496416896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330c005020 con 0x7f331c162260 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.129+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f331c0cae40 con 0x7f331c162260 2024-09-10T20:44:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.129+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f331c0a1960 con 0x7f331c162260 2024-09-10T20:44:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.130+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f331c0d2b90 con 0x7f331c162260 2024-09-10T20:44:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.130+0000 7f3313fff640 1 -- 172.21.15.28:0/3496416896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f330c0051c0 con 0x7f331c162260 2024-09-10T20:44:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.131+0000 7f3313fff640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32fc0785a0 0x7f32fc07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.131+0000 7f3313fff640 1 -- 172.21.15.28:0/3496416896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7f330c09b2d0 con 0x7f331c162260 2024-09-10T20:44:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.131+0000 7f331affd640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32fc0785a0 0x7f32fc07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.131+0000 7f331affd640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32fc0785a0 0x7f32fc07aa60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3314035000 tx=0x7f3314039040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.135+0000 7f3313fff640 1 -- 172.21.15.28:0/3496416896 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f331c0cae40 con 0x7f331c162260 2024-09-10T20:44:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.324+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f331c0b9f30 con 0x7f331c162260 2024-09-10T20:44:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.325+0000 7f3313fff640 1 -- 172.21.15.28:0/3496416896 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+16843 (secure 0 0 0) 0x7f331c1f5c20 con 0x7f331c162260 2024-09-10T20:44:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":116,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:07.461146+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:43:49.411437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-10T20:43:50.719111+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]},{"pool":19,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32fc0785a0 msgr2=0x7f32fc07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32fc0785a0 0x7f32fc07aa60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3314035000 tx=0x7f3314039040 comp rx=0 tx=0).stop 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c162260 msgr2=0x7f331c0ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c162260 0x7f331c0ca900 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f330c00e700 tx=0x7f330c00ebc0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 shutdown_connections 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32fc0785a0 0x7f32fc07aa60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c162260 0x7f331c0ca900 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f331c15de70 0x7f331c1f56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.327+0000 7f332182a640 1 --2- 172.21.15.28:0/3496416896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f331c0cf780 0x7f331c0a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.328+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 >> 172.21.15.28:0/3496416896 conn(0x7f331c0d3fb0 msgr2=0x7f331c0b8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.328+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 shutdown_connections 2024-09-10T20:44:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.328+0000 7f332182a640 1 -- 172.21.15.28:0/3496416896 wait complete. 2024-09-10T20:44:09.343 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:44:09.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.446+0000 7f372d82a640 1 -- 172.21.15.28:0/861277204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 msgr2=0x7f37100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.446+0000 7f372d82a640 1 --2- 172.21.15.28:0/861277204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 0x7f37100051c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f3720005fd0 tx=0x7f3710005830 comp rx=0 tx=0).stop 2024-09-10T20:44:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.446+0000 7f37277fe640 1 -- 172.21.15.28:0/861277204 reap_dead start 2024-09-10T20:44:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.446+0000 7f372d82a640 1 -- 172.21.15.28:0/861277204 shutdown_connections 2024-09-10T20:44:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.446+0000 7f372d82a640 1 -- 172.21.15.28:0/861277204 >> 172.21.15.28:0/861277204 conn(0x7f37280b1c40 msgr2=0x7f37280b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.446+0000 7f372d82a640 1 -- 172.21.15.28:0/861277204 shutdown_connections 2024-09-10T20:44:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.446+0000 7f372d82a640 1 -- 172.21.15.28:0/861277204 wait complete. 2024-09-10T20:44:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.447+0000 7f372d82a640 1 Processor -- start 2024-09-10T20:44:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f372d82a640 1 -- start start 2024-09-10T20:44:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f372d82a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 0x7f37281e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f372d82a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372814f740 0x7f372814ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f372d82a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3728159170 0x7f37281595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f372d82a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f372815c590 con 0x7f3710004d40 2024-09-10T20:44:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f372d82a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f372815c410 con 0x7f3728159170 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f372d82a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f372815c710 con 0x7f372814f740 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f3726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 0x7f37281e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f3726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 0x7f37281e6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53096/0 (socket says 172.21.15.28:53096) 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.448+0000 7f3726ffd640 1 -- 172.21.15.28:0/348785659 learned_addr learned my addr 172.21.15.28:0/348785659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f37267fc640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372814f740 0x7f372814ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f37277fe640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3728159170 0x7f37281595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f3726ffd640 1 -- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372814f740 msgr2=0x7f372814ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f3726ffd640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372814f740 0x7f372814ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f3726ffd640 1 -- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3728159170 msgr2=0x7f37281595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f3726ffd640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3728159170 0x7f37281595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f3726ffd640 1 -- 172.21.15.28:0/348785659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37281576e0 con 0x7f3710004d40 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f3726ffd640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 0x7f37281e6f00 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f37180065e0 tx=0x7f37180039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f370ffff640 1 -- 172.21.15.28:0/348785659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3718015070 con 0x7f3710004d40 2024-09-10T20:44:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f370ffff640 1 -- 172.21.15.28:0/348785659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f371800f980 con 0x7f3710004d40 2024-09-10T20:44:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f370ffff640 1 -- 172.21.15.28:0/348785659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371800fdd0 con 0x7f3710004d40 2024-09-10T20:44:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37281579c0 con 0x7f3710004d40 2024-09-10T20:44:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.449+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3728157ca0 con 0x7f3710004d40 2024-09-10T20:44:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.450+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f372806eb10 con 0x7f3710004d40 2024-09-10T20:44:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.451+0000 7f370ffff640 1 -- 172.21.15.28:0/348785659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f3718011430 con 0x7f3710004d40 2024-09-10T20:44:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.452+0000 7f370ffff640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37140785d0 0x7f371407aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.452+0000 7f37267fc640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37140785d0 0x7f371407aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.452+0000 7f370ffff640 1 -- 172.21.15.28:0/348785659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7f371809b640 con 0x7f3710004d40 2024-09-10T20:44:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.452+0000 7f37267fc640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37140785d0 0x7f371407aa90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3728158f60 tx=0x7f371c005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.456+0000 7f370ffff640 1 -- 172.21.15.28:0/348785659 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f37180637a0 con 0x7f3710004d40 2024-09-10T20:44:09.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.677+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3728070960 con 0x7f3710004d40 2024-09-10T20:44:09.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.678+0000 7f370ffff640 1 -- 172.21.15.28:0/348785659 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7f37180630a0 con 0x7f3710004d40 2024-09-10T20:44:09.682 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:09.682 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"btime":"2024-09-10T20:44:01:436697+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3157854991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3157854991},{"type":"v1","addr":"172.21.15.159:6835","nonce":3157854991}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/3291396887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3291396887},{"type":"v1","addr":"172.21.15.159:6801","nonce":3291396887}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7461,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3500302836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3500302836},{"type":"v1","addr":"172.21.15.28:6833","nonce":3500302836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:43:51.763823+0000","modified":"2024-09-10T20:43:54.783424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7482":{"gid":7482,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1765205934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1765205934},{"type":"v1","addr":"172.21.15.28:6835","nonce":1765205934}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7482,"qdb_cluster":[7482]},"id":9}]} 2024-09-10T20:44:09.682 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 77 2024-09-10T20:44:09.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37140785d0 msgr2=0x7f371407aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37140785d0 0x7f371407aa90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3728158f60 tx=0x7f371c005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:09.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 msgr2=0x7f37281e6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 0x7f37281e6f00 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f37180065e0 tx=0x7f37180039c0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 shutdown_connections 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37140785d0 0x7f371407aa90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3728159170 0x7f37281595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f372814f740 0x7f372814ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 --2- 172.21.15.28:0/348785659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3710004d40 0x7f37281e6f00 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 >> 172.21.15.28:0/348785659 conn(0x7f37280b1c40 msgr2=0x7f37280b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.680+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 shutdown_connections 2024-09-10T20:44:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.681+0000 7f372d82a640 1 -- 172.21.15.28:0/348785659 wait complete. 2024-09-10T20:44:09.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.796+0000 7fc865952640 1 -- 172.21.15.28:0/2527715997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86015a190 msgr2=0x7fc8600b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.796+0000 7fc865952640 1 --2- 172.21.15.28:0/2527715997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86015a190 0x7fc8600b2ba0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc85000a940 tx=0x7fc85002b850 comp rx=0 tx=0).stop 2024-09-10T20:44:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.796+0000 7fc865952640 1 -- 172.21.15.28:0/2527715997 shutdown_connections 2024-09-10T20:44:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.796+0000 7fc865952640 1 --2- 172.21.15.28:0/2527715997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc86015a190 0x7fc8600b2ba0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.796+0000 7fc865952640 1 -- 172.21.15.28:0/2527715997 >> 172.21.15.28:0/2527715997 conn(0x7fc86006cf00 msgr2=0x7fc86006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.796+0000 7fc865952640 1 -- 172.21.15.28:0/2527715997 shutdown_connections 2024-09-10T20:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.796+0000 7fc865952640 1 -- 172.21.15.28:0/2527715997 wait complete. 2024-09-10T20:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.797+0000 7fc865952640 1 Processor -- start 2024-09-10T20:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.797+0000 7fc865952640 1 -- start start 2024-09-10T20:44:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc865952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8600b4d10 0x7fc860150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc865952640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc86015a190 0x7fc860072e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc865952640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86015df70 0x7fc8600adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc865952640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600d0480 con 0x7fc8600b4d10 2024-09-10T20:44:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc865952640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600d0300 con 0x7fc86015a190 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc865952640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc8600d0600 con 0x7fc86015df70 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc85effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8600b4d10 0x7fc860150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc85effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8600b4d10 0x7fc860150770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53118/0 (socket says 172.21.15.28:53118) 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc85effd640 1 -- 172.21.15.28:0/2236577147 learned_addr learned my addr 172.21.15.28:0/2236577147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc857fff640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc86015a190 0x7fc860072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.798+0000 7fc85f7fe640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86015df70 0x7fc8600adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85effd640 1 -- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86015df70 msgr2=0x7fc8600adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85effd640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86015df70 0x7fc8600adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85effd640 1 -- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc86015a190 msgr2=0x7fc860072e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85effd640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc86015a190 0x7fc860072e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85effd640 1 -- 172.21.15.28:0/2236577147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc860073820 con 0x7fc8600b4d10 2024-09-10T20:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc857fff640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc86015a190 0x7fc860072e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85f7fe640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86015df70 0x7fc8600adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85effd640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8600b4d10 0x7fc860150770 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc850000c00 tx=0x7fc850003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85cff9640 1 -- 172.21.15.28:0/2236577147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850003f40 con 0x7fc8600b4d10 2024-09-10T20:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85cff9640 1 -- 172.21.15.28:0/2236577147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc850031a50 con 0x7fc8600b4d10 2024-09-10T20:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc85cff9640 1 -- 172.21.15.28:0/2236577147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850031ea0 con 0x7fc8600b4d10 2024-09-10T20:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc860073b00 con 0x7fc8600b4d10 2024-09-10T20:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.799+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8600740c0 con 0x7fc8600b4d10 2024-09-10T20:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.800+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8600b2ba0 con 0x7fc8600b4d10 2024-09-10T20:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.801+0000 7fc85cff9640 1 -- 172.21.15.28:0/2236577147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fc85004d030 con 0x7fc8600b4d10 2024-09-10T20:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.801+0000 7fc85cff9640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8400785a0 0x7fc84007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.801+0000 7fc85cff9640 1 -- 172.21.15.28:0/2236577147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7fc8500bbef0 con 0x7fc8600b4d10 2024-09-10T20:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.801+0000 7fc857fff640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8400785a0 0x7fc84007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.802+0000 7fc857fff640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8400785a0 0x7fc84007aa60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc86009d780 tx=0x7fc848008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:09.805+0000 7fc85cff9640 1 -- 172.21.15.28:0/2236577147 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc850041030 con 0x7fc8600b4d10 2024-09-10T20:44:10.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.026+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8601598c0 con 0x7fc8600b4d10 2024-09-10T20:44:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.027+0000 7fc85cff9640 1 -- 172.21.15.28:0/2236577147 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7fc850083fd0 con 0x7fc8600b4d10 2024-09-10T20:44:10.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:10.031 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"btime":"2024-09-10T20:44:01:436697+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3157854991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3157854991},{"type":"v1","addr":"172.21.15.159:6835","nonce":3157854991}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/3291396887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3291396887},{"type":"v1","addr":"172.21.15.159:6801","nonce":3291396887}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7461,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3500302836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3500302836},{"type":"v1","addr":"172.21.15.28:6833","nonce":3500302836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:43:51.763823+0000","modified":"2024-09-10T20:43:54.783424+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7482":{"gid":7482,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1765205934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1765205934},{"type":"v1","addr":"172.21.15.28:6835","nonce":1765205934}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7482,"qdb_cluster":[7482]},"id":9}]} 2024-09-10T20:44:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 77 2024-09-10T20:44:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8400785a0 msgr2=0x7fc84007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8400785a0 0x7fc84007aa60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc86009d780 tx=0x7fc848008040 comp rx=0 tx=0).stop 2024-09-10T20:44:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8600b4d10 msgr2=0x7fc860150770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8600b4d10 0x7fc860150770 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc850000c00 tx=0x7fc850003d50 comp rx=0 tx=0).stop 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 shutdown_connections 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8400785a0 0x7fc84007aa60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc86015df70 0x7fc8600adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc86015a190 0x7fc860072e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 --2- 172.21.15.28:0/2236577147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8600b4d10 0x7fc860150770 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 >> 172.21.15.28:0/2236577147 conn(0x7fc86006cf00 msgr2=0x7fc86015b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 shutdown_connections 2024-09-10T20:44:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.029+0000 7fc865952640 1 -- 172.21.15.28:0/2236577147 wait complete. 2024-09-10T20:44:10.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 -- 172.21.15.28:0/1674814801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 msgr2=0x7f27c00d48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 --2- 172.21.15.28:0/1674814801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c00d48f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009830 tx=0x7f27b002a560 comp rx=0 tx=0).stop 2024-09-10T20:44:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 -- 172.21.15.28:0/1674814801 shutdown_connections 2024-09-10T20:44:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 --2- 172.21.15.28:0/1674814801 >> v2:172.21.15.159:3300/0 conn(0x7f27c00cf3f0 0x7f27c00d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 --2- 172.21.15.28:0/1674814801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c00d48f0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 -- 172.21.15.28:0/1674814801 >> 172.21.15.28:0/1674814801 conn(0x7f27c00d3dd0 msgr2=0x7f27c00d41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 -- 172.21.15.28:0/1674814801 shutdown_connections 2024-09-10T20:44:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.146+0000 7f27bffff640 1 -- 172.21.15.28:0/1674814801 wait complete. 2024-09-10T20:44:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.147+0000 7f27bffff640 1 Processor -- start 2024-09-10T20:44:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.147+0000 7f27bffff640 1 -- start start 2024-09-10T20:44:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bffff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27c00cf3f0 0x7f27c0155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bffff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27c00d2390 0x7f27c009fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c01f8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bffff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27c00d0ef0 con 0x7f27c015dec0 2024-09-10T20:44:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bffff640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f27c00b88e0 con 0x7f27c00cf3f0 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bffff640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f27c00d1070 con 0x7f27c00d2390 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c01f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c01f8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53142/0 (socket says 172.21.15.28:53142) 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bf7fe640 1 -- 172.21.15.28:0/1606620452 learned_addr learned my addr 172.21.15.28:0/1606620452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27beffd640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27c00cf3f0 0x7f27c0155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27be7fc640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27c00d2390 0x7f27c009fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bf7fe640 1 -- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27c00d2390 msgr2=0x7f27c009fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bf7fe640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27c00d2390 0x7f27c009fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bf7fe640 1 -- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27c00cf3f0 msgr2=0x7f27c0155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.148+0000 7f27bf7fe640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27c00cf3f0 0x7f27c0155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.149+0000 7f27bf7fe640 1 -- 172.21.15.28:0/1606620452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c00c8a90 con 0x7f27c015dec0 2024-09-10T20:44:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.149+0000 7f27bf7fe640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c01f8400 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f27b800c770 tx=0x7f27b800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.149+0000 7f27a77fe640 1 -- 172.21.15.28:0/1606620452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b8003da0 con 0x7f27c015dec0 2024-09-10T20:44:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.149+0000 7f27a77fe640 1 -- 172.21.15.28:0/1606620452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27b800ce20 con 0x7f27c015dec0 2024-09-10T20:44:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.149+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c00c8d70 con 0x7f27c015dec0 2024-09-10T20:44:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.149+0000 7f27a77fe640 1 -- 172.21.15.28:0/1606620452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b8005290 con 0x7f27c015dec0 2024-09-10T20:44:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.149+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c009dc50 con 0x7f27c015dec0 2024-09-10T20:44:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.150+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c00d1520 con 0x7f27c015dec0 2024-09-10T20:44:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.151+0000 7f27a77fe640 1 -- 172.21.15.28:0/1606620452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f27b8005430 con 0x7f27c015dec0 2024-09-10T20:44:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.151+0000 7f27a77fe640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27ac0785a0 0x7f27ac07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.151+0000 7f27beffd640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27ac0785a0 0x7f27ac07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.151+0000 7f27a77fe640 1 -- 172.21.15.28:0/1606620452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7f27b809a6a0 con 0x7f27c015dec0 2024-09-10T20:44:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.152+0000 7f27beffd640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27ac0785a0 0x7f27ac07aa60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f27b0034000 tx=0x7f27b0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.155+0000 7f27a77fe640 1 -- 172.21.15.28:0/1606620452 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f27b8062130 con 0x7f27c015dec0 2024-09-10T20:44:10.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.341+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27c00d1c30 con 0x7f27c015dec0 2024-09-10T20:44:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.342+0000 7f27a77fe640 1 -- 172.21.15.28:0/1606620452 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+16843 (secure 0 0 0) 0x7f27b8066b60 con 0x7f27c015dec0 2024-09-10T20:44:10.346 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:10.347 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":116,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:07.461146+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:43:49.411437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-10T20:43:50.719111+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]},{"pool":19,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27ac0785a0 msgr2=0x7f27ac07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27ac0785a0 0x7f27ac07aa60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f27b0034000 tx=0x7f27b0004730 comp rx=0 tx=0).stop 2024-09-10T20:44:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 msgr2=0x7f27c01f8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c01f8400 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f27b800c770 tx=0x7f27b800cc30 comp rx=0 tx=0).stop 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 shutdown_connections 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27ac0785a0 0x7f27ac07aa60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c015dec0 0x7f27c01f8400 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27c00d2390 0x7f27c009fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 --2- 172.21.15.28:0/1606620452 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27c00cf3f0 0x7f27c0155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 >> 172.21.15.28:0/1606620452 conn(0x7f27c00d3dd0 msgr2=0x7f27c006e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 shutdown_connections 2024-09-10T20:44:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.344+0000 7f27bffff640 1 -- 172.21.15.28:0/1606620452 wait complete. 2024-09-10T20:44:10.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:44:10.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 -- 172.21.15.28:0/3962078978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d40cf780 msgr2=0x7f83d40cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 --2- 172.21.15.28:0/3962078978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d40cf780 0x7f83d40cfbf0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f83d408c020 tx=0x7f83bc02a530 comp rx=0 tx=0).stop 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 -- 172.21.15.28:0/3962078978 shutdown_connections 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 --2- 172.21.15.28:0/3962078978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d40cf780 0x7f83d40cfbf0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 --2- 172.21.15.28:0/3962078978 >> v2:172.21.15.159:3300/0 conn(0x7f83d415de70 0x7f83d40d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 -- 172.21.15.28:0/3962078978 >> 172.21.15.28:0/3962078978 conn(0x7f83d40d3fb0 msgr2=0x7f83d40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 -- 172.21.15.28:0/3962078978 shutdown_connections 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.461+0000 7f83daa16640 1 -- 172.21.15.28:0/3962078978 wait complete. 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 Processor -- start 2024-09-10T20:44:10.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 -- start start 2024-09-10T20:44:10.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83d40cf780 0x7f83d409dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d415de70 0x7f83d409e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83d4161b50 0x7f83d409fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83d40b7690 con 0x7f83d415de70 2024-09-10T20:44:10.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f83d40b7510 con 0x7f83d4161b50 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83daa16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f83d40b7810 con 0x7f83d40cf780 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d415de70 0x7f83d409e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d415de70 0x7f83d409e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33608/0 (socket says 172.21.15.28:33608) 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.463+0000 7f83cbfff640 1 -- 172.21.15.28:0/2696708806 learned_addr learned my addr 172.21.15.28:0/2696708806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83d3fff640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83d40cf780 0x7f83d409dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83d8fb3640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83d4161b50 0x7f83d409fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83cbfff640 1 -- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83d40cf780 msgr2=0x7f83d409dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83cbfff640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83d40cf780 0x7f83d409dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83cbfff640 1 -- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83d4161b50 msgr2=0x7f83d409fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83cbfff640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83d4161b50 0x7f83d409fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83cbfff640 1 -- 172.21.15.28:0/2696708806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d41f4db0 con 0x7f83d415de70 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83cbfff640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d415de70 0x7f83d409e8f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f83bc02ab20 tx=0x7f83bc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83d17fa640 1 -- 172.21.15.28:0/2696708806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc040070 con 0x7f83d415de70 2024-09-10T20:44:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83d17fa640 1 -- 172.21.15.28:0/2696708806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83bc005310 con 0x7f83d415de70 2024-09-10T20:44:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d41f5090 con 0x7f83d415de70 2024-09-10T20:44:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83d17fa640 1 -- 172.21.15.28:0/2696708806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0047a0 con 0x7f83d415de70 2024-09-10T20:44:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.464+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d41f5650 con 0x7f83d415de70 2024-09-10T20:44:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.465+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d40cc180 con 0x7f83d415de70 2024-09-10T20:44:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.466+0000 7f83d17fa640 1 -- 172.21.15.28:0/2696708806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f83bc04c030 con 0x7f83d415de70 2024-09-10T20:44:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.466+0000 7f83d17fa640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f83b40785a0 0x7f83b407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.467+0000 7f83d17fa640 1 -- 172.21.15.28:0/2696708806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5793+0+0 (secure 0 0 0) 0x7f83bc0bba80 con 0x7f83d415de70 2024-09-10T20:44:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.467+0000 7f83d3fff640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f83b40785a0 0x7f83b407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.467+0000 7f83d3fff640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f83b40785a0 0x7f83b407aa60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f83c400bbe0 tx=0x7f83c400b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:10.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.471+0000 7f83d17fa640 1 -- 172.21.15.28:0/2696708806 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f83bc083680 con 0x7f83d415de70 2024-09-10T20:44:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.694+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f83d40b9f30 con 0x7f83d415de70 2024-09-10T20:44:10.698 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:44:10.694+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:44:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.695+0000 7f83d17fa640 1 -- 172.21.15.28:0/2696708806 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) ==== 161+0+0 (secure 0 0 0) 0x7f83bc0880b0 con 0x7f83d415de70 2024-09-10T20:44:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:44:10.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f83b40785a0 msgr2=0x7f83b407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f83b40785a0 0x7f83b407aa60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f83c400bbe0 tx=0x7f83c400b390 comp rx=0 tx=0).stop 2024-09-10T20:44:10.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d415de70 msgr2=0x7f83d409e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d415de70 0x7f83d409e8f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f83bc02ab20 tx=0x7f83bc032c10 comp rx=0 tx=0).stop 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 shutdown_connections 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f83b40785a0 0x7f83b407aa60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83d4161b50 0x7f83d409fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d415de70 0x7f83d409e8f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 --2- 172.21.15.28:0/2696708806 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83d40cf780 0x7f83d409dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.698+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 >> 172.21.15.28:0/2696708806 conn(0x7f83d40d3fb0 msgr2=0x7f83d40b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.699+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 shutdown_connections 2024-09-10T20:44:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.699+0000 7f83daa16640 1 -- 172.21.15.28:0/2696708806 wait complete. 2024-09-10T20:44:10.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:44:10.740 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:44:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.820+0000 7fb5a62b0640 1 -- 172.21.15.28:0/2207385113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb58c004800 msgr2=0x7fb58c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.820+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/2207385113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb58c004800 0x7fb58c004c80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb58800bb20 tx=0x7fb58c005830 comp rx=0 tx=0).stop 2024-09-10T20:44:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.820+0000 7fb5a62b0640 1 -- 172.21.15.28:0/2207385113 shutdown_connections 2024-09-10T20:44:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.820+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/2207385113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb58c004800 0x7fb58c004c80 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.820+0000 7fb5a62b0640 1 -- 172.21.15.28:0/2207385113 >> 172.21.15.28:0/2207385113 conn(0x7fb5a00b21d0 msgr2=0x7fb5a00b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.820+0000 7fb5a62b0640 1 -- 172.21.15.28:0/2207385113 shutdown_connections 2024-09-10T20:44:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.820+0000 7fb5a62b0640 1 -- 172.21.15.28:0/2207385113 wait complete. 2024-09-10T20:44:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.821+0000 7fb5a62b0640 1 Processor -- start 2024-09-10T20:44:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.821+0000 7fb5a62b0640 1 -- start start 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a62b0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb58c004800 0x7fb5a0154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a62b0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb5a0159c60 0x7fb5a014c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a62b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a00cf2d0 0x7fb5a00cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a62b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a0073cb0 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a62b0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a0073b30 con 0x7fb58c004800 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a62b0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a0073e30 con 0x7fb5a0159c60 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a00cf2d0 0x7fb5a00cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a00cf2d0 0x7fb5a00cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33644/0 (socket says 172.21.15.28:33644) 2024-09-10T20:44:10.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 -- 172.21.15.28:0/3363497171 learned_addr learned my addr 172.21.15.28:0/3363497171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb59ffff640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb58c004800 0x7fb5a0154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb597fff640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb5a0159c60 0x7fb5a014c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 -- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb5a0159c60 msgr2=0x7fb5a014c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb5a0159c60 0x7fb5a014c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 -- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb58c004800 msgr2=0x7fb5a0154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb58c004800 0x7fb5a0154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.822+0000 7fb5a484d640 1 -- 172.21.15.28:0/3363497171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a0157ff0 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.823+0000 7fb5a484d640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a00cf2d0 0x7fb5a00cf750 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00b57e0 tx=0x7fb598006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.823+0000 7fb59d7fa640 1 -- 172.21.15.28:0/3363497171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598015070 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.823+0000 7fb59d7fa640 1 -- 172.21.15.28:0/3363497171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb598011900 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.823+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a01f7ec0 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.823+0000 7fb59d7fa640 1 -- 172.21.15.28:0/3363497171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598011d50 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.823+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a01f8480 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.824+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a006a3c0 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.825+0000 7fb59d7fa640 1 -- 172.21.15.28:0/3363497171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fb5980103c0 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.825+0000 7fb59d7fa640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb5800785a0 0x7fb58007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:10.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.825+0000 7fb59d7fa640 1 -- 172.21.15.28:0/3363497171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5851+0+0 (secure 0 0 0) 0x7fb59809b8e0 con 0x7fb5a00cf2d0 2024-09-10T20:44:10.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.825+0000 7fb59ffff640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb5800785a0 0x7fb58007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:10.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.826+0000 7fb59ffff640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb5800785a0 0x7fb58007aa60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00668e0 tx=0x7fb5900067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:10.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:10.829+0000 7fb59d7fa640 1 -- 172.21.15.28:0/3363497171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb598063340 con 0x7fb5a00cf2d0 2024-09-10T20:44:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.054+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb5a006f050 con 0x7fb5a00cf2d0 2024-09-10T20:44:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.697+0000 7fb59d7fa640 1 -- 172.21.15.28:0/3363497171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) ==== 104+0+0 (secure 0 0 0) 0x7fb598067d70 con 0x7fb5a00cf2d0 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb5800785a0 msgr2=0x7fb58007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb5800785a0 0x7fb58007aa60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00668e0 tx=0x7fb5900067a0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a00cf2d0 msgr2=0x7fb5a00cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a00cf2d0 0x7fb5a00cf750 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00b57e0 tx=0x7fb598006430 comp rx=0 tx=0).stop 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 shutdown_connections 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb5800785a0 0x7fb58007aa60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a00cf2d0 0x7fb5a00cf750 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb5a0159c60 0x7fb5a014c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 --2- 172.21.15.28:0/3363497171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb58c004800 0x7fb5a0154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 >> 172.21.15.28:0/3363497171 conn(0x7fb5a00b21d0 msgr2=0x7fb5a00b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 shutdown_connections 2024-09-10T20:44:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.700+0000 7fb5a62b0640 1 -- 172.21.15.28:0/3363497171 wait complete. 2024-09-10T20:44:11.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.872+0000 7f47869c6640 1 -- 172.21.15.28:0/1696381009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 msgr2=0x7f47800d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.872+0000 7f47869c6640 1 --2- 172.21.15.28:0/1696381009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f47800d1c60 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f478008d390 tx=0x7f477802aac0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.873+0000 7f47869c6640 1 -- 172.21.15.28:0/1696381009 shutdown_connections 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.873+0000 7f47869c6640 1 --2- 172.21.15.28:0/1696381009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f47800d1c60 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.873+0000 7f47869c6640 1 --2- 172.21.15.28:0/1696381009 >> v2:172.21.15.159:3300/0 conn(0x7f4780161850 0x7f47800d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.873+0000 7f47869c6640 1 -- 172.21.15.28:0/1696381009 >> 172.21.15.28:0/1696381009 conn(0x7f47800d3f90 msgr2=0x7f47800d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.873+0000 7f47869c6640 1 -- 172.21.15.28:0/1696381009 shutdown_connections 2024-09-10T20:44:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.873+0000 7f47869c6640 1 -- 172.21.15.28:0/1696381009 wait complete. 2024-09-10T20:44:11.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.874+0000 7f47869c6640 1 Processor -- start 2024-09-10T20:44:11.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.874+0000 7f47869c6640 1 -- start start 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f47869c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f4780155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f47869c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4780161850 0x7f478015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f47869c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4780162000 0x7f478015a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f47869c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47800d21a0 con 0x7f47800b4c20 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f47869c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4780162740 con 0x7f4780162000 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f47869c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f47800d2320 con 0x7f4780161850 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f4780155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f4780155660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33662/0 (socket says 172.21.15.28:33662) 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 -- 172.21.15.28:0/2250802034 learned_addr learned my addr 172.21.15.28:0/2250802034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f4784f63640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4780162000 0x7f478015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477f7fe640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4780161850 0x7f478015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 -- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4780161850 msgr2=0x7f478015a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4780161850 0x7f478015a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 -- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4780162000 msgr2=0x7f478015a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4780162000 0x7f478015a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 -- 172.21.15.28:0/2250802034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47801f4c90 con 0x7f47800b4c20 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.875+0000 7f477ffff640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f4780155660 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f477000b3f0 tx=0x7f477000b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.876+0000 7f477cff9640 1 -- 172.21.15.28:0/2250802034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770015020 con 0x7f47800b4c20 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.876+0000 7f477cff9640 1 -- 172.21.15.28:0/2250802034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4770011920 con 0x7f47800b4c20 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.876+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47801f4f10 con 0x7f47800b4c20 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.876+0000 7f477cff9640 1 -- 172.21.15.28:0/2250802034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770011d70 con 0x7f47800b4c20 2024-09-10T20:44:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.876+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47801f54d0 con 0x7f47800b4c20 2024-09-10T20:44:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.877+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47800d1c60 con 0x7f47800b4c20 2024-09-10T20:44:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.877+0000 7f477cff9640 1 -- 172.21.15.28:0/2250802034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f477001a400 con 0x7f47800b4c20 2024-09-10T20:44:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.878+0000 7f477cff9640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f476c0785a0 0x7f476c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.878+0000 7f477f7fe640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f476c0785a0 0x7f476c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.878+0000 7f477cff9640 1 -- 172.21.15.28:0/2250802034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5851+0+0 (secure 0 0 0) 0x7f477009a700 con 0x7f47800b4c20 2024-09-10T20:44:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.879+0000 7f477f7fe640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f476c0785a0 0x7f476c07aa60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f478015a050 tx=0x7f477400a2d0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:11.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:11.883+0000 7f477cff9640 1 -- 172.21.15.28:0/2250802034 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f477001f030 con 0x7f47800b4c20 2024-09-10T20:44:12.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.068+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f47800d2bf0 con 0x7f47800b4c20 2024-09-10T20:44:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.457+0000 7f477cff9640 1 -- 172.21.15.28:0/2250802034 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) ==== 181+0+0 (secure 0 0 0) 0x7f47801f4c90 con 0x7f47800b4c20 2024-09-10T20:44:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:44:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f476c0785a0 msgr2=0x7f476c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f476c0785a0 0x7f476c07aa60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f478015a050 tx=0x7f477400a2d0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 msgr2=0x7f4780155660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f4780155660 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f477000b3f0 tx=0x7f477000b7f0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 shutdown_connections 2024-09-10T20:44:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f476c0785a0 0x7f476c07aa60 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f478015a050 tx=0x7f477400a2d0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4780162000 0x7f478015a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4780161850 0x7f478015a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 --2- 172.21.15.28:0/2250802034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47800b4c20 0x7f4780155660 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 >> 172.21.15.28:0/2250802034 conn(0x7f47800d3f90 msgr2=0x7f478015f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 shutdown_connections 2024-09-10T20:44:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.461+0000 7f47869c6640 1 -- 172.21.15.28:0/2250802034 wait complete. 2024-09-10T20:44:12.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:44:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.587+0000 7f94a467f640 1 -- 172.21.15.28:0/1387672749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 msgr2=0x7f949c15f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.587+0000 7f94a467f640 1 --2- 172.21.15.28:0/1387672749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c15f230 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f949c0ca630 tx=0x7f949802a4a0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.587+0000 7f94a467f640 1 -- 172.21.15.28:0/1387672749 shutdown_connections 2024-09-10T20:44:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.587+0000 7f94a467f640 1 --2- 172.21.15.28:0/1387672749 >> v2:172.21.15.159:3300/0 conn(0x7f949c0cf810 0x7f949c06ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.587+0000 7f94a467f640 1 --2- 172.21.15.28:0/1387672749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c15f230 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.587+0000 7f94a467f640 1 -- 172.21.15.28:0/1387672749 >> 172.21.15.28:0/1387672749 conn(0x7f949c0ccb60 msgr2=0x7f949c0ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.588+0000 7f94a467f640 1 -- 172.21.15.28:0/1387672749 shutdown_connections 2024-09-10T20:44:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.588+0000 7f94a467f640 1 -- 172.21.15.28:0/1387672749 wait complete. 2024-09-10T20:44:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.589+0000 7f94a467f640 1 Processor -- start 2024-09-10T20:44:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.589+0000 7f94a467f640 1 -- start start 2024-09-10T20:44:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.589+0000 7f94a467f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a467f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f949c0b4790 0x7f949c0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a467f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f949c0cf810 0x7f949c09d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a467f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f949c06f340 con 0x7f949c0b43b0 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a467f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f949c06f1c0 con 0x7f949c0cf810 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a467f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f949c09db50 con 0x7f949c0b4790 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c0c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33688/0 (socket says 172.21.15.28:33688) 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 -- 172.21.15.28:0/591272347 learned_addr learned my addr 172.21.15.28:0/591272347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a1c1a640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f949c0b4790 0x7f949c0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a2c1c640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f949c0cf810 0x7f949c09d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 -- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f949c0b4790 msgr2=0x7f949c0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f949c0b4790 0x7f949c0c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 -- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f949c0cf810 msgr2=0x7f949c09d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f949c0cf810 0x7f949c09d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 -- 172.21.15.28:0/591272347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949c0c8880 con 0x7f949c0b43b0 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a241b640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c0c7ce0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f94980098f0 tx=0x7f9498032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f9492ffd640 1 -- 172.21.15.28:0/591272347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498040070 con 0x7f949c0b43b0 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f9492ffd640 1 -- 172.21.15.28:0/591272347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9498005310 con 0x7f949c0b43b0 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949c0c82c0 con 0x7f949c0b43b0 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f9492ffd640 1 -- 172.21.15.28:0/591272347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94980047a0 con 0x7f949c0b43b0 2024-09-10T20:44:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.590+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949c09abd0 con 0x7f949c0b43b0 2024-09-10T20:44:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.592+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f949c15f230 con 0x7f949c0b43b0 2024-09-10T20:44:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.592+0000 7f9492ffd640 1 -- 172.21.15.28:0/591272347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f949804c030 con 0x7f949c0b43b0 2024-09-10T20:44:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.592+0000 7f9492ffd640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94800785a0 0x7f948007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.593+0000 7f9492ffd640 1 -- 172.21.15.28:0/591272347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5413+0+0 (secure 0 0 0) 0x7f94980bb910 con 0x7f949c0b43b0 2024-09-10T20:44:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.593+0000 7f94a1c1a640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94800785a0 0x7f948007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.593+0000 7f94a1c1a640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94800785a0 0x7f948007aa60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f949c15ad80 tx=0x7f948c007d40 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.597+0000 7f9492ffd640 1 -- 172.21.15.28:0/591272347 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9498083600 con 0x7f949c0b43b0 2024-09-10T20:44:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:12.790+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f949c0b4db0 con 0x7f949c0b43b0 2024-09-10T20:44:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.701+0000 7f9492ffd640 1 -- 172.21.15.28:0/591272347 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) ==== 169+0+0 (secure 0 0 0) 0x7f9498088030 con 0x7f949c0b43b0 2024-09-10T20:44:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:44:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.705+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94800785a0 msgr2=0x7f948007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.705+0000 7f94a467f640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94800785a0 0x7f948007aa60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f949c15ad80 tx=0x7f948c007d40 comp rx=0 tx=0).stop 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.705+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 msgr2=0x7f949c0c7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.705+0000 7f94a467f640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c0c7ce0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f94980098f0 tx=0x7f9498032c10 comp rx=0 tx=0).stop 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 shutdown_connections 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94800785a0 0x7f948007aa60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f949c0cf810 0x7f949c09d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f949c0b4790 0x7f949c0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 --2- 172.21.15.28:0/591272347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c0b43b0 0x7f949c0c7ce0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 >> 172.21.15.28:0/591272347 conn(0x7f949c0ccb60 msgr2=0x7f949c075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 shutdown_connections 2024-09-10T20:44:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.706+0000 7f94a467f640 1 -- 172.21.15.28:0/591272347 wait complete. 2024-09-10T20:44:13.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:44:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 -- 172.21.15.28:0/47897045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 msgr2=0x7fc6900d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 --2- 172.21.15.28:0/47897045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900d0330 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fc690073cb0 tx=0x7fc68402a560 comp rx=0 tx=0).stop 2024-09-10T20:44:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 -- 172.21.15.28:0/47897045 shutdown_connections 2024-09-10T20:44:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 --2- 172.21.15.28:0/47897045 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc6900ce5f0 0x7fc6900cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 --2- 172.21.15.28:0/47897045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900d0330 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 --2- 172.21.15.28:0/47897045 >> v2:172.21.15.159:3300/0 conn(0x7fc69015bf30 0x7fc6900d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 -- 172.21.15.28:0/47897045 >> 172.21.15.28:0/47897045 conn(0x7fc6900d1e40 msgr2=0x7fc6900d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 -- 172.21.15.28:0/47897045 shutdown_connections 2024-09-10T20:44:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.833+0000 7fc696b0e640 1 -- 172.21.15.28:0/47897045 wait complete. 2024-09-10T20:44:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.834+0000 7fc696b0e640 1 Processor -- start 2024-09-10T20:44:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.834+0000 7fc696b0e640 1 -- start start 2024-09-10T20:44:13.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc696b0e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6900ce5f0 0x7fc6900c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:13.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc696b0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc696b0e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc69015bf30 0x7fc69006e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc696b0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6900d0cc0 con 0x7fc6900cfec0 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc696b0e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc6900d0b40 con 0x7fc6900ce5f0 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc696b0e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc6900cefa0 con 0x7fc69015bf30 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc68ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc68ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33698/0 (socket says 172.21.15.28:33698) 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc68ffff640 1 -- 172.21.15.28:0/599839179 learned_addr learned my addr 172.21.15.28:0/599839179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc6950ab640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc69015bf30 0x7fc69006e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.835+0000 7fc6948aa640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6900ce5f0 0x7fc6900c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68ffff640 1 -- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc69015bf30 msgr2=0x7fc69006e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68ffff640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc69015bf30 0x7fc69006e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68ffff640 1 -- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6900ce5f0 msgr2=0x7fc6900c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68ffff640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6900ce5f0 0x7fc6900c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68ffff640 1 -- 172.21.15.28:0/599839179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6900b28f0 con 0x7fc6900cfec0 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc6948aa640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6900ce5f0 0x7fc6900c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68ffff640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900aea20 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fc684009850 tx=0x7fc6840039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68d7fa640 1 -- 172.21.15.28:0/599839179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68403f070 con 0x7fc6900cfec0 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68d7fa640 1 -- 172.21.15.28:0/599839179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc684004890 con 0x7fc6900cfec0 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6900ca5a0 con 0x7fc6900cfec0 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc68d7fa640 1 -- 172.21.15.28:0/599839179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684035030 con 0x7fc6900cfec0 2024-09-10T20:44:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.836+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6900cab60 con 0x7fc6900cfec0 2024-09-10T20:44:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.837+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6900d0330 con 0x7fc6900cfec0 2024-09-10T20:44:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.838+0000 7fc68d7fa640 1 -- 172.21.15.28:0/599839179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fc684004ca0 con 0x7fc6900cfec0 2024-09-10T20:44:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.838+0000 7fc68d7fa640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc67c0785a0 0x7fc67c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.838+0000 7fc68d7fa640 1 -- 172.21.15.28:0/599839179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7fc6840bb830 con 0x7fc6900cfec0 2024-09-10T20:44:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.838+0000 7fc6948aa640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc67c0785a0 0x7fc67c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.839+0000 7fc6948aa640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc67c0785a0 0x7fc67c07aa60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc69006e610 tx=0x7fc680008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:13.842+0000 7fc68d7fa640 1 -- 172.21.15.28:0/599839179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc684083d30 con 0x7fc6900cfec0 2024-09-10T20:44:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.026+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fc6900ce260 con 0x7fc6900cfec0 2024-09-10T20:44:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.030+0000 7fc68d7fa640 1 -- 172.21.15.28:0/599839179 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v41) ==== 71+0+0 (secure 0 0 0) 0x7fc684083630 con 0x7fc6900cfec0 2024-09-10T20:44:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc67c0785a0 msgr2=0x7fc67c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc67c0785a0 0x7fc67c07aa60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc69006e610 tx=0x7fc680008040 comp rx=0 tx=0).stop 2024-09-10T20:44:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 msgr2=0x7fc6900aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900aea20 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fc684009850 tx=0x7fc6840039e0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 shutdown_connections 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc67c0785a0 0x7fc67c07aa60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc69015bf30 0x7fc69006e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6900cfec0 0x7fc6900aea20 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 --2- 172.21.15.28:0/599839179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc6900ce5f0 0x7fc6900c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 >> 172.21.15.28:0/599839179 conn(0x7fc6900d1e40 msgr2=0x7fc69015e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 shutdown_connections 2024-09-10T20:44:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.032+0000 7fc696b0e640 1 -- 172.21.15.28:0/599839179 wait complete. 2024-09-10T20:44:14.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2' 2024-09-10T20:44:14.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 -- 172.21.15.28:0/4189025439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 msgr2=0x7fcddc0d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 --2- 172.21.15.28:0/4189025439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0d0140 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4005fd0 tx=0x7fcdd402aa90 comp rx=0 tx=0).stop 2024-09-10T20:44:14.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 -- 172.21.15.28:0/4189025439 shutdown_connections 2024-09-10T20:44:14.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 --2- 172.21.15.28:0/4189025439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0d0140 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 --2- 172.21.15.28:0/4189025439 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcddc1554e0 0x7fcddc15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 -- 172.21.15.28:0/4189025439 >> 172.21.15.28:0/4189025439 conn(0x7fcddc06cf00 msgr2=0x7fcddc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 -- 172.21.15.28:0/4189025439 shutdown_connections 2024-09-10T20:44:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.148+0000 7fcde45ec640 1 -- 172.21.15.28:0/4189025439 wait complete. 2024-09-10T20:44:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.149+0000 7fcde45ec640 1 Processor -- start 2024-09-10T20:44:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.149+0000 7fcde45ec640 1 -- start start 2024-09-10T20:44:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde45ec640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcddc155100 0x7fcddc1f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde45ec640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcddc1554e0 0x7fcddc14b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde45ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde45ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc072f60 con 0x7fcddc15ee70 2024-09-10T20:44:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde45ec640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc072de0 con 0x7fcddc1554e0 2024-09-10T20:44:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde45ec640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcddc0730e0 con 0x7fcddc155100 2024-09-10T20:44:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0b1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33720/0 (socket says 172.21.15.28:33720) 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 -- 172.21.15.28:0/3165268347 learned_addr learned my addr 172.21.15.28:0/3165268347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde1b87640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcddc1554e0 0x7fcddc14b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2388640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcddc155100 0x7fcddc1f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 -- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcddc155100 msgr2=0x7fcddc1f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcddc155100 0x7fcddc1f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 -- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcddc1554e0 msgr2=0x7fcddc14b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcddc1554e0 0x7fcddc14b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2b89640 1 -- 172.21.15.28:0/3165268347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc1f5fa0 con 0x7fcddc15ee70 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.150+0000 7fcde2388640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcddc155100 0x7fcddc1f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.151+0000 7fcde2b89640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0b1ba0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009bf0 tx=0x7fcdd4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.151+0000 7fcdd2ffd640 1 -- 172.21.15.28:0/3165268347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4046070 con 0x7fcddc15ee70 2024-09-10T20:44:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.151+0000 7fcdd2ffd640 1 -- 172.21.15.28:0/3165268347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdd4004250 con 0x7fcddc15ee70 2024-09-10T20:44:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.151+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcddc1f6280 con 0x7fcddc15ee70 2024-09-10T20:44:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.151+0000 7fcdd2ffd640 1 -- 172.21.15.28:0/3165268347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4004660 con 0x7fcddc15ee70 2024-09-10T20:44:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.151+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcddc1593f0 con 0x7fcddc15ee70 2024-09-10T20:44:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.152+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcddc1595f0 con 0x7fcddc15ee70 2024-09-10T20:44:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.152+0000 7fcdd2ffd640 1 -- 172.21.15.28:0/3165268347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fcdd404b440 con 0x7fcddc15ee70 2024-09-10T20:44:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.153+0000 7fcdd2ffd640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcdc00785a0 0x7fcdc007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.153+0000 7fcdd2ffd640 1 -- 172.21.15.28:0/3165268347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7fcdd40bb6e0 con 0x7fcddc15ee70 2024-09-10T20:44:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.153+0000 7fcde2388640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcdc00785a0 0x7fcdc007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.153+0000 7fcde2388640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcdc00785a0 0x7fcdc007aa60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007fd0 tx=0x7fcdd8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.157+0000 7fcdd2ffd640 1 -- 172.21.15.28:0/3165268347 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcddc1595f0 con 0x7fcddc15ee70 2024-09-10T20:44:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.415+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]} v 0) -- 0x7fcddc073590 con 0x7fcddc15ee70 2024-09-10T20:44:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.710+0000 7fcdd2ffd640 1 -- 172.21.15.28:0/3165268347 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]}]=0 v333) ==== 148+0+0 (secure 0 0 0) 0x7fcdd4032030 con 0x7fcddc15ee70 2024-09-10T20:44:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcdc00785a0 msgr2=0x7fcdc007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcdc00785a0 0x7fcdc007aa60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007fd0 tx=0x7fcdd8009040 comp rx=0 tx=0).stop 2024-09-10T20:44:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 msgr2=0x7fcddc0b1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0b1ba0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009bf0 tx=0x7fcdd4003ce0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 shutdown_connections 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcdc00785a0 0x7fcdc007aa60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcddc15ee70 0x7fcddc0b1ba0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcddc1554e0 0x7fcddc14b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 --2- 172.21.15.28:0/3165268347 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcddc155100 0x7fcddc1f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 >> 172.21.15.28:0/3165268347 conn(0x7fcddc06cf00 msgr2=0x7fcddc09d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 shutdown_connections 2024-09-10T20:44:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.712+0000 7fcde45ec640 1 -- 172.21.15.28:0/3165268347 wait complete. 2024-09-10T20:44:14.728 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-10T20:44:14.728 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_ops_on_nonexistent_vol (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:44:14.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:14.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.831+0000 7f7449247640 1 -- 172.21.15.28:0/2069045376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 msgr2=0x7f7430004f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.831+0000 7f7449247640 1 --2- 172.21.15.28:0/2069045376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 0x7f7430004f60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f743400bb20 tx=0x7f7430006940 comp rx=0 tx=0).stop 2024-09-10T20:44:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.831+0000 7f744359e640 1 -- 172.21.15.28:0/2069045376 reap_dead start 2024-09-10T20:44:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.831+0000 7f7449247640 1 -- 172.21.15.28:0/2069045376 shutdown_connections 2024-09-10T20:44:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.831+0000 7f7449247640 1 -- 172.21.15.28:0/2069045376 >> 172.21.15.28:0/2069045376 conn(0x7f744406cf00 msgr2=0x7f744406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.831+0000 7f7449247640 1 -- 172.21.15.28:0/2069045376 shutdown_connections 2024-09-10T20:44:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.831+0000 7f7449247640 1 -- 172.21.15.28:0/2069045376 wait complete. 2024-09-10T20:44:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.833+0000 7f7449247640 1 Processor -- start 2024-09-10T20:44:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.833+0000 7f7449247640 1 -- start start 2024-09-10T20:44:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.833+0000 7f7449247640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 0x7f744415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.833+0000 7f7449247640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74441f8760 0x7f74440b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7449247640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74440c90b0 0x7f744406f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7449247640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74440b2f70 con 0x7f7430004b40 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7449247640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f74440b2df0 con 0x7f74440c90b0 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7449247640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f74440b2770 con 0x7f74441f8760 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 0x7f744415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 0x7f744415cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33740/0 (socket says 172.21.15.28:33740) 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 -- 172.21.15.28:0/1836827611 learned_addr learned my addr 172.21.15.28:0/1836827611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f744359e640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74440c90b0 0x7f744406f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f743a59c640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74441f8760 0x7f74440b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 -- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74441f8760 msgr2=0x7f74440b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74441f8760 0x7f74440b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 -- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74440c90b0 msgr2=0x7f744406f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74440c90b0 0x7f744406f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 -- 172.21.15.28:0/1836827611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744415d400 con 0x7f7430004b40 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f743a59c640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74441f8760 0x7f74440b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7442d9d640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 0x7f744415cdb0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f743402b000 tx=0x7f74340058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f743bfff640 1 -- 172.21.15.28:0/1836827611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434040070 con 0x7f7430004b40 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f743bfff640 1 -- 172.21.15.28:0/1836827611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f743403a920 con 0x7f7430004b40 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744415d6e0 con 0x7f7430004b40 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.834+0000 7f743bfff640 1 -- 172.21.15.28:0/1836827611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434035040 con 0x7f7430004b40 2024-09-10T20:44:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.835+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74441f7f60 con 0x7f7430004b40 2024-09-10T20:44:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.835+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74441f8b40 con 0x7f7430004b40 2024-09-10T20:44:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.836+0000 7f743bfff640 1 -- 172.21.15.28:0/1836827611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f7434044400 con 0x7f7430004b40 2024-09-10T20:44:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.837+0000 7f743bfff640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74240785d0 0x7f742407aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.837+0000 7f743bfff640 1 -- 172.21.15.28:0/1836827611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7f74340bb340 con 0x7f7430004b40 2024-09-10T20:44:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.837+0000 7f743a59c640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74240785d0 0x7f742407aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.837+0000 7f743a59c640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74240785d0 0x7f742407aa90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f742c001010 tx=0x7f742c0067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:14.841+0000 7f743bfff640 1 -- 172.21.15.28:0/1836827611 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f74441f8b40 con 0x7f7430004b40 2024-09-10T20:44:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.025+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74440d0500 con 0x7f7430004b40 2024-09-10T20:44:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.026+0000 7f743bfff640 1 -- 172.21.15.28:0/1836827611 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) ==== 74+0+13086 (secure 0 0 0) 0x7f74340831e0 con 0x7f7430004b40 2024-09-10T20:44:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:15.030 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":119,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:13.700513+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1765205934":"2024-09-11T20:44:10.695018+0000","172.21.15.28:6834/1765205934":"2024-09-11T20:44:10.695018+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74240785d0 msgr2=0x7f742407aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74240785d0 0x7f742407aa90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f742c001010 tx=0x7f742c0067a0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 msgr2=0x7f744415cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 0x7f744415cdb0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f743402b000 tx=0x7f74340058d0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 shutdown_connections 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74240785d0 0x7f742407aa90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74440c90b0 0x7f744406f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74441f8760 0x7f74440b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 --2- 172.21.15.28:0/1836827611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7430004b40 0x7f744415cdb0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 >> 172.21.15.28:0/1836827611 conn(0x7f744406cf00 msgr2=0x7f7444075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 shutdown_connections 2024-09-10T20:44:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.028+0000 7f7449247640 1 -- 172.21.15.28:0/1836827611 wait complete. 2024-09-10T20:44:15.043 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:44:15.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:44:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 -- 172.21.15.28:0/3256564081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 msgr2=0x7f3ba8074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 --2- 172.21.15.28:0/3256564081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba8074340 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3ba80af610 tx=0x7f3b9802b910 comp rx=0 tx=0).stop 2024-09-10T20:44:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 -- 172.21.15.28:0/3256564081 shutdown_connections 2024-09-10T20:44:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 --2- 172.21.15.28:0/3256564081 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba806e890 0x7f3ba806ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 --2- 172.21.15.28:0/3256564081 >> v2:172.21.15.159:3300/0 conn(0x7f3ba80b2f50 0x7f3ba80b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 --2- 172.21.15.28:0/3256564081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba8074340 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 -- 172.21.15.28:0/3256564081 >> 172.21.15.28:0/3256564081 conn(0x7f3ba806d110 msgr2=0x7f3ba80b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 -- 172.21.15.28:0/3256564081 shutdown_connections 2024-09-10T20:44:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.148+0000 7f3bac86a640 1 -- 172.21.15.28:0/3256564081 wait complete. 2024-09-10T20:44:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.149+0000 7f3bac86a640 1 Processor -- start 2024-09-10T20:44:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.149+0000 7f3bac86a640 1 -- start start 2024-09-10T20:44:15.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3bac86a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba806e890 0x7f3ba81f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3bac86a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba814e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3bac86a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba80b2f50 0x7f3ba81f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3bac86a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba80b2100 con 0x7f3ba80b2b70 2024-09-10T20:44:15.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3bac86a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba80b1f80 con 0x7f3ba80b2f50 2024-09-10T20:44:15.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3bac86a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba80b2280 con 0x7f3ba806e890 2024-09-10T20:44:15.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba814e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba814e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33748/0 (socket says 172.21.15.28:33748) 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 -- 172.21.15.28:0/2494445378 learned_addr learned my addr 172.21.15.28:0/2494445378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba659c640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba806e890 0x7f3ba81f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba6d9d640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba80b2f50 0x7f3ba81f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 -- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba806e890 msgr2=0x7f3ba81f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba806e890 0x7f3ba81f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 -- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba80b2f50 msgr2=0x7f3ba81f6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba80b2f50 0x7f3ba81f6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.150+0000 7f3ba5d9b640 1 -- 172.21.15.28:0/2494445378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba81f6eb0 con 0x7f3ba80b2b70 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.151+0000 7f3ba5d9b640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba814e660 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000d950 tx=0x7f3b9000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.151+0000 7f3b9effd640 1 -- 172.21.15.28:0/2494445378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90017070 con 0x7f3ba80b2b70 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.151+0000 7f3b9effd640 1 -- 172.21.15.28:0/2494445378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b90009d20 con 0x7f3ba80b2b70 2024-09-10T20:44:15.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.151+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba814d770 con 0x7f3ba80b2b70 2024-09-10T20:44:15.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.151+0000 7f3b9effd640 1 -- 172.21.15.28:0/2494445378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b900106c0 con 0x7f3ba80b2b70 2024-09-10T20:44:15.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.151+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba81e2b20 con 0x7f3ba80b2b70 2024-09-10T20:44:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.152+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba80d0bd0 con 0x7f3ba80b2b70 2024-09-10T20:44:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.152+0000 7f3b9effd640 1 -- 172.21.15.28:0/2494445378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f3b90010860 con 0x7f3ba80b2b70 2024-09-10T20:44:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.153+0000 7f3b9effd640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b880785a0 0x7f3b8807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.153+0000 7f3ba659c640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b880785a0 0x7f3b8807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.153+0000 7f3b9effd640 1 -- 172.21.15.28:0/2494445378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7f3b9009aad0 con 0x7f3ba80b2b70 2024-09-10T20:44:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.154+0000 7f3ba659c640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b880785a0 0x7f3b8807aa60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3b98036000 tx=0x7f3b98004ab0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.157+0000 7f3b9effd640 1 -- 172.21.15.28:0/2494445378 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3ba80d0bd0 con 0x7f3ba80b2b70 2024-09-10T20:44:15.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.341+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ba8074340 con 0x7f3ba80b2b70 2024-09-10T20:44:15.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.342+0000 7f3b9effd640 1 -- 172.21.15.28:0/2494445378 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) ==== 93+0+10 (secure 0 0 0) 0x7f3b900627d0 con 0x7f3ba80b2b70 2024-09-10T20:44:15.346 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b880785a0 msgr2=0x7f3b8807aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b880785a0 0x7f3b8807aa60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3b98036000 tx=0x7f3b98004ab0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 msgr2=0x7f3ba814e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba814e660 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000d950 tx=0x7f3b9000de10 comp rx=0 tx=0).stop 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 shutdown_connections 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b880785a0 0x7f3b8807aa60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ba80b2f50 0x7f3ba81f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba80b2b70 0x7f3ba814e660 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 --2- 172.21.15.28:0/2494445378 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ba806e890 0x7f3ba81f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 >> 172.21.15.28:0/2494445378 conn(0x7f3ba806d110 msgr2=0x7f3ba8158230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 shutdown_connections 2024-09-10T20:44:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.344+0000 7f3bac86a640 1 -- 172.21.15.28:0/2494445378 wait complete. 2024-09-10T20:44:15.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol' 2024-09-10T20:44:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.460+0000 7fd107247640 1 -- 172.21.15.28:0/1737432801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 msgr2=0x7fd1000d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.460+0000 7fd107247640 1 --2- 172.21.15.28:0/1737432801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 0x7fd1000d0140 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8005fd0 tx=0x7fd0f802aa90 comp rx=0 tx=0).stop 2024-09-10T20:44:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.460+0000 7fd107247640 1 -- 172.21.15.28:0/1737432801 shutdown_connections 2024-09-10T20:44:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.460+0000 7fd107247640 1 --2- 172.21.15.28:0/1737432801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 0x7fd1000d0140 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.460+0000 7fd107247640 1 --2- 172.21.15.28:0/1737432801 >> v2:172.21.15.159:3300/0 conn(0x7fd1001554e0 0x7fd10015e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.460+0000 7fd107247640 1 -- 172.21.15.28:0/1737432801 >> 172.21.15.28:0/1737432801 conn(0x7fd10006cf00 msgr2=0x7fd10006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.461+0000 7fd107247640 1 -- 172.21.15.28:0/1737432801 shutdown_connections 2024-09-10T20:44:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.461+0000 7fd107247640 1 -- 172.21.15.28:0/1737432801 wait complete. 2024-09-10T20:44:15.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.462+0000 7fd107247640 1 Processor -- start 2024-09-10T20:44:15.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.462+0000 7fd107247640 1 -- start start 2024-09-10T20:44:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd107247640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd100155100 0x7fd1001f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd107247640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1001554e0 0x7fd100159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd107247640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 0x7fd10015a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd107247640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd100072f60 con 0x7fd1001554e0 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd107247640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd100072de0 con 0x7fd100155100 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd107247640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd1000730e0 con 0x7fd10015ee70 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1001554e0 0x7fd100159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1001554e0 0x7fd100159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33758/0 (socket says 172.21.15.28:33758) 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 -- 172.21.15.28:0/2367787118 learned_addr learned my addr 172.21.15.28:0/2367787118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd104fe3640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd100155100 0x7fd1001f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd1057e4640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 0x7fd10015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 -- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 msgr2=0x7fd10015a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 0x7fd10015a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 -- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd100155100 msgr2=0x7fd1001f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd100155100 0x7fd1001f3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 -- 172.21.15.28:0/2367787118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10014a170 con 0x7fd1001554e0 2024-09-10T20:44:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.463+0000 7fd0fffff640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1001554e0 0x7fd100159d40 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400e9f0 tx=0x7fd0f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.464+0000 7fd0fd7fa640 1 -- 172.21.15.28:0/2367787118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f400cd80 con 0x7fd1001554e0 2024-09-10T20:44:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.464+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10014a450 con 0x7fd1001554e0 2024-09-10T20:44:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.464+0000 7fd0fd7fa640 1 -- 172.21.15.28:0/2367787118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0f4010430 con 0x7fd1001554e0 2024-09-10T20:44:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.464+0000 7fd0fd7fa640 1 -- 172.21.15.28:0/2367787118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4010840 con 0x7fd1001554e0 2024-09-10T20:44:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.464+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10014aa10 con 0x7fd1001554e0 2024-09-10T20:44:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.465+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd100073590 con 0x7fd1001554e0 2024-09-10T20:44:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.465+0000 7fd0fd7fa640 1 -- 172.21.15.28:0/2367787118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fd0f40109e0 con 0x7fd1001554e0 2024-09-10T20:44:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.466+0000 7fd0fd7fa640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd0ec0785a0 0x7fd0ec07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.466+0000 7fd104fe3640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd0ec0785a0 0x7fd0ec07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.466+0000 7fd0fd7fa640 1 -- 172.21.15.28:0/2367787118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7fd0f409b2b0 con 0x7fd1001554e0 2024-09-10T20:44:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.466+0000 7fd104fe3640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd0ec0785a0 0x7fd0ec07aa60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0001720 tx=0x7fd0f0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.470+0000 7fd0fd7fa640 1 -- 172.21.15.28:0/2367787118 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd0f4063030 con 0x7fd1001554e0 2024-09-10T20:44:15.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:15.731+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]} v 0) -- 0x7fd100069a80 con 0x7fd1001554e0 2024-09-10T20:44:16.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.716+0000 7fd0fd7fa640 1 -- 172.21.15.28:0/2367787118 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]}]=0 v335) ==== 154+0+0 (secure 0 0 0) 0x7fd0f4067a60 con 0x7fd1001554e0 2024-09-10T20:44:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd0ec0785a0 msgr2=0x7fd0ec07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd0ec0785a0 0x7fd0ec07aa60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0001720 tx=0x7fd0f0009040 comp rx=0 tx=0).stop 2024-09-10T20:44:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1001554e0 msgr2=0x7fd100159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1001554e0 0x7fd100159d40 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400e9f0 tx=0x7fd0f400eeb0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 shutdown_connections 2024-09-10T20:44:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd0ec0785a0 0x7fd0ec07aa60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd10015ee70 0x7fd10015a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1001554e0 0x7fd100159d40 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 --2- 172.21.15.28:0/2367787118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd100155100 0x7fd1001f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 >> 172.21.15.28:0/2367787118 conn(0x7fd10006cf00 msgr2=0x7fd10015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 shutdown_connections 2024-09-10T20:44:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.719+0000 7fd107247640 1 -- 172.21.15.28:0/2367787118 wait complete. 2024-09-10T20:44:16.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.844+0000 7f34abdca640 1 -- 172.21.15.28:0/1053302507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a40d2960 msgr2=0x7f34a40ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.844+0000 7f34abdca640 1 --2- 172.21.15.28:0/1053302507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a40d2960 0x7f34a40ce850 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3498005fd0 tx=0x7f349802a4a0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.844+0000 7f34abdca640 1 -- 172.21.15.28:0/1053302507 shutdown_connections 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.844+0000 7f34abdca640 1 --2- 172.21.15.28:0/1053302507 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34a40cd9b0 0x7f34a40ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.844+0000 7f34abdca640 1 --2- 172.21.15.28:0/1053302507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a40d2960 0x7f34a40ce850 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.844+0000 7f34abdca640 1 -- 172.21.15.28:0/1053302507 >> 172.21.15.28:0/1053302507 conn(0x7f34a40d1e40 msgr2=0x7f34a40d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.845+0000 7f34abdca640 1 -- 172.21.15.28:0/1053302507 shutdown_connections 2024-09-10T20:44:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.845+0000 7f34abdca640 1 -- 172.21.15.28:0/1053302507 wait complete. 2024-09-10T20:44:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.846+0000 7f34abdca640 1 Processor -- start 2024-09-10T20:44:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.846+0000 7f34abdca640 1 -- start start 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.846+0000 7f34abdca640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34a40cd9b0 0x7f34a41577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34abdca640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34a40d2960 0x7f34a409f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34abdca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a415c050 0x7f34a40cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34abdca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34a40b1960 con 0x7f34a415c050 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34abdca640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f34a40b17e0 con 0x7f34a40cd9b0 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34abdca640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f34a40b1ae0 con 0x7f34a40d2960 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a415c050 0x7f34a40cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a415c050 0x7f34a40cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33784/0 (socket says 172.21.15.28:33784) 2024-09-10T20:44:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 -- 172.21.15.28:0/2942317481 learned_addr learned my addr 172.21.15.28:0/2942317481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34a9365640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34a40d2960 0x7f34a409f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34a9b66640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34a40cd9b0 0x7f34a41577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 -- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34a40d2960 msgr2=0x7f34a409f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34a40d2960 0x7f34a409f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 -- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34a40cd9b0 msgr2=0x7f34a41577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34a40cd9b0 0x7f34a41577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 -- 172.21.15.28:0/2942317481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a409dbb0 con 0x7f34a415c050 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34aa367640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a415c050 0x7f34a40cb0e0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c970 tx=0x7f349c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34a27fc640 1 -- 172.21.15.28:0/2942317481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c007bb0 con 0x7f34a415c050 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34a27fc640 1 -- 172.21.15.28:0/2942317481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f349c007e40 con 0x7f34a415c050 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a409fae0 con 0x7f34a415c050 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34a27fc640 1 -- 172.21.15.28:0/2942317481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c005280 con 0x7f34a415c050 2024-09-10T20:44:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.847+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a41f5390 con 0x7f34a415c050 2024-09-10T20:44:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.848+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a40d1bb0 con 0x7f34a415c050 2024-09-10T20:44:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.849+0000 7f34a27fc640 1 -- 172.21.15.28:0/2942317481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f349c005420 con 0x7f34a415c050 2024-09-10T20:44:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.850+0000 7f34a27fc640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34900785a0 0x7f349007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.850+0000 7f34a27fc640 1 -- 172.21.15.28:0/2942317481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7f349c09ac60 con 0x7f34a415c050 2024-09-10T20:44:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.850+0000 7f34a9b66640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34900785a0 0x7f349007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.850+0000 7f34a9b66640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34900785a0 0x7f349007aa60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3494009f60 tx=0x7f34940094f0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:16.854+0000 7f34a27fc640 1 -- 172.21.15.28:0/2942317481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f34a40d1bb0 con 0x7f34a415c050 2024-09-10T20:44:17.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.038+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f34a40b2070 con 0x7f34a415c050 2024-09-10T20:44:17.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.038+0000 7f34a27fc640 1 -- 172.21.15.28:0/2942317481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v41) ==== 59+0+1998 (secure 0 0 0) 0x7f349c014030 con 0x7f34a415c050 2024-09-10T20:44:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:44:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:44:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:44:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.040+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34900785a0 msgr2=0x7f349007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.040+0000 7f34abdca640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34900785a0 0x7f349007aa60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3494009f60 tx=0x7f34940094f0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.040+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a415c050 msgr2=0x7f34a40cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.040+0000 7f34abdca640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a415c050 0x7f34a40cb0e0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c970 tx=0x7f349c00ce30 comp rx=0 tx=0).stop 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 shutdown_connections 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f34900785a0 0x7f349007aa60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34a415c050 0x7f34a40cb0e0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34a40d2960 0x7f34a409f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 --2- 172.21.15.28:0/2942317481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34a40cd9b0 0x7f34a41577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 >> 172.21.15.28:0/2942317481 conn(0x7f34a40d1e40 msgr2=0x7f34a40b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 shutdown_connections 2024-09-10T20:44:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.041+0000 7f34abdca640 1 -- 172.21.15.28:0/2942317481 wait complete. 2024-09-10T20:44:17.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:44:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.160+0000 7f58f70d6640 1 -- 172.21.15.28:0/56651130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e0004800 msgr2=0x7f58e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.160+0000 7f58f70d6640 1 --2- 172.21.15.28:0/56651130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e0004800 0x7f58e0004c80 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00c310 tx=0x7f58e0005830 comp rx=0 tx=0).stop 2024-09-10T20:44:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.160+0000 7f58f5673640 1 -- 172.21.15.28:0/56651130 reap_dead start 2024-09-10T20:44:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.160+0000 7f58f70d6640 1 -- 172.21.15.28:0/56651130 shutdown_connections 2024-09-10T20:44:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.160+0000 7f58f70d6640 1 -- 172.21.15.28:0/56651130 >> 172.21.15.28:0/56651130 conn(0x7f58f00d4070 msgr2=0x7f58f00d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.160+0000 7f58f70d6640 1 -- 172.21.15.28:0/56651130 shutdown_connections 2024-09-10T20:44:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.160+0000 7f58f70d6640 1 -- 172.21.15.28:0/56651130 wait complete. 2024-09-10T20:44:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.161+0000 7f58f70d6640 1 Processor -- start 2024-09-10T20:44:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.161+0000 7f58f70d6640 1 -- start start 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f70d6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58e0004800 0x7f58f00cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f70d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f00b9a60 0x7f58f006d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f70d6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ac0 0x7f58f00b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f70d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58f00b2dd0 con 0x7f58f00b9a60 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f70d6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f58f00d4e80 con 0x7f58f00b8ac0 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f70d6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f58f00b2f50 con 0x7f58e0004800 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f00b9a60 0x7f58f006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f00b9a60 0x7f58f006d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33812/0 (socket says 172.21.15.28:33812) 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58effff640 1 -- 172.21.15.28:0/1675701885 learned_addr learned my addr 172.21.15.28:0/1675701885 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f4e72640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58e0004800 0x7f58f00cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58f5673640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ac0 0x7f58f00b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58effff640 1 -- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58e0004800 msgr2=0x7f58f00cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58effff640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58e0004800 0x7f58f00cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58effff640 1 -- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ac0 msgr2=0x7f58f00b8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.162+0000 7f58effff640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ac0 0x7f58f00b8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.163+0000 7f58effff640 1 -- 172.21.15.28:0/1675701885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f00b9ee0 con 0x7f58f00b9a60 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.163+0000 7f58effff640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f00b9a60 0x7f58f006d700 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f58dc000c00 tx=0x7f58dc001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.163+0000 7f58ed7fa640 1 -- 172.21.15.28:0/1675701885 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc031040 con 0x7f58f00b9a60 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.163+0000 7f58ed7fa640 1 -- 172.21.15.28:0/1675701885 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58dc0035f0 con 0x7f58f00b9a60 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.163+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f00cd850 con 0x7f58f00b9a60 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.163+0000 7f58ed7fa640 1 -- 172.21.15.28:0/1675701885 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc003a20 con 0x7f58f00b9a60 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.163+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f00cde10 con 0x7f58f00b9a60 2024-09-10T20:44:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.164+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f00b1830 con 0x7f58f00b9a60 2024-09-10T20:44:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.165+0000 7f58ed7fa640 1 -- 172.21.15.28:0/1675701885 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f58dc048030 con 0x7f58f00b9a60 2024-09-10T20:44:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.165+0000 7f58ed7fa640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f58d40785a0 0x7f58d407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.165+0000 7f58ed7fa640 1 -- 172.21.15.28:0/1675701885 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7f58dc0ba350 con 0x7f58f00b9a60 2024-09-10T20:44:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.165+0000 7f58f4e72640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f58d40785a0 0x7f58d407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.166+0000 7f58f4e72640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f58d40785a0 0x7f58d407aa60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f58e40036b0 tx=0x7f58e4006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.169+0000 7f58ed7fa640 1 -- 172.21.15.28:0/1675701885 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f58dc082190 con 0x7f58f00b9a60 2024-09-10T20:44:17.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.353+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f58f00ce400 con 0x7f58f00b9a60 2024-09-10T20:44:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.357+0000 7f58ed7fa640 1 -- 172.21.15.28:0/1675701885 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) ==== 122+0+0 (secure 0 0 0) 0x7f58dc086bc0 con 0x7f58f00b9a60 2024-09-10T20:44:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f58d40785a0 msgr2=0x7f58d407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f58d40785a0 0x7f58d407aa60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f58e40036b0 tx=0x7f58e4006b70 comp rx=0 tx=0).stop 2024-09-10T20:44:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f00b9a60 msgr2=0x7f58f006d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f00b9a60 0x7f58f006d700 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f58dc000c00 tx=0x7f58dc001cf0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 shutdown_connections 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f58d40785a0 0x7f58d407aa60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f58f00b8ac0 0x7f58f00b8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f00b9a60 0x7f58f006d700 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 --2- 172.21.15.28:0/1675701885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f58e0004800 0x7f58f00cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 >> 172.21.15.28:0/1675701885 conn(0x7f58f00d4070 msgr2=0x7f58f01605f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 shutdown_connections 2024-09-10T20:44:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.359+0000 7f58f70d6640 1 -- 172.21.15.28:0/1675701885 wait complete. 2024-09-10T20:44:17.374 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:44:17.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:17.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:17.466 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:44:17.466 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:17.466 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:17.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:44:17.483 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:17.483 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:17.501 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:44:17.501 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:17.501 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:17.539 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:44:17.540 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:17.540 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:17.559 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:44:17.559 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:44:17.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 -- 172.21.15.28:0/162622009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 msgr2=0x7fed280b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 --2- 172.21.15.28:0/162622009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280b47a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fed18009830 tx=0x7fed1802a510 comp rx=0 tx=0).stop 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 -- 172.21.15.28:0/162622009 shutdown_connections 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 --2- 172.21.15.28:0/162622009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280b47a0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 --2- 172.21.15.28:0/162622009 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed280cd3d0 0x7fed280cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 -- 172.21.15.28:0/162622009 >> 172.21.15.28:0/162622009 conn(0x7fed2806cf00 msgr2=0x7fed2806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 -- 172.21.15.28:0/162622009 shutdown_connections 2024-09-10T20:44:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.680+0000 7fed2f35d640 1 -- 172.21.15.28:0/162622009 wait complete. 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.682+0000 7fed2f35d640 1 Processor -- start 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.682+0000 7fed2f35d640 1 -- start start 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.682+0000 7fed2f35d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2f35d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed280cd3d0 0x7fed28156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2f35d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed280d0090 0x7fed28156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2f35d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed280d09a0 con 0x7fed280b4340 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2f35d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fed280d0820 con 0x7fed280d0090 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2f35d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fed280b3590 con 0x7fed280cd3d0 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280ae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33836/0 (socket says 172.21.15.28:33836) 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 -- 172.21.15.28:0/3663608602 learned_addr learned my addr 172.21.15.28:0/3663608602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2c8f8640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed280cd3d0 0x7fed28156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d8fa640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed280d0090 0x7fed28156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 -- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed280cd3d0 msgr2=0x7fed28156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed280cd3d0 0x7fed28156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 -- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed280d0090 msgr2=0x7fed28156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed280d0090 0x7fed28156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 -- 172.21.15.28:0/3663608602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed281f36e0 con 0x7fed280b4340 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2d0f9640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280ae960 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fed2400d950 tx=0x7fed2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed1dffb640 1 -- 172.21.15.28:0/3663608602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed24017070 con 0x7fed280b4340 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed1dffb640 1 -- 172.21.15.28:0/3663608602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed24009d20 con 0x7fed280b4340 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed281f2940 con 0x7fed280b4340 2024-09-10T20:44:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.683+0000 7fed1dffb640 1 -- 172.21.15.28:0/3663608602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed240106c0 con 0x7fed280b4340 2024-09-10T20:44:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.684+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed281f2f00 con 0x7fed280b4340 2024-09-10T20:44:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.685+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed280b3ab0 con 0x7fed280b4340 2024-09-10T20:44:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.685+0000 7fed1dffb640 1 -- 172.21.15.28:0/3663608602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fed24010860 con 0x7fed280b4340 2024-09-10T20:44:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.686+0000 7fed1dffb640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed0c0785a0 0x7fed0c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.686+0000 7fed2c8f8640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed0c0785a0 0x7fed0c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.686+0000 7fed1dffb640 1 -- 172.21.15.28:0/3663608602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7fed2409a9d0 con 0x7fed280b4340 2024-09-10T20:44:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.686+0000 7fed2c8f8640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed0c0785a0 0x7fed0c07aa60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fed18034000 tx=0x7fed18004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.690+0000 7fed1dffb640 1 -- 172.21.15.28:0/3663608602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fed240627a0 con 0x7fed280b4340 2024-09-10T20:44:17.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.911+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed280cf5c0 con 0x7fed280b4340 2024-09-10T20:44:17.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.911+0000 7fed1dffb640 1 -- 172.21.15.28:0/3663608602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+3917 (secure 0 0 0) 0x7fed240671d0 con 0x7fed280b4340 2024-09-10T20:44:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:17.916 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":79,"btime":"2024-09-10T20:44:11:696472+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3157854991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3157854991},{"type":"v1","addr":"172.21.15.159:6835","nonce":3157854991}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/3291396887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3291396887},{"type":"v1","addr":"172.21.15.159:6801","nonce":3291396887}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7461,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3500302836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3500302836},{"type":"v1","addr":"172.21.15.28:6833","nonce":3500302836}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70},{"gid":7788,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1677326947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1677326947},{"type":"v1","addr":"172.21.15.28:6835","nonce":1677326947}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[]} 2024-09-10T20:44:17.916 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 79 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.913+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed0c0785a0 msgr2=0x7fed0c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.913+0000 7fed2f35d640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed0c0785a0 0x7fed0c07aa60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fed18034000 tx=0x7fed18004730 comp rx=0 tx=0).stop 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.913+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 msgr2=0x7fed280ae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.913+0000 7fed2f35d640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280ae960 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fed2400d950 tx=0x7fed2400de10 comp rx=0 tx=0).stop 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 shutdown_connections 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fed0c0785a0 0x7fed0c07aa60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fed280d0090 0x7fed28156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fed280cd3d0 0x7fed28156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 --2- 172.21.15.28:0/3663608602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280b4340 0x7fed280ae960 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 >> 172.21.15.28:0/3663608602 conn(0x7fed2806cf00 msgr2=0x7fed28075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:17.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 shutdown_connections 2024-09-10T20:44:17.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:17.914+0000 7fed2f35d640 1 -- 172.21.15.28:0/3663608602 wait complete. 2024-09-10T20:44:17.929 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:44:17.929 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:44:17.929 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:44:17.929 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:17.929 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:44:17.929 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:44:17.930 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:44:17.930 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:17.930 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:44:17.930 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:44:17.930 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:44:17.930 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:17.931 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:44:17.931 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:44:17.931 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:44:17.931 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:17.939 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:44:17.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:44:17.940 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:44:17.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:44:17.942 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:44:17.942 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:44:17.942 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:44:17.943 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:44:17.982 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:44:17.982 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:17.984 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:44:17.984 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:17.984 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:44:17.985 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:17.985 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:44:17.985 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:17.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:44:18.059 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:44:18.060 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:44:18.065 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:44:18.066 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:44:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 -- 172.21.15.28:0/332752538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b800cf780 msgr2=0x7f4b800cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/332752538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b800cf780 0x7f4b800cfbc0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f4b68005fd0 tx=0x7f4b6802a5d0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 -- 172.21.15.28:0/332752538 shutdown_connections 2024-09-10T20:44:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/332752538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b800cf780 0x7f4b800cfbc0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/332752538 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b8015de70 0x7f4b800d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 -- 172.21.15.28:0/332752538 >> 172.21.15.28:0/332752538 conn(0x7f4b800d3fb0 msgr2=0x7f4b800d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 -- 172.21.15.28:0/332752538 shutdown_connections 2024-09-10T20:44:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.091+0000 7f4b85a7f640 1 -- 172.21.15.28:0/332752538 wait complete. 2024-09-10T20:44:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.092+0000 7f4b85a7f640 1 Processor -- start 2024-09-10T20:44:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.092+0000 7f4b85a7f640 1 -- start start 2024-09-10T20:44:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b85a7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b800cf780 0x7f4b801f8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b85a7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8015de70 0x7f4b800cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b85a7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b80161b50 0x7f4b80160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b85a7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b80162350 con 0x7f4b8015de70 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b85a7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801621d0 con 0x7f4b800cf780 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b85a7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4b801624d0 con 0x7f4b80161b50 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8015de70 0x7f4b800cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8015de70 0x7f4b800cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33862/0 (socket says 172.21.15.28:33862) 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b76ffd640 1 -- 172.21.15.28:0/4050706371 learned_addr learned my addr 172.21.15.28:0/4050706371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b7ffff640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b80161b50 0x7f4b80160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b7f7fe640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b800cf780 0x7f4b801f8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b76ffd640 1 -- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b80161b50 msgr2=0x7f4b80160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.093+0000 7f4b76ffd640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b80161b50 0x7f4b80160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b76ffd640 1 -- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b800cf780 msgr2=0x7f4b801f8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b76ffd640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b800cf780 0x7f4b801f8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b76ffd640 1 -- 172.21.15.28:0/4050706371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b800ce370 con 0x7f4b8015de70 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b76ffd640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8015de70 0x7f4b800cde30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006010 tx=0x7f4b680038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b7cff9640 1 -- 172.21.15.28:0/4050706371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6803f070 con 0x7f4b8015de70 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b7cff9640 1 -- 172.21.15.28:0/4050706371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b68005600 con 0x7f4b8015de70 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b801f52e0 con 0x7f4b8015de70 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b7cff9640 1 -- 172.21.15.28:0/4050706371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68035030 con 0x7f4b8015de70 2024-09-10T20:44:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.094+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b801f58a0 con 0x7f4b8015de70 2024-09-10T20:44:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.095+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b800cfbc0 con 0x7f4b8015de70 2024-09-10T20:44:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.095+0000 7f4b7cff9640 1 -- 172.21.15.28:0/4050706371 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f4b680049e0 con 0x7f4b8015de70 2024-09-10T20:44:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.096+0000 7f4b7cff9640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4b600785a0 0x7f4b6007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.096+0000 7f4b7cff9640 1 -- 172.21.15.28:0/4050706371 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 4971+0+0 (secure 0 0 0) 0x7f4b680bb360 con 0x7f4b8015de70 2024-09-10T20:44:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.096+0000 7f4b7f7fe640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4b600785a0 0x7f4b6007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.096+0000 7f4b7f7fe640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4b600785a0 0x7f4b6007aa60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4b70005fd0 tx=0x7f4b7000b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.100+0000 7f4b7cff9640 1 -- 172.21.15.28:0/4050706371 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4b680832a0 con 0x7f4b8015de70 2024-09-10T20:44:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.296+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f4b800b9f30 con 0x7f4b8015de70 2024-09-10T20:44:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.731+0000 7f4b7cff9640 1 -- 172.21.15.28:0/4050706371 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v121) ==== 97+0+0 (secure 0 0 0) 0x7f4b68087cd0 con 0x7f4b8015de70 2024-09-10T20:44:18.735 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:44:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4b600785a0 msgr2=0x7f4b6007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4b600785a0 0x7f4b6007aa60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4b70005fd0 tx=0x7f4b7000b390 comp rx=0 tx=0).stop 2024-09-10T20:44:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8015de70 msgr2=0x7f4b800cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8015de70 0x7f4b800cde30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006010 tx=0x7f4b680038a0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 shutdown_connections 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4b600785a0 0x7f4b6007aa60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4b80161b50 0x7f4b80160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8015de70 0x7f4b800cde30 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 --2- 172.21.15.28:0/4050706371 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4b800cf780 0x7f4b801f8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 >> 172.21.15.28:0/4050706371 conn(0x7f4b800d3fb0 msgr2=0x7f4b8006ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.734+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 shutdown_connections 2024-09-10T20:44:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.735+0000 7f4b85a7f640 1 -- 172.21.15.28:0/4050706371 wait complete. 2024-09-10T20:44:18.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:44:18.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.853+0000 7f59d1b33640 1 -- 172.21.15.28:0/145828913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b4004d90 msgr2=0x7f59b4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.853+0000 7f59d1b33640 1 --2- 172.21.15.28:0/145828913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b4004d90 0x7f59b4005210 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f59cc0d0bb0 tx=0x7f59b4005830 comp rx=0 tx=0).stop 2024-09-10T20:44:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.853+0000 7f59cbfff640 1 -- 172.21.15.28:0/145828913 reap_dead start 2024-09-10T20:44:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.853+0000 7f59d1b33640 1 -- 172.21.15.28:0/145828913 shutdown_connections 2024-09-10T20:44:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.853+0000 7f59d1b33640 1 -- 172.21.15.28:0/145828913 >> 172.21.15.28:0/145828913 conn(0x7f59cc06cf00 msgr2=0x7f59cc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.853+0000 7f59d1b33640 1 -- 172.21.15.28:0/145828913 shutdown_connections 2024-09-10T20:44:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.853+0000 7f59d1b33640 1 -- 172.21.15.28:0/145828913 wait complete. 2024-09-10T20:44:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.855+0000 7f59d1b33640 1 Processor -- start 2024-09-10T20:44:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.855+0000 7f59d1b33640 1 -- start start 2024-09-10T20:44:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59b4004d90 0x7f59cc0cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc1f3050 0x7f59cc0b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59cc0c6b50 0x7f59cc0ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc0b3000 con 0x7f59cc1f3050 2024-09-10T20:44:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc0b2e80 con 0x7f59b4004d90 2024-09-10T20:44:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f59cc0b3180 con 0x7f59cc0c6b50 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc1f3050 0x7f59cc0b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc1f3050 0x7f59cc0b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33878/0 (socket says 172.21.15.28:33878) 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 -- 172.21.15.28:0/2703863466 learned_addr learned my addr 172.21.15.28:0/2703863466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59cbfff640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59cc0c6b50 0x7f59cc0ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59cb7fe640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59b4004d90 0x7f59cc0cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 -- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59cc0c6b50 msgr2=0x7f59cc0ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59cc0c6b50 0x7f59cc0ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 -- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59b4004d90 msgr2=0x7f59cc0cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59b4004d90 0x7f59cc0cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 -- 172.21.15.28:0/2703863466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc0c7410 con 0x7f59cc1f3050 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59cb7fe640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59b4004d90 0x7f59cc0cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59caffd640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc1f3050 0x7f59cc0b13e0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f59c000e610 tx=0x7f59c000ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59b3fff640 1 -- 172.21.15.28:0/2703863466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0015070 con 0x7f59cc1f3050 2024-09-10T20:44:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59b3fff640 1 -- 172.21.15.28:0/2703863466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59c000ca60 con 0x7f59cc1f3050 2024-09-10T20:44:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59cc0b0900 con 0x7f59cc1f3050 2024-09-10T20:44:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59b3fff640 1 -- 172.21.15.28:0/2703863466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0011460 con 0x7f59cc1f3050 2024-09-10T20:44:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.856+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59cc0b1920 con 0x7f59cc1f3050 2024-09-10T20:44:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.857+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59cc0d0090 con 0x7f59cc1f3050 2024-09-10T20:44:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.858+0000 7f59b3fff640 1 -- 172.21.15.28:0/2703863466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f59c0011600 con 0x7f59cc1f3050 2024-09-10T20:44:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.858+0000 7f59b3fff640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f59b80785a0 0x7f59b807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.859+0000 7f59b3fff640 1 -- 172.21.15.28:0/2703863466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4885+0+0 (secure 0 0 0) 0x7f59c009add0 con 0x7f59cc1f3050 2024-09-10T20:44:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.859+0000 7f59cb7fe640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f59b80785a0 0x7f59b807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.859+0000 7f59cb7fe640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f59b80785a0 0x7f59b807aa60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f59bc000c00 tx=0x7f59bc006c40 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:18.863+0000 7f59b3fff640 1 -- 172.21.15.28:0/2703863466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f59c00631b0 con 0x7f59cc1f3050 2024-09-10T20:44:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.116+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f59cc0af040 con 0x7f59cc1f3050 2024-09-10T20:44:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.116+0000 7f59b3fff640 1 -- 172.21.15.28:0/2703863466 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) ==== 80+0+5304 (secure 0 0 0) 0x7f59c0062ab0 con 0x7f59cc1f3050 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:44:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:44:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:44:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:44:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:44:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:44:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:44:19.135 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f59b80785a0 msgr2=0x7f59b807aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f59b80785a0 0x7f59b807aa60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f59bc000c00 tx=0x7f59bc006c40 comp rx=0 tx=0).stop 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc1f3050 msgr2=0x7f59cc0b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc1f3050 0x7f59cc0b13e0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f59c000e610 tx=0x7f59c000ead0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 shutdown_connections 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f59b80785a0 0x7f59b807aa60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f59cc0c6b50 0x7f59cc0ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc1f3050 0x7f59cc0b13e0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 --2- 172.21.15.28:0/2703863466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f59b4004d90 0x7f59cc0cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 >> 172.21.15.28:0/2703863466 conn(0x7f59cc06cf00 msgr2=0x7f59cc15d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 shutdown_connections 2024-09-10T20:44:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.119+0000 7f59d1b33640 1 -- 172.21.15.28:0/2703863466 wait complete. 2024-09-10T20:44:19.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.239+0000 7fa045d64640 1 -- 172.21.15.28:0/853517062 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 msgr2=0x7fa0400b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.239+0000 7fa045d64640 1 --2- 172.21.15.28:0/853517062 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 0x7fa0400b2e80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa038009be0 tx=0x7fa03802a8d0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.239+0000 7fa045d64640 1 -- 172.21.15.28:0/853517062 shutdown_connections 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.239+0000 7fa045d64640 1 --2- 172.21.15.28:0/853517062 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0400cc570 0x7fa0400d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.239+0000 7fa045d64640 1 --2- 172.21.15.28:0/853517062 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 0x7fa0400b2e80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.239+0000 7fa045d64640 1 -- 172.21.15.28:0/853517062 >> 172.21.15.28:0/853517062 conn(0x7fa04006cf00 msgr2=0x7fa04006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.240+0000 7fa045d64640 1 -- 172.21.15.28:0/853517062 shutdown_connections 2024-09-10T20:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.240+0000 7fa045d64640 1 -- 172.21.15.28:0/853517062 wait complete. 2024-09-10T20:44:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.241+0000 7fa045d64640 1 Processor -- start 2024-09-10T20:44:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.241+0000 7fa045d64640 1 -- start start 2024-09-10T20:44:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa045d64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0400cc570 0x7fa04014fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa045d64640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0400cd3d0 0x7fa0400b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa045d64640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 0x7fa0400c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa045d64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0400b33c0 con 0x7fa0400cc570 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa045d64640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa0400b4b60 con 0x7fa0400d05d0 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa045d64640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa0400b3540 con 0x7fa0400cd3d0 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0400cc570 0x7fa04014fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0400cc570 0x7fa04014fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33882/0 (socket says 172.21.15.28:33882) 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 -- 172.21.15.28:0/2834269563 learned_addr learned my addr 172.21.15.28:0/2834269563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03effd640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0400cd3d0 0x7fa0400b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03ffff640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 0x7fa0400c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 -- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0400cd3d0 msgr2=0x7fa0400b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0400cd3d0 0x7fa0400b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 -- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 msgr2=0x7fa0400c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 0x7fa0400c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 -- 172.21.15.28:0/2834269563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa040150180 con 0x7fa0400cc570 2024-09-10T20:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa03f7fe640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0400cc570 0x7fa04014fc40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c450 tx=0x7fa03000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.242+0000 7fa027fff640 1 -- 172.21.15.28:0/2834269563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030016020 con 0x7fa0400cc570 2024-09-10T20:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.243+0000 7fa027fff640 1 -- 172.21.15.28:0/2834269563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa030012920 con 0x7fa0400cc570 2024-09-10T20:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.243+0000 7fa027fff640 1 -- 172.21.15.28:0/2834269563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030012d70 con 0x7fa0400cc570 2024-09-10T20:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.243+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0400726c0 con 0x7fa0400cc570 2024-09-10T20:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.243+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa040072f40 con 0x7fa0400cc570 2024-09-10T20:44:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.244+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0400b39d0 con 0x7fa0400cc570 2024-09-10T20:44:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.244+0000 7fa027fff640 1 -- 172.21.15.28:0/2834269563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fa03001b400 con 0x7fa0400cc570 2024-09-10T20:44:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.245+0000 7fa027fff640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa02c0785a0 0x7fa02c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.245+0000 7fa027fff640 1 -- 172.21.15.28:0/2834269563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4885+0+0 (secure 0 0 0) 0x7fa03009a590 con 0x7fa0400cc570 2024-09-10T20:44:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.245+0000 7fa03effd640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa02c0785a0 0x7fa02c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.245+0000 7fa03effd640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa02c0785a0 0x7fa02c07aa60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa0340095f0 tx=0x7fa034005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:19.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.249+0000 7fa027fff640 1 -- 172.21.15.28:0/2834269563 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa030020030 con 0x7fa0400cc570 2024-09-10T20:44:19.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.436+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa040069a80 con 0x7fa0400cc570 2024-09-10T20:44:19.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.436+0000 7fa027fff640 1 -- 172.21.15.28:0/2834269563 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v123) ==== 84+0+65 (secure 0 0 0) 0x7fa040150180 con 0x7fa0400cc570 2024-09-10T20:44:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:19.440 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:44:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:44:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:44:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:19.441 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:44:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa02c0785a0 msgr2=0x7fa02c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa02c0785a0 0x7fa02c07aa60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa0340095f0 tx=0x7fa034005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0400cc570 msgr2=0x7fa04014fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0400cc570 0x7fa04014fc40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c450 tx=0x7fa03000c910 comp rx=0 tx=0).stop 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 shutdown_connections 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa02c0785a0 0x7fa02c07aa60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa0400d05d0 0x7fa0400c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa0400cd3d0 0x7fa0400b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 --2- 172.21.15.28:0/2834269563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0400cc570 0x7fa04014fc40 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 >> 172.21.15.28:0/2834269563 conn(0x7fa04006cf00 msgr2=0x7fa040075920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 shutdown_connections 2024-09-10T20:44:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.438+0000 7fa045d64640 1 -- 172.21.15.28:0/2834269563 wait complete. 2024-09-10T20:44:19.453 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:44:19.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:44:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.560+0000 7fdd248ed640 1 -- 172.21.15.28:0/1976904365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 msgr2=0x7fdd0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.560+0000 7fdd248ed640 1 --2- 172.21.15.28:0/1976904365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd0c004c80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800c310 tx=0x7fdd0c005830 comp rx=0 tx=0).stop 2024-09-10T20:44:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.560+0000 7fdd248ed640 1 -- 172.21.15.28:0/1976904365 shutdown_connections 2024-09-10T20:44:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.560+0000 7fdd248ed640 1 --2- 172.21.15.28:0/1976904365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd0c004c80 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.560+0000 7fdd248ed640 1 -- 172.21.15.28:0/1976904365 >> 172.21.15.28:0/1976904365 conn(0x7fdd200d3f90 msgr2=0x7fdd200d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.560+0000 7fdd248ed640 1 -- 172.21.15.28:0/1976904365 shutdown_connections 2024-09-10T20:44:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.560+0000 7fdd248ed640 1 -- 172.21.15.28:0/1976904365 wait complete. 2024-09-10T20:44:19.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.561+0000 7fdd248ed640 1 Processor -- start 2024-09-10T20:44:19.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.561+0000 7fdd248ed640 1 -- start start 2024-09-10T20:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd248ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd200d0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd248ed640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015f6c0 0x7fdd2015fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd248ed640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd200a0ea0 0x7fdd200a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd248ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd200d4a80 con 0x7fdd0c004800 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd248ed640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201626b0 con 0x7fdd2015f6c0 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd248ed640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdd200d4c00 con 0x7fdd200a0ea0 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd200d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd200d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33904/0 (socket says 172.21.15.28:33904) 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 -- 172.21.15.28:0/757920054 learned_addr learned my addr 172.21.15.28:0/757920054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1dd9b640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015f6c0 0x7fdd2015fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1ed9d640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd200a0ea0 0x7fdd200a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 -- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd200a0ea0 msgr2=0x7fdd200a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd200a0ea0 0x7fdd200a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 -- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015f6c0 msgr2=0x7fdd2015fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015f6c0 0x7fdd2015fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1e59c640 1 -- 172.21.15.28:0/757920054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd201f4a60 con 0x7fdd0c004800 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.562+0000 7fdd1ed9d640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd200a0ea0 0x7fdd200a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.563+0000 7fdd1e59c640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd200d0740 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fdd100018c0 tx=0x7fdd10001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.563+0000 7fdd16ffd640 1 -- 172.21.15.28:0/757920054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10015070 con 0x7fdd0c004800 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.563+0000 7fdd16ffd640 1 -- 172.21.15.28:0/757920054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd1000ebb0 con 0x7fdd0c004800 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.563+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd200a1820 con 0x7fdd0c004800 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.563+0000 7fdd16ffd640 1 -- 172.21.15.28:0/757920054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd100020c0 con 0x7fdd0c004800 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.563+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd201f4fe0 con 0x7fdd0c004800 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.564+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd201f5700 con 0x7fdd0c004800 2024-09-10T20:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.564+0000 7fdd16ffd640 1 -- 172.21.15.28:0/757920054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fdd10011460 con 0x7fdd0c004800 2024-09-10T20:44:19.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.565+0000 7fdd16ffd640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdd000785a0 0x7fdd0007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:19.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.565+0000 7fdd1dd9b640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdd000785a0 0x7fdd0007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:19.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.565+0000 7fdd16ffd640 1 -- 172.21.15.28:0/757920054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4885+0+0 (secure 0 0 0) 0x7fdd1009b1f0 con 0x7fdd0c004800 2024-09-10T20:44:19.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.566+0000 7fdd1dd9b640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdd000785a0 0x7fdd0007aa60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdd08033000 tx=0x7fdd08038040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.569+0000 7fdd16ffd640 1 -- 172.21.15.28:0/757920054 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdd10062f80 con 0x7fdd0c004800 2024-09-10T20:44:19.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:19.756+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fdd200b9f10 con 0x7fdd0c004800 2024-09-10T20:44:20.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.746+0000 7fdd16ffd640 1 -- 172.21.15.28:0/757920054 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v124) ==== 138+0+0 (secure 0 0 0) 0x7fdd100679b0 con 0x7fdd0c004800 2024-09-10T20:44:20.750 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:44:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdd000785a0 msgr2=0x7fdd0007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdd000785a0 0x7fdd0007aa60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdd08033000 tx=0x7fdd08038040 comp rx=0 tx=0).stop 2024-09-10T20:44:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 msgr2=0x7fdd200d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd200d0740 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fdd100018c0 tx=0x7fdd10001d80 comp rx=0 tx=0).stop 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 shutdown_connections 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdd000785a0 0x7fdd0007aa60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdd200a0ea0 0x7fdd200a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdd2015f6c0 0x7fdd2015fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 --2- 172.21.15.28:0/757920054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd0c004800 0x7fdd200d0740 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 >> 172.21.15.28:0/757920054 conn(0x7fdd200d3f90 msgr2=0x7fdd200d2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 shutdown_connections 2024-09-10T20:44:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.750+0000 7fdd248ed640 1 -- 172.21.15.28:0/757920054 wait complete. 2024-09-10T20:44:20.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:44:20.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.887+0000 7f7954860640 1 -- 172.21.15.28:0/123050283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793c004800 msgr2=0x7f793c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.887+0000 7f7954860640 1 --2- 172.21.15.28:0/123050283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793c004800 0x7f793c004c80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f7938001740 tx=0x7f793c005830 comp rx=0 tx=0).stop 2024-09-10T20:44:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.887+0000 7f794ed9d640 1 -- 172.21.15.28:0/123050283 reap_dead start 2024-09-10T20:44:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.887+0000 7f7954860640 1 -- 172.21.15.28:0/123050283 shutdown_connections 2024-09-10T20:44:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.887+0000 7f7954860640 1 -- 172.21.15.28:0/123050283 >> 172.21.15.28:0/123050283 conn(0x7f795009d240 msgr2=0x7f79500d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.887+0000 7f7954860640 1 -- 172.21.15.28:0/123050283 shutdown_connections 2024-09-10T20:44:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.887+0000 7f7954860640 1 -- 172.21.15.28:0/123050283 wait complete. 2024-09-10T20:44:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 Processor -- start 2024-09-10T20:44:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 -- start start 2024-09-10T20:44:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f793c004800 0x7f79501534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79500b9390 0x7f79500b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500b73a0 0x7f79500cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79500b16d0 con 0x7f79500b9390 2024-09-10T20:44:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f79500b1550 con 0x7f793c004800 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f7954860640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f79500b1850 con 0x7f79500b73a0 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f794dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79500b9390 0x7f79500b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.889+0000 7f794dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79500b9390 0x7f79500b6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49388/0 (socket says 172.21.15.28:49388) 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794dd9b640 1 -- 172.21.15.28:0/4032876132 learned_addr learned my addr 172.21.15.28:0/4032876132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794ed9d640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500b73a0 0x7f79500cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794e59c640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f793c004800 0x7f79501534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794dd9b640 1 -- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500b73a0 msgr2=0x7f79500cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794dd9b640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500b73a0 0x7f79500cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794dd9b640 1 -- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f793c004800 msgr2=0x7f79501534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794dd9b640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f793c004800 0x7f79501534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794dd9b640 1 -- 172.21.15.28:0/4032876132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79501f8ae0 con 0x7f79500b9390 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f794dd9b640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79500b9390 0x7f79500b6e60 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f7938001830 tx=0x7f7938006150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f7946ffd640 1 -- 172.21.15.28:0/4032876132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938037040 con 0x7f79500b9390 2024-09-10T20:44:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f7946ffd640 1 -- 172.21.15.28:0/4032876132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7938041070 con 0x7f79500b9390 2024-09-10T20:44:20.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79501f8dc0 con 0x7f79500b9390 2024-09-10T20:44:20.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f7946ffd640 1 -- 172.21.15.28:0/4032876132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938003a50 con 0x7f79500b9390 2024-09-10T20:44:20.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.890+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79500cb480 con 0x7f79500b9390 2024-09-10T20:44:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.891+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79500b1db0 con 0x7f79500b9390 2024-09-10T20:44:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.892+0000 7f7946ffd640 1 -- 172.21.15.28:0/4032876132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f7938003bf0 con 0x7f79500b9390 2024-09-10T20:44:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.893+0000 7f7946ffd640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f79300785a0 0x7f793007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.893+0000 7f794e59c640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f79300785a0 0x7f793007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.893+0000 7f7946ffd640 1 -- 172.21.15.28:0/4032876132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5255+0+0 (secure 0 0 0) 0x7f79380ba720 con 0x7f79500b9390 2024-09-10T20:44:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.893+0000 7f794e59c640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f79300785a0 0x7f793007aa60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7948005fd0 tx=0x7f7948009500 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:20.897+0000 7f7946ffd640 1 -- 172.21.15.28:0/4032876132 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7938082470 con 0x7f79500b9390 2024-09-10T20:44:21.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.088+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f79500b9b00 con 0x7f79500b9390 2024-09-10T20:44:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.752+0000 7f7946ffd640 1 -- 172.21.15.28:0/4032876132 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v125) ==== 170+0+0 (secure 0 0 0) 0x7f7938086ea0 con 0x7f79500b9390 2024-09-10T20:44:21.757 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:44:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.756+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f79300785a0 msgr2=0x7f793007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:21.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.756+0000 7f7954860640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f79300785a0 0x7f793007aa60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7948005fd0 tx=0x7f7948009500 comp rx=0 tx=0).stop 2024-09-10T20:44:21.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.756+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79500b9390 msgr2=0x7f79500b6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:21.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.756+0000 7f7954860640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79500b9390 0x7f79500b6e60 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f7938001830 tx=0x7f7938006150 comp rx=0 tx=0).stop 2024-09-10T20:44:21.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.769+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 shutdown_connections 2024-09-10T20:44:21.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.769+0000 7f7954860640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f79300785a0 0x7f793007aa60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.769+0000 7f7954860640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79500b73a0 0x7f79500cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.769+0000 7f7954860640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79500b9390 0x7f79500b6e60 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.769+0000 7f7954860640 1 --2- 172.21.15.28:0/4032876132 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f793c004800 0x7f79501534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.769+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 >> 172.21.15.28:0/4032876132 conn(0x7f795009d240 msgr2=0x7f7950075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:21.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.771+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 shutdown_connections 2024-09-10T20:44:21.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.771+0000 7f7954860640 1 -- 172.21.15.28:0/4032876132 wait complete. 2024-09-10T20:44:21.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:44:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 -- 172.21.15.28:0/488252574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 msgr2=0x7fb96806e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 --2- 172.21.15.28:0/488252574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 0x7fb96806e940 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb960005fd0 tx=0x7fb96002aa90 comp rx=0 tx=0).stop 2024-09-10T20:44:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 -- 172.21.15.28:0/488252574 shutdown_connections 2024-09-10T20:44:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 --2- 172.21.15.28:0/488252574 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 0x7fb96806e940 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 --2- 172.21.15.28:0/488252574 >> v2:172.21.15.159:3300/0 conn(0x7fb9680b3590 0x7fb96806e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 -- 172.21.15.28:0/488252574 >> 172.21.15.28:0/488252574 conn(0x7fb96806d110 msgr2=0x7fb9680b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 -- 172.21.15.28:0/488252574 shutdown_connections 2024-09-10T20:44:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.901+0000 7fb96fa72640 1 -- 172.21.15.28:0/488252574 wait complete. 2024-09-10T20:44:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.902+0000 7fb96fa72640 1 Processor -- start 2024-09-10T20:44:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.902+0000 7fb96fa72640 1 -- start start 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96fa72640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 0x7fb9681f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96fa72640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9680b3590 0x7fb9681f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96fa72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb968143830 0x7fb9680ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96fa72640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9680708f0 con 0x7fb968143830 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96fa72640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb968070770 con 0x7fb9680b3590 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96fa72640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb96815da80 con 0x7fb968072d20 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96e00f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb968143830 0x7fb9680ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96e00f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb968143830 0x7fb9680ae390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49396/0 (socket says 172.21.15.28:49396) 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96e00f640 1 -- 172.21.15.28:0/2608023510 learned_addr learned my addr 172.21.15.28:0/2608023510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96d00d640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9680b3590 0x7fb9681f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.903+0000 7fb96d80e640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 0x7fb9681f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96e00f640 1 -- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 msgr2=0x7fb9681f7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96e00f640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 0x7fb9681f7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96e00f640 1 -- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9680b3590 msgr2=0x7fb9681f84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96e00f640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9680b3590 0x7fb9681f84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96e00f640 1 -- 172.21.15.28:0/2608023510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9681f8e60 con 0x7fb968143830 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96e00f640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb968143830 0x7fb9680ae390 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fb960009d90 tx=0x7fb960003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb95e7fc640 1 -- 172.21.15.28:0/2608023510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb960046070 con 0x7fb968143830 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb95e7fc640 1 -- 172.21.15.28:0/2608023510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb960004040 con 0x7fb968143830 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96806f270 con 0x7fb968143830 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb95e7fc640 1 -- 172.21.15.28:0/2608023510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb960004450 con 0x7fb968143830 2024-09-10T20:44:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.904+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96815cb40 con 0x7fb968143830 2024-09-10T20:44:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.905+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9680b2e80 con 0x7fb968143830 2024-09-10T20:44:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.906+0000 7fb95e7fc640 1 -- 172.21.15.28:0/2608023510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fb9600045f0 con 0x7fb968143830 2024-09-10T20:44:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.907+0000 7fb95e7fc640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb94c0785a0 0x7fb94c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.907+0000 7fb95e7fc640 1 -- 172.21.15.28:0/2608023510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5637+0+0 (secure 0 0 0) 0x7fb9600bbc70 con 0x7fb968143830 2024-09-10T20:44:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.907+0000 7fb96d80e640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb94c0785a0 0x7fb94c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.907+0000 7fb96d80e640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb94c0785a0 0x7fb94c07aa60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb964001720 tx=0x7fb964009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:21.910+0000 7fb95e7fc640 1 -- 172.21.15.28:0/2608023510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb960050080 con 0x7fb968143830 2024-09-10T20:44:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.132+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb96815bc60 con 0x7fb968143830 2024-09-10T20:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.762+0000 7fb95e7fc640 1 -- 172.21.15.28:0/2608023510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v82) ==== 174+0+0 (secure 0 0 0) 0x7fb960032030 con 0x7fb968143830 2024-09-10T20:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 20 and data pool 21 2024-09-10T20:44:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb94c0785a0 msgr2=0x7fb94c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb94c0785a0 0x7fb94c07aa60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb964001720 tx=0x7fb964009040 comp rx=0 tx=0).stop 2024-09-10T20:44:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb968143830 msgr2=0x7fb9680ae390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb968143830 0x7fb9680ae390 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fb960009d90 tx=0x7fb960003ba0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 shutdown_connections 2024-09-10T20:44:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb94c0785a0 0x7fb94c07aa60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb968143830 0x7fb9680ae390 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9680b3590 0x7fb9681f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 --2- 172.21.15.28:0/2608023510 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb968072d20 0x7fb9681f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.769+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 >> 172.21.15.28:0/2608023510 conn(0x7fb96806d110 msgr2=0x7fb96815b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.770+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 shutdown_connections 2024-09-10T20:44:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.770+0000 7fb96fa72640 1 -- 172.21.15.28:0/2608023510 wait complete. 2024-09-10T20:44:22.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.906+0000 7fb645fd3640 1 -- 172.21.15.28:0/2235442792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb64006d1f0 msgr2=0x7fb6400caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.906+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2235442792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb64006d1f0 0x7fb6400caa70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb634005fd0 tx=0x7fb63402a670 comp rx=0 tx=0).stop 2024-09-10T20:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.906+0000 7fb645fd3640 1 -- 172.21.15.28:0/2235442792 shutdown_connections 2024-09-10T20:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.906+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2235442792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb64006d1f0 0x7fb6400caa70 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.906+0000 7fb645fd3640 1 -- 172.21.15.28:0/2235442792 >> 172.21.15.28:0/2235442792 conn(0x7fb6400d0420 msgr2=0x7fb6400d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.906+0000 7fb645fd3640 1 -- 172.21.15.28:0/2235442792 shutdown_connections 2024-09-10T20:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.906+0000 7fb645fd3640 1 -- 172.21.15.28:0/2235442792 wait complete. 2024-09-10T20:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.907+0000 7fb645fd3640 1 Processor -- start 2024-09-10T20:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb645fd3640 1 -- start start 2024-09-10T20:44:22.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb645fd3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb64006d1f0 0x7fb640157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb645fd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb640156130 0x7fb64014be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb645fd3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb640156510 0x7fb6400ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb645fd3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb64006f2c0 con 0x7fb640156130 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb645fd3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb64006f140 con 0x7fb640156510 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb645fd3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb640073900 con 0x7fb64006d1f0 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb63effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb640156130 0x7fb64014be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.908+0000 7fb63effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb640156130 0x7fb64014be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49402/0 (socket says 172.21.15.28:49402) 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63effd640 1 -- 172.21.15.28:0/2918675189 learned_addr learned my addr 172.21.15.28:0/2918675189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63f7fe640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb64006d1f0 0x7fb640157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63ffff640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb640156510 0x7fb6400ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63effd640 1 -- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb64006d1f0 msgr2=0x7fb640157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63effd640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb64006d1f0 0x7fb640157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63effd640 1 -- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb640156510 msgr2=0x7fb6400ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63effd640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb640156510 0x7fb6400ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63effd640 1 -- 172.21.15.28:0/2918675189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6401f4f80 con 0x7fb640156130 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63f7fe640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb64006d1f0 0x7fb640157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb63effd640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb640156130 0x7fb64014be70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb634009850 tx=0x7fb6340039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb627fff640 1 -- 172.21.15.28:0/2918675189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63403f070 con 0x7fb640156130 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb627fff640 1 -- 172.21.15.28:0/2918675189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb634004890 con 0x7fb640156130 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6401f8a10 con 0x7fb640156130 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb627fff640 1 -- 172.21.15.28:0/2918675189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634035030 con 0x7fb640156130 2024-09-10T20:44:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.909+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6401f8fd0 con 0x7fb640156130 2024-09-10T20:44:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.910+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6401f5150 con 0x7fb640156130 2024-09-10T20:44:22.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.911+0000 7fb627fff640 1 -- 172.21.15.28:0/2918675189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fb634004c60 con 0x7fb640156130 2024-09-10T20:44:22.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.911+0000 7fb627fff640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb62c0785a0 0x7fb62c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:22.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.911+0000 7fb627fff640 1 -- 172.21.15.28:0/2918675189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5737+0+0 (secure 0 0 0) 0x7fb6340bba30 con 0x7fb640156130 2024-09-10T20:44:22.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.911+0000 7fb63f7fe640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb62c0785a0 0x7fb62c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:22.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.912+0000 7fb63f7fe640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb62c0785a0 0x7fb62c07aa60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb630007fd0 tx=0x7fb630009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:22.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:22.915+0000 7fb627fff640 1 -- 172.21.15.28:0/2918675189 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb634083cc0 con 0x7fb640156130 2024-09-10T20:44:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.102+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb640073db0 con 0x7fb640156130 2024-09-10T20:44:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.103+0000 7fb627fff640 1 -- 172.21.15.28:0/2918675189 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+16796 (secure 0 0 0) 0x7fb6340835c0 con 0x7fb640156130 2024-09-10T20:44:23.108 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:23.108 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":126,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:22.133713+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:19.761497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:21.093337+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb62c0785a0 msgr2=0x7fb62c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb62c0785a0 0x7fb62c07aa60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb630007fd0 tx=0x7fb630009040 comp rx=0 tx=0).stop 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb640156130 msgr2=0x7fb64014be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb640156130 0x7fb64014be70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb634009850 tx=0x7fb6340039e0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 shutdown_connections 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb62c0785a0 0x7fb62c07aa60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb640156510 0x7fb6400ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb640156130 0x7fb64014be70 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 --2- 172.21.15.28:0/2918675189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb64006d1f0 0x7fb640157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 >> 172.21.15.28:0/2918675189 conn(0x7fb6400d0420 msgr2=0x7fb6400d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 shutdown_connections 2024-09-10T20:44:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.106+0000 7fb645fd3640 1 -- 172.21.15.28:0/2918675189 wait complete. 2024-09-10T20:44:23.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 -- 172.21.15.28:0/4203589372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb00738b0 msgr2=0x7f3cb015e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 --2- 172.21.15.28:0/4203589372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb00738b0 0x7f3cb015e7e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0074360 tx=0x7f3ca402a4d0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 -- 172.21.15.28:0/4203589372 shutdown_connections 2024-09-10T20:44:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 --2- 172.21.15.28:0/4203589372 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cb0073cf0 0x7f3cb015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 --2- 172.21.15.28:0/4203589372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb00738b0 0x7f3cb015e7e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 --2- 172.21.15.28:0/4203589372 >> v2:172.21.15.159:3300/0 conn(0x7f3cb0151cc0 0x7f3cb00cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 -- 172.21.15.28:0/4203589372 >> 172.21.15.28:0/4203589372 conn(0x7f3cb00d0420 msgr2=0x7f3cb00d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 -- 172.21.15.28:0/4203589372 shutdown_connections 2024-09-10T20:44:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.222+0000 7f3cb849b640 1 -- 172.21.15.28:0/4203589372 wait complete. 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.223+0000 7f3cb849b640 1 Processor -- start 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.223+0000 7f3cb849b640 1 -- start start 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb849b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cb00738b0 0x7f3cb0147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb849b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cb0073cf0 0x7f3cb00b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb849b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb0151cc0 0x7f3cb006f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb849b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb00cb210 con 0x7f3cb0151cc0 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb849b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb015f640 con 0x7f3cb00738b0 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb849b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb00cb390 con 0x7f3cb0073cf0 2024-09-10T20:44:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb6a38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb0151cc0 0x7f3cb006f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb6a38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb0151cc0 0x7f3cb006f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49422/0 (socket says 172.21.15.28:49422) 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb6a38640 1 -- 172.21.15.28:0/3989532451 learned_addr learned my addr 172.21.15.28:0/3989532451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb5a36640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cb0073cf0 0x7f3cb00b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb6237640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cb00738b0 0x7f3cb0147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb6a38640 1 -- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cb0073cf0 msgr2=0x7f3cb00b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.224+0000 7f3cb6a38640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cb0073cf0 0x7f3cb00b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3cb6a38640 1 -- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cb00738b0 msgr2=0x7f3cb0147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3cb6a38640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cb00738b0 0x7f3cb0147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3cb6a38640 1 -- 172.21.15.28:0/3989532451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb00b17f0 con 0x7f3cb0151cc0 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3cb6237640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cb00738b0 0x7f3cb0147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3cb6a38640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb0151cc0 0x7f3cb006f700 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800d650 tx=0x7f3ca800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3caeffd640 1 -- 172.21.15.28:0/3989532451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8017070 con 0x7f3cb0151cc0 2024-09-10T20:44:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3caeffd640 1 -- 172.21.15.28:0/3989532451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ca8005330 con 0x7f3cb0151cc0 2024-09-10T20:44:23.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3caeffd640 1 -- 172.21.15.28:0/3989532451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8009820 con 0x7f3cb0151cc0 2024-09-10T20:44:23.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb00b1ad0 con 0x7f3cb0151cc0 2024-09-10T20:44:23.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.225+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb00b0060 con 0x7f3cb0151cc0 2024-09-10T20:44:23.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.226+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb00cee20 con 0x7f3cb0151cc0 2024-09-10T20:44:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.227+0000 7f3caeffd640 1 -- 172.21.15.28:0/3989532451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f3ca80099c0 con 0x7f3cb0151cc0 2024-09-10T20:44:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.227+0000 7f3caeffd640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c9c0785a0 0x7f3c9c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.227+0000 7f3caeffd640 1 -- 172.21.15.28:0/3989532451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5737+0+0 (secure 0 0 0) 0x7f3ca809a900 con 0x7f3cb0151cc0 2024-09-10T20:44:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.227+0000 7f3cb6237640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c9c0785a0 0x7f3c9c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.228+0000 7f3cb6237640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c9c0785a0 0x7f3c9c07aa60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00b1280 tx=0x7f3ca000a410 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:23.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.231+0000 7f3caeffd640 1 -- 172.21.15.28:0/3989532451 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3cb00cee20 con 0x7f3cb0151cc0 2024-09-10T20:44:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.417+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3cb00cf0f0 con 0x7f3cb0151cc0 2024-09-10T20:44:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.418+0000 7f3caeffd640 1 -- 172.21.15.28:0/3989532451 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+16796 (secure 0 0 0) 0x7f3ca806b140 con 0x7f3cb0151cc0 2024-09-10T20:44:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:23.423 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":126,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:22.133713+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:19.761497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:21.093337+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.420+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c9c0785a0 msgr2=0x7f3c9c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.420+0000 7f3cb849b640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c9c0785a0 0x7f3c9c07aa60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00b1280 tx=0x7f3ca000a410 comp rx=0 tx=0).stop 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.420+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb0151cc0 msgr2=0x7f3cb006f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.420+0000 7f3cb849b640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb0151cc0 0x7f3cb006f700 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800d650 tx=0x7f3ca800db10 comp rx=0 tx=0).stop 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 shutdown_connections 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3c9c0785a0 0x7f3c9c07aa60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cb0151cc0 0x7f3cb006f700 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cb0073cf0 0x7f3cb00b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 --2- 172.21.15.28:0/3989532451 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cb00738b0 0x7f3cb0147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 >> 172.21.15.28:0/3989532451 conn(0x7f3cb00d0420 msgr2=0x7f3cb006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 shutdown_connections 2024-09-10T20:44:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.421+0000 7f3cb849b640 1 -- 172.21.15.28:0/3989532451 wait complete. 2024-09-10T20:44:23.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:44:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.540+0000 7f3b12b82640 1 -- 172.21.15.28:0/3226036186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af8004800 msgr2=0x7f3af8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.540+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3226036186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af8004800 0x7f3af8004c80 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f3afc006fd0 tx=0x7f3af8005830 comp rx=0 tx=0).stop 2024-09-10T20:44:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.540+0000 7f3b12b82640 1 -- 172.21.15.28:0/3226036186 shutdown_connections 2024-09-10T20:44:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.540+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3226036186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af8004800 0x7f3af8004c80 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.541+0000 7f3b12b82640 1 -- 172.21.15.28:0/3226036186 >> 172.21.15.28:0/3226036186 conn(0x7f3b0c06d110 msgr2=0x7f3b0c0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.541+0000 7f3b12b82640 1 -- 172.21.15.28:0/3226036186 shutdown_connections 2024-09-10T20:44:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.541+0000 7f3b12b82640 1 -- 172.21.15.28:0/3226036186 wait complete. 2024-09-10T20:44:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.542+0000 7f3b12b82640 1 Processor -- start 2024-09-10T20:44:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.542+0000 7f3b12b82640 1 -- start start 2024-09-10T20:44:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b12b82640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3af8004800 0x7f3b0c159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b12b82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b0c1576e0 0x7f3b0c14b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b12b82640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b0c14c2b0 0x7f3b0c14c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b12b82640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c0b1f80 con 0x7f3b0c1576e0 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b12b82640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c0cf470 con 0x7f3b0c14c2b0 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b12b82640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0c0b2100 con 0x7f3af8004800 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b0c1576e0 0x7f3b0c14b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b0c1576e0 0x7f3b0c14b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49446/0 (socket says 172.21.15.28:49446) 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 -- 172.21.15.28:0/3153687490 learned_addr learned my addr 172.21.15.28:0/3153687490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b1091e640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3af8004800 0x7f3b0c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 -- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3af8004800 msgr2=0x7f3b0c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b1111f640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b0c14c2b0 0x7f3b0c14c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3af8004800 0x7f3b0c159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 -- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b0c14c2b0 msgr2=0x7f3b0c14c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b0c14c2b0 0x7f3b0c14c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 -- 172.21.15.28:0/3153687490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0c156990 con 0x7f3b0c1576e0 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b0bfff640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b0c1576e0 0x7f3b0c14b510 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c6d0 tx=0x7f3afc004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.543+0000 7f3b097fa640 1 -- 172.21.15.28:0/3153687490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc039070 con 0x7f3b0c1576e0 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.544+0000 7f3b097fa640 1 -- 172.21.15.28:0/3153687490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3afc02ba90 con 0x7f3b0c1576e0 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.544+0000 7f3b097fa640 1 -- 172.21.15.28:0/3153687490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc006070 con 0x7f3b0c1576e0 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.544+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0c156c00 con 0x7f3b0c1576e0 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.544+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0c159550 con 0x7f3b0c1576e0 2024-09-10T20:44:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.544+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b0c06e400 con 0x7f3b0c1576e0 2024-09-10T20:44:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.545+0000 7f3b097fa640 1 -- 172.21.15.28:0/3153687490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f3afc006210 con 0x7f3b0c1576e0 2024-09-10T20:44:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.546+0000 7f3b097fa640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3aec0785d0 0x7f3aec07aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.546+0000 7f3b097fa640 1 -- 172.21.15.28:0/3153687490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5737+0+0 (secure 0 0 0) 0x7f3afc0babb0 con 0x7f3b0c1576e0 2024-09-10T20:44:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.546+0000 7f3b1091e640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3aec0785d0 0x7f3aec07aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.546+0000 7f3b1091e640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3aec0785d0 0x7f3aec07aa90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c151eb0 tx=0x7f3b04006e10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.550+0000 7f3b097fa640 1 -- 172.21.15.28:0/3153687490 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3afc082c40 con 0x7f3b0c1576e0 2024-09-10T20:44:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:23.772+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3b0c069a40 con 0x7f3b0c1576e0 2024-09-10T20:44:24.742 INFO:tasks.ceph.mds.b.smithi159.stderr:2024-09-10T20:44:24.738+0000 7f7715542640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:44:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.783+0000 7f3b097fa640 1 -- 172.21.15.28:0/3153687490 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v85) ==== 118+0+0 (secure 0 0 0) 0x7f3afc089070 con 0x7f3b0c1576e0 2024-09-10T20:44:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3aec0785d0 msgr2=0x7f3aec07aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3aec0785d0 0x7f3aec07aa90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c151eb0 tx=0x7f3b04006e10 comp rx=0 tx=0).stop 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b0c1576e0 msgr2=0x7f3b0c14b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b0c1576e0 0x7f3b0c14b510 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c6d0 tx=0x7f3afc004bb0 comp rx=0 tx=0).stop 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 shutdown_connections 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3aec0785d0 0x7f3aec07aa90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b0c14c2b0 0x7f3b0c14c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b0c1576e0 0x7f3b0c14b510 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 --2- 172.21.15.28:0/3153687490 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3af8004800 0x7f3b0c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.786+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 >> 172.21.15.28:0/3153687490 conn(0x7f3b0c06d110 msgr2=0x7f3b0c15bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.787+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 shutdown_connections 2024-09-10T20:44:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.787+0000 7f3b12b82640 1 -- 172.21.15.28:0/3153687490 wait complete. 2024-09-10T20:44:24.804 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:44:24.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:44:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.915+0000 7fbd0f194640 1 -- 172.21.15.28:0/531204385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0004800 msgr2=0x7fbcf0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.915+0000 7fbd0f194640 1 --2- 172.21.15.28:0/531204385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0004800 0x7fbcf0004c80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fbd08159f30 tx=0x7fbcf0005830 comp rx=0 tx=0).stop 2024-09-10T20:44:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.915+0000 7fbd0d731640 1 -- 172.21.15.28:0/531204385 reap_dead start 2024-09-10T20:44:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.916+0000 7fbd0f194640 1 -- 172.21.15.28:0/531204385 shutdown_connections 2024-09-10T20:44:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.916+0000 7fbd0f194640 1 -- 172.21.15.28:0/531204385 >> 172.21.15.28:0/531204385 conn(0x7fbd080d0d50 msgr2=0x7fbd080d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.916+0000 7fbd0f194640 1 -- 172.21.15.28:0/531204385 shutdown_connections 2024-09-10T20:44:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.916+0000 7fbd0f194640 1 -- 172.21.15.28:0/531204385 wait complete. 2024-09-10T20:44:24.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.917+0000 7fbd0f194640 1 Processor -- start 2024-09-10T20:44:24.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.917+0000 7fbd0f194640 1 -- start start 2024-09-10T20:44:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0f194640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcf0004800 0x7fbd080c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0f194640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd080c41c0 0x7fbd081f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0f194640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd080c4670 0x7fbd081f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0f194640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd080cb8e0 con 0x7fbd080c41c0 2024-09-10T20:44:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0f194640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbd080742a0 con 0x7fbd080c4670 2024-09-10T20:44:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0f194640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbd080cba60 con 0x7fbcf0004800 2024-09-10T20:44:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd080c41c0 0x7fbd081f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd080c41c0 0x7fbd081f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49464/0 (socket says 172.21.15.28:49464) 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 -- 172.21.15.28:0/1982742141 learned_addr learned my addr 172.21.15.28:0/1982742141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0cf30640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcf0004800 0x7fbd080c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbd0d731640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd080c4670 0x7fbd081f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 -- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcf0004800 msgr2=0x7fbd080c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcf0004800 0x7fbd080c7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 -- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd080c4670 msgr2=0x7fbd081f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd080c4670 0x7fbd081f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 -- 172.21.15.28:0/1982742141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd080c4b50 con 0x7fbd080c41c0 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.918+0000 7fbcfffff640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd080c41c0 0x7fbd081f5b70 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8035040 tx=0x7fbcf802baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.919+0000 7fbcfd7fa640 1 -- 172.21.15.28:0/1982742141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8041070 con 0x7fbd080c41c0 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.919+0000 7fbcfd7fa640 1 -- 172.21.15.28:0/1982742141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcf8004310 con 0x7fbd080c41c0 2024-09-10T20:44:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.919+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd080b66b0 con 0x7fbd080c41c0 2024-09-10T20:44:24.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.919+0000 7fbcfd7fa640 1 -- 172.21.15.28:0/1982742141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8003850 con 0x7fbd080c41c0 2024-09-10T20:44:24.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.919+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd080c7680 con 0x7fbd080c41c0 2024-09-10T20:44:24.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.920+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd080c1210 con 0x7fbd080c41c0 2024-09-10T20:44:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.921+0000 7fbcfd7fa640 1 -- 172.21.15.28:0/1982742141 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fbcf80039f0 con 0x7fbd080c41c0 2024-09-10T20:44:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.921+0000 7fbcfd7fa640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbcf40785a0 0x7fbcf407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.921+0000 7fbcfd7fa640 1 -- 172.21.15.28:0/1982742141 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fbcf80bbe70 con 0x7fbd080c41c0 2024-09-10T20:44:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.921+0000 7fbd0cf30640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbcf40785a0 0x7fbcf407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.922+0000 7fbd0cf30640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbcf40785a0 0x7fbcf407aa60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbd080b12f0 tx=0x7fbd04001b90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:24.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:24.925+0000 7fbcfd7fa640 1 -- 172.21.15.28:0/1982742141 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbcf8083a40 con 0x7fbd080c41c0 2024-09-10T20:44:25.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.146+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fbd0815c4e0 con 0x7fbd080c41c0 2024-09-10T20:44:25.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.785+0000 7fbcfd7fa640 1 -- 172.21.15.28:0/1982742141 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v86) ==== 122+0+0 (secure 0 0 0) 0x7fbcf8088470 con 0x7fbd080c41c0 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbcf40785a0 msgr2=0x7fbcf407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbcf40785a0 0x7fbcf407aa60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbd080b12f0 tx=0x7fbd04001b90 comp rx=0 tx=0).stop 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd080c41c0 msgr2=0x7fbd081f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd080c41c0 0x7fbd081f5b70 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8035040 tx=0x7fbcf802baf0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 shutdown_connections 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbcf40785a0 0x7fbcf407aa60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbd080c4670 0x7fbd081f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd080c41c0 0x7fbd081f5b70 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 --2- 172.21.15.28:0/1982742141 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcf0004800 0x7fbd080c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 >> 172.21.15.28:0/1982742141 conn(0x7fbd080d0d50 msgr2=0x7fbd0806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 shutdown_connections 2024-09-10T20:44:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.788+0000 7fbd0f194640 1 -- 172.21.15.28:0/1982742141 wait complete. 2024-09-10T20:44:25.805 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:44:25.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 -- 172.21.15.28:0/444429522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68400d2b90 msgr2=0x7f6840161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 --2- 172.21.15.28:0/444429522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68400d2b90 0x7f6840161b50 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f6830005fd0 tx=0x7f683002a530 comp rx=0 tx=0).stop 2024-09-10T20:44:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 -- 172.21.15.28:0/444429522 shutdown_connections 2024-09-10T20:44:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 --2- 172.21.15.28:0/444429522 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6840162090 0x7f68400b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 --2- 172.21.15.28:0/444429522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68400d2b90 0x7f6840161b50 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 -- 172.21.15.28:0/444429522 >> 172.21.15.28:0/444429522 conn(0x7f68400d3fb0 msgr2=0x7f68400d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 -- 172.21.15.28:0/444429522 shutdown_connections 2024-09-10T20:44:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.914+0000 7f6848408640 1 -- 172.21.15.28:0/444429522 wait complete. 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.915+0000 7f6848408640 1 Processor -- start 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.915+0000 7f6848408640 1 -- start start 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f6848408640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68400d2b90 0x7f684014c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f6848408640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684015de70 0x7f684015f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f6848408640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6840162090 0x7f684015fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f6848408640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f684015e3d0 con 0x7f6840162090 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f6848408640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f684015e250 con 0x7f684015de70 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f6848408640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f68400b7340 con 0x7f68400d2b90 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6840162090 0x7f684015fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6840162090 0x7f684015fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49490/0 (socket says 172.21.15.28:49490) 2024-09-10T20:44:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 -- 172.21.15.28:0/3548897048 learned_addr learned my addr 172.21.15.28:0/3548897048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68459a3640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684015de70 0x7f684015f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68461a4640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68400d2b90 0x7f684014c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 -- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68400d2b90 msgr2=0x7f684014c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68400d2b90 0x7f684014c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 -- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684015de70 msgr2=0x7f684015f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684015de70 0x7f684015f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.916+0000 7f68469a5640 1 -- 172.21.15.28:0/3548897048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68400b1b60 con 0x7f6840162090 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.917+0000 7f68469a5640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6840162090 0x7f684015fae0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f683800e990 tx=0x7f683800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.917+0000 7f6836ffd640 1 -- 172.21.15.28:0/3548897048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838004d20 con 0x7f6840162090 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.917+0000 7f6836ffd640 1 -- 172.21.15.28:0/3548897048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f683800cb00 con 0x7f6840162090 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.917+0000 7f6836ffd640 1 -- 172.21.15.28:0/3548897048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838005020 con 0x7f6840162090 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.917+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68400b1e40 con 0x7f6840162090 2024-09-10T20:44:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.917+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68400b2400 con 0x7f6840162090 2024-09-10T20:44:25.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.918+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68400b77a0 con 0x7f6840162090 2024-09-10T20:44:25.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.919+0000 7f6836ffd640 1 -- 172.21.15.28:0/3548897048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f683800f040 con 0x7f6840162090 2024-09-10T20:44:25.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.919+0000 7f6836ffd640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68240785a0 0x7f682407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.919+0000 7f68461a4640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68240785a0 0x7f682407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.919+0000 7f6836ffd640 1 -- 172.21.15.28:0/3548897048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f683809af40 con 0x7f6840162090 2024-09-10T20:44:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.920+0000 7f68461a4640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68240785a0 0x7f682407aa60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f683c000f60 tx=0x7f683c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:25.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:25.923+0000 7f6836ffd640 1 -- 172.21.15.28:0/3548897048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6838016030 con 0x7f6840162090 2024-09-10T20:44:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.144+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68400b9f30 con 0x7f6840162090 2024-09-10T20:44:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.144+0000 7f6836ffd640 1 -- 172.21.15.28:0/3548897048 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5227 (secure 0 0 0) 0x7f6838062980 con 0x7f6840162090 2024-09-10T20:44:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:26.149 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"btime":"2024-09-10T20:44:25:784364+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1041363523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1041363523},{"type":"v1","addr":"172.21.15.28:6834","nonce":1041363523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1561732564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1561732564},{"type":"v1","addr":"172.21.15.28:6835","nonce":1561732564}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6521,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/97213244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":97213244},{"type":"v1","addr":"172.21.15.159:6835","nonce":97213244}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:22.133677+0000","modified":"2024-09-10T20:44:25.147825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6964":{"gid":6964,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/3268698079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3268698079},{"type":"v1","addr":"172.21.15.159:6801","nonce":3268698079}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6964,"qdb_cluster":[6964]},"id":10}]} 2024-09-10T20:44:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-10T20:44:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68240785a0 msgr2=0x7f682407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68240785a0 0x7f682407aa60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f683c000f60 tx=0x7f683c009040 comp rx=0 tx=0).stop 2024-09-10T20:44:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6840162090 msgr2=0x7f684015fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6840162090 0x7f684015fae0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f683800e990 tx=0x7f683800ee50 comp rx=0 tx=0).stop 2024-09-10T20:44:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 shutdown_connections 2024-09-10T20:44:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68240785a0 0x7f682407aa60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6840162090 0x7f684015fae0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f684015de70 0x7f684015f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 --2- 172.21.15.28:0/3548897048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68400d2b90 0x7f684014c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 >> 172.21.15.28:0/3548897048 conn(0x7f68400d3fb0 msgr2=0x7f68400cf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 shutdown_connections 2024-09-10T20:44:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.147+0000 7f6848408640 1 -- 172.21.15.28:0/3548897048 wait complete. 2024-09-10T20:44:26.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 -- 172.21.15.28:0/3974873989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74006d1f0 msgr2=0x7fb74006ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 --2- 172.21.15.28:0/3974873989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74006d1f0 0x7fb74006ef30 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fb738005fd0 tx=0x7fb73802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 -- 172.21.15.28:0/3974873989 shutdown_connections 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 --2- 172.21.15.28:0/3974873989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74006d1f0 0x7fb74006ef30 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 --2- 172.21.15.28:0/3974873989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb740156130 0x7fb7400caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 -- 172.21.15.28:0/3974873989 >> 172.21.15.28:0/3974873989 conn(0x7fb7400d0420 msgr2=0x7fb7400d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 -- 172.21.15.28:0/3974873989 shutdown_connections 2024-09-10T20:44:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.263+0000 7fb747b16640 1 -- 172.21.15.28:0/3974873989 wait complete. 2024-09-10T20:44:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.264+0000 7fb747b16640 1 Processor -- start 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.264+0000 7fb747b16640 1 -- start start 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb747b16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb74006d1f0 0x7fb7400b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb747b16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740156130 0x7fb7400b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb747b16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb740156510 0x7fb7400cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb747b16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7400733b0 con 0x7fb740156130 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb747b16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb740073230 con 0x7fb74006d1f0 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb747b16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb740073530 con 0x7fb740156510 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740156130 0x7fb7400b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740156130 0x7fb7400b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49498/0 (socket says 172.21.15.28:49498) 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 -- 172.21.15.28:0/3407946478 learned_addr learned my addr 172.21.15.28:0/3407946478 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 -- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb740156510 msgr2=0x7fb7400cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7458b2640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb74006d1f0 0x7fb7400b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7460b3640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb740156510 0x7fb7400cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb740156510 0x7fb7400cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 -- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb74006d1f0 msgr2=0x7fb7400b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb74006d1f0 0x7fb7400b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 -- 172.21.15.28:0/3407946478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7400b3930 con 0x7fb740156130 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7460b3640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb740156510 0x7fb7400cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7458b2640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb74006d1f0 0x7fb7400b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7450b1640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740156130 0x7fb7400b5530 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb73000c3a0 tx=0x7fb73000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7367fc640 1 -- 172.21.15.28:0/3407946478 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb730011790 con 0x7fb740156130 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7367fc640 1 -- 172.21.15.28:0/3407946478 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb730017070 con 0x7fb740156130 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7401f83c0 con 0x7fb740156130 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.265+0000 7fb7367fc640 1 -- 172.21.15.28:0/3407946478 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb730011cc0 con 0x7fb740156130 2024-09-10T20:44:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.266+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7401f8980 con 0x7fb740156130 2024-09-10T20:44:26.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.267+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb74006ef30 con 0x7fb740156130 2024-09-10T20:44:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.267+0000 7fb7367fc640 1 -- 172.21.15.28:0/3407946478 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fb73001c460 con 0x7fb740156130 2024-09-10T20:44:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.268+0000 7fb7367fc640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb7240785a0 0x7fb72407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.268+0000 7fb7367fc640 1 -- 172.21.15.28:0/3407946478 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fb73009a600 con 0x7fb740156130 2024-09-10T20:44:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.268+0000 7fb7458b2640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb7240785a0 0x7fb72407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.268+0000 7fb7458b2640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb7240785a0 0x7fb72407aa60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009b40 tx=0x7fb73c005e50 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.272+0000 7fb7367fc640 1 -- 172.21.15.28:0/3407946478 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb730062820 con 0x7fb740156130 2024-09-10T20:44:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.462+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7400cc4f0 con 0x7fb740156130 2024-09-10T20:44:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.463+0000 7fb7367fc640 1 -- 172.21.15.28:0/3407946478 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16783 (secure 0 0 0) 0x7fb730062120 con 0x7fb740156130 2024-09-10T20:44:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:26.467 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:23.754781+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:19.761497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:21.093337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb7240785a0 msgr2=0x7fb72407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb7240785a0 0x7fb72407aa60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009b40 tx=0x7fb73c005e50 comp rx=0 tx=0).stop 2024-09-10T20:44:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740156130 msgr2=0x7fb7400b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740156130 0x7fb7400b5530 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb73000c3a0 tx=0x7fb73000c860 comp rx=0 tx=0).stop 2024-09-10T20:44:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 shutdown_connections 2024-09-10T20:44:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb7240785a0 0x7fb72407aa60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb740156510 0x7fb7400cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740156130 0x7fb7400b5530 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 --2- 172.21.15.28:0/3407946478 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb74006d1f0 0x7fb7400b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 >> 172.21.15.28:0/3407946478 conn(0x7fb7400d0420 msgr2=0x7fb7401601d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 shutdown_connections 2024-09-10T20:44:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.465+0000 7fb747b16640 1 -- 172.21.15.28:0/3407946478 wait complete. 2024-09-10T20:44:26.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:44:26.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:44:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.583+0000 7fec233f8640 1 -- 172.21.15.28:0/2711381139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 msgr2=0x7fec04005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.583+0000 7fec233f8640 1 --2- 172.21.15.28:0/2711381139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 0x7fec04005210 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0d1200 tx=0x7fec04005830 comp rx=0 tx=0).stop 2024-09-10T20:44:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.583+0000 7fec21995640 1 -- 172.21.15.28:0/2711381139 reap_dead start 2024-09-10T20:44:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.583+0000 7fec233f8640 1 -- 172.21.15.28:0/2711381139 shutdown_connections 2024-09-10T20:44:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.583+0000 7fec233f8640 1 -- 172.21.15.28:0/2711381139 >> 172.21.15.28:0/2711381139 conn(0x7fec1c06dbf0 msgr2=0x7fec1c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.583+0000 7fec233f8640 1 -- 172.21.15.28:0/2711381139 shutdown_connections 2024-09-10T20:44:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.583+0000 7fec233f8640 1 -- 172.21.15.28:0/2711381139 wait complete. 2024-09-10T20:44:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.584+0000 7fec233f8640 1 Processor -- start 2024-09-10T20:44:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.584+0000 7fec233f8640 1 -- start start 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec233f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 0x7fec1c1f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec233f8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec1c0c5b90 0x7fec1c160310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec233f8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec1c160850 0x7fec1c140620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec233f8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c0d1b50 con 0x7fec04004d90 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec233f8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c0d19d0 con 0x7fec1c0c5b90 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec233f8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fec1c06f550 con 0x7fec1c160850 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 0x7fec1c1f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 0x7fec1c1f86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49512/0 (socket says 172.21.15.28:49512) 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 -- 172.21.15.28:0/3326788606 learned_addr learned my addr 172.21.15.28:0/3326788606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21995640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec1c160850 0x7fec1c140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec20993640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec1c0c5b90 0x7fec1c160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 -- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec1c160850 msgr2=0x7fec1c140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec1c160850 0x7fec1c140620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 -- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec1c0c5b90 msgr2=0x7fec1c160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec1c0c5b90 0x7fec1c160310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 -- 172.21.15.28:0/3326788606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1c0c6830 con 0x7fec04004d90 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21995640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec1c160850 0x7fec1c140620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec20993640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec1c0c5b90 0x7fec1c160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec21194640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 0x7fec1c1f86a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fec14003f50 tx=0x7fec140016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec19ffb640 1 -- 172.21.15.28:0/3326788606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14011640 con 0x7fec04004d90 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec19ffb640 1 -- 172.21.15.28:0/3326788606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec14016020 con 0x7fec04004d90 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.585+0000 7fec19ffb640 1 -- 172.21.15.28:0/3326788606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14011a60 con 0x7fec04004d90 2024-09-10T20:44:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.586+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1c1f4660 con 0x7fec04004d90 2024-09-10T20:44:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.586+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1c1f48f0 con 0x7fec04004d90 2024-09-10T20:44:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.587+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec1c161950 con 0x7fec04004d90 2024-09-10T20:44:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.587+0000 7fec19ffb640 1 -- 172.21.15.28:0/3326788606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fec14011c00 con 0x7fec04004d90 2024-09-10T20:44:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.588+0000 7fec19ffb640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec080785a0 0x7fec0807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.588+0000 7fec19ffb640 1 -- 172.21.15.28:0/3326788606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fec1409b610 con 0x7fec04004d90 2024-09-10T20:44:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.588+0000 7fec20993640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec080785a0 0x7fec0807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.588+0000 7fec20993640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec080785a0 0x7fec0807aa60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec10001220 tx=0x7fec100067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.592+0000 7fec19ffb640 1 -- 172.21.15.28:0/3326788606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fec14020020 con 0x7fec04004d90 2024-09-10T20:44:26.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.773+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec1c1460e0 con 0x7fec080785a0 2024-09-10T20:44:26.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.777+0000 7fec19ffb640 1 -- 172.21.15.28:0/3326788606 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7fec1c1460e0 con 0x7fec080785a0 2024-09-10T20:44:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:26.785 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:44:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec080785a0 msgr2=0x7fec0807aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec080785a0 0x7fec0807aa60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec10001220 tx=0x7fec100067a0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 msgr2=0x7fec1c1f86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 0x7fec1c1f86a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fec14003f50 tx=0x7fec140016f0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 shutdown_connections 2024-09-10T20:44:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec080785a0 0x7fec0807aa60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec1c160850 0x7fec1c140620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec1c0c5b90 0x7fec1c160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 --2- 172.21.15.28:0/3326788606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec04004d90 0x7fec1c1f86a0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 >> 172.21.15.28:0/3326788606 conn(0x7fec1c06dbf0 msgr2=0x7fec1c070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 shutdown_connections 2024-09-10T20:44:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.780+0000 7fec233f8640 1 -- 172.21.15.28:0/3326788606 wait complete. 2024-09-10T20:44:26.795 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-10T20:44:26.319582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279700,"kb_used_data":13524,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754695020,"statfs":{"total":773094113280,"available":772807700480,"internally_reserved":0,"allocated":13848576,"data_stored":6741396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.571909"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087576+0000","last_change":"2024-09-10T20:44:22.784196+0000","last_active":"2024-09-10T20:44:24.087576+0000","last_peered":"2024-09-10T20:44:24.087576+0000","last_clean":"2024-09-10T20:44:24.087576+0000","last_became_active":"2024-09-10T20:44:22.783532+0000","last_became_peered":"2024-09-10T20:44:22.783532+0000","last_unstale":"2024-09-10T20:44:24.087576+0000","last_undegraded":"2024-09-10T20:44:24.087576+0000","last_fullsized":"2024-09-10T20:44:24.087576+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:13.310561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780872+0000","last_change":"2024-09-10T20:44:22.770726+0000","last_active":"2024-09-10T20:44:23.780872+0000","last_peered":"2024-09-10T20:44:23.780872+0000","last_clean":"2024-09-10T20:44:23.780872+0000","last_became_active":"2024-09-10T20:44:22.769964+0000","last_became_peered":"2024-09-10T20:44:22.769964+0000","last_unstale":"2024-09-10T20:44:23.780872+0000","last_undegraded":"2024-09-10T20:44:23.780872+0000","last_fullsized":"2024-09-10T20:44:23.780872+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:20.622191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087667+0000","last_change":"2024-09-10T20:44:22.773512+0000","last_active":"2024-09-10T20:44:24.087667+0000","last_peered":"2024-09-10T20:44:24.087667+0000","last_clean":"2024-09-10T20:44:24.087667+0000","last_became_active":"2024-09-10T20:44:22.773229+0000","last_became_peered":"2024-09-10T20:44:22.773229+0000","last_unstale":"2024-09-10T20:44:24.087667+0000","last_undegraded":"2024-09-10T20:44:24.087667+0000","last_fullsized":"2024-09-10T20:44:24.087667+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:13:53.798166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758415+0000","last_change":"2024-09-10T20:44:22.772619+0000","last_active":"2024-09-10T20:44:23.758415+0000","last_peered":"2024-09-10T20:44:23.758415+0000","last_clean":"2024-09-10T20:44:23.758415+0000","last_became_active":"2024-09-10T20:44:22.772321+0000","last_became_peered":"2024-09-10T20:44:22.772321+0000","last_unstale":"2024-09-10T20:44:23.758415+0000","last_undegraded":"2024-09-10T20:44:23.758415+0000","last_fullsized":"2024-09-10T20:44:23.758415+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:45.686643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087075+0000","last_change":"2024-09-10T20:44:22.773955+0000","last_active":"2024-09-10T20:44:24.087075+0000","last_peered":"2024-09-10T20:44:24.087075+0000","last_clean":"2024-09-10T20:44:24.087075+0000","last_became_active":"2024-09-10T20:44:22.773613+0000","last_became_peered":"2024-09-10T20:44:22.773613+0000","last_unstale":"2024-09-10T20:44:24.087075+0000","last_undegraded":"2024-09-10T20:44:24.087075+0000","last_fullsized":"2024-09-10T20:44:24.087075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:40.210289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759150+0000","last_change":"2024-09-10T20:44:22.771240+0000","last_active":"2024-09-10T20:44:23.759150+0000","last_peered":"2024-09-10T20:44:23.759150+0000","last_clean":"2024-09-10T20:44:23.759150+0000","last_became_active":"2024-09-10T20:44:22.770955+0000","last_became_peered":"2024-09-10T20:44:22.770955+0000","last_unstale":"2024-09-10T20:44:23.759150+0000","last_undegraded":"2024-09-10T20:44:23.759150+0000","last_fullsized":"2024-09-10T20:44:23.759150+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:41.015623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759178+0000","last_change":"2024-09-10T20:44:22.778100+0000","last_active":"2024-09-10T20:44:23.759178+0000","last_peered":"2024-09-10T20:44:23.759178+0000","last_clean":"2024-09-10T20:44:23.759178+0000","last_became_active":"2024-09-10T20:44:22.777800+0000","last_became_peered":"2024-09-10T20:44:22.777800+0000","last_unstale":"2024-09-10T20:44:23.759178+0000","last_undegraded":"2024-09-10T20:44:23.759178+0000","last_fullsized":"2024-09-10T20:44:23.759178+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:56.442874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087060+0000","last_change":"2024-09-10T20:44:22.767533+0000","last_active":"2024-09-10T20:44:24.087060+0000","last_peered":"2024-09-10T20:44:24.087060+0000","last_clean":"2024-09-10T20:44:24.087060+0000","last_became_active":"2024-09-10T20:44:22.767126+0000","last_became_peered":"2024-09-10T20:44:22.767126+0000","last_unstale":"2024-09-10T20:44:24.087060+0000","last_undegraded":"2024-09-10T20:44:24.087060+0000","last_fullsized":"2024-09-10T20:44:24.087060+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:39:05.891827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759124+0000","last_change":"2024-09-10T20:44:22.779905+0000","last_active":"2024-09-10T20:44:23.759124+0000","last_peered":"2024-09-10T20:44:23.759124+0000","last_clean":"2024-09-10T20:44:23.759124+0000","last_became_active":"2024-09-10T20:44:22.779645+0000","last_became_peered":"2024-09-10T20:44:22.779645+0000","last_unstale":"2024-09-10T20:44:23.759124+0000","last_undegraded":"2024-09-10T20:44:23.759124+0000","last_fullsized":"2024-09-10T20:44:23.759124+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:22.561965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.088000+0000","last_change":"2024-09-10T20:44:22.777545+0000","last_active":"2024-09-10T20:44:24.088000+0000","last_peered":"2024-09-10T20:44:24.088000+0000","last_clean":"2024-09-10T20:44:24.088000+0000","last_became_active":"2024-09-10T20:44:22.777243+0000","last_became_peered":"2024-09-10T20:44:22.777243+0000","last_unstale":"2024-09-10T20:44:24.088000+0000","last_undegraded":"2024-09-10T20:44:24.088000+0000","last_fullsized":"2024-09-10T20:44:24.088000+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:54.165792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759444+0000","last_change":"2024-09-10T20:44:22.770149+0000","last_active":"2024-09-10T20:44:23.759444+0000","last_peered":"2024-09-10T20:44:23.759444+0000","last_clean":"2024-09-10T20:44:23.759444+0000","last_became_active":"2024-09-10T20:44:22.769838+0000","last_became_peered":"2024-09-10T20:44:22.769838+0000","last_unstale":"2024-09-10T20:44:23.759444+0000","last_undegraded":"2024-09-10T20:44:23.759444+0000","last_fullsized":"2024-09-10T20:44:23.759444+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:39.976689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087208+0000","last_change":"2024-09-10T20:44:22.776151+0000","last_active":"2024-09-10T20:44:24.087208+0000","last_peered":"2024-09-10T20:44:24.087208+0000","last_clean":"2024-09-10T20:44:24.087208+0000","last_became_active":"2024-09-10T20:44:22.775791+0000","last_became_peered":"2024-09-10T20:44:22.775791+0000","last_unstale":"2024-09-10T20:44:24.087208+0000","last_undegraded":"2024-09-10T20:44:24.087208+0000","last_fullsized":"2024-09-10T20:44:24.087208+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:01.456087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087349+0000","last_change":"2024-09-10T20:44:22.774050+0000","last_active":"2024-09-10T20:44:24.087349+0000","last_peered":"2024-09-10T20:44:24.087349+0000","last_clean":"2024-09-10T20:44:24.087349+0000","last_became_active":"2024-09-10T20:44:22.773758+0000","last_became_peered":"2024-09-10T20:44:22.773758+0000","last_unstale":"2024-09-10T20:44:24.087349+0000","last_undegraded":"2024-09-10T20:44:24.087349+0000","last_fullsized":"2024-09-10T20:44:24.087349+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:50.319442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086516+0000","last_change":"2024-09-10T20:44:22.764932+0000","last_active":"2024-09-10T20:44:24.086516+0000","last_peered":"2024-09-10T20:44:24.086516+0000","last_clean":"2024-09-10T20:44:24.086516+0000","last_became_active":"2024-09-10T20:44:22.764628+0000","last_became_peered":"2024-09-10T20:44:22.764628+0000","last_unstale":"2024-09-10T20:44:24.086516+0000","last_undegraded":"2024-09-10T20:44:24.086516+0000","last_fullsized":"2024-09-10T20:44:24.086516+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:59.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087619+0000","last_change":"2024-09-10T20:44:22.768129+0000","last_active":"2024-09-10T20:44:24.087619+0000","last_peered":"2024-09-10T20:44:24.087619+0000","last_clean":"2024-09-10T20:44:24.087619+0000","last_became_active":"2024-09-10T20:44:22.767839+0000","last_became_peered":"2024-09-10T20:44:22.767839+0000","last_unstale":"2024-09-10T20:44:24.087619+0000","last_undegraded":"2024-09-10T20:44:24.087619+0000","last_fullsized":"2024-09-10T20:44:24.087619+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:46:24.032732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758742+0000","last_change":"2024-09-10T20:44:22.780060+0000","last_active":"2024-09-10T20:44:23.758742+0000","last_peered":"2024-09-10T20:44:23.758742+0000","last_clean":"2024-09-10T20:44:23.758742+0000","last_became_active":"2024-09-10T20:44:22.779790+0000","last_became_peered":"2024-09-10T20:44:22.779790+0000","last_unstale":"2024-09-10T20:44:23.758742+0000","last_undegraded":"2024-09-10T20:44:23.758742+0000","last_fullsized":"2024-09-10T20:44:23.758742+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:57.653715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087858+0000","last_change":"2024-09-10T20:44:22.784439+0000","last_active":"2024-09-10T20:44:24.087858+0000","last_peered":"2024-09-10T20:44:24.087858+0000","last_clean":"2024-09-10T20:44:24.087858+0000","last_became_active":"2024-09-10T20:44:22.784117+0000","last_became_peered":"2024-09-10T20:44:22.784117+0000","last_unstale":"2024-09-10T20:44:24.087858+0000","last_undegraded":"2024-09-10T20:44:24.087858+0000","last_fullsized":"2024-09-10T20:44:24.087858+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:48.312610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087143+0000","last_change":"2024-09-10T20:44:22.775215+0000","last_active":"2024-09-10T20:44:24.087143+0000","last_peered":"2024-09-10T20:44:24.087143+0000","last_clean":"2024-09-10T20:44:24.087143+0000","last_became_active":"2024-09-10T20:44:22.774902+0000","last_became_peered":"2024-09-10T20:44:22.774902+0000","last_unstale":"2024-09-10T20:44:24.087143+0000","last_undegraded":"2024-09-10T20:44:24.087143+0000","last_fullsized":"2024-09-10T20:44:24.087143+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:56:33.106935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087347+0000","last_change":"2024-09-10T20:44:22.776108+0000","last_active":"2024-09-10T20:44:24.087347+0000","last_peered":"2024-09-10T20:44:24.087347+0000","last_clean":"2024-09-10T20:44:24.087347+0000","last_became_active":"2024-09-10T20:44:22.775515+0000","last_became_peered":"2024-09-10T20:44:22.775515+0000","last_unstale":"2024-09-10T20:44:24.087347+0000","last_undegraded":"2024-09-10T20:44:24.087347+0000","last_fullsized":"2024-09-10T20:44:24.087347+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:25.495179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087394+0000","last_change":"2024-09-10T20:44:22.779148+0000","last_active":"2024-09-10T20:44:24.087394+0000","last_peered":"2024-09-10T20:44:24.087394+0000","last_clean":"2024-09-10T20:44:24.087394+0000","last_became_active":"2024-09-10T20:44:22.775647+0000","last_became_peered":"2024-09-10T20:44:22.775647+0000","last_unstale":"2024-09-10T20:44:24.087394+0000","last_undegraded":"2024-09-10T20:44:24.087394+0000","last_fullsized":"2024-09-10T20:44:24.087394+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:12.364430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.088083+0000","last_change":"2024-09-10T20:44:22.782593+0000","last_active":"2024-09-10T20:44:24.088083+0000","last_peered":"2024-09-10T20:44:24.088083+0000","last_clean":"2024-09-10T20:44:24.088083+0000","last_became_active":"2024-09-10T20:44:22.781859+0000","last_became_peered":"2024-09-10T20:44:22.781859+0000","last_unstale":"2024-09-10T20:44:24.088083+0000","last_undegraded":"2024-09-10T20:44:24.088083+0000","last_fullsized":"2024-09-10T20:44:24.088083+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:35:54.281377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086753+0000","last_change":"2024-09-10T20:44:22.781283+0000","last_active":"2024-09-10T20:44:24.086753+0000","last_peered":"2024-09-10T20:44:24.086753+0000","last_clean":"2024-09-10T20:44:24.086753+0000","last_became_active":"2024-09-10T20:44:22.776499+0000","last_became_peered":"2024-09-10T20:44:22.776499+0000","last_unstale":"2024-09-10T20:44:24.086753+0000","last_undegraded":"2024-09-10T20:44:24.086753+0000","last_fullsized":"2024-09-10T20:44:24.086753+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:41.015623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757854+0000","last_change":"2024-09-10T20:44:22.775649+0000","last_active":"2024-09-10T20:44:23.757854+0000","last_peered":"2024-09-10T20:44:23.757854+0000","last_clean":"2024-09-10T20:44:23.757854+0000","last_became_active":"2024-09-10T20:44:22.775306+0000","last_became_peered":"2024-09-10T20:44:22.775306+0000","last_unstale":"2024-09-10T20:44:23.757854+0000","last_undegraded":"2024-09-10T20:44:23.757854+0000","last_fullsized":"2024-09-10T20:44:23.757854+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:55:04.556498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086519+0000","last_change":"2024-09-10T20:44:22.781266+0000","last_active":"2024-09-10T20:44:24.086519+0000","last_peered":"2024-09-10T20:44:24.086519+0000","last_clean":"2024-09-10T20:44:24.086519+0000","last_became_active":"2024-09-10T20:44:22.780968+0000","last_became_peered":"2024-09-10T20:44:22.780968+0000","last_unstale":"2024-09-10T20:44:24.086519+0000","last_undegraded":"2024-09-10T20:44:24.086519+0000","last_fullsized":"2024-09-10T20:44:24.086519+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:50:21.686992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780732+0000","last_change":"2024-09-10T20:44:22.774185+0000","last_active":"2024-09-10T20:44:23.780732+0000","last_peered":"2024-09-10T20:44:23.780732+0000","last_clean":"2024-09-10T20:44:23.780732+0000","last_became_active":"2024-09-10T20:44:22.773880+0000","last_became_peered":"2024-09-10T20:44:22.773880+0000","last_unstale":"2024-09-10T20:44:23.780732+0000","last_undegraded":"2024-09-10T20:44:23.780732+0000","last_fullsized":"2024-09-10T20:44:23.780732+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:05:16.012861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087532+0000","last_change":"2024-09-10T20:44:22.771997+0000","last_active":"2024-09-10T20:44:24.087532+0000","last_peered":"2024-09-10T20:44:24.087532+0000","last_clean":"2024-09-10T20:44:24.087532+0000","last_became_active":"2024-09-10T20:44:22.770349+0000","last_became_peered":"2024-09-10T20:44:22.770349+0000","last_unstale":"2024-09-10T20:44:24.087532+0000","last_undegraded":"2024-09-10T20:44:24.087532+0000","last_fullsized":"2024-09-10T20:44:24.087532+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:14.942711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780786+0000","last_change":"2024-09-10T20:44:22.775232+0000","last_active":"2024-09-10T20:44:23.780786+0000","last_peered":"2024-09-10T20:44:23.780786+0000","last_clean":"2024-09-10T20:44:23.780786+0000","last_became_active":"2024-09-10T20:44:22.774938+0000","last_became_peered":"2024-09-10T20:44:22.774938+0000","last_unstale":"2024-09-10T20:44:23.780786+0000","last_undegraded":"2024-09-10T20:44:23.780786+0000","last_fullsized":"2024-09-10T20:44:23.780786+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:34.555897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.781000+0000","last_change":"2024-09-10T20:44:22.771794+0000","last_active":"2024-09-10T20:44:23.781000+0000","last_peered":"2024-09-10T20:44:23.781000+0000","last_clean":"2024-09-10T20:44:23.781000+0000","last_became_active":"2024-09-10T20:44:22.771485+0000","last_became_peered":"2024-09-10T20:44:22.771485+0000","last_unstale":"2024-09-10T20:44:23.781000+0000","last_undegraded":"2024-09-10T20:44:23.781000+0000","last_fullsized":"2024-09-10T20:44:23.781000+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:59:25.428047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087163+0000","last_change":"2024-09-10T20:44:22.771437+0000","last_active":"2024-09-10T20:44:24.087163+0000","last_peered":"2024-09-10T20:44:24.087163+0000","last_clean":"2024-09-10T20:44:24.087163+0000","last_became_active":"2024-09-10T20:44:22.771175+0000","last_became_peered":"2024-09-10T20:44:22.771175+0000","last_unstale":"2024-09-10T20:44:24.087163+0000","last_undegraded":"2024-09-10T20:44:24.087163+0000","last_fullsized":"2024-09-10T20:44:24.087163+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:43:10.184638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086766+0000","last_change":"2024-09-10T20:44:22.776740+0000","last_active":"2024-09-10T20:44:24.086766+0000","last_peered":"2024-09-10T20:44:24.086766+0000","last_clean":"2024-09-10T20:44:24.086766+0000","last_became_active":"2024-09-10T20:44:22.776349+0000","last_became_peered":"2024-09-10T20:44:22.776349+0000","last_unstale":"2024-09-10T20:44:24.086766+0000","last_undegraded":"2024-09-10T20:44:24.086766+0000","last_fullsized":"2024-09-10T20:44:24.086766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:39.976689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780692+0000","last_change":"2024-09-10T20:44:22.774165+0000","last_active":"2024-09-10T20:44:23.780692+0000","last_peered":"2024-09-10T20:44:23.780692+0000","last_clean":"2024-09-10T20:44:23.780692+0000","last_became_active":"2024-09-10T20:44:22.773261+0000","last_became_peered":"2024-09-10T20:44:22.773261+0000","last_unstale":"2024-09-10T20:44:23.780692+0000","last_undegraded":"2024-09-10T20:44:23.780692+0000","last_fullsized":"2024-09-10T20:44:23.780692+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:21:27.439538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780587+0000","last_change":"2024-09-10T20:44:22.768754+0000","last_active":"2024-09-10T20:44:23.780587+0000","last_peered":"2024-09-10T20:44:23.780587+0000","last_clean":"2024-09-10T20:44:23.780587+0000","last_became_active":"2024-09-10T20:44:22.768386+0000","last_became_peered":"2024-09-10T20:44:22.768386+0000","last_unstale":"2024-09-10T20:44:23.780587+0000","last_undegraded":"2024-09-10T20:44:23.780587+0000","last_fullsized":"2024-09-10T20:44:23.780587+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:17:01.263759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087360+0000","last_change":"2024-09-10T20:44:22.783800+0000","last_active":"2024-09-10T20:44:24.087360+0000","last_peered":"2024-09-10T20:44:24.087360+0000","last_clean":"2024-09-10T20:44:24.087360+0000","last_became_active":"2024-09-10T20:44:22.783379+0000","last_became_peered":"2024-09-10T20:44:22.783379+0000","last_unstale":"2024-09-10T20:44:24.087360+0000","last_undegraded":"2024-09-10T20:44:24.087360+0000","last_fullsized":"2024-09-10T20:44:24.087360+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:09:47.184726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"126'4","reported_seq":19,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087802+0000","last_change":"2024-09-10T20:44:21.767756+0000","last_active":"2024-09-10T20:44:24.087802+0000","last_peered":"2024-09-10T20:44:24.087802+0000","last_clean":"2024-09-10T20:44:24.087802+0000","last_became_active":"2024-09-10T20:44:21.767447+0000","last_became_peered":"2024-09-10T20:44:21.767447+0000","last_unstale":"2024-09-10T20:44:24.087802+0000","last_undegraded":"2024-09-10T20:44:24.087802+0000","last_fullsized":"2024-09-10T20:44:24.087802+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:22.064675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759665+0000","last_change":"2024-09-10T20:44:22.773682+0000","last_active":"2024-09-10T20:44:23.759665+0000","last_peered":"2024-09-10T20:44:23.759665+0000","last_clean":"2024-09-10T20:44:23.759665+0000","last_became_active":"2024-09-10T20:44:22.773398+0000","last_became_peered":"2024-09-10T20:44:22.773398+0000","last_unstale":"2024-09-10T20:44:23.759665+0000","last_undegraded":"2024-09-10T20:44:23.759665+0000","last_fullsized":"2024-09-10T20:44:23.759665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:35.528603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.1e","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759085+0000","last_change":"2024-09-10T20:44:21.770325+0000","last_active":"2024-09-10T20:44:23.759085+0000","last_peered":"2024-09-10T20:44:23.759085+0000","last_clean":"2024-09-10T20:44:23.759085+0000","last_became_active":"2024-09-10T20:44:21.769922+0000","last_became_peered":"2024-09-10T20:44:21.769922+0000","last_unstale":"2024-09-10T20:44:23.759085+0000","last_undegraded":"2024-09-10T20:44:23.759085+0000","last_fullsized":"2024-09-10T20:44:23.759085+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:27.854358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758851+0000","last_change":"2024-09-10T20:44:22.773347+0000","last_active":"2024-09-10T20:44:23.758851+0000","last_peered":"2024-09-10T20:44:23.758851+0000","last_clean":"2024-09-10T20:44:23.758851+0000","last_became_active":"2024-09-10T20:44:22.773095+0000","last_became_peered":"2024-09-10T20:44:22.773095+0000","last_unstale":"2024-09-10T20:44:23.758851+0000","last_undegraded":"2024-09-10T20:44:23.758851+0000","last_fullsized":"2024-09-10T20:44:23.758851+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:59.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087597+0000","last_change":"2024-09-10T20:44:21.771518+0000","last_active":"2024-09-10T20:44:24.087597+0000","last_peered":"2024-09-10T20:44:24.087597+0000","last_clean":"2024-09-10T20:44:24.087597+0000","last_became_active":"2024-09-10T20:44:21.771197+0000","last_became_peered":"2024-09-10T20:44:21.771197+0000","last_unstale":"2024-09-10T20:44:24.087597+0000","last_undegraded":"2024-09-10T20:44:24.087597+0000","last_fullsized":"2024-09-10T20:44:24.087597+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:41:24.612492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758432+0000","last_change":"2024-09-10T20:44:22.778714+0000","last_active":"2024-09-10T20:44:23.758432+0000","last_peered":"2024-09-10T20:44:23.758432+0000","last_clean":"2024-09-10T20:44:23.758432+0000","last_became_active":"2024-09-10T20:44:22.778415+0000","last_became_peered":"2024-09-10T20:44:22.778415+0000","last_unstale":"2024-09-10T20:44:23.758432+0000","last_undegraded":"2024-09-10T20:44:23.758432+0000","last_fullsized":"2024-09-10T20:44:23.758432+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:49.195233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1c","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087444+0000","last_change":"2024-09-10T20:44:21.768550+0000","last_active":"2024-09-10T20:44:24.087444+0000","last_peered":"2024-09-10T20:44:24.087444+0000","last_clean":"2024-09-10T20:44:24.087444+0000","last_became_active":"2024-09-10T20:44:21.768282+0000","last_became_peered":"2024-09-10T20:44:21.768282+0000","last_unstale":"2024-09-10T20:44:24.087444+0000","last_undegraded":"2024-09-10T20:44:24.087444+0000","last_fullsized":"2024-09-10T20:44:24.087444+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:25.954092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780485+0000","last_change":"2024-09-10T20:44:22.774338+0000","last_active":"2024-09-10T20:44:23.780485+0000","last_peered":"2024-09-10T20:44:23.780485+0000","last_clean":"2024-09-10T20:44:23.780485+0000","last_became_active":"2024-09-10T20:44:22.774054+0000","last_became_peered":"2024-09-10T20:44:22.774054+0000","last_unstale":"2024-09-10T20:44:23.780485+0000","last_undegraded":"2024-09-10T20:44:23.780485+0000","last_fullsized":"2024-09-10T20:44:23.780485+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:09:26.979922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087327+0000","last_change":"2024-09-10T20:44:21.770014+0000","last_active":"2024-09-10T20:44:24.087327+0000","last_peered":"2024-09-10T20:44:24.087327+0000","last_clean":"2024-09-10T20:44:24.087327+0000","last_became_active":"2024-09-10T20:44:21.769731+0000","last_became_peered":"2024-09-10T20:44:21.769731+0000","last_unstale":"2024-09-10T20:44:24.087327+0000","last_undegraded":"2024-09-10T20:44:24.087327+0000","last_fullsized":"2024-09-10T20:44:24.087327+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:31:14.863733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757950+0000","last_change":"2024-09-10T20:44:22.780480+0000","last_active":"2024-09-10T20:44:23.757950+0000","last_peered":"2024-09-10T20:44:23.757950+0000","last_clean":"2024-09-10T20:44:23.757950+0000","last_became_active":"2024-09-10T20:44:22.780199+0000","last_became_peered":"2024-09-10T20:44:22.780199+0000","last_unstale":"2024-09-10T20:44:23.757950+0000","last_undegraded":"2024-09-10T20:44:23.757950+0000","last_fullsized":"2024-09-10T20:44:23.757950+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:50:06.791869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1a","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086534+0000","last_change":"2024-09-10T20:44:21.773854+0000","last_active":"2024-09-10T20:44:24.086534+0000","last_peered":"2024-09-10T20:44:24.086534+0000","last_clean":"2024-09-10T20:44:24.086534+0000","last_became_active":"2024-09-10T20:44:21.773367+0000","last_became_peered":"2024-09-10T20:44:21.773367+0000","last_unstale":"2024-09-10T20:44:24.086534+0000","last_undegraded":"2024-09-10T20:44:24.086534+0000","last_fullsized":"2024-09-10T20:44:24.086534+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:24.363905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780547+0000","last_change":"2024-09-10T20:44:22.768979+0000","last_active":"2024-09-10T20:44:23.780547+0000","last_peered":"2024-09-10T20:44:23.780547+0000","last_clean":"2024-09-10T20:44:23.780547+0000","last_became_active":"2024-09-10T20:44:22.768730+0000","last_became_peered":"2024-09-10T20:44:22.768730+0000","last_unstale":"2024-09-10T20:44:23.780547+0000","last_undegraded":"2024-09-10T20:44:23.780547+0000","last_fullsized":"2024-09-10T20:44:23.780547+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:31.187021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758876+0000","last_change":"2024-09-10T20:44:21.770425+0000","last_active":"2024-09-10T20:44:23.758876+0000","last_peered":"2024-09-10T20:44:23.758876+0000","last_clean":"2024-09-10T20:44:23.758876+0000","last_became_active":"2024-09-10T20:44:21.770132+0000","last_became_peered":"2024-09-10T20:44:21.770132+0000","last_unstale":"2024-09-10T20:44:23.758876+0000","last_undegraded":"2024-09-10T20:44:23.758876+0000","last_fullsized":"2024-09-10T20:44:23.758876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:26.284038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757902+0000","last_change":"2024-09-10T20:44:22.781934+0000","last_active":"2024-09-10T20:44:23.757902+0000","last_peered":"2024-09-10T20:44:23.757902+0000","last_clean":"2024-09-10T20:44:23.757902+0000","last_became_active":"2024-09-10T20:44:22.781666+0000","last_became_peered":"2024-09-10T20:44:22.781666+0000","last_unstale":"2024-09-10T20:44:23.757902+0000","last_undegraded":"2024-09-10T20:44:23.757902+0000","last_fullsized":"2024-09-10T20:44:23.757902+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:45.108544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086506+0000","last_change":"2024-09-10T20:44:21.766731+0000","last_active":"2024-09-10T20:44:24.086506+0000","last_peered":"2024-09-10T20:44:24.086506+0000","last_clean":"2024-09-10T20:44:24.086506+0000","last_became_active":"2024-09-10T20:44:21.766436+0000","last_became_peered":"2024-09-10T20:44:21.766436+0000","last_unstale":"2024-09-10T20:44:24.086506+0000","last_undegraded":"2024-09-10T20:44:24.086506+0000","last_fullsized":"2024-09-10T20:44:24.086506+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:23.402304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758146+0000","last_change":"2024-09-10T20:44:22.775317+0000","last_active":"2024-09-10T20:44:23.758146+0000","last_peered":"2024-09-10T20:44:23.758146+0000","last_clean":"2024-09-10T20:44:23.758146+0000","last_became_active":"2024-09-10T20:44:22.775035+0000","last_became_peered":"2024-09-10T20:44:22.775035+0000","last_unstale":"2024-09-10T20:44:23.758146+0000","last_undegraded":"2024-09-10T20:44:23.758146+0000","last_fullsized":"2024-09-10T20:44:23.758146+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:38:00.722488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759111+0000","last_change":"2024-09-10T20:44:21.768095+0000","last_active":"2024-09-10T20:44:23.759111+0000","last_peered":"2024-09-10T20:44:23.759111+0000","last_clean":"2024-09-10T20:44:23.759111+0000","last_became_active":"2024-09-10T20:44:21.767828+0000","last_became_peered":"2024-09-10T20:44:21.767828+0000","last_unstale":"2024-09-10T20:44:23.759111+0000","last_undegraded":"2024-09-10T20:44:23.759111+0000","last_fullsized":"2024-09-10T20:44:23.759111+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:41.247860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087294+0000","last_change":"2024-09-10T20:44:22.763611+0000","last_active":"2024-09-10T20:44:24.087294+0000","last_peered":"2024-09-10T20:44:24.087294+0000","last_clean":"2024-09-10T20:44:24.087294+0000","last_became_active":"2024-09-10T20:44:22.763283+0000","last_became_peered":"2024-09-10T20:44:22.763283+0000","last_unstale":"2024-09-10T20:44:24.087294+0000","last_undegraded":"2024-09-10T20:44:24.087294+0000","last_fullsized":"2024-09-10T20:44:24.087294+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:30.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087693+0000","last_change":"2024-09-10T20:44:21.764013+0000","last_active":"2024-09-10T20:44:24.087693+0000","last_peered":"2024-09-10T20:44:24.087693+0000","last_clean":"2024-09-10T20:44:24.087693+0000","last_became_active":"2024-09-10T20:44:21.763708+0000","last_became_peered":"2024-09-10T20:44:21.763708+0000","last_unstale":"2024-09-10T20:44:24.087693+0000","last_undegraded":"2024-09-10T20:44:24.087693+0000","last_fullsized":"2024-09-10T20:44:24.087693+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:39:23.629301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.760222+0000","last_change":"2024-09-10T20:44:22.781062+0000","last_active":"2024-09-10T20:44:23.760222+0000","last_peered":"2024-09-10T20:44:23.760222+0000","last_clean":"2024-09-10T20:44:23.760222+0000","last_became_active":"2024-09-10T20:44:22.780769+0000","last_became_peered":"2024-09-10T20:44:22.780769+0000","last_unstale":"2024-09-10T20:44:23.760222+0000","last_undegraded":"2024-09-10T20:44:23.760222+0000","last_fullsized":"2024-09-10T20:44:23.760222+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:03.693931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"126'6","reported_seq":21,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759890+0000","last_change":"2024-09-10T20:44:21.767335+0000","last_active":"2024-09-10T20:44:23.759890+0000","last_peered":"2024-09-10T20:44:23.759890+0000","last_clean":"2024-09-10T20:44:23.759890+0000","last_became_active":"2024-09-10T20:44:21.766311+0000","last_became_peered":"2024-09-10T20:44:21.766311+0000","last_unstale":"2024-09-10T20:44:23.759890+0000","last_undegraded":"2024-09-10T20:44:23.759890+0000","last_fullsized":"2024-09-10T20:44:23.759890+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:58.465551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758379+0000","last_change":"2024-09-10T20:44:22.779030+0000","last_active":"2024-09-10T20:44:23.758379+0000","last_peered":"2024-09-10T20:44:23.758379+0000","last_clean":"2024-09-10T20:44:23.758379+0000","last_became_active":"2024-09-10T20:44:22.778695+0000","last_became_peered":"2024-09-10T20:44:22.778695+0000","last_unstale":"2024-09-10T20:44:23.758379+0000","last_undegraded":"2024-09-10T20:44:23.758379+0000","last_fullsized":"2024-09-10T20:44:23.758379+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:27:05.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.13","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.088018+0000","last_change":"2024-09-10T20:44:21.772059+0000","last_active":"2024-09-10T20:44:24.088018+0000","last_peered":"2024-09-10T20:44:24.088018+0000","last_clean":"2024-09-10T20:44:24.088018+0000","last_became_active":"2024-09-10T20:44:21.771119+0000","last_became_peered":"2024-09-10T20:44:21.771119+0000","last_unstale":"2024-09-10T20:44:24.088018+0000","last_undegraded":"2024-09-10T20:44:24.088018+0000","last_fullsized":"2024-09-10T20:44:24.088018+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:34:49.033713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087168+0000","last_change":"2024-09-10T20:44:22.771216+0000","last_active":"2024-09-10T20:44:24.087168+0000","last_peered":"2024-09-10T20:44:24.087168+0000","last_clean":"2024-09-10T20:44:24.087168+0000","last_became_active":"2024-09-10T20:44:22.770820+0000","last_became_peered":"2024-09-10T20:44:22.770820+0000","last_unstale":"2024-09-10T20:44:24.087168+0000","last_undegraded":"2024-09-10T20:44:24.087168+0000","last_fullsized":"2024-09-10T20:44:24.087168+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:16:15.090111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087076+0000","last_change":"2024-09-10T20:44:21.768468+0000","last_active":"2024-09-10T20:44:24.087076+0000","last_peered":"2024-09-10T20:44:24.087076+0000","last_clean":"2024-09-10T20:44:24.087076+0000","last_became_active":"2024-09-10T20:44:21.767611+0000","last_became_peered":"2024-09-10T20:44:21.767611+0000","last_unstale":"2024-09-10T20:44:24.087076+0000","last_undegraded":"2024-09-10T20:44:24.087076+0000","last_fullsized":"2024-09-10T20:44:24.087076+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:27.854358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757771+0000","last_change":"2024-09-10T20:44:22.776823+0000","last_active":"2024-09-10T20:44:23.757771+0000","last_peered":"2024-09-10T20:44:23.757771+0000","last_clean":"2024-09-10T20:44:23.757771+0000","last_became_active":"2024-09-10T20:44:22.776247+0000","last_became_peered":"2024-09-10T20:44:22.776247+0000","last_unstale":"2024-09-10T20:44:23.757771+0000","last_undegraded":"2024-09-10T20:44:23.757771+0000","last_fullsized":"2024-09-10T20:44:23.757771+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:26.529905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780806+0000","last_change":"2024-09-10T20:44:21.767847+0000","last_active":"2024-09-10T20:44:23.780806+0000","last_peered":"2024-09-10T20:44:23.780806+0000","last_clean":"2024-09-10T20:44:23.780806+0000","last_became_active":"2024-09-10T20:44:21.767558+0000","last_became_peered":"2024-09-10T20:44:21.767558+0000","last_unstale":"2024-09-10T20:44:23.780806+0000","last_undegraded":"2024-09-10T20:44:23.780806+0000","last_fullsized":"2024-09-10T20:44:23.780806+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:25:12.814210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758069+0000","last_change":"2024-09-10T20:44:22.774676+0000","last_active":"2024-09-10T20:44:23.758069+0000","last_peered":"2024-09-10T20:44:23.758069+0000","last_clean":"2024-09-10T20:44:23.758069+0000","last_became_active":"2024-09-10T20:44:22.773386+0000","last_became_peered":"2024-09-10T20:44:22.773386+0000","last_unstale":"2024-09-10T20:44:23.758069+0000","last_undegraded":"2024-09-10T20:44:23.758069+0000","last_fullsized":"2024-09-10T20:44:23.758069+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:36.620341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.10","version":"126'8","reported_seq":23,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087052+0000","last_change":"2024-09-10T20:44:21.767744+0000","last_active":"2024-09-10T20:44:24.087052+0000","last_peered":"2024-09-10T20:44:24.087052+0000","last_clean":"2024-09-10T20:44:24.087052+0000","last_became_active":"2024-09-10T20:44:21.767330+0000","last_became_peered":"2024-09-10T20:44:21.767330+0000","last_unstale":"2024-09-10T20:44:24.087052+0000","last_undegraded":"2024-09-10T20:44:24.087052+0000","last_fullsized":"2024-09-10T20:44:24.087052+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:35:14.023341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087584+0000","last_change":"2024-09-10T20:44:22.777494+0000","last_active":"2024-09-10T20:44:24.087584+0000","last_peered":"2024-09-10T20:44:24.087584+0000","last_clean":"2024-09-10T20:44:24.087584+0000","last_became_active":"2024-09-10T20:44:22.777185+0000","last_became_peered":"2024-09-10T20:44:22.777185+0000","last_unstale":"2024-09-10T20:44:24.087584+0000","last_undegraded":"2024-09-10T20:44:24.087584+0000","last_fullsized":"2024-09-10T20:44:24.087584+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:21:48.026564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.f","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758354+0000","last_change":"2024-09-10T20:44:21.768344+0000","last_active":"2024-09-10T20:44:23.758354+0000","last_peered":"2024-09-10T20:44:23.758354+0000","last_clean":"2024-09-10T20:44:23.758354+0000","last_became_active":"2024-09-10T20:44:21.767681+0000","last_became_peered":"2024-09-10T20:44:21.767681+0000","last_unstale":"2024-09-10T20:44:23.758354+0000","last_undegraded":"2024-09-10T20:44:23.758354+0000","last_fullsized":"2024-09-10T20:44:23.758354+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:31.805375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780931+0000","last_change":"2024-09-10T20:44:22.773318+0000","last_active":"2024-09-10T20:44:23.780931+0000","last_peered":"2024-09-10T20:44:23.780931+0000","last_clean":"2024-09-10T20:44:23.780931+0000","last_became_active":"2024-09-10T20:44:22.772418+0000","last_became_peered":"2024-09-10T20:44:22.772418+0000","last_unstale":"2024-09-10T20:44:23.780931+0000","last_undegraded":"2024-09-10T20:44:23.780931+0000","last_fullsized":"2024-09-10T20:44:23.780931+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:27:30.959623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.e","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087452+0000","last_change":"2024-09-10T20:44:21.768392+0000","last_active":"2024-09-10T20:44:24.087452+0000","last_peered":"2024-09-10T20:44:24.087452+0000","last_clean":"2024-09-10T20:44:24.087452+0000","last_became_active":"2024-09-10T20:44:21.768129+0000","last_became_peered":"2024-09-10T20:44:21.768129+0000","last_unstale":"2024-09-10T20:44:24.087452+0000","last_undegraded":"2024-09-10T20:44:24.087452+0000","last_fullsized":"2024-09-10T20:44:24.087452+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:25:06.406109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087864+0000","last_change":"2024-09-10T20:44:22.781340+0000","last_active":"2024-09-10T20:44:24.087864+0000","last_peered":"2024-09-10T20:44:24.087864+0000","last_clean":"2024-09-10T20:44:24.087864+0000","last_became_active":"2024-09-10T20:44:22.781067+0000","last_became_peered":"2024-09-10T20:44:22.781067+0000","last_unstale":"2024-09-10T20:44:24.087864+0000","last_undegraded":"2024-09-10T20:44:24.087864+0000","last_fullsized":"2024-09-10T20:44:24.087864+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:44.365744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.d","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.760138+0000","last_change":"2024-09-10T20:44:21.771314+0000","last_active":"2024-09-10T20:44:23.760138+0000","last_peered":"2024-09-10T20:44:23.760138+0000","last_clean":"2024-09-10T20:44:23.760138+0000","last_became_active":"2024-09-10T20:44:21.771004+0000","last_became_peered":"2024-09-10T20:44:21.771004+0000","last_unstale":"2024-09-10T20:44:23.760138+0000","last_undegraded":"2024-09-10T20:44:23.760138+0000","last_fullsized":"2024-09-10T20:44:23.760138+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:59:44.503229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759448+0000","last_change":"2024-09-10T20:44:22.778142+0000","last_active":"2024-09-10T20:44:23.759448+0000","last_peered":"2024-09-10T20:44:23.759448+0000","last_clean":"2024-09-10T20:44:23.759448+0000","last_became_active":"2024-09-10T20:44:22.777895+0000","last_became_peered":"2024-09-10T20:44:22.777895+0000","last_unstale":"2024-09-10T20:44:23.759448+0000","last_undegraded":"2024-09-10T20:44:23.759448+0000","last_fullsized":"2024-09-10T20:44:23.759448+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:32:18.646983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.781048+0000","last_change":"2024-09-10T20:44:21.768618+0000","last_active":"2024-09-10T20:44:23.781048+0000","last_peered":"2024-09-10T20:44:23.781048+0000","last_clean":"2024-09-10T20:44:23.781048+0000","last_became_active":"2024-09-10T20:44:21.768079+0000","last_became_peered":"2024-09-10T20:44:21.768079+0000","last_unstale":"2024-09-10T20:44:23.781048+0000","last_undegraded":"2024-09-10T20:44:23.781048+0000","last_fullsized":"2024-09-10T20:44:23.781048+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:38.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086967+0000","last_change":"2024-09-10T20:44:22.761854+0000","last_active":"2024-09-10T20:44:24.086967+0000","last_peered":"2024-09-10T20:44:24.086967+0000","last_clean":"2024-09-10T20:44:24.086967+0000","last_became_active":"2024-09-10T20:44:22.761480+0000","last_became_peered":"2024-09-10T20:44:22.761480+0000","last_unstale":"2024-09-10T20:44:24.086967+0000","last_undegraded":"2024-09-10T20:44:24.086967+0000","last_fullsized":"2024-09-10T20:44:24.086967+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:48.845611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.b","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758792+0000","last_change":"2024-09-10T20:44:21.769450+0000","last_active":"2024-09-10T20:44:23.758792+0000","last_peered":"2024-09-10T20:44:23.758792+0000","last_clean":"2024-09-10T20:44:23.758792+0000","last_became_active":"2024-09-10T20:44:21.769186+0000","last_became_peered":"2024-09-10T20:44:21.769186+0000","last_unstale":"2024-09-10T20:44:23.758792+0000","last_undegraded":"2024-09-10T20:44:23.758792+0000","last_fullsized":"2024-09-10T20:44:23.758792+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:38.953342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780878+0000","last_change":"2024-09-10T20:44:22.773918+0000","last_active":"2024-09-10T20:44:23.780878+0000","last_peered":"2024-09-10T20:44:23.780878+0000","last_clean":"2024-09-10T20:44:23.780878+0000","last_became_active":"2024-09-10T20:44:22.772469+0000","last_became_peered":"2024-09-10T20:44:22.772469+0000","last_unstale":"2024-09-10T20:44:23.780878+0000","last_undegraded":"2024-09-10T20:44:23.780878+0000","last_fullsized":"2024-09-10T20:44:23.780878+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:58.912329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758488+0000","last_change":"2024-09-10T20:44:21.776181+0000","last_active":"2024-09-10T20:44:23.758488+0000","last_peered":"2024-09-10T20:44:23.758488+0000","last_clean":"2024-09-10T20:44:23.758488+0000","last_became_active":"2024-09-10T20:44:21.775856+0000","last_became_peered":"2024-09-10T20:44:21.775856+0000","last_unstale":"2024-09-10T20:44:23.758488+0000","last_undegraded":"2024-09-10T20:44:23.758488+0000","last_fullsized":"2024-09-10T20:44:23.758488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:14.459746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086683+0000","last_change":"2024-09-10T20:44:22.769457+0000","last_active":"2024-09-10T20:44:24.086683+0000","last_peered":"2024-09-10T20:44:24.086683+0000","last_clean":"2024-09-10T20:44:24.086683+0000","last_became_active":"2024-09-10T20:44:22.767195+0000","last_became_peered":"2024-09-10T20:44:22.767195+0000","last_unstale":"2024-09-10T20:44:24.086683+0000","last_undegraded":"2024-09-10T20:44:24.086683+0000","last_fullsized":"2024-09-10T20:44:24.086683+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:22.561965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758003+0000","last_change":"2024-09-10T20:44:21.772028+0000","last_active":"2024-09-10T20:44:23.758003+0000","last_peered":"2024-09-10T20:44:23.758003+0000","last_clean":"2024-09-10T20:44:23.758003+0000","last_became_active":"2024-09-10T20:44:21.771773+0000","last_became_peered":"2024-09-10T20:44:21.771773+0000","last_unstale":"2024-09-10T20:44:23.758003+0000","last_undegraded":"2024-09-10T20:44:23.758003+0000","last_fullsized":"2024-09-10T20:44:23.758003+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:25.914659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780894+0000","last_change":"2024-09-10T20:44:22.774964+0000","last_active":"2024-09-10T20:44:23.780894+0000","last_peered":"2024-09-10T20:44:23.780894+0000","last_clean":"2024-09-10T20:44:23.780894+0000","last_became_active":"2024-09-10T20:44:22.774655+0000","last_became_peered":"2024-09-10T20:44:22.774655+0000","last_unstale":"2024-09-10T20:44:23.780894+0000","last_undegraded":"2024-09-10T20:44:23.780894+0000","last_fullsized":"2024-09-10T20:44:23.780894+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:42.729183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758576+0000","last_change":"2024-09-10T20:44:21.776165+0000","last_active":"2024-09-10T20:44:23.758576+0000","last_peered":"2024-09-10T20:44:23.758576+0000","last_clean":"2024-09-10T20:44:23.758576+0000","last_became_active":"2024-09-10T20:44:21.775792+0000","last_became_peered":"2024-09-10T20:44:21.775792+0000","last_unstale":"2024-09-10T20:44:23.758576+0000","last_undegraded":"2024-09-10T20:44:23.758576+0000","last_fullsized":"2024-09-10T20:44:23.758576+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:52:40.496377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758529+0000","last_change":"2024-09-10T20:44:22.779705+0000","last_active":"2024-09-10T20:44:23.758529+0000","last_peered":"2024-09-10T20:44:23.758529+0000","last_clean":"2024-09-10T20:44:23.758529+0000","last_became_active":"2024-09-10T20:44:22.779425+0000","last_became_peered":"2024-09-10T20:44:22.779425+0000","last_unstale":"2024-09-10T20:44:23.758529+0000","last_undegraded":"2024-09-10T20:44:23.758529+0000","last_fullsized":"2024-09-10T20:44:23.758529+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:44:37.176432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.7","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758485+0000","last_change":"2024-09-10T20:44:21.769893+0000","last_active":"2024-09-10T20:44:23.758485+0000","last_peered":"2024-09-10T20:44:23.758485+0000","last_clean":"2024-09-10T20:44:23.758485+0000","last_became_active":"2024-09-10T20:44:21.769596+0000","last_became_peered":"2024-09-10T20:44:21.769596+0000","last_unstale":"2024-09-10T20:44:23.758485+0000","last_undegraded":"2024-09-10T20:44:23.758485+0000","last_fullsized":"2024-09-10T20:44:23.758485+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:49.801277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758171+0000","last_change":"2024-09-10T20:44:22.780498+0000","last_active":"2024-09-10T20:44:23.758171+0000","last_peered":"2024-09-10T20:44:23.758171+0000","last_clean":"2024-09-10T20:44:23.758171+0000","last_became_active":"2024-09-10T20:44:22.780188+0000","last_became_peered":"2024-09-10T20:44:22.780188+0000","last_unstale":"2024-09-10T20:44:23.758171+0000","last_undegraded":"2024-09-10T20:44:23.758171+0000","last_fullsized":"2024-09-10T20:44:23.758171+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:52:26.615517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"127'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.782119+0000","last_change":"2024-09-10T20:44:21.776259+0000","last_active":"2024-09-10T20:44:23.782119+0000","last_peered":"2024-09-10T20:44:23.782119+0000","last_clean":"2024-09-10T20:44:23.782119+0000","last_became_active":"2024-09-10T20:44:21.775980+0000","last_became_peered":"2024-09-10T20:44:21.775980+0000","last_unstale":"2024-09-10T20:44:23.782119+0000","last_undegraded":"2024-09-10T20:44:23.782119+0000","last_fullsized":"2024-09-10T20:44:23.782119+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:24:49.802885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087450+0000","last_change":"2024-09-10T20:44:22.779447+0000","last_active":"2024-09-10T20:44:24.087450+0000","last_peered":"2024-09-10T20:44:24.087450+0000","last_clean":"2024-09-10T20:44:24.087450+0000","last_became_active":"2024-09-10T20:44:22.778761+0000","last_became_peered":"2024-09-10T20:44:22.778761+0000","last_unstale":"2024-09-10T20:44:24.087450+0000","last_undegraded":"2024-09-10T20:44:24.087450+0000","last_fullsized":"2024-09-10T20:44:24.087450+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:21.670485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.760228+0000","last_change":"2024-09-10T20:44:21.765124+0000","last_active":"2024-09-10T20:44:23.760228+0000","last_peered":"2024-09-10T20:44:23.760228+0000","last_clean":"2024-09-10T20:44:23.760228+0000","last_became_active":"2024-09-10T20:44:21.764859+0000","last_became_peered":"2024-09-10T20:44:21.764859+0000","last_unstale":"2024-09-10T20:44:23.760228+0000","last_undegraded":"2024-09-10T20:44:23.760228+0000","last_fullsized":"2024-09-10T20:44:23.760228+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:40.381619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087272+0000","last_change":"2024-09-10T20:44:22.774054+0000","last_active":"2024-09-10T20:44:24.087272+0000","last_peered":"2024-09-10T20:44:24.087272+0000","last_clean":"2024-09-10T20:44:24.087272+0000","last_became_active":"2024-09-10T20:44:22.773753+0000","last_became_peered":"2024-09-10T20:44:22.773753+0000","last_unstale":"2024-09-10T20:44:24.087272+0000","last_undegraded":"2024-09-10T20:44:24.087272+0000","last_fullsized":"2024-09-10T20:44:24.087272+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:12:40.348430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759983+0000","last_change":"2024-09-10T20:44:21.766619+0000","last_active":"2024-09-10T20:44:23.759983+0000","last_peered":"2024-09-10T20:44:23.759983+0000","last_clean":"2024-09-10T20:44:23.759983+0000","last_became_active":"2024-09-10T20:44:21.765793+0000","last_became_peered":"2024-09-10T20:44:21.765793+0000","last_unstale":"2024-09-10T20:44:23.759983+0000","last_undegraded":"2024-09-10T20:44:23.759983+0000","last_fullsized":"2024-09-10T20:44:23.759983+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:21.551486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759443+0000","last_change":"2024-09-10T20:44:22.771960+0000","last_active":"2024-09-10T20:44:23.759443+0000","last_peered":"2024-09-10T20:44:23.759443+0000","last_clean":"2024-09-10T20:44:23.759443+0000","last_became_active":"2024-09-10T20:44:22.771475+0000","last_became_peered":"2024-09-10T20:44:22.771475+0000","last_unstale":"2024-09-10T20:44:23.759443+0000","last_undegraded":"2024-09-10T20:44:23.759443+0000","last_fullsized":"2024-09-10T20:44:23.759443+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:15:56.464407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780708+0000","last_change":"2024-09-10T20:44:21.770622+0000","last_active":"2024-09-10T20:44:23.780708+0000","last_peered":"2024-09-10T20:44:23.780708+0000","last_clean":"2024-09-10T20:44:23.780708+0000","last_became_active":"2024-09-10T20:44:21.770334+0000","last_became_peered":"2024-09-10T20:44:21.770334+0000","last_unstale":"2024-09-10T20:44:23.780708+0000","last_undegraded":"2024-09-10T20:44:23.780708+0000","last_fullsized":"2024-09-10T20:44:23.780708+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:08:36.823038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758160+0000","last_change":"2024-09-10T20:44:22.774927+0000","last_active":"2024-09-10T20:44:23.758160+0000","last_peered":"2024-09-10T20:44:23.758160+0000","last_clean":"2024-09-10T20:44:23.758160+0000","last_became_active":"2024-09-10T20:44:22.773467+0000","last_became_peered":"2024-09-10T20:44:22.773467+0000","last_unstale":"2024-09-10T20:44:23.758160+0000","last_undegraded":"2024-09-10T20:44:23.758160+0000","last_fullsized":"2024-09-10T20:44:23.758160+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:38.997424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086770+0000","last_change":"2024-09-10T20:44:21.771876+0000","last_active":"2024-09-10T20:44:24.086770+0000","last_peered":"2024-09-10T20:44:24.086770+0000","last_clean":"2024-09-10T20:44:24.086770+0000","last_became_active":"2024-09-10T20:44:21.771586+0000","last_became_peered":"2024-09-10T20:44:21.771586+0000","last_unstale":"2024-09-10T20:44:24.086770+0000","last_undegraded":"2024-09-10T20:44:24.086770+0000","last_fullsized":"2024-09-10T20:44:24.086770+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:45.602221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757816+0000","last_change":"2024-09-10T20:44:22.776508+0000","last_active":"2024-09-10T20:44:23.757816+0000","last_peered":"2024-09-10T20:44:23.757816+0000","last_clean":"2024-09-10T20:44:23.757816+0000","last_became_active":"2024-09-10T20:44:22.775881+0000","last_became_peered":"2024-09-10T20:44:22.775881+0000","last_unstale":"2024-09-10T20:44:23.757816+0000","last_undegraded":"2024-09-10T20:44:23.757816+0000","last_fullsized":"2024-09-10T20:44:23.757816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:22:36.366298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086276+0000","last_change":"2024-09-10T20:44:21.767491+0000","last_active":"2024-09-10T20:44:24.086276+0000","last_peered":"2024-09-10T20:44:24.086276+0000","last_clean":"2024-09-10T20:44:24.086276+0000","last_became_active":"2024-09-10T20:44:21.767165+0000","last_became_peered":"2024-09-10T20:44:21.767165+0000","last_unstale":"2024-09-10T20:44:24.086276+0000","last_undegraded":"2024-09-10T20:44:24.086276+0000","last_fullsized":"2024-09-10T20:44:24.086276+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:38.953342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087033+0000","last_change":"2024-09-10T20:44:22.771646+0000","last_active":"2024-09-10T20:44:24.087033+0000","last_peered":"2024-09-10T20:44:24.087033+0000","last_clean":"2024-09-10T20:44:24.087033+0000","last_became_active":"2024-09-10T20:44:22.771301+0000","last_became_peered":"2024-09-10T20:44:22.771301+0000","last_unstale":"2024-09-10T20:44:24.087033+0000","last_undegraded":"2024-09-10T20:44:24.087033+0000","last_fullsized":"2024-09-10T20:44:24.087033+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:28:35.662737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758607+0000","last_change":"2024-09-10T20:44:21.771204+0000","last_active":"2024-09-10T20:44:23.758607+0000","last_peered":"2024-09-10T20:44:23.758607+0000","last_clean":"2024-09-10T20:44:23.758607+0000","last_became_active":"2024-09-10T20:44:21.770891+0000","last_became_peered":"2024-09-10T20:44:21.770891+0000","last_unstale":"2024-09-10T20:44:23.758607+0000","last_undegraded":"2024-09-10T20:44:23.758607+0000","last_fullsized":"2024-09-10T20:44:23.758607+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:40.339751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759777+0000","last_change":"2024-09-10T20:44:22.772704+0000","last_active":"2024-09-10T20:44:23.759777+0000","last_peered":"2024-09-10T20:44:23.759777+0000","last_clean":"2024-09-10T20:44:23.759777+0000","last_became_active":"2024-09-10T20:44:22.771819+0000","last_became_peered":"2024-09-10T20:44:22.771819+0000","last_unstale":"2024-09-10T20:44:23.759777+0000","last_undegraded":"2024-09-10T20:44:23.759777+0000","last_fullsized":"2024-09-10T20:44:23.759777+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:38.237580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.15","version":"126'3","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087580+0000","last_change":"2024-09-10T20:44:21.773614+0000","last_active":"2024-09-10T20:44:24.087580+0000","last_peered":"2024-09-10T20:44:24.087580+0000","last_clean":"2024-09-10T20:44:24.087580+0000","last_became_active":"2024-09-10T20:44:21.773299+0000","last_became_peered":"2024-09-10T20:44:21.773299+0000","last_unstale":"2024-09-10T20:44:24.087580+0000","last_undegraded":"2024-09-10T20:44:24.087580+0000","last_fullsized":"2024-09-10T20:44:24.087580+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:41.247860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":335,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086582+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:44:24.086582+0000","last_peered":"2024-09-10T20:44:24.086582+0000","last_clean":"2024-09-10T20:44:24.086582+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:44:24.086582+0000","last_undegraded":"2024-09-10T20:44:24.086582+0000","last_fullsized":"2024-09-10T20:44:24.086582+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1572864,"data_stored":693689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1585152,"data_stored":700017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542223,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2191360,"data_stored":1285863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1593344,"data_stored":695905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509521,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36896,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334944,"statfs":{"total":96636764160,"available":96598982656,"internally_reserved":0,"allocated":1581056,"data_stored":694305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1576960,"data_stored":693711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509520,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37476,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334364,"statfs":{"total":96636764160,"available":96598388736,"internally_reserved":0,"allocated":2170880,"data_stored":1284127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509520,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36896,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334944,"statfs":{"total":96636764160,"available":96598982656,"internally_reserved":0,"allocated":1576960,"data_stored":693779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:44:26.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.937+0000 7fa69a872640 1 -- 172.21.15.28:0/935156205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940b4d10 msgr2=0x7fa6940ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.937+0000 7fa69a872640 1 --2- 172.21.15.28:0/935156205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940b4d10 0x7fa6940ce030 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa68c005fd0 tx=0x7fa68c02bb30 comp rx=0 tx=0).stop 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.937+0000 7fa69a872640 1 -- 172.21.15.28:0/935156205 shutdown_connections 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.937+0000 7fa69a872640 1 --2- 172.21.15.28:0/935156205 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa6940b4d10 0x7fa6940ce030 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.937+0000 7fa69a872640 1 --2- 172.21.15.28:0/935156205 >> v2:172.21.15.159:3300/0 conn(0x7fa69415df70 0x7fa6940b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.937+0000 7fa69a872640 1 -- 172.21.15.28:0/935156205 >> 172.21.15.28:0/935156205 conn(0x7fa69406cf00 msgr2=0x7fa69406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.938+0000 7fa69a872640 1 -- 172.21.15.28:0/935156205 shutdown_connections 2024-09-10T20:44:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.938+0000 7fa69a872640 1 -- 172.21.15.28:0/935156205 wait complete. 2024-09-10T20:44:26.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.939+0000 7fa69a872640 1 Processor -- start 2024-09-10T20:44:26.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.939+0000 7fa69a872640 1 -- start start 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa69a872640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940b4d10 0x7fa69415bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa69a872640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa69415a190 0x7fa6941f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa69a872640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa69415df70 0x7fa6940c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa69a872640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6940d0270 con 0x7fa69415a190 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa69a872640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa6940d00f0 con 0x7fa6940b4d10 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa69a872640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa6940d03f0 con 0x7fa69415df70 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa69415a190 0x7fa6941f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa69415a190 0x7fa6941f4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49528/0 (socket says 172.21.15.28:49528) 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 -- 172.21.15.28:0/3825230265 learned_addr learned my addr 172.21.15.28:0/3825230265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa693fff640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940b4d10 0x7fa69415bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa698e0f640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa69415df70 0x7fa6940c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 -- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa69415df70 msgr2=0x7fa6940c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa69415df70 0x7fa6940c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 -- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940b4d10 msgr2=0x7fa69415bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940b4d10 0x7fa69415bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.940+0000 7fa6937fe640 1 -- 172.21.15.28:0/3825230265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6940ca460 con 0x7fa69415a190 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.941+0000 7fa6937fe640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa69415a190 0x7fa6941f4fb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fa68800ea20 tx=0x7fa68800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.941+0000 7fa690ff9640 1 -- 172.21.15.28:0/3825230265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68800cd50 con 0x7fa69415a190 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.941+0000 7fa690ff9640 1 -- 172.21.15.28:0/3825230265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa688004020 con 0x7fa69415a190 2024-09-10T20:44:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.941+0000 7fa690ff9640 1 -- 172.21.15.28:0/3825230265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688004450 con 0x7fa69415a190 2024-09-10T20:44:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.941+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69406e690 con 0x7fa69415a190 2024-09-10T20:44:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.941+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6941f54f0 con 0x7fa69415a190 2024-09-10T20:44:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.942+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6940ce030 con 0x7fa69415a190 2024-09-10T20:44:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.942+0000 7fa690ff9640 1 -- 172.21.15.28:0/3825230265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fa6880035f0 con 0x7fa69415a190 2024-09-10T20:44:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.943+0000 7fa690ff9640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa6800785a0 0x7fa68007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.943+0000 7fa690ff9640 1 -- 172.21.15.28:0/3825230265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fa68809b1a0 con 0x7fa69415a190 2024-09-10T20:44:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.943+0000 7fa693fff640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa6800785a0 0x7fa68007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.943+0000 7fa693fff640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa6800785a0 0x7fa68007aa60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa684007fd0 tx=0x7fa684001410 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:26.947+0000 7fa690ff9640 1 -- 172.21.15.28:0/3825230265 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa688062c70 con 0x7fa69415a190 2024-09-10T20:44:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.131+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa6940ad0e0 con 0x7fa6800785a0 2024-09-10T20:44:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.135+0000 7fa690ff9640 1 -- 172.21.15.28:0/3825230265 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7fa6940ad0e0 con 0x7fa6800785a0 2024-09-10T20:44:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:44:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.138+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa6800785a0 msgr2=0x7fa68007aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.138+0000 7fa69a872640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa6800785a0 0x7fa68007aa60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa684007fd0 tx=0x7fa684001410 comp rx=0 tx=0).stop 2024-09-10T20:44:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa69415a190 msgr2=0x7fa6941f4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa69415a190 0x7fa6941f4fb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fa68800ea20 tx=0x7fa68800eee0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 shutdown_connections 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa6800785a0 0x7fa68007aa60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa69415df70 0x7fa6940c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa69415a190 0x7fa6941f4fb0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 --2- 172.21.15.28:0/3825230265 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa6940b4d10 0x7fa69415bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 >> 172.21.15.28:0/3825230265 conn(0x7fa69406cf00 msgr2=0x7fa694074320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 shutdown_connections 2024-09-10T20:44:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.139+0000 7fa69a872640 1 -- 172.21.15.28:0/3825230265 wait complete. 2024-09-10T20:44:27.154 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-10T20:44:26.319582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279700,"kb_used_data":13524,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754695020,"statfs":{"total":773094113280,"available":772807700480,"internally_reserved":0,"allocated":13848576,"data_stored":6741396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.571909"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087576+0000","last_change":"2024-09-10T20:44:22.784196+0000","last_active":"2024-09-10T20:44:24.087576+0000","last_peered":"2024-09-10T20:44:24.087576+0000","last_clean":"2024-09-10T20:44:24.087576+0000","last_became_active":"2024-09-10T20:44:22.783532+0000","last_became_peered":"2024-09-10T20:44:22.783532+0000","last_unstale":"2024-09-10T20:44:24.087576+0000","last_undegraded":"2024-09-10T20:44:24.087576+0000","last_fullsized":"2024-09-10T20:44:24.087576+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:13.310561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780872+0000","last_change":"2024-09-10T20:44:22.770726+0000","last_active":"2024-09-10T20:44:23.780872+0000","last_peered":"2024-09-10T20:44:23.780872+0000","last_clean":"2024-09-10T20:44:23.780872+0000","last_became_active":"2024-09-10T20:44:22.769964+0000","last_became_peered":"2024-09-10T20:44:22.769964+0000","last_unstale":"2024-09-10T20:44:23.780872+0000","last_undegraded":"2024-09-10T20:44:23.780872+0000","last_fullsized":"2024-09-10T20:44:23.780872+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:20.622191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087667+0000","last_change":"2024-09-10T20:44:22.773512+0000","last_active":"2024-09-10T20:44:24.087667+0000","last_peered":"2024-09-10T20:44:24.087667+0000","last_clean":"2024-09-10T20:44:24.087667+0000","last_became_active":"2024-09-10T20:44:22.773229+0000","last_became_peered":"2024-09-10T20:44:22.773229+0000","last_unstale":"2024-09-10T20:44:24.087667+0000","last_undegraded":"2024-09-10T20:44:24.087667+0000","last_fullsized":"2024-09-10T20:44:24.087667+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:13:53.798166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758415+0000","last_change":"2024-09-10T20:44:22.772619+0000","last_active":"2024-09-10T20:44:23.758415+0000","last_peered":"2024-09-10T20:44:23.758415+0000","last_clean":"2024-09-10T20:44:23.758415+0000","last_became_active":"2024-09-10T20:44:22.772321+0000","last_became_peered":"2024-09-10T20:44:22.772321+0000","last_unstale":"2024-09-10T20:44:23.758415+0000","last_undegraded":"2024-09-10T20:44:23.758415+0000","last_fullsized":"2024-09-10T20:44:23.758415+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:45.686643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087075+0000","last_change":"2024-09-10T20:44:22.773955+0000","last_active":"2024-09-10T20:44:24.087075+0000","last_peered":"2024-09-10T20:44:24.087075+0000","last_clean":"2024-09-10T20:44:24.087075+0000","last_became_active":"2024-09-10T20:44:22.773613+0000","last_became_peered":"2024-09-10T20:44:22.773613+0000","last_unstale":"2024-09-10T20:44:24.087075+0000","last_undegraded":"2024-09-10T20:44:24.087075+0000","last_fullsized":"2024-09-10T20:44:24.087075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:40.210289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759150+0000","last_change":"2024-09-10T20:44:22.771240+0000","last_active":"2024-09-10T20:44:23.759150+0000","last_peered":"2024-09-10T20:44:23.759150+0000","last_clean":"2024-09-10T20:44:23.759150+0000","last_became_active":"2024-09-10T20:44:22.770955+0000","last_became_peered":"2024-09-10T20:44:22.770955+0000","last_unstale":"2024-09-10T20:44:23.759150+0000","last_undegraded":"2024-09-10T20:44:23.759150+0000","last_fullsized":"2024-09-10T20:44:23.759150+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:41.015623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759178+0000","last_change":"2024-09-10T20:44:22.778100+0000","last_active":"2024-09-10T20:44:23.759178+0000","last_peered":"2024-09-10T20:44:23.759178+0000","last_clean":"2024-09-10T20:44:23.759178+0000","last_became_active":"2024-09-10T20:44:22.777800+0000","last_became_peered":"2024-09-10T20:44:22.777800+0000","last_unstale":"2024-09-10T20:44:23.759178+0000","last_undegraded":"2024-09-10T20:44:23.759178+0000","last_fullsized":"2024-09-10T20:44:23.759178+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:56.442874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087060+0000","last_change":"2024-09-10T20:44:22.767533+0000","last_active":"2024-09-10T20:44:24.087060+0000","last_peered":"2024-09-10T20:44:24.087060+0000","last_clean":"2024-09-10T20:44:24.087060+0000","last_became_active":"2024-09-10T20:44:22.767126+0000","last_became_peered":"2024-09-10T20:44:22.767126+0000","last_unstale":"2024-09-10T20:44:24.087060+0000","last_undegraded":"2024-09-10T20:44:24.087060+0000","last_fullsized":"2024-09-10T20:44:24.087060+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:39:05.891827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759124+0000","last_change":"2024-09-10T20:44:22.779905+0000","last_active":"2024-09-10T20:44:23.759124+0000","last_peered":"2024-09-10T20:44:23.759124+0000","last_clean":"2024-09-10T20:44:23.759124+0000","last_became_active":"2024-09-10T20:44:22.779645+0000","last_became_peered":"2024-09-10T20:44:22.779645+0000","last_unstale":"2024-09-10T20:44:23.759124+0000","last_undegraded":"2024-09-10T20:44:23.759124+0000","last_fullsized":"2024-09-10T20:44:23.759124+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:22.561965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.088000+0000","last_change":"2024-09-10T20:44:22.777545+0000","last_active":"2024-09-10T20:44:24.088000+0000","last_peered":"2024-09-10T20:44:24.088000+0000","last_clean":"2024-09-10T20:44:24.088000+0000","last_became_active":"2024-09-10T20:44:22.777243+0000","last_became_peered":"2024-09-10T20:44:22.777243+0000","last_unstale":"2024-09-10T20:44:24.088000+0000","last_undegraded":"2024-09-10T20:44:24.088000+0000","last_fullsized":"2024-09-10T20:44:24.088000+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:54.165792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759444+0000","last_change":"2024-09-10T20:44:22.770149+0000","last_active":"2024-09-10T20:44:23.759444+0000","last_peered":"2024-09-10T20:44:23.759444+0000","last_clean":"2024-09-10T20:44:23.759444+0000","last_became_active":"2024-09-10T20:44:22.769838+0000","last_became_peered":"2024-09-10T20:44:22.769838+0000","last_unstale":"2024-09-10T20:44:23.759444+0000","last_undegraded":"2024-09-10T20:44:23.759444+0000","last_fullsized":"2024-09-10T20:44:23.759444+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:39.976689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087208+0000","last_change":"2024-09-10T20:44:22.776151+0000","last_active":"2024-09-10T20:44:24.087208+0000","last_peered":"2024-09-10T20:44:24.087208+0000","last_clean":"2024-09-10T20:44:24.087208+0000","last_became_active":"2024-09-10T20:44:22.775791+0000","last_became_peered":"2024-09-10T20:44:22.775791+0000","last_unstale":"2024-09-10T20:44:24.087208+0000","last_undegraded":"2024-09-10T20:44:24.087208+0000","last_fullsized":"2024-09-10T20:44:24.087208+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:43:01.456087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087349+0000","last_change":"2024-09-10T20:44:22.774050+0000","last_active":"2024-09-10T20:44:24.087349+0000","last_peered":"2024-09-10T20:44:24.087349+0000","last_clean":"2024-09-10T20:44:24.087349+0000","last_became_active":"2024-09-10T20:44:22.773758+0000","last_became_peered":"2024-09-10T20:44:22.773758+0000","last_unstale":"2024-09-10T20:44:24.087349+0000","last_undegraded":"2024-09-10T20:44:24.087349+0000","last_fullsized":"2024-09-10T20:44:24.087349+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:50.319442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086516+0000","last_change":"2024-09-10T20:44:22.764932+0000","last_active":"2024-09-10T20:44:24.086516+0000","last_peered":"2024-09-10T20:44:24.086516+0000","last_clean":"2024-09-10T20:44:24.086516+0000","last_became_active":"2024-09-10T20:44:22.764628+0000","last_became_peered":"2024-09-10T20:44:22.764628+0000","last_unstale":"2024-09-10T20:44:24.086516+0000","last_undegraded":"2024-09-10T20:44:24.086516+0000","last_fullsized":"2024-09-10T20:44:24.086516+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:59.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087619+0000","last_change":"2024-09-10T20:44:22.768129+0000","last_active":"2024-09-10T20:44:24.087619+0000","last_peered":"2024-09-10T20:44:24.087619+0000","last_clean":"2024-09-10T20:44:24.087619+0000","last_became_active":"2024-09-10T20:44:22.767839+0000","last_became_peered":"2024-09-10T20:44:22.767839+0000","last_unstale":"2024-09-10T20:44:24.087619+0000","last_undegraded":"2024-09-10T20:44:24.087619+0000","last_fullsized":"2024-09-10T20:44:24.087619+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:46:24.032732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758742+0000","last_change":"2024-09-10T20:44:22.780060+0000","last_active":"2024-09-10T20:44:23.758742+0000","last_peered":"2024-09-10T20:44:23.758742+0000","last_clean":"2024-09-10T20:44:23.758742+0000","last_became_active":"2024-09-10T20:44:22.779790+0000","last_became_peered":"2024-09-10T20:44:22.779790+0000","last_unstale":"2024-09-10T20:44:23.758742+0000","last_undegraded":"2024-09-10T20:44:23.758742+0000","last_fullsized":"2024-09-10T20:44:23.758742+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:57.653715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087858+0000","last_change":"2024-09-10T20:44:22.784439+0000","last_active":"2024-09-10T20:44:24.087858+0000","last_peered":"2024-09-10T20:44:24.087858+0000","last_clean":"2024-09-10T20:44:24.087858+0000","last_became_active":"2024-09-10T20:44:22.784117+0000","last_became_peered":"2024-09-10T20:44:22.784117+0000","last_unstale":"2024-09-10T20:44:24.087858+0000","last_undegraded":"2024-09-10T20:44:24.087858+0000","last_fullsized":"2024-09-10T20:44:24.087858+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:48.312610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087143+0000","last_change":"2024-09-10T20:44:22.775215+0000","last_active":"2024-09-10T20:44:24.087143+0000","last_peered":"2024-09-10T20:44:24.087143+0000","last_clean":"2024-09-10T20:44:24.087143+0000","last_became_active":"2024-09-10T20:44:22.774902+0000","last_became_peered":"2024-09-10T20:44:22.774902+0000","last_unstale":"2024-09-10T20:44:24.087143+0000","last_undegraded":"2024-09-10T20:44:24.087143+0000","last_fullsized":"2024-09-10T20:44:24.087143+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:56:33.106935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087347+0000","last_change":"2024-09-10T20:44:22.776108+0000","last_active":"2024-09-10T20:44:24.087347+0000","last_peered":"2024-09-10T20:44:24.087347+0000","last_clean":"2024-09-10T20:44:24.087347+0000","last_became_active":"2024-09-10T20:44:22.775515+0000","last_became_peered":"2024-09-10T20:44:22.775515+0000","last_unstale":"2024-09-10T20:44:24.087347+0000","last_undegraded":"2024-09-10T20:44:24.087347+0000","last_fullsized":"2024-09-10T20:44:24.087347+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:25.495179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087394+0000","last_change":"2024-09-10T20:44:22.779148+0000","last_active":"2024-09-10T20:44:24.087394+0000","last_peered":"2024-09-10T20:44:24.087394+0000","last_clean":"2024-09-10T20:44:24.087394+0000","last_became_active":"2024-09-10T20:44:22.775647+0000","last_became_peered":"2024-09-10T20:44:22.775647+0000","last_unstale":"2024-09-10T20:44:24.087394+0000","last_undegraded":"2024-09-10T20:44:24.087394+0000","last_fullsized":"2024-09-10T20:44:24.087394+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:12.364430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.088083+0000","last_change":"2024-09-10T20:44:22.782593+0000","last_active":"2024-09-10T20:44:24.088083+0000","last_peered":"2024-09-10T20:44:24.088083+0000","last_clean":"2024-09-10T20:44:24.088083+0000","last_became_active":"2024-09-10T20:44:22.781859+0000","last_became_peered":"2024-09-10T20:44:22.781859+0000","last_unstale":"2024-09-10T20:44:24.088083+0000","last_undegraded":"2024-09-10T20:44:24.088083+0000","last_fullsized":"2024-09-10T20:44:24.088083+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:35:54.281377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086753+0000","last_change":"2024-09-10T20:44:22.781283+0000","last_active":"2024-09-10T20:44:24.086753+0000","last_peered":"2024-09-10T20:44:24.086753+0000","last_clean":"2024-09-10T20:44:24.086753+0000","last_became_active":"2024-09-10T20:44:22.776499+0000","last_became_peered":"2024-09-10T20:44:22.776499+0000","last_unstale":"2024-09-10T20:44:24.086753+0000","last_undegraded":"2024-09-10T20:44:24.086753+0000","last_fullsized":"2024-09-10T20:44:24.086753+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:18:41.015623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757854+0000","last_change":"2024-09-10T20:44:22.775649+0000","last_active":"2024-09-10T20:44:23.757854+0000","last_peered":"2024-09-10T20:44:23.757854+0000","last_clean":"2024-09-10T20:44:23.757854+0000","last_became_active":"2024-09-10T20:44:22.775306+0000","last_became_peered":"2024-09-10T20:44:22.775306+0000","last_unstale":"2024-09-10T20:44:23.757854+0000","last_undegraded":"2024-09-10T20:44:23.757854+0000","last_fullsized":"2024-09-10T20:44:23.757854+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:55:04.556498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086519+0000","last_change":"2024-09-10T20:44:22.781266+0000","last_active":"2024-09-10T20:44:24.086519+0000","last_peered":"2024-09-10T20:44:24.086519+0000","last_clean":"2024-09-10T20:44:24.086519+0000","last_became_active":"2024-09-10T20:44:22.780968+0000","last_became_peered":"2024-09-10T20:44:22.780968+0000","last_unstale":"2024-09-10T20:44:24.086519+0000","last_undegraded":"2024-09-10T20:44:24.086519+0000","last_fullsized":"2024-09-10T20:44:24.086519+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:50:21.686992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780732+0000","last_change":"2024-09-10T20:44:22.774185+0000","last_active":"2024-09-10T20:44:23.780732+0000","last_peered":"2024-09-10T20:44:23.780732+0000","last_clean":"2024-09-10T20:44:23.780732+0000","last_became_active":"2024-09-10T20:44:22.773880+0000","last_became_peered":"2024-09-10T20:44:22.773880+0000","last_unstale":"2024-09-10T20:44:23.780732+0000","last_undegraded":"2024-09-10T20:44:23.780732+0000","last_fullsized":"2024-09-10T20:44:23.780732+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:05:16.012861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087532+0000","last_change":"2024-09-10T20:44:22.771997+0000","last_active":"2024-09-10T20:44:24.087532+0000","last_peered":"2024-09-10T20:44:24.087532+0000","last_clean":"2024-09-10T20:44:24.087532+0000","last_became_active":"2024-09-10T20:44:22.770349+0000","last_became_peered":"2024-09-10T20:44:22.770349+0000","last_unstale":"2024-09-10T20:44:24.087532+0000","last_undegraded":"2024-09-10T20:44:24.087532+0000","last_fullsized":"2024-09-10T20:44:24.087532+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:14.942711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780786+0000","last_change":"2024-09-10T20:44:22.775232+0000","last_active":"2024-09-10T20:44:23.780786+0000","last_peered":"2024-09-10T20:44:23.780786+0000","last_clean":"2024-09-10T20:44:23.780786+0000","last_became_active":"2024-09-10T20:44:22.774938+0000","last_became_peered":"2024-09-10T20:44:22.774938+0000","last_unstale":"2024-09-10T20:44:23.780786+0000","last_undegraded":"2024-09-10T20:44:23.780786+0000","last_fullsized":"2024-09-10T20:44:23.780786+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:34.555897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.781000+0000","last_change":"2024-09-10T20:44:22.771794+0000","last_active":"2024-09-10T20:44:23.781000+0000","last_peered":"2024-09-10T20:44:23.781000+0000","last_clean":"2024-09-10T20:44:23.781000+0000","last_became_active":"2024-09-10T20:44:22.771485+0000","last_became_peered":"2024-09-10T20:44:22.771485+0000","last_unstale":"2024-09-10T20:44:23.781000+0000","last_undegraded":"2024-09-10T20:44:23.781000+0000","last_fullsized":"2024-09-10T20:44:23.781000+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:59:25.428047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087163+0000","last_change":"2024-09-10T20:44:22.771437+0000","last_active":"2024-09-10T20:44:24.087163+0000","last_peered":"2024-09-10T20:44:24.087163+0000","last_clean":"2024-09-10T20:44:24.087163+0000","last_became_active":"2024-09-10T20:44:22.771175+0000","last_became_peered":"2024-09-10T20:44:22.771175+0000","last_unstale":"2024-09-10T20:44:24.087163+0000","last_undegraded":"2024-09-10T20:44:24.087163+0000","last_fullsized":"2024-09-10T20:44:24.087163+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:43:10.184638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086766+0000","last_change":"2024-09-10T20:44:22.776740+0000","last_active":"2024-09-10T20:44:24.086766+0000","last_peered":"2024-09-10T20:44:24.086766+0000","last_clean":"2024-09-10T20:44:24.086766+0000","last_became_active":"2024-09-10T20:44:22.776349+0000","last_became_peered":"2024-09-10T20:44:22.776349+0000","last_unstale":"2024-09-10T20:44:24.086766+0000","last_undegraded":"2024-09-10T20:44:24.086766+0000","last_fullsized":"2024-09-10T20:44:24.086766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:39.976689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780692+0000","last_change":"2024-09-10T20:44:22.774165+0000","last_active":"2024-09-10T20:44:23.780692+0000","last_peered":"2024-09-10T20:44:23.780692+0000","last_clean":"2024-09-10T20:44:23.780692+0000","last_became_active":"2024-09-10T20:44:22.773261+0000","last_became_peered":"2024-09-10T20:44:22.773261+0000","last_unstale":"2024-09-10T20:44:23.780692+0000","last_undegraded":"2024-09-10T20:44:23.780692+0000","last_fullsized":"2024-09-10T20:44:23.780692+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:21:27.439538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780587+0000","last_change":"2024-09-10T20:44:22.768754+0000","last_active":"2024-09-10T20:44:23.780587+0000","last_peered":"2024-09-10T20:44:23.780587+0000","last_clean":"2024-09-10T20:44:23.780587+0000","last_became_active":"2024-09-10T20:44:22.768386+0000","last_became_peered":"2024-09-10T20:44:22.768386+0000","last_unstale":"2024-09-10T20:44:23.780587+0000","last_undegraded":"2024-09-10T20:44:23.780587+0000","last_fullsized":"2024-09-10T20:44:23.780587+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:17:01.263759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087360+0000","last_change":"2024-09-10T20:44:22.783800+0000","last_active":"2024-09-10T20:44:24.087360+0000","last_peered":"2024-09-10T20:44:24.087360+0000","last_clean":"2024-09-10T20:44:24.087360+0000","last_became_active":"2024-09-10T20:44:22.783379+0000","last_became_peered":"2024-09-10T20:44:22.783379+0000","last_unstale":"2024-09-10T20:44:24.087360+0000","last_undegraded":"2024-09-10T20:44:24.087360+0000","last_fullsized":"2024-09-10T20:44:24.087360+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:09:47.184726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"126'4","reported_seq":19,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087802+0000","last_change":"2024-09-10T20:44:21.767756+0000","last_active":"2024-09-10T20:44:24.087802+0000","last_peered":"2024-09-10T20:44:24.087802+0000","last_clean":"2024-09-10T20:44:24.087802+0000","last_became_active":"2024-09-10T20:44:21.767447+0000","last_became_peered":"2024-09-10T20:44:21.767447+0000","last_unstale":"2024-09-10T20:44:24.087802+0000","last_undegraded":"2024-09-10T20:44:24.087802+0000","last_fullsized":"2024-09-10T20:44:24.087802+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:22.064675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759665+0000","last_change":"2024-09-10T20:44:22.773682+0000","last_active":"2024-09-10T20:44:23.759665+0000","last_peered":"2024-09-10T20:44:23.759665+0000","last_clean":"2024-09-10T20:44:23.759665+0000","last_became_active":"2024-09-10T20:44:22.773398+0000","last_became_peered":"2024-09-10T20:44:22.773398+0000","last_unstale":"2024-09-10T20:44:23.759665+0000","last_undegraded":"2024-09-10T20:44:23.759665+0000","last_fullsized":"2024-09-10T20:44:23.759665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:35.528603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.1e","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759085+0000","last_change":"2024-09-10T20:44:21.770325+0000","last_active":"2024-09-10T20:44:23.759085+0000","last_peered":"2024-09-10T20:44:23.759085+0000","last_clean":"2024-09-10T20:44:23.759085+0000","last_became_active":"2024-09-10T20:44:21.769922+0000","last_became_peered":"2024-09-10T20:44:21.769922+0000","last_unstale":"2024-09-10T20:44:23.759085+0000","last_undegraded":"2024-09-10T20:44:23.759085+0000","last_fullsized":"2024-09-10T20:44:23.759085+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:27.854358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758851+0000","last_change":"2024-09-10T20:44:22.773347+0000","last_active":"2024-09-10T20:44:23.758851+0000","last_peered":"2024-09-10T20:44:23.758851+0000","last_clean":"2024-09-10T20:44:23.758851+0000","last_became_active":"2024-09-10T20:44:22.773095+0000","last_became_peered":"2024-09-10T20:44:22.773095+0000","last_unstale":"2024-09-10T20:44:23.758851+0000","last_undegraded":"2024-09-10T20:44:23.758851+0000","last_fullsized":"2024-09-10T20:44:23.758851+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:59.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087597+0000","last_change":"2024-09-10T20:44:21.771518+0000","last_active":"2024-09-10T20:44:24.087597+0000","last_peered":"2024-09-10T20:44:24.087597+0000","last_clean":"2024-09-10T20:44:24.087597+0000","last_became_active":"2024-09-10T20:44:21.771197+0000","last_became_peered":"2024-09-10T20:44:21.771197+0000","last_unstale":"2024-09-10T20:44:24.087597+0000","last_undegraded":"2024-09-10T20:44:24.087597+0000","last_fullsized":"2024-09-10T20:44:24.087597+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:41:24.612492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758432+0000","last_change":"2024-09-10T20:44:22.778714+0000","last_active":"2024-09-10T20:44:23.758432+0000","last_peered":"2024-09-10T20:44:23.758432+0000","last_clean":"2024-09-10T20:44:23.758432+0000","last_became_active":"2024-09-10T20:44:22.778415+0000","last_became_peered":"2024-09-10T20:44:22.778415+0000","last_unstale":"2024-09-10T20:44:23.758432+0000","last_undegraded":"2024-09-10T20:44:23.758432+0000","last_fullsized":"2024-09-10T20:44:23.758432+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:49.195233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1c","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087444+0000","last_change":"2024-09-10T20:44:21.768550+0000","last_active":"2024-09-10T20:44:24.087444+0000","last_peered":"2024-09-10T20:44:24.087444+0000","last_clean":"2024-09-10T20:44:24.087444+0000","last_became_active":"2024-09-10T20:44:21.768282+0000","last_became_peered":"2024-09-10T20:44:21.768282+0000","last_unstale":"2024-09-10T20:44:24.087444+0000","last_undegraded":"2024-09-10T20:44:24.087444+0000","last_fullsized":"2024-09-10T20:44:24.087444+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:25.954092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780485+0000","last_change":"2024-09-10T20:44:22.774338+0000","last_active":"2024-09-10T20:44:23.780485+0000","last_peered":"2024-09-10T20:44:23.780485+0000","last_clean":"2024-09-10T20:44:23.780485+0000","last_became_active":"2024-09-10T20:44:22.774054+0000","last_became_peered":"2024-09-10T20:44:22.774054+0000","last_unstale":"2024-09-10T20:44:23.780485+0000","last_undegraded":"2024-09-10T20:44:23.780485+0000","last_fullsized":"2024-09-10T20:44:23.780485+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:09:26.979922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087327+0000","last_change":"2024-09-10T20:44:21.770014+0000","last_active":"2024-09-10T20:44:24.087327+0000","last_peered":"2024-09-10T20:44:24.087327+0000","last_clean":"2024-09-10T20:44:24.087327+0000","last_became_active":"2024-09-10T20:44:21.769731+0000","last_became_peered":"2024-09-10T20:44:21.769731+0000","last_unstale":"2024-09-10T20:44:24.087327+0000","last_undegraded":"2024-09-10T20:44:24.087327+0000","last_fullsized":"2024-09-10T20:44:24.087327+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:31:14.863733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757950+0000","last_change":"2024-09-10T20:44:22.780480+0000","last_active":"2024-09-10T20:44:23.757950+0000","last_peered":"2024-09-10T20:44:23.757950+0000","last_clean":"2024-09-10T20:44:23.757950+0000","last_became_active":"2024-09-10T20:44:22.780199+0000","last_became_peered":"2024-09-10T20:44:22.780199+0000","last_unstale":"2024-09-10T20:44:23.757950+0000","last_undegraded":"2024-09-10T20:44:23.757950+0000","last_fullsized":"2024-09-10T20:44:23.757950+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:50:06.791869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1a","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086534+0000","last_change":"2024-09-10T20:44:21.773854+0000","last_active":"2024-09-10T20:44:24.086534+0000","last_peered":"2024-09-10T20:44:24.086534+0000","last_clean":"2024-09-10T20:44:24.086534+0000","last_became_active":"2024-09-10T20:44:21.773367+0000","last_became_peered":"2024-09-10T20:44:21.773367+0000","last_unstale":"2024-09-10T20:44:24.086534+0000","last_undegraded":"2024-09-10T20:44:24.086534+0000","last_fullsized":"2024-09-10T20:44:24.086534+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:24.363905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780547+0000","last_change":"2024-09-10T20:44:22.768979+0000","last_active":"2024-09-10T20:44:23.780547+0000","last_peered":"2024-09-10T20:44:23.780547+0000","last_clean":"2024-09-10T20:44:23.780547+0000","last_became_active":"2024-09-10T20:44:22.768730+0000","last_became_peered":"2024-09-10T20:44:22.768730+0000","last_unstale":"2024-09-10T20:44:23.780547+0000","last_undegraded":"2024-09-10T20:44:23.780547+0000","last_fullsized":"2024-09-10T20:44:23.780547+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:31.187021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758876+0000","last_change":"2024-09-10T20:44:21.770425+0000","last_active":"2024-09-10T20:44:23.758876+0000","last_peered":"2024-09-10T20:44:23.758876+0000","last_clean":"2024-09-10T20:44:23.758876+0000","last_became_active":"2024-09-10T20:44:21.770132+0000","last_became_peered":"2024-09-10T20:44:21.770132+0000","last_unstale":"2024-09-10T20:44:23.758876+0000","last_undegraded":"2024-09-10T20:44:23.758876+0000","last_fullsized":"2024-09-10T20:44:23.758876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:26.284038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757902+0000","last_change":"2024-09-10T20:44:22.781934+0000","last_active":"2024-09-10T20:44:23.757902+0000","last_peered":"2024-09-10T20:44:23.757902+0000","last_clean":"2024-09-10T20:44:23.757902+0000","last_became_active":"2024-09-10T20:44:22.781666+0000","last_became_peered":"2024-09-10T20:44:22.781666+0000","last_unstale":"2024-09-10T20:44:23.757902+0000","last_undegraded":"2024-09-10T20:44:23.757902+0000","last_fullsized":"2024-09-10T20:44:23.757902+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:45.108544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086506+0000","last_change":"2024-09-10T20:44:21.766731+0000","last_active":"2024-09-10T20:44:24.086506+0000","last_peered":"2024-09-10T20:44:24.086506+0000","last_clean":"2024-09-10T20:44:24.086506+0000","last_became_active":"2024-09-10T20:44:21.766436+0000","last_became_peered":"2024-09-10T20:44:21.766436+0000","last_unstale":"2024-09-10T20:44:24.086506+0000","last_undegraded":"2024-09-10T20:44:24.086506+0000","last_fullsized":"2024-09-10T20:44:24.086506+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:23.402304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758146+0000","last_change":"2024-09-10T20:44:22.775317+0000","last_active":"2024-09-10T20:44:23.758146+0000","last_peered":"2024-09-10T20:44:23.758146+0000","last_clean":"2024-09-10T20:44:23.758146+0000","last_became_active":"2024-09-10T20:44:22.775035+0000","last_became_peered":"2024-09-10T20:44:22.775035+0000","last_unstale":"2024-09-10T20:44:23.758146+0000","last_undegraded":"2024-09-10T20:44:23.758146+0000","last_fullsized":"2024-09-10T20:44:23.758146+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:38:00.722488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759111+0000","last_change":"2024-09-10T20:44:21.768095+0000","last_active":"2024-09-10T20:44:23.759111+0000","last_peered":"2024-09-10T20:44:23.759111+0000","last_clean":"2024-09-10T20:44:23.759111+0000","last_became_active":"2024-09-10T20:44:21.767828+0000","last_became_peered":"2024-09-10T20:44:21.767828+0000","last_unstale":"2024-09-10T20:44:23.759111+0000","last_undegraded":"2024-09-10T20:44:23.759111+0000","last_fullsized":"2024-09-10T20:44:23.759111+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:41.247860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087294+0000","last_change":"2024-09-10T20:44:22.763611+0000","last_active":"2024-09-10T20:44:24.087294+0000","last_peered":"2024-09-10T20:44:24.087294+0000","last_clean":"2024-09-10T20:44:24.087294+0000","last_became_active":"2024-09-10T20:44:22.763283+0000","last_became_peered":"2024-09-10T20:44:22.763283+0000","last_unstale":"2024-09-10T20:44:24.087294+0000","last_undegraded":"2024-09-10T20:44:24.087294+0000","last_fullsized":"2024-09-10T20:44:24.087294+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:30.738717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087693+0000","last_change":"2024-09-10T20:44:21.764013+0000","last_active":"2024-09-10T20:44:24.087693+0000","last_peered":"2024-09-10T20:44:24.087693+0000","last_clean":"2024-09-10T20:44:24.087693+0000","last_became_active":"2024-09-10T20:44:21.763708+0000","last_became_peered":"2024-09-10T20:44:21.763708+0000","last_unstale":"2024-09-10T20:44:24.087693+0000","last_undegraded":"2024-09-10T20:44:24.087693+0000","last_fullsized":"2024-09-10T20:44:24.087693+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:39:23.629301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.760222+0000","last_change":"2024-09-10T20:44:22.781062+0000","last_active":"2024-09-10T20:44:23.760222+0000","last_peered":"2024-09-10T20:44:23.760222+0000","last_clean":"2024-09-10T20:44:23.760222+0000","last_became_active":"2024-09-10T20:44:22.780769+0000","last_became_peered":"2024-09-10T20:44:22.780769+0000","last_unstale":"2024-09-10T20:44:23.760222+0000","last_undegraded":"2024-09-10T20:44:23.760222+0000","last_fullsized":"2024-09-10T20:44:23.760222+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:03.693931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"126'6","reported_seq":21,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759890+0000","last_change":"2024-09-10T20:44:21.767335+0000","last_active":"2024-09-10T20:44:23.759890+0000","last_peered":"2024-09-10T20:44:23.759890+0000","last_clean":"2024-09-10T20:44:23.759890+0000","last_became_active":"2024-09-10T20:44:21.766311+0000","last_became_peered":"2024-09-10T20:44:21.766311+0000","last_unstale":"2024-09-10T20:44:23.759890+0000","last_undegraded":"2024-09-10T20:44:23.759890+0000","last_fullsized":"2024-09-10T20:44:23.759890+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:58.465551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758379+0000","last_change":"2024-09-10T20:44:22.779030+0000","last_active":"2024-09-10T20:44:23.758379+0000","last_peered":"2024-09-10T20:44:23.758379+0000","last_clean":"2024-09-10T20:44:23.758379+0000","last_became_active":"2024-09-10T20:44:22.778695+0000","last_became_peered":"2024-09-10T20:44:22.778695+0000","last_unstale":"2024-09-10T20:44:23.758379+0000","last_undegraded":"2024-09-10T20:44:23.758379+0000","last_fullsized":"2024-09-10T20:44:23.758379+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:27:05.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.13","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.088018+0000","last_change":"2024-09-10T20:44:21.772059+0000","last_active":"2024-09-10T20:44:24.088018+0000","last_peered":"2024-09-10T20:44:24.088018+0000","last_clean":"2024-09-10T20:44:24.088018+0000","last_became_active":"2024-09-10T20:44:21.771119+0000","last_became_peered":"2024-09-10T20:44:21.771119+0000","last_unstale":"2024-09-10T20:44:24.088018+0000","last_undegraded":"2024-09-10T20:44:24.088018+0000","last_fullsized":"2024-09-10T20:44:24.088018+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:34:49.033713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087168+0000","last_change":"2024-09-10T20:44:22.771216+0000","last_active":"2024-09-10T20:44:24.087168+0000","last_peered":"2024-09-10T20:44:24.087168+0000","last_clean":"2024-09-10T20:44:24.087168+0000","last_became_active":"2024-09-10T20:44:22.770820+0000","last_became_peered":"2024-09-10T20:44:22.770820+0000","last_unstale":"2024-09-10T20:44:24.087168+0000","last_undegraded":"2024-09-10T20:44:24.087168+0000","last_fullsized":"2024-09-10T20:44:24.087168+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:16:15.090111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087076+0000","last_change":"2024-09-10T20:44:21.768468+0000","last_active":"2024-09-10T20:44:24.087076+0000","last_peered":"2024-09-10T20:44:24.087076+0000","last_clean":"2024-09-10T20:44:24.087076+0000","last_became_active":"2024-09-10T20:44:21.767611+0000","last_became_peered":"2024-09-10T20:44:21.767611+0000","last_unstale":"2024-09-10T20:44:24.087076+0000","last_undegraded":"2024-09-10T20:44:24.087076+0000","last_fullsized":"2024-09-10T20:44:24.087076+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:27.854358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757771+0000","last_change":"2024-09-10T20:44:22.776823+0000","last_active":"2024-09-10T20:44:23.757771+0000","last_peered":"2024-09-10T20:44:23.757771+0000","last_clean":"2024-09-10T20:44:23.757771+0000","last_became_active":"2024-09-10T20:44:22.776247+0000","last_became_peered":"2024-09-10T20:44:22.776247+0000","last_unstale":"2024-09-10T20:44:23.757771+0000","last_undegraded":"2024-09-10T20:44:23.757771+0000","last_fullsized":"2024-09-10T20:44:23.757771+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:26.529905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780806+0000","last_change":"2024-09-10T20:44:21.767847+0000","last_active":"2024-09-10T20:44:23.780806+0000","last_peered":"2024-09-10T20:44:23.780806+0000","last_clean":"2024-09-10T20:44:23.780806+0000","last_became_active":"2024-09-10T20:44:21.767558+0000","last_became_peered":"2024-09-10T20:44:21.767558+0000","last_unstale":"2024-09-10T20:44:23.780806+0000","last_undegraded":"2024-09-10T20:44:23.780806+0000","last_fullsized":"2024-09-10T20:44:23.780806+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:25:12.814210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758069+0000","last_change":"2024-09-10T20:44:22.774676+0000","last_active":"2024-09-10T20:44:23.758069+0000","last_peered":"2024-09-10T20:44:23.758069+0000","last_clean":"2024-09-10T20:44:23.758069+0000","last_became_active":"2024-09-10T20:44:22.773386+0000","last_became_peered":"2024-09-10T20:44:22.773386+0000","last_unstale":"2024-09-10T20:44:23.758069+0000","last_undegraded":"2024-09-10T20:44:23.758069+0000","last_fullsized":"2024-09-10T20:44:23.758069+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:36.620341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.10","version":"126'8","reported_seq":23,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087052+0000","last_change":"2024-09-10T20:44:21.767744+0000","last_active":"2024-09-10T20:44:24.087052+0000","last_peered":"2024-09-10T20:44:24.087052+0000","last_clean":"2024-09-10T20:44:24.087052+0000","last_became_active":"2024-09-10T20:44:21.767330+0000","last_became_peered":"2024-09-10T20:44:21.767330+0000","last_unstale":"2024-09-10T20:44:24.087052+0000","last_undegraded":"2024-09-10T20:44:24.087052+0000","last_fullsized":"2024-09-10T20:44:24.087052+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:35:14.023341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087584+0000","last_change":"2024-09-10T20:44:22.777494+0000","last_active":"2024-09-10T20:44:24.087584+0000","last_peered":"2024-09-10T20:44:24.087584+0000","last_clean":"2024-09-10T20:44:24.087584+0000","last_became_active":"2024-09-10T20:44:22.777185+0000","last_became_peered":"2024-09-10T20:44:22.777185+0000","last_unstale":"2024-09-10T20:44:24.087584+0000","last_undegraded":"2024-09-10T20:44:24.087584+0000","last_fullsized":"2024-09-10T20:44:24.087584+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:21:48.026564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.f","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758354+0000","last_change":"2024-09-10T20:44:21.768344+0000","last_active":"2024-09-10T20:44:23.758354+0000","last_peered":"2024-09-10T20:44:23.758354+0000","last_clean":"2024-09-10T20:44:23.758354+0000","last_became_active":"2024-09-10T20:44:21.767681+0000","last_became_peered":"2024-09-10T20:44:21.767681+0000","last_unstale":"2024-09-10T20:44:23.758354+0000","last_undegraded":"2024-09-10T20:44:23.758354+0000","last_fullsized":"2024-09-10T20:44:23.758354+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:31.805375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780931+0000","last_change":"2024-09-10T20:44:22.773318+0000","last_active":"2024-09-10T20:44:23.780931+0000","last_peered":"2024-09-10T20:44:23.780931+0000","last_clean":"2024-09-10T20:44:23.780931+0000","last_became_active":"2024-09-10T20:44:22.772418+0000","last_became_peered":"2024-09-10T20:44:22.772418+0000","last_unstale":"2024-09-10T20:44:23.780931+0000","last_undegraded":"2024-09-10T20:44:23.780931+0000","last_fullsized":"2024-09-10T20:44:23.780931+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:27:30.959623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.e","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087452+0000","last_change":"2024-09-10T20:44:21.768392+0000","last_active":"2024-09-10T20:44:24.087452+0000","last_peered":"2024-09-10T20:44:24.087452+0000","last_clean":"2024-09-10T20:44:24.087452+0000","last_became_active":"2024-09-10T20:44:21.768129+0000","last_became_peered":"2024-09-10T20:44:21.768129+0000","last_unstale":"2024-09-10T20:44:24.087452+0000","last_undegraded":"2024-09-10T20:44:24.087452+0000","last_fullsized":"2024-09-10T20:44:24.087452+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:25:06.406109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087864+0000","last_change":"2024-09-10T20:44:22.781340+0000","last_active":"2024-09-10T20:44:24.087864+0000","last_peered":"2024-09-10T20:44:24.087864+0000","last_clean":"2024-09-10T20:44:24.087864+0000","last_became_active":"2024-09-10T20:44:22.781067+0000","last_became_peered":"2024-09-10T20:44:22.781067+0000","last_unstale":"2024-09-10T20:44:24.087864+0000","last_undegraded":"2024-09-10T20:44:24.087864+0000","last_fullsized":"2024-09-10T20:44:24.087864+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:44.365744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.d","version":"126'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.760138+0000","last_change":"2024-09-10T20:44:21.771314+0000","last_active":"2024-09-10T20:44:23.760138+0000","last_peered":"2024-09-10T20:44:23.760138+0000","last_clean":"2024-09-10T20:44:23.760138+0000","last_became_active":"2024-09-10T20:44:21.771004+0000","last_became_peered":"2024-09-10T20:44:21.771004+0000","last_unstale":"2024-09-10T20:44:23.760138+0000","last_undegraded":"2024-09-10T20:44:23.760138+0000","last_fullsized":"2024-09-10T20:44:23.760138+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:59:44.503229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759448+0000","last_change":"2024-09-10T20:44:22.778142+0000","last_active":"2024-09-10T20:44:23.759448+0000","last_peered":"2024-09-10T20:44:23.759448+0000","last_clean":"2024-09-10T20:44:23.759448+0000","last_became_active":"2024-09-10T20:44:22.777895+0000","last_became_peered":"2024-09-10T20:44:22.777895+0000","last_unstale":"2024-09-10T20:44:23.759448+0000","last_undegraded":"2024-09-10T20:44:23.759448+0000","last_fullsized":"2024-09-10T20:44:23.759448+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:32:18.646983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.781048+0000","last_change":"2024-09-10T20:44:21.768618+0000","last_active":"2024-09-10T20:44:23.781048+0000","last_peered":"2024-09-10T20:44:23.781048+0000","last_clean":"2024-09-10T20:44:23.781048+0000","last_became_active":"2024-09-10T20:44:21.768079+0000","last_became_peered":"2024-09-10T20:44:21.768079+0000","last_unstale":"2024-09-10T20:44:23.781048+0000","last_undegraded":"2024-09-10T20:44:23.781048+0000","last_fullsized":"2024-09-10T20:44:23.781048+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:38.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086967+0000","last_change":"2024-09-10T20:44:22.761854+0000","last_active":"2024-09-10T20:44:24.086967+0000","last_peered":"2024-09-10T20:44:24.086967+0000","last_clean":"2024-09-10T20:44:24.086967+0000","last_became_active":"2024-09-10T20:44:22.761480+0000","last_became_peered":"2024-09-10T20:44:22.761480+0000","last_unstale":"2024-09-10T20:44:24.086967+0000","last_undegraded":"2024-09-10T20:44:24.086967+0000","last_fullsized":"2024-09-10T20:44:24.086967+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:48.845611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.b","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758792+0000","last_change":"2024-09-10T20:44:21.769450+0000","last_active":"2024-09-10T20:44:23.758792+0000","last_peered":"2024-09-10T20:44:23.758792+0000","last_clean":"2024-09-10T20:44:23.758792+0000","last_became_active":"2024-09-10T20:44:21.769186+0000","last_became_peered":"2024-09-10T20:44:21.769186+0000","last_unstale":"2024-09-10T20:44:23.758792+0000","last_undegraded":"2024-09-10T20:44:23.758792+0000","last_fullsized":"2024-09-10T20:44:23.758792+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:38.953342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780878+0000","last_change":"2024-09-10T20:44:22.773918+0000","last_active":"2024-09-10T20:44:23.780878+0000","last_peered":"2024-09-10T20:44:23.780878+0000","last_clean":"2024-09-10T20:44:23.780878+0000","last_became_active":"2024-09-10T20:44:22.772469+0000","last_became_peered":"2024-09-10T20:44:22.772469+0000","last_unstale":"2024-09-10T20:44:23.780878+0000","last_undegraded":"2024-09-10T20:44:23.780878+0000","last_fullsized":"2024-09-10T20:44:23.780878+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:58.912329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758488+0000","last_change":"2024-09-10T20:44:21.776181+0000","last_active":"2024-09-10T20:44:23.758488+0000","last_peered":"2024-09-10T20:44:23.758488+0000","last_clean":"2024-09-10T20:44:23.758488+0000","last_became_active":"2024-09-10T20:44:21.775856+0000","last_became_peered":"2024-09-10T20:44:21.775856+0000","last_unstale":"2024-09-10T20:44:23.758488+0000","last_undegraded":"2024-09-10T20:44:23.758488+0000","last_fullsized":"2024-09-10T20:44:23.758488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:14.459746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086683+0000","last_change":"2024-09-10T20:44:22.769457+0000","last_active":"2024-09-10T20:44:24.086683+0000","last_peered":"2024-09-10T20:44:24.086683+0000","last_clean":"2024-09-10T20:44:24.086683+0000","last_became_active":"2024-09-10T20:44:22.767195+0000","last_became_peered":"2024-09-10T20:44:22.767195+0000","last_unstale":"2024-09-10T20:44:24.086683+0000","last_undegraded":"2024-09-10T20:44:24.086683+0000","last_fullsized":"2024-09-10T20:44:24.086683+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:22.561965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758003+0000","last_change":"2024-09-10T20:44:21.772028+0000","last_active":"2024-09-10T20:44:23.758003+0000","last_peered":"2024-09-10T20:44:23.758003+0000","last_clean":"2024-09-10T20:44:23.758003+0000","last_became_active":"2024-09-10T20:44:21.771773+0000","last_became_peered":"2024-09-10T20:44:21.771773+0000","last_unstale":"2024-09-10T20:44:23.758003+0000","last_undegraded":"2024-09-10T20:44:23.758003+0000","last_fullsized":"2024-09-10T20:44:23.758003+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:25.914659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780894+0000","last_change":"2024-09-10T20:44:22.774964+0000","last_active":"2024-09-10T20:44:23.780894+0000","last_peered":"2024-09-10T20:44:23.780894+0000","last_clean":"2024-09-10T20:44:23.780894+0000","last_became_active":"2024-09-10T20:44:22.774655+0000","last_became_peered":"2024-09-10T20:44:22.774655+0000","last_unstale":"2024-09-10T20:44:23.780894+0000","last_undegraded":"2024-09-10T20:44:23.780894+0000","last_fullsized":"2024-09-10T20:44:23.780894+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:42.729183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758576+0000","last_change":"2024-09-10T20:44:21.776165+0000","last_active":"2024-09-10T20:44:23.758576+0000","last_peered":"2024-09-10T20:44:23.758576+0000","last_clean":"2024-09-10T20:44:23.758576+0000","last_became_active":"2024-09-10T20:44:21.775792+0000","last_became_peered":"2024-09-10T20:44:21.775792+0000","last_unstale":"2024-09-10T20:44:23.758576+0000","last_undegraded":"2024-09-10T20:44:23.758576+0000","last_fullsized":"2024-09-10T20:44:23.758576+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:52:40.496377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758529+0000","last_change":"2024-09-10T20:44:22.779705+0000","last_active":"2024-09-10T20:44:23.758529+0000","last_peered":"2024-09-10T20:44:23.758529+0000","last_clean":"2024-09-10T20:44:23.758529+0000","last_became_active":"2024-09-10T20:44:22.779425+0000","last_became_peered":"2024-09-10T20:44:22.779425+0000","last_unstale":"2024-09-10T20:44:23.758529+0000","last_undegraded":"2024-09-10T20:44:23.758529+0000","last_fullsized":"2024-09-10T20:44:23.758529+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:44:37.176432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.7","version":"126'2","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758485+0000","last_change":"2024-09-10T20:44:21.769893+0000","last_active":"2024-09-10T20:44:23.758485+0000","last_peered":"2024-09-10T20:44:23.758485+0000","last_clean":"2024-09-10T20:44:23.758485+0000","last_became_active":"2024-09-10T20:44:21.769596+0000","last_became_peered":"2024-09-10T20:44:21.769596+0000","last_unstale":"2024-09-10T20:44:23.758485+0000","last_undegraded":"2024-09-10T20:44:23.758485+0000","last_fullsized":"2024-09-10T20:44:23.758485+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:01:49.801277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758171+0000","last_change":"2024-09-10T20:44:22.780498+0000","last_active":"2024-09-10T20:44:23.758171+0000","last_peered":"2024-09-10T20:44:23.758171+0000","last_clean":"2024-09-10T20:44:23.758171+0000","last_became_active":"2024-09-10T20:44:22.780188+0000","last_became_peered":"2024-09-10T20:44:22.780188+0000","last_unstale":"2024-09-10T20:44:23.758171+0000","last_undegraded":"2024-09-10T20:44:23.758171+0000","last_fullsized":"2024-09-10T20:44:23.758171+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:52:26.615517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"127'1","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.782119+0000","last_change":"2024-09-10T20:44:21.776259+0000","last_active":"2024-09-10T20:44:23.782119+0000","last_peered":"2024-09-10T20:44:23.782119+0000","last_clean":"2024-09-10T20:44:23.782119+0000","last_became_active":"2024-09-10T20:44:21.775980+0000","last_became_peered":"2024-09-10T20:44:21.775980+0000","last_unstale":"2024-09-10T20:44:23.782119+0000","last_undegraded":"2024-09-10T20:44:23.782119+0000","last_fullsized":"2024-09-10T20:44:23.782119+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:24:49.802885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087450+0000","last_change":"2024-09-10T20:44:22.779447+0000","last_active":"2024-09-10T20:44:24.087450+0000","last_peered":"2024-09-10T20:44:24.087450+0000","last_clean":"2024-09-10T20:44:24.087450+0000","last_became_active":"2024-09-10T20:44:22.778761+0000","last_became_peered":"2024-09-10T20:44:22.778761+0000","last_unstale":"2024-09-10T20:44:24.087450+0000","last_undegraded":"2024-09-10T20:44:24.087450+0000","last_fullsized":"2024-09-10T20:44:24.087450+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:21.670485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.760228+0000","last_change":"2024-09-10T20:44:21.765124+0000","last_active":"2024-09-10T20:44:23.760228+0000","last_peered":"2024-09-10T20:44:23.760228+0000","last_clean":"2024-09-10T20:44:23.760228+0000","last_became_active":"2024-09-10T20:44:21.764859+0000","last_became_peered":"2024-09-10T20:44:21.764859+0000","last_unstale":"2024-09-10T20:44:23.760228+0000","last_undegraded":"2024-09-10T20:44:23.760228+0000","last_fullsized":"2024-09-10T20:44:23.760228+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:40.381619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087272+0000","last_change":"2024-09-10T20:44:22.774054+0000","last_active":"2024-09-10T20:44:24.087272+0000","last_peered":"2024-09-10T20:44:24.087272+0000","last_clean":"2024-09-10T20:44:24.087272+0000","last_became_active":"2024-09-10T20:44:22.773753+0000","last_became_peered":"2024-09-10T20:44:22.773753+0000","last_unstale":"2024-09-10T20:44:24.087272+0000","last_undegraded":"2024-09-10T20:44:24.087272+0000","last_fullsized":"2024-09-10T20:44:24.087272+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:12:40.348430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759983+0000","last_change":"2024-09-10T20:44:21.766619+0000","last_active":"2024-09-10T20:44:23.759983+0000","last_peered":"2024-09-10T20:44:23.759983+0000","last_clean":"2024-09-10T20:44:23.759983+0000","last_became_active":"2024-09-10T20:44:21.765793+0000","last_became_peered":"2024-09-10T20:44:21.765793+0000","last_unstale":"2024-09-10T20:44:23.759983+0000","last_undegraded":"2024-09-10T20:44:23.759983+0000","last_fullsized":"2024-09-10T20:44:23.759983+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:21.551486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759443+0000","last_change":"2024-09-10T20:44:22.771960+0000","last_active":"2024-09-10T20:44:23.759443+0000","last_peered":"2024-09-10T20:44:23.759443+0000","last_clean":"2024-09-10T20:44:23.759443+0000","last_became_active":"2024-09-10T20:44:22.771475+0000","last_became_peered":"2024-09-10T20:44:22.771475+0000","last_unstale":"2024-09-10T20:44:23.759443+0000","last_undegraded":"2024-09-10T20:44:23.759443+0000","last_fullsized":"2024-09-10T20:44:23.759443+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:15:56.464407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.780708+0000","last_change":"2024-09-10T20:44:21.770622+0000","last_active":"2024-09-10T20:44:23.780708+0000","last_peered":"2024-09-10T20:44:23.780708+0000","last_clean":"2024-09-10T20:44:23.780708+0000","last_became_active":"2024-09-10T20:44:21.770334+0000","last_became_peered":"2024-09-10T20:44:21.770334+0000","last_unstale":"2024-09-10T20:44:23.780708+0000","last_undegraded":"2024-09-10T20:44:23.780708+0000","last_fullsized":"2024-09-10T20:44:23.780708+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:08:36.823038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758160+0000","last_change":"2024-09-10T20:44:22.774927+0000","last_active":"2024-09-10T20:44:23.758160+0000","last_peered":"2024-09-10T20:44:23.758160+0000","last_clean":"2024-09-10T20:44:23.758160+0000","last_became_active":"2024-09-10T20:44:22.773467+0000","last_became_peered":"2024-09-10T20:44:22.773467+0000","last_unstale":"2024-09-10T20:44:23.758160+0000","last_undegraded":"2024-09-10T20:44:23.758160+0000","last_fullsized":"2024-09-10T20:44:23.758160+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:38.997424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086770+0000","last_change":"2024-09-10T20:44:21.771876+0000","last_active":"2024-09-10T20:44:24.086770+0000","last_peered":"2024-09-10T20:44:24.086770+0000","last_clean":"2024-09-10T20:44:24.086770+0000","last_became_active":"2024-09-10T20:44:21.771586+0000","last_became_peered":"2024-09-10T20:44:21.771586+0000","last_unstale":"2024-09-10T20:44:24.086770+0000","last_undegraded":"2024-09-10T20:44:24.086770+0000","last_fullsized":"2024-09-10T20:44:24.086770+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:45.602221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.757816+0000","last_change":"2024-09-10T20:44:22.776508+0000","last_active":"2024-09-10T20:44:23.757816+0000","last_peered":"2024-09-10T20:44:23.757816+0000","last_clean":"2024-09-10T20:44:23.757816+0000","last_became_active":"2024-09-10T20:44:22.775881+0000","last_became_peered":"2024-09-10T20:44:22.775881+0000","last_unstale":"2024-09-10T20:44:23.757816+0000","last_undegraded":"2024-09-10T20:44:23.757816+0000","last_fullsized":"2024-09-10T20:44:23.757816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:22:36.366298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086276+0000","last_change":"2024-09-10T20:44:21.767491+0000","last_active":"2024-09-10T20:44:24.086276+0000","last_peered":"2024-09-10T20:44:24.086276+0000","last_clean":"2024-09-10T20:44:24.086276+0000","last_became_active":"2024-09-10T20:44:21.767165+0000","last_became_peered":"2024-09-10T20:44:21.767165+0000","last_unstale":"2024-09-10T20:44:24.086276+0000","last_undegraded":"2024-09-10T20:44:24.086276+0000","last_fullsized":"2024-09-10T20:44:24.086276+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:38.953342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087033+0000","last_change":"2024-09-10T20:44:22.771646+0000","last_active":"2024-09-10T20:44:24.087033+0000","last_peered":"2024-09-10T20:44:24.087033+0000","last_clean":"2024-09-10T20:44:24.087033+0000","last_became_active":"2024-09-10T20:44:22.771301+0000","last_became_peered":"2024-09-10T20:44:22.771301+0000","last_unstale":"2024-09-10T20:44:24.087033+0000","last_undegraded":"2024-09-10T20:44:24.087033+0000","last_fullsized":"2024-09-10T20:44:24.087033+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:28:35.662737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.758607+0000","last_change":"2024-09-10T20:44:21.771204+0000","last_active":"2024-09-10T20:44:23.758607+0000","last_peered":"2024-09-10T20:44:23.758607+0000","last_clean":"2024-09-10T20:44:23.758607+0000","last_became_active":"2024-09-10T20:44:21.770891+0000","last_became_peered":"2024-09-10T20:44:21.770891+0000","last_unstale":"2024-09-10T20:44:23.758607+0000","last_undegraded":"2024-09-10T20:44:23.758607+0000","last_fullsized":"2024-09-10T20:44:23.758607+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:10:40.339751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:23.759777+0000","last_change":"2024-09-10T20:44:22.772704+0000","last_active":"2024-09-10T20:44:23.759777+0000","last_peered":"2024-09-10T20:44:23.759777+0000","last_clean":"2024-09-10T20:44:23.759777+0000","last_became_active":"2024-09-10T20:44:22.771819+0000","last_became_peered":"2024-09-10T20:44:22.771819+0000","last_unstale":"2024-09-10T20:44:23.759777+0000","last_undegraded":"2024-09-10T20:44:23.759777+0000","last_fullsized":"2024-09-10T20:44:23.759777+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:21.747375+0000","last_clean_scrub_stamp":"2024-09-10T20:44:21.747375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:38.237580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.15","version":"126'3","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.087580+0000","last_change":"2024-09-10T20:44:21.773614+0000","last_active":"2024-09-10T20:44:24.087580+0000","last_peered":"2024-09-10T20:44:24.087580+0000","last_clean":"2024-09-10T20:44:24.087580+0000","last_became_active":"2024-09-10T20:44:21.773299+0000","last_became_peered":"2024-09-10T20:44:21.773299+0000","last_unstale":"2024-09-10T20:44:24.087580+0000","last_undegraded":"2024-09-10T20:44:24.087580+0000","last_fullsized":"2024-09-10T20:44:24.087580+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:20.736895+0000","last_clean_scrub_stamp":"2024-09-10T20:44:20.736895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:33:41.247860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":335,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T20:44:24.086582+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:44:24.086582+0000","last_peered":"2024-09-10T20:44:24.086582+0000","last_clean":"2024-09-10T20:44:24.086582+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:44:24.086582+0000","last_undegraded":"2024-09-10T20:44:24.086582+0000","last_fullsized":"2024-09-10T20:44:24.086582+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1572864,"data_stored":693689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1585152,"data_stored":700017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542223,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2191360,"data_stored":1285863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1593344,"data_stored":695905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509521,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36896,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334944,"statfs":{"total":96636764160,"available":96598982656,"internally_reserved":0,"allocated":1581056,"data_stored":694305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1576960,"data_stored":693711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509520,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37476,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334364,"statfs":{"total":96636764160,"available":96598388736,"internally_reserved":0,"allocated":2170880,"data_stored":1284127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509520,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36896,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334944,"statfs":{"total":96636764160,"available":96598982656,"internally_reserved":0,"allocated":1576960,"data_stored":693779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:44:27.158 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:44:27.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:44:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.303+0000 7f0eb6f52640 1 -- 172.21.15.28:0/137958337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98004800 msgr2=0x7f0e98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.303+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/137958337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98004800 0x7f0e98004c80 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400c2e0 tx=0x7f0e98005830 comp rx=0 tx=0).stop 2024-09-10T20:44:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.303+0000 7f0eb6f52640 1 -- 172.21.15.28:0/137958337 shutdown_connections 2024-09-10T20:44:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.303+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/137958337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e98004800 0x7f0e98004c80 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.303+0000 7f0eb6f52640 1 -- 172.21.15.28:0/137958337 >> 172.21.15.28:0/137958337 conn(0x7f0eb00d4040 msgr2=0x7f0eb00d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.303+0000 7f0eb6f52640 1 -- 172.21.15.28:0/137958337 shutdown_connections 2024-09-10T20:44:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.303+0000 7f0eb6f52640 1 -- 172.21.15.28:0/137958337 wait complete. 2024-09-10T20:44:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.304+0000 7f0eb6f52640 1 Processor -- start 2024-09-10T20:44:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.304+0000 7f0eb6f52640 1 -- start start 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb6f52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e98004800 0x7f0eb00c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb6f52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb0154720 0x7f0eb00b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb6f52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb01541d0 0x7f0eb00b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb6f52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb00d4ce0 con 0x7f0eb01541d0 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb6f52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb00d4b60 con 0x7f0eb0154720 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb6f52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb00d4e60 con 0x7f0e98004800 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb54ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb01541d0 0x7f0eb00b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb54ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb01541d0 0x7f0eb00b73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49542/0 (socket says 172.21.15.28:49542) 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb54ef640 1 -- 172.21.15.28:0/1618761745 learned_addr learned my addr 172.21.15.28:0/1618761745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eaffff640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb0154720 0x7f0eb00b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.305+0000 7f0eb4cee640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e98004800 0x7f0eb00c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb54ef640 1 -- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e98004800 msgr2=0x7f0eb00c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb54ef640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e98004800 0x7f0eb00c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb54ef640 1 -- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb0154720 msgr2=0x7f0eb00b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb54ef640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb0154720 0x7f0eb00b6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb54ef640 1 -- 172.21.15.28:0/1618761745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb00ca200 con 0x7f0eb01541d0 2024-09-10T20:44:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb54ef640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb01541d0 0x7f0eb00b73d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00b30b0 tx=0x7f0ea80019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1618761745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8017070 con 0x7f0eb01541d0 2024-09-10T20:44:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1618761745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ea80108e0 con 0x7f0eb01541d0 2024-09-10T20:44:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1618761745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8010d30 con 0x7f0eb01541d0 2024-09-10T20:44:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb00c9c80 con 0x7f0eb01541d0 2024-09-10T20:44:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.306+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb00c9f20 con 0x7f0eb01541d0 2024-09-10T20:44:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.307+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb00cf830 con 0x7f0eb01541d0 2024-09-10T20:44:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.308+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1618761745 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f0ea801b3b0 con 0x7f0eb01541d0 2024-09-10T20:44:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.308+0000 7f0ead7fa640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c0785a0 0x7f0e9c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.308+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1618761745 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f0ea809a9f0 con 0x7f0eb01541d0 2024-09-10T20:44:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.308+0000 7f0eb4cee640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c0785a0 0x7f0e9c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.309+0000 7f0eb4cee640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c0785a0 0x7f0e9c07aa60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009d70 tx=0x7f0ea0005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.313+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1618761745 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0ea8062430 con 0x7f0eb01541d0 2024-09-10T20:44:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.569+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0eb00b4800 con 0x7f0eb01541d0 2024-09-10T20:44:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.572+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1618761745 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v34) ==== 190+0+147 (secure 0 0 0) 0x7f0ea8066e60 con 0x7f0eb01541d0 2024-09-10T20:44:27.576 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:44:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:44:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:44:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:44:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:44:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:44:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c0785a0 msgr2=0x7f0e9c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c0785a0 0x7f0e9c07aa60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009d70 tx=0x7f0ea0005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb01541d0 msgr2=0x7f0eb00b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb01541d0 0x7f0eb00b73d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00b30b0 tx=0x7f0ea80019b0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 shutdown_connections 2024-09-10T20:44:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c0785a0 0x7f0e9c07aa60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb01541d0 0x7f0eb00b73d0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb0154720 0x7f0eb00b6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 --2- 172.21.15.28:0/1618761745 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0e98004800 0x7f0eb00c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 >> 172.21.15.28:0/1618761745 conn(0x7f0eb00d4040 msgr2=0x7f0eb00d4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 shutdown_connections 2024-09-10T20:44:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.575+0000 7f0eb6f52640 1 -- 172.21.15.28:0/1618761745 wait complete. 2024-09-10T20:44:27.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.692+0000 7f6d1dadd640 1 -- 172.21.15.28:0/2858886904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d00004080 msgr2=0x7f6d00004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.692+0000 7f6d1dadd640 1 --2- 172.21.15.28:0/2858886904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d00004080 0x7f6d00004800 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c260 tx=0x7f6d00005830 comp rx=0 tx=0).stop 2024-09-10T20:44:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.692+0000 7f6d17fff640 1 -- 172.21.15.28:0/2858886904 reap_dead start 2024-09-10T20:44:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.692+0000 7f6d1dadd640 1 -- 172.21.15.28:0/2858886904 shutdown_connections 2024-09-10T20:44:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.692+0000 7f6d1dadd640 1 -- 172.21.15.28:0/2858886904 >> 172.21.15.28:0/2858886904 conn(0x7f6d180d3fb0 msgr2=0x7f6d180d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.692+0000 7f6d1dadd640 1 -- 172.21.15.28:0/2858886904 shutdown_connections 2024-09-10T20:44:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.692+0000 7f6d1dadd640 1 -- 172.21.15.28:0/2858886904 wait complete. 2024-09-10T20:44:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.693+0000 7f6d1dadd640 1 Processor -- start 2024-09-10T20:44:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.693+0000 7f6d1dadd640 1 -- start start 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d1dadd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d00004080 0x7f6d1815b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d1dadd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d180b5ab0 0x7f6d180b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d1dadd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d18160790 0x7f6d18160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d1dadd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d180cfd70 con 0x7f6d180b5ab0 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d1dadd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6d180cfbf0 con 0x7f6d00004080 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d1dadd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6d180cfef0 con 0x7f6d18160790 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d16ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d180b5ab0 0x7f6d180b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d16ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d180b5ab0 0x7f6d180b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49562/0 (socket says 172.21.15.28:49562) 2024-09-10T20:44:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d16ffd640 1 -- 172.21.15.28:0/3564882569 learned_addr learned my addr 172.21.15.28:0/3564882569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d17fff640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d18160790 0x7f6d18160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.694+0000 7f6d177fe640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d00004080 0x7f6d1815b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d16ffd640 1 -- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d18160790 msgr2=0x7f6d18160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d16ffd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d18160790 0x7f6d18160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d16ffd640 1 -- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d00004080 msgr2=0x7f6d1815b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d16ffd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d00004080 0x7f6d1815b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d16ffd640 1 -- 172.21.15.28:0/3564882569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d18161410 con 0x7f6d180b5ab0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d16ffd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d180b5ab0 0x7f6d180b5f30 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c290 tx=0x7f6d0c0064a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6cfffff640 1 -- 172.21.15.28:0/3564882569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c003a00 con 0x7f6d180b5ab0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6cfffff640 1 -- 172.21.15.28:0/3564882569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d0c003c90 con 0x7f6d180b5ab0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6cfffff640 1 -- 172.21.15.28:0/3564882569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c02e6a0 con 0x7f6d180b5ab0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d181f4d70 con 0x7f6d180b5ab0 2024-09-10T20:44:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.695+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d18161160 con 0x7f6d180b5ab0 2024-09-10T20:44:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.696+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d180d2b90 con 0x7f6d180b5ab0 2024-09-10T20:44:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.697+0000 7f6cfffff640 1 -- 172.21.15.28:0/3564882569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f6d0c02e840 con 0x7f6d180b5ab0 2024-09-10T20:44:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.697+0000 7f6cfffff640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6d040785d0 0x7f6d0407aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.697+0000 7f6cfffff640 1 -- 172.21.15.28:0/3564882569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f6d0c0bb3c0 con 0x7f6d180b5ab0 2024-09-10T20:44:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.697+0000 7f6d177fe640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6d040785d0 0x7f6d0407aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.698+0000 7f6d177fe640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6d040785d0 0x7f6d0407aa90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6d08003700 tx=0x7f6d0800b260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.702+0000 7f6cfffff640 1 -- 172.21.15.28:0/3564882569 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6d0c047020 con 0x7f6d180b5ab0 2024-09-10T20:44:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.925+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d180b9f30 con 0x7f6d180b5ab0 2024-09-10T20:44:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.926+0000 7f6cfffff640 1 -- 172.21.15.28:0/3564882569 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5227 (secure 0 0 0) 0x7f6d0c03e070 con 0x7f6d180b5ab0 2024-09-10T20:44:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:27.930 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"btime":"2024-09-10T20:44:25:784364+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1041363523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1041363523},{"type":"v1","addr":"172.21.15.28:6834","nonce":1041363523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1561732564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1561732564},{"type":"v1","addr":"172.21.15.28:6835","nonce":1561732564}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6521,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/97213244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":97213244},{"type":"v1","addr":"172.21.15.159:6835","nonce":97213244}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:22.133677+0000","modified":"2024-09-10T20:44:25.147825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6964":{"gid":6964,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/3268698079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3268698079},{"type":"v1","addr":"172.21.15.159:6801","nonce":3268698079}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6964,"qdb_cluster":[6964]},"id":10}]} 2024-09-10T20:44:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-10T20:44:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6d040785d0 msgr2=0x7f6d0407aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6d040785d0 0x7f6d0407aa90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6d08003700 tx=0x7f6d0800b260 comp rx=0 tx=0).stop 2024-09-10T20:44:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d180b5ab0 msgr2=0x7f6d180b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d180b5ab0 0x7f6d180b5f30 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c290 tx=0x7f6d0c0064a0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 shutdown_connections 2024-09-10T20:44:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6d040785d0 0x7f6d0407aa90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6d18160790 0x7f6d18160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d180b5ab0 0x7f6d180b5f30 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 --2- 172.21.15.28:0/3564882569 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6d00004080 0x7f6d1815b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:27.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 >> 172.21.15.28:0/3564882569 conn(0x7f6d180d3fb0 msgr2=0x7f6d180ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:27.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 shutdown_connections 2024-09-10T20:44:27.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:27.928+0000 7f6d1dadd640 1 -- 172.21.15.28:0/3564882569 wait complete. 2024-09-10T20:44:28.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:44:22.133677+0000', 'modified': '2024-09-10T20:44:25.147825+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6964}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6964': {'gid': 6964, 'name': 'b', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.159:6801/3268698079', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 3268698079}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 3268698079}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6964, 'qdb_cluster': [6964]} 2024-09-10T20:44:28.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:44:28.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-10T20:44:28.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.452+0000 7f449da5a640 1 -- 172.21.15.28:0/3928822894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840051c0 msgr2=0x7f4484005910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.452+0000 7f449da5a640 1 --2- 172.21.15.28:0/3928822894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840051c0 0x7f4484005910 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f448000c260 tx=0x7f4484006940 comp rx=0 tx=0).stop 2024-09-10T20:44:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.452+0000 7f44977fe640 1 -- 172.21.15.28:0/3928822894 reap_dead start 2024-09-10T20:44:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.452+0000 7f449da5a640 1 -- 172.21.15.28:0/3928822894 shutdown_connections 2024-09-10T20:44:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.452+0000 7f449da5a640 1 -- 172.21.15.28:0/3928822894 >> 172.21.15.28:0/3928822894 conn(0x7f449806cf00 msgr2=0x7f449806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.452+0000 7f449da5a640 1 -- 172.21.15.28:0/3928822894 shutdown_connections 2024-09-10T20:44:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.452+0000 7f449da5a640 1 -- 172.21.15.28:0/3928822894 wait complete. 2024-09-10T20:44:28.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 Processor -- start 2024-09-10T20:44:28.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 -- start start 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44840051c0 0x7f449809aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44981f8ce0 0x7f4498158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44981f83b0 0x7f449814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4498073ac0 con 0x7f44981f83b0 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4498073940 con 0x7f44981f8ce0 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.454+0000 7f449da5a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f44980707b0 con 0x7f44840051c0 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44981f83b0 0x7f449814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44981f83b0 0x7f449814b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49590/0 (socket says 172.21.15.28:49590) 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 -- 172.21.15.28:0/2961069903 learned_addr learned my addr 172.21.15.28:0/2961069903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f448ffff640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44981f8ce0 0x7f4498158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f4496ffd640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44840051c0 0x7f449809aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 -- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44840051c0 msgr2=0x7f449809aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44840051c0 0x7f449809aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 -- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44981f8ce0 msgr2=0x7f4498158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44981f8ce0 0x7f4498158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 -- 172.21.15.28:0/2961069903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449815a840 con 0x7f44981f83b0 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f448ffff640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44981f8ce0 0x7f4498158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f4496ffd640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44840051c0 0x7f449809aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f44977fe640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44981f83b0 0x7f449814b2f0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f44980d0c20 tx=0x7f44900019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f4494ff9640 1 -- 172.21.15.28:0/2961069903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490011870 con 0x7f44981f83b0 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f4494ff9640 1 -- 172.21.15.28:0/2961069903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4490017070 con 0x7f44981f83b0 2024-09-10T20:44:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f449815ab20 con 0x7f44981f83b0 2024-09-10T20:44:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f4494ff9640 1 -- 172.21.15.28:0/2961069903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490011da0 con 0x7f44981f83b0 2024-09-10T20:44:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.455+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f449815b0e0 con 0x7f44981f83b0 2024-09-10T20:44:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.457+0000 7f4494ff9640 1 -- 172.21.15.28:0/2961069903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f449001c460 con 0x7f44981f83b0 2024-09-10T20:44:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.458+0000 7f4494ff9640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f44780785d0 0x7f447807aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.458+0000 7f4496ffd640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f44780785d0 0x7f447807aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.458+0000 7f4494ff9640 1 -- 172.21.15.28:0/2961069903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f449009b340 con 0x7f44981f83b0 2024-09-10T20:44:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.458+0000 7f4496ffd640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f44780785d0 0x7f447807aa90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4488003700 tx=0x7f448800b260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.459+0000 7f44a0824740 1 Processor -- start 2024-09-10T20:44:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.459+0000 7f44a0824740 1 -- start start 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.459+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa06d1b0 0x559afa044b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.459+0000 7f44a0824740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa08d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa0425f0 0x559afa042a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f44a0824740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559af9fb9820 con 0x559afa08cdf0 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x559af9fa8c50 con 0x559afa0425f0 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x559af9faca60 con 0x559afa06d1b0 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa08d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa08d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49594/0 (socket says 172.21.15.28:49594) 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 -- 172.21.15.28:0/920591277 learned_addr learned my addr 172.21.15.28:0/920591277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f44977fe640 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa0425f0 0x559afa042a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f4496ffd640 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa06d1b0 0x559afa044b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 -- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa06d1b0 msgr2=0x559afa044b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa06d1b0 0x559afa044b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 -- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa0425f0 msgr2=0x559afa042a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa0425f0 0x559afa042a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 -- 172.21.15.28:0/920591277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af9d0a850 con 0x559afa08cdf0 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f44977fe640 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa0425f0 0x559afa042a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448ffff640 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa08d1f0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f448002b0f0 tx=0x7f4498074220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3bd57abd8a669d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448d7fa640 1 -- 172.21.15.28:0/920591277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480031070 con 0x559afa08cdf0 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448d7fa640 1 -- 172.21.15.28:0/920591277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44800016b0 con 0x559afa08cdf0 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.460+0000 7f448d7fa640 1 -- 172.21.15.28:0/920591277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44800062b0 con 0x559afa08cdf0 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 -- 172.21.15.28:0/920591277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 msgr2=0x559afa08d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa08d1f0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f448002b0f0 tx=0x7f4498074220 comp rx=0 tx=0).stop 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 -- 172.21.15.28:0/920591277 shutdown_connections 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa0425f0 0x559afa042a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa08d1f0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 --2- 172.21.15.28:0/920591277 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa06d1b0 0x559afa044b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 -- 172.21.15.28:0/920591277 >> 172.21.15.28:0/920591277 conn(0x559afa0436a0 msgr2=0x559afa0470c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 -- 172.21.15.28:0/920591277 shutdown_connections 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.461+0000 7f44a0824740 1 -- 172.21.15.28:0/920591277 wait complete. 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.462+0000 7f44a0824740 1 -- start start 2024-09-10T20:44:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.462+0000 7f44a0824740 1 Processor -- start 2024-09-10T20:44:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa06d1b0 0x559afa059de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa068950 0x559afa068db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559af9d19620 con 0x559afa06d1b0 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x559af9f4e5f0 con 0x559afa068950 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x559afa041890 con 0x559afa08cdf0 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 10 client.0 _ll_register_callbacks cb 0x559af9f520a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa06d1b0 0x559afa059de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa06d1b0 0x559afa059de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49604/0 (socket says 172.21.15.28:49604) 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 -- 172.21.15.28:0/334714399 learned_addr learned my addr 172.21.15.28:0/334714399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44977fe640 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa068950 0x559afa068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f448ffff640 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 -- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 msgr2=0x559afa068410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 -- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa068950 msgr2=0x559afa068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa068950 0x559afa068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 -- 172.21.15.28:0/334714399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559af9d816b0 con 0x559afa06d1b0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f4496ffd640 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa06d1b0 0x559afa059de0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f448800be90 tx=0x7f44880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a0db383eaeb18c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44767fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44767fc640 1 -- 172.21.15.28:0/334714399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448801b450 con 0x559afa06d1b0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44767fc640 1 -- 172.21.15.28:0/334714399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f4488028af0 con 0x559afa06d1b0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.463+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44800016b0 con 0x559afa06d1b0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44767fc640 1 -- 172.21.15.28:0/334714399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f4488028d30 con 0x559afa06d1b0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 10 client.7989 fetch_fsmap learned FSMap version 86 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) -- 0x559afa05a850 con 0x559afa06d1b0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44767fc640 1 -- 172.21.15.28:0/334714399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44800016b0 con 0x559afa06d1b0 2024-09-10T20:44:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44767fc640 1 -- 172.21.15.28:0/334714399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 86) ==== 3502+0+0 (secure 0 0 0) 0x7f448802f050 con 0x559afa06d1b0 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 10 client.7989 fetch_fsmap finished waiting for FSMap version 86 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 10 client.7989 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 20 client.7989 populate_metadata read hostname 'smithi028' 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa083430 0x559afa06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 4 client.7989 mds_command: new command op to 6964 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 --> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559af9fbb770 con 0x559afa083430 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.464+0000 7f448ffff640 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa083430 0x559afa06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.465+0000 7f44767fc640 10 client.7989 ms_handle_connect on v2:172.21.15.159:6800/3268698079 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.465+0000 7f448ffff640 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa083430 0x559afa06ba00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=186aa39ea08e03a9 server_cookie=6aeafcd4a74f88d7 in_seq=0 out_seq=0 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44767fc640 1 -- 172.21.15.28:0/334714399 <== mds.0 v2:172.21.15.159:6800/3268698079 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559af9fbb770 con 0x559afa083430 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44767fc640 10 client.7989 handle_command_reply: tid=0 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 client.7989 shutdown 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa06d1b0 msgr2=0x559afa059de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa06d1b0 0x559afa059de0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f448800be90 tx=0x7f44880046c0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 shutdown_connections 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa083430 0x559afa06ba00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa068950 0x559afa068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 --2- 172.21.15.28:0/334714399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa06d1b0 0x559afa059de0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44767fc640 0 client.7989 ms_handle_reset on v2:172.21.15.159:6800/3268698079 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 >> 172.21.15.28:0/334714399 conn(0x559afa0425f0 msgr2=0x559afa046ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 shutdown_connections 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 1 -- 172.21.15.28:0/334714399 wait complete. 2024-09-10T20:44:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.466+0000 7f44a0824740 20 client.7989 trim_cache size 0 max 16384 2024-09-10T20:44:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.485+0000 7f44a0824740 1 Processor -- start 2024-09-10T20:44:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.485+0000 7f44a0824740 1 -- start start 2024-09-10T20:44:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa068950 0x559afa059fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa068d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44a0824740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa069270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44a0824740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4498200900 con 0x559afa08cdf0 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f449815c6e0 con 0x559afa06d1b0 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4498156820 con 0x559afa068950 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa069270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa069270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49620/0 (socket says 172.21.15.28:49620) 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 -- 172.21.15.28:0/1168460004 learned_addr learned my addr 172.21.15.28:0/1168460004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f4496ffd640 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa068950 0x559afa059fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f448ffff640 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 -- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa068950 msgr2=0x559afa059fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa068950 0x559afa059fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 -- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 msgr2=0x559afa068d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa068d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f44977fe640 1 -- 172.21.15.28:0/1168460004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af9fdee30 con 0x559afa08cdf0 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.486+0000 7f448ffff640 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa068d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44977fe640 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa069270 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4490001330 tx=0x7f4490099080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66a4927fe2b5574 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f448cff9640 1 -- 172.21.15.28:0/1168460004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490062710 con 0x559afa08cdf0 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f448cff9640 1 -- 172.21.15.28:0/1168460004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4490062c00 con 0x559afa08cdf0 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f448cff9640 1 -- 172.21.15.28:0/1168460004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44900676e0 con 0x559afa08cdf0 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 -- 172.21.15.28:0/1168460004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 msgr2=0x559afa069270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa069270 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4490001330 tx=0x7f4490099080 comp rx=0 tx=0).stop 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 -- 172.21.15.28:0/1168460004 shutdown_connections 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa08cdf0 0x559afa069270 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa068d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 --2- 172.21.15.28:0/1168460004 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa068950 0x559afa059fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 -- 172.21.15.28:0/1168460004 >> 172.21.15.28:0/1168460004 conn(0x559afa0425f0 msgr2=0x559afa042e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 -- 172.21.15.28:0/1168460004 shutdown_connections 2024-09-10T20:44:28.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.487+0000 7f44a0824740 1 -- 172.21.15.28:0/1168460004 wait complete. 2024-09-10T20:44:28.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.488+0000 7f44a0824740 1 -- start start 2024-09-10T20:44:28.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.488+0000 7f44a0824740 1 Processor -- start 2024-09-10T20:44:28.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44a0824740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa068950 0x559afa1075f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44a0824740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44a0824740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559afa0832b0 con 0x559afa068950 2024-09-10T20:44:28.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x559afa06c540 con 0x559afa06d1b0 2024-09-10T20:44:28.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44a0824740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x559afa0838a0 con 0x559afa08cdf0 2024-09-10T20:44:28.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44a0824740 10 client.0 _ll_register_callbacks cb 0x7f4488004380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:44:28.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa068950 0x559afa1075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa068950 0x559afa1075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49630/0 (socket says 172.21.15.28:49630) 2024-09-10T20:44:28.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 -- 172.21.15.28:0/511909846 learned_addr learned my addr 172.21.15.28:0/511909846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:28.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f44977fe640 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f448ffff640 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 -- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 msgr2=0x559afa068d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 -- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 msgr2=0x559afa0b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:44:28.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa0b0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.489+0000 7f4496ffd640 1 -- 172.21.15.28:0/511909846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559afa044cc0 con 0x559afa068950 2024-09-10T20:44:28.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f4496ffd640 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa068950 0x559afa1075f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4488003f90 tx=0x7f448801b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13844628542356e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44767fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:44:28.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44767fc640 1 -- 172.21.15.28:0/511909846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488031680 con 0x559afa068950 2024-09-10T20:44:28.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44767fc640 1 -- 172.21.15.28:0/511909846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f4488031820 con 0x559afa068950 2024-09-10T20:44:28.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4490062c00 con 0x559afa068950 2024-09-10T20:44:28.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44767fc640 1 -- 172.21.15.28:0/511909846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f4488042050 con 0x559afa068950 2024-09-10T20:44:28.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 10 client.7995 fetch_fsmap learned FSMap version 86 2024-09-10T20:44:28.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) -- 0x559afa045840 con 0x559afa068950 2024-09-10T20:44:28.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44767fc640 1 -- 172.21.15.28:0/511909846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4490062c00 con 0x559afa068950 2024-09-10T20:44:28.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44767fc640 1 -- 172.21.15.28:0/511909846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 86) ==== 3502+0+0 (secure 0 0 0) 0x7f4488045020 con 0x559afa068950 2024-09-10T20:44:28.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 10 client.7995 fetch_fsmap finished waiting for FSMap version 86 2024-09-10T20:44:28.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 10 client.7995 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-10T20:44:28.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 20 client.7995 populate_metadata read hostname 'smithi028' 2024-09-10T20:44:28.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa10e220 0x559afa0663f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 4 client.7995 mds_command: new command op to 6964 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:44:28.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 --> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] -- command(tid 0: {"prefix": "status"}) -- 0x559afa069cd0 con 0x559afa10e220 2024-09-10T20:44:28.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.490+0000 7f448ffff640 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa10e220 0x559afa0663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44767fc640 10 client.7995 ms_handle_connect on v2:172.21.15.159:6800/3268698079 2024-09-10T20:44:28.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f448ffff640 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa10e220 0x559afa0663f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afbb4a1d231304d4 server_cookie=cdbf3fba56e83ef2 in_seq=0 out_seq=0 2024-09-10T20:44:28.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44767fc640 1 -- 172.21.15.28:0/511909846 <== mds.0 v2:172.21.15.159:6800/3268698079 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559afa069cd0 con 0x559afa10e220 2024-09-10T20:44:28.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44767fc640 10 client.7995 handle_command_reply: tid=0 2024-09-10T20:44:28.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 client.7995 shutdown 2024-09-10T20:44:28.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa068950 msgr2=0x559afa1075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa068950 0x559afa1075f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4488003f90 tx=0x7f448801b450 comp rx=0 tx=0).stop 2024-09-10T20:44:28.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 shutdown_connections 2024-09-10T20:44:28.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:6800/3268698079,v1:172.21.15.159:6801/3268698079] conn(0x559afa10e220 0x559afa0663f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x559afa08cdf0 0x559afa068d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x559afa06d1b0 0x559afa0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44767fc640 0 client.7995 ms_handle_reset on v2:172.21.15.159:6800/3268698079 2024-09-10T20:44:28.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 --2- 172.21.15.28:0/511909846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afa068950 0x559afa1075f0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 >> 172.21.15.28:0/511909846 conn(0x559afa0425f0 msgr2=0x559afa042c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 shutdown_connections 2024-09-10T20:44:28.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 1 -- 172.21.15.28:0/511909846 wait complete. 2024-09-10T20:44:28.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.491+0000 7f44a0824740 20 client.7995 trim_cache size 0 max 16384 2024-09-10T20:44:28.509 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:44:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:44:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:44:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6964, 2024-09-10T20:44:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:44:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:44:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:44:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7246935160000003, 2024-09-10T20:44:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 86, 2024-09-10T20:44:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 127, 2024-09-10T20:44:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 127, 2024-09-10T20:44:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.428846663 2024-09-10T20:44:28.511 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:44:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f44780785d0 msgr2=0x7f447807aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f44780785d0 0x7f447807aa90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4488003700 tx=0x7f448800b260 comp rx=0 tx=0).stop 2024-09-10T20:44:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44981f83b0 msgr2=0x7f449814b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44981f83b0 0x7f449814b2f0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f44980d0c20 tx=0x7f44900019b0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 shutdown_connections 2024-09-10T20:44:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f44780785d0 0x7f447807aa90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44981f83b0 0x7f449814b2f0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f44981f8ce0 0x7f4498158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 --2- 172.21.15.28:0/2961069903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f44840051c0 0x7f449809aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 >> 172.21.15.28:0/2961069903 conn(0x7f449806cf00 msgr2=0x7f449814c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 shutdown_connections 2024-09-10T20:44:28.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.492+0000 7f449da5a640 1 -- 172.21.15.28:0/2961069903 wait complete. 2024-09-10T20:44:28.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:44:28.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 -- 172.21.15.28:0/2939496183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146a80 msgr2=0x7f613c0d16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 --2- 172.21.15.28:0/2939496183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146a80 0x7f613c0d16e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6138009830 tx=0x7f613802a510 comp rx=0 tx=0).stop 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 -- 172.21.15.28:0/2939496183 shutdown_connections 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 --2- 172.21.15.28:0/2939496183 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f613c162060 0x7f613c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 --2- 172.21.15.28:0/2939496183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146a80 0x7f613c0d16e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 -- 172.21.15.28:0/2939496183 >> 172.21.15.28:0/2939496183 conn(0x7f613c06dbf0 msgr2=0x7f613c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 -- 172.21.15.28:0/2939496183 shutdown_connections 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.619+0000 7f614419b640 1 -- 172.21.15.28:0/2939496183 wait complete. 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.620+0000 7f614419b640 1 Processor -- start 2024-09-10T20:44:28.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.620+0000 7f614419b640 1 -- start start 2024-09-10T20:44:28.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f614419b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f613c146a80 0x7f613c145e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f614419b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146e60 0x7f613c1f86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f614419b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c162060 0x7f613c13ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f614419b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f613c06f760 con 0x7f613c162060 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f614419b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f613c06d490 con 0x7f613c146e60 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f614419b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f613c06f8e0 con 0x7f613c146a80 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f6142738640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c162060 0x7f613c13ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f6142738640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c162060 0x7f613c13ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49644/0 (socket says 172.21.15.28:49644) 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f6142738640 1 -- 172.21.15.28:0/3143635962 learned_addr learned my addr 172.21.15.28:0/3143635962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f6141f37640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f613c146a80 0x7f613c145e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.621+0000 7f6141736640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146e60 0x7f613c1f86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f6142738640 1 -- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f613c146a80 msgr2=0x7f613c145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f6142738640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f613c146a80 0x7f613c145e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f6142738640 1 -- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146e60 msgr2=0x7f613c1f86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f6142738640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146e60 0x7f613c1f86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f6142738640 1 -- 172.21.15.28:0/3143635962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c0c5de0 con 0x7f613c162060 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f6142738640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c162060 0x7f613c13ee00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f613400ea80 tx=0x7f613400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f61327fc640 1 -- 172.21.15.28:0/3143635962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134004b20 con 0x7f613c162060 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f61327fc640 1 -- 172.21.15.28:0/3143635962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6134004ce0 con 0x7f613c162060 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c143640 con 0x7f613c162060 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f61327fc640 1 -- 172.21.15.28:0/3143635962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134005280 con 0x7f613c162060 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.622+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c143c00 con 0x7f613c162060 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.623+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f613c0d16c0 con 0x7f613c162060 2024-09-10T20:44:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.624+0000 7f61327fc640 1 -- 172.21.15.28:0/3143635962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f6134005420 con 0x7f613c162060 2024-09-10T20:44:28.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.624+0000 7f61327fc640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f61200785d0 0x7f612007aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:28.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.624+0000 7f61327fc640 1 -- 172.21.15.28:0/3143635962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f613409b4a0 con 0x7f613c162060 2024-09-10T20:44:28.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.624+0000 7f6141f37640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f61200785d0 0x7f612007aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:28.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.625+0000 7f6141f37640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f61200785d0 0x7f612007aa90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6138034000 tx=0x7f6138004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:28.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.628+0000 7f61327fc640 1 -- 172.21.15.28:0/3143635962 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6134062f70 con 0x7f613c162060 2024-09-10T20:44:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.852+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f613c15e500 con 0x7f613c162060 2024-09-10T20:44:28.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.853+0000 7f61327fc640 1 -- 172.21.15.28:0/3143635962 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) ==== 53+0+73 (secure 0 0 0) 0x7f61340679a0 con 0x7f613c162060 2024-09-10T20:44:28.857 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f61200785d0 msgr2=0x7f612007aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f61200785d0 0x7f612007aa90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6138034000 tx=0x7f6138004730 comp rx=0 tx=0).stop 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c162060 msgr2=0x7f613c13ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c162060 0x7f613c13ee00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f613400ea80 tx=0x7f613400ef40 comp rx=0 tx=0).stop 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 shutdown_connections 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f61200785d0 0x7f612007aa90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f613c162060 0x7f613c13ee00 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f613c146e60 0x7f613c1f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 --2- 172.21.15.28:0/3143635962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f613c146a80 0x7f613c145e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:28.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 >> 172.21.15.28:0/3143635962 conn(0x7f613c06dbf0 msgr2=0x7f613c06fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:28.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 shutdown_connections 2024-09-10T20:44:28.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:28.855+0000 7f614419b640 1 -- 172.21.15.28:0/3143635962 wait complete. 2024-09-10T20:44:28.869 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:44:28.869 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:44:28.869 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:44:28.870 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:44:28.870 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:44:28.870 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:44:28.870 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:44:28.870 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:44:28.870 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:44:28.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:28.870 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:44:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42362sec preferred_lft 42362sec 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:44:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:44:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:28.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:28.890 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:44:28.952 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:44:28.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:28.953 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:44:29.011 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:44:29.011 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:44:29.011 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:44:29.011 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:44:29.011 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:44:29.011 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:44:29.012 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:44:29.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:44:29.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:44:29.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:44:29.012 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:44:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:44:29.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:44:29.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:29.139 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:44:29.155 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:44:29.155 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:44:29.155 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:44:29.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:29.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:44:29.221 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:44:29.222 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:44:29.278 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:44:29.278 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:44:29.278 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:44:29.333 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:44:29.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:29.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:44:29.418 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:44:29.418 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:44:29.418 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:44:29.427 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:44:29.427 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:44:29.427 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:44:29.427 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:44:29.427 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:44:29.445 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:44:29.445 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:44:29.445 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:44:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:44:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:44:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:44:29.448 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:44:29.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:29.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:44:29.475 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:44:29.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:44:29.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:44:29.579 INFO:teuthology.orchestra.run.smithi028.stdout:client7090 2024-09-10T20:44:29.579 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:29.579 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:29.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:44:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.736+0000 7f32cc6b8640 1 -- 172.21.15.28:0/891607799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 msgr2=0x7f32c415e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.736+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/891607799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c415e930 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f32b4005fd0 tx=0x7f32b402a540 comp rx=0 tx=0).stop 2024-09-10T20:44:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.736+0000 7f32cc6b8640 1 -- 172.21.15.28:0/891607799 shutdown_connections 2024-09-10T20:44:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.736+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/891607799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c415e930 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.736+0000 7f32cc6b8640 1 -- 172.21.15.28:0/891607799 >> 172.21.15.28:0/891607799 conn(0x7f32c406cf00 msgr2=0x7f32c406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.736+0000 7f32cc6b8640 1 -- 172.21.15.28:0/891607799 shutdown_connections 2024-09-10T20:44:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.736+0000 7f32cc6b8640 1 -- 172.21.15.28:0/891607799 wait complete. 2024-09-10T20:44:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.738+0000 7f32cc6b8640 1 Processor -- start 2024-09-10T20:44:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.738+0000 7f32cc6b8640 1 -- start start 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.738+0000 7f32cc6b8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32c4155100 0x7f32c40ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.738+0000 7f32cc6b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c4152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32cc6b8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32c415ee70 0x7f32c4152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32cc6b8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32c4072f60 con 0x7f32c41554e0 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32cc6b8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f32c4072de0 con 0x7f32c4155100 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32cc6b8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f32c40730e0 con 0x7f32c415ee70 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c4152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c4152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49664/0 (socket says 172.21.15.28:49664) 2024-09-10T20:44:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 -- 172.21.15.28:0/3605509530 learned_addr learned my addr 172.21.15.28:0/3605509530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32ca454640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32c4155100 0x7f32c40ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32cac55640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32c415ee70 0x7f32c4152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 -- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32c415ee70 msgr2=0x7f32c4152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32c415ee70 0x7f32c4152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 -- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32c4155100 msgr2=0x7f32c40ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32c4155100 0x7f32c40ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 -- 172.21.15.28:0/3605509530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c41e7520 con 0x7f32c41554e0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32c9c53640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c4152240 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f32b4000c00 tx=0x7f32b40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32baffd640 1 -- 172.21.15.28:0/3605509530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b403f070 con 0x7f32c41554e0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32baffd640 1 -- 172.21.15.28:0/3605509530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32b4004890 con 0x7f32c41554e0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c41e7800 con 0x7f32c41554e0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32baffd640 1 -- 172.21.15.28:0/3605509530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b4035030 con 0x7f32c41554e0 2024-09-10T20:44:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.739+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c40adc70 con 0x7f32c41554e0 2024-09-10T20:44:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.741+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c4074760 con 0x7f32c41554e0 2024-09-10T20:44:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.741+0000 7f32baffd640 1 -- 172.21.15.28:0/3605509530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f32b4004ce0 con 0x7f32c41554e0 2024-09-10T20:44:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.742+0000 7f32baffd640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32a80785a0 0x7f32a807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.742+0000 7f32baffd640 1 -- 172.21.15.28:0/3605509530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f32b40bb950 con 0x7f32c41554e0 2024-09-10T20:44:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.742+0000 7f32ca454640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32a80785a0 0x7f32a807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.742+0000 7f32ca454640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32a80785a0 0x7f32a807aa60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f32c41e7310 tx=0x7f32c0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.746+0000 7f32baffd640 1 -- 172.21.15.28:0/3605509530 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f32b4083590 con 0x7f32c41554e0 2024-09-10T20:44:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.974+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f32c40cd5c0 con 0x7f32c41554e0 2024-09-10T20:44:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.975+0000 7f32baffd640 1 -- 172.21.15.28:0/3605509530 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) ==== 53+0+73 (secure 0 0 0) 0x7f32b4087fc0 con 0x7f32c41554e0 2024-09-10T20:44:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:44:29.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32a80785a0 msgr2=0x7f32a807aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32a80785a0 0x7f32a807aa60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f32c41e7310 tx=0x7f32c0009040 comp rx=0 tx=0).stop 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 msgr2=0x7f32c4152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c4152240 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f32b4000c00 tx=0x7f32b40039e0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 shutdown_connections 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f32a80785a0 0x7f32a807aa60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f32c415ee70 0x7f32c4152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c41554e0 0x7f32c4152240 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 --2- 172.21.15.28:0/3605509530 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f32c4155100 0x7f32c40ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 >> 172.21.15.28:0/3605509530 conn(0x7f32c406cf00 msgr2=0x7f32c4153270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 shutdown_connections 2024-09-10T20:44:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:29.977+0000 7f32cc6b8640 1 -- 172.21.15.28:0/3605509530 wait complete. 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:44:29.992 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:44:29.993 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:44:29.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:29.993 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:44:30.013 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:44:30.016 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42379sec preferred_lft 42379sec 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:30.017 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:44:30.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:30.018 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:44:30.081 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:44:30.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:30.083 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:44:30.141 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:44:30.141 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:44:30.141 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:44:30.141 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:44:30.142 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:44:30.142 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:44:30.142 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:44:30.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:44:30.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:44:30.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:44:30.142 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:44:30.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:44:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:44:30.268 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:44:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:44:30.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:30.275 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:44:30.290 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:44:30.290 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:44:30.290 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:44:30.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:30.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:44:30.355 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:44:30.355 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:30.412 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:44:30.412 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:44:30.412 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:30.467 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:44:30.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:30.467 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:30.550 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:30.550 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:44:30.550 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:44:30.559 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:30.559 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:44:30.559 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:44:30.559 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:44:30.559 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:44:30.564 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:44:30.565 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:44:30.565 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:44:30.565 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:44:30.565 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:44:30.565 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:44:30.566 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:44:30.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:30.567 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:30.632 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:44:30.721 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:44:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:44:30.723 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:44:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:44:30.729 INFO:teuthology.orchestra.run.smithi159.stdout:client6626 2024-09-10T20:44:30.729 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:30.729 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:30.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 -- 172.21.15.28:0/675955474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300b5a80 msgr2=0x7f7d30073700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 --2- 172.21.15.28:0/675955474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300b5a80 0x7f7d30073700 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7d300749f0 tx=0x7f7d2402a5f0 comp rx=0 tx=0).stop 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 -- 172.21.15.28:0/675955474 shutdown_connections 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 --2- 172.21.15.28:0/675955474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300b5a80 0x7f7d30073700 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 --2- 172.21.15.28:0/675955474 >> v2:172.21.15.159:3300/0 conn(0x7f7d30073c40 0x7f7d300add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 -- 172.21.15.28:0/675955474 >> 172.21.15.28:0/675955474 conn(0x7f7d300b2920 msgr2=0x7f7d300b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 -- 172.21.15.28:0/675955474 shutdown_connections 2024-09-10T20:44:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.870+0000 7f7d36c08640 1 -- 172.21.15.28:0/675955474 wait complete. 2024-09-10T20:44:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.872+0000 7f7d36c08640 1 Processor -- start 2024-09-10T20:44:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.872+0000 7f7d36c08640 1 -- start start 2024-09-10T20:44:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.872+0000 7f7d36c08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d30073c40 0x7f7d301530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d36c08640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d300b5a80 0x7f7d3014ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d36c08640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3015ab00 0x7f7d30151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d36c08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3015aee0 con 0x7f7d30073c40 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d36c08640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7d300b4630 con 0x7f7d3015ab00 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d36c08640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3015b060 con 0x7f7d300b5a80 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d30073c40 0x7f7d301530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d30073c40 0x7f7d301530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52604/0 (socket says 172.21.15.28:52604) 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 -- 172.21.15.28:0/2550425615 learned_addr learned my addr 172.21.15.28:0/2550425615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d351a5640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3015ab00 0x7f7d30151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d300b5a80 0x7f7d3014ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 -- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d300b5a80 msgr2=0x7f7d3014ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d300b5a80 0x7f7d3014ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 -- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3015ab00 msgr2=0x7f7d30151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3015ab00 0x7f7d30151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 -- 172.21.15.28:0/2550425615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d301507d0 con 0x7f7d30073c40 2024-09-10T20:44:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d349a4640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d30073c40 0x7f7d301530e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000c3a0 tx=0x7f7d2000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/2550425615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20017070 con 0x7f7d30073c40 2024-09-10T20:44:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/2550425615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d200053c0 con 0x7f7d30073c40 2024-09-10T20:44:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d30150ab0 con 0x7f7d30073c40 2024-09-10T20:44:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/2550425615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20003850 con 0x7f7d30073c40 2024-09-10T20:44:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.873+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3015c8b0 con 0x7f7d30073c40 2024-09-10T20:44:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.874+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d300ae470 con 0x7f7d30073c40 2024-09-10T20:44:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.875+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/2550425615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f7d200039f0 con 0x7f7d30073c40 2024-09-10T20:44:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.876+0000 7f7d2d7fa640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d1c0785a0 0x7f7d1c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.876+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/2550425615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f7d20099f00 con 0x7f7d30073c40 2024-09-10T20:44:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.876+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d1c0785a0 0x7f7d1c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.876+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d1c0785a0 0x7f7d1c07aa60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d24033000 tx=0x7f7d24004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:30.880+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/2550425615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7d200619d0 con 0x7f7d30073c40 2024-09-10T20:44:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.102+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f7d300b4e50 con 0x7f7d30073c40 2024-09-10T20:44:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.810+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/2550425615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v87) ==== 136+0+0 (secure 0 0 0) 0x7f7d20066400 con 0x7f7d30073c40 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.813+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d1c0785a0 msgr2=0x7f7d1c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d1c0785a0 0x7f7d1c07aa60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d24033000 tx=0x7f7d24004730 comp rx=0 tx=0).stop 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d30073c40 msgr2=0x7f7d301530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d30073c40 0x7f7d301530e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000c3a0 tx=0x7f7d2000c860 comp rx=0 tx=0).stop 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 shutdown_connections 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d1c0785a0 0x7f7d1c07aa60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3015ab00 0x7f7d30151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d300b5a80 0x7f7d3014ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 --2- 172.21.15.28:0/2550425615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d30073c40 0x7f7d301530e0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 >> 172.21.15.28:0/2550425615 conn(0x7f7d300b2920 msgr2=0x7f7d301580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 shutdown_connections 2024-09-10T20:44:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.814+0000 7f7d36c08640 1 -- 172.21.15.28:0/2550425615 wait complete. 2024-09-10T20:44:31.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 -- 172.21.15.28:0/2491618368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bb40 msgr2=0x7fa5c00af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 --2- 172.21.15.28:0/2491618368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bb40 0x7fa5c00af660 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0073780 tx=0x7fa5bc02a560 comp rx=0 tx=0).stop 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 -- 172.21.15.28:0/2491618368 shutdown_connections 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 --2- 172.21.15.28:0/2491618368 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c006cae0 0x7fa5c006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 --2- 172.21.15.28:0/2491618368 >> v2:172.21.15.159:3300/0 conn(0x7fa5c015bf20 0x7fa5c00afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 --2- 172.21.15.28:0/2491618368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bb40 0x7fa5c00af660 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 -- 172.21.15.28:0/2491618368 >> 172.21.15.28:0/2491618368 conn(0x7fa5c006dbf0 msgr2=0x7fa5c006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 -- 172.21.15.28:0/2491618368 shutdown_connections 2024-09-10T20:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.941+0000 7fa5c7918640 1 -- 172.21.15.28:0/2491618368 wait complete. 2024-09-10T20:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.942+0000 7fa5c7918640 1 Processor -- start 2024-09-10T20:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.942+0000 7fa5c7918640 1 -- start start 2024-09-10T20:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c7918640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c006cae0 0x7fa5c014a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c7918640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c015bb40 0x7fa5c00ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c7918640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bf20 0x7fa5c009ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c7918640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c00b4280 con 0x7fa5c015bf20 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c7918640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c00b4100 con 0x7fa5c015bb40 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c7918640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c00b4400 con 0x7fa5c006cae0 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bf20 0x7fa5c009ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bf20 0x7fa5c009ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52626/0 (socket says 172.21.15.28:52626) 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 -- 172.21.15.28:0/490683314 learned_addr learned my addr 172.21.15.28:0/490683314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c56b4640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c006cae0 0x7fa5c014a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c4eb3640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c015bb40 0x7fa5c00ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 -- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c006cae0 msgr2=0x7fa5c014a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c006cae0 0x7fa5c014a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 -- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c015bb40 msgr2=0x7fa5c00ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c015bb40 0x7fa5c00ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 -- 172.21.15.28:0/490683314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c00cd860 con 0x7fa5c015bf20 2024-09-10T20:44:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c4eb3640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c015bb40 0x7fa5c00ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.943+0000 7fa5c5eb5640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bf20 0x7fa5c009ba80 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800cce0 tx=0x7fa5b800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.944+0000 7fa5b5ffb640 1 -- 172.21.15.28:0/490683314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800f450 con 0x7fa5c015bf20 2024-09-10T20:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.944+0000 7fa5b5ffb640 1 -- 172.21.15.28:0/490683314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5b800f700 con 0x7fa5c015bf20 2024-09-10T20:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.944+0000 7fa5b5ffb640 1 -- 172.21.15.28:0/490683314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800fb50 con 0x7fa5c015bf20 2024-09-10T20:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.944+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c00cdb40 con 0x7fa5c015bf20 2024-09-10T20:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.944+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c00ce100 con 0x7fa5c015bf20 2024-09-10T20:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.945+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c00ce300 con 0x7fa5c015bf20 2024-09-10T20:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.945+0000 7fa5b5ffb640 1 -- 172.21.15.28:0/490683314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fa5b80213f0 con 0x7fa5c015bf20 2024-09-10T20:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.946+0000 7fa5b5ffb640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa5a40785a0 0x7fa5a407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.946+0000 7fa5b5ffb640 1 -- 172.21.15.28:0/490683314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fa5b809b600 con 0x7fa5c015bf20 2024-09-10T20:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.946+0000 7fa5c56b4640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa5a40785a0 0x7fa5a407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.946+0000 7fa5c56b4640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa5a40785a0 0x7fa5a407aa60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc033000 tx=0x7fa5bc004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:31.950+0000 7fa5b5ffb640 1 -- 172.21.15.28:0/490683314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa5b8014030 con 0x7fa5c015bf20 2024-09-10T20:44:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.159+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fa5c00b48b0 con 0x7fa5a40785a0 2024-09-10T20:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.161+0000 7fa5b5ffb640 1 -- 172.21.15.28:0/490683314 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fa5c00b48b0 con 0x7fa5a40785a0 2024-09-10T20:44:32.166 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:44:32.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:32.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:44:32.166 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:32.166 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa5a40785a0 msgr2=0x7fa5a407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa5a40785a0 0x7fa5a407aa60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc033000 tx=0x7fa5bc004730 comp rx=0 tx=0).stop 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bf20 msgr2=0x7fa5c009ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bf20 0x7fa5c009ba80 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800cce0 tx=0x7fa5b800a620 comp rx=0 tx=0).stop 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 shutdown_connections 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa5a40785a0 0x7fa5a407aa60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c015bf20 0x7fa5c009ba80 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa5c015bb40 0x7fa5c00ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 --2- 172.21.15.28:0/490683314 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa5c006cae0 0x7fa5c014a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 >> 172.21.15.28:0/490683314 conn(0x7fa5c006dbf0 msgr2=0x7fa5c015aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 shutdown_connections 2024-09-10T20:44:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.163+0000 7fa5c7918640 1 -- 172.21.15.28:0/490683314 wait complete. 2024-09-10T20:44:32.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:44:32.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.282+0000 7f0c40ad6640 1 -- 172.21.15.28:0/511397345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c161770 msgr2=0x7f0c3c0d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.282+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/511397345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c161770 0x7f0c3c0d20e0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009830 tx=0x7f0c2c02a480 comp rx=0 tx=0).stop 2024-09-10T20:44:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.282+0000 7f0c40ad6640 1 -- 172.21.15.28:0/511397345 shutdown_connections 2024-09-10T20:44:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.282+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/511397345 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c3c161b50 0x7f0c3c0d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.282+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/511397345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c161770 0x7f0c3c0d20e0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.282+0000 7f0c40ad6640 1 -- 172.21.15.28:0/511397345 >> 172.21.15.28:0/511397345 conn(0x7f0c3c0d4070 msgr2=0x7f0c3c0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.283+0000 7f0c40ad6640 1 -- 172.21.15.28:0/511397345 shutdown_connections 2024-09-10T20:44:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.283+0000 7f0c40ad6640 1 -- 172.21.15.28:0/511397345 wait complete. 2024-09-10T20:44:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.284+0000 7f0c40ad6640 1 Processor -- start 2024-09-10T20:44:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.284+0000 7f0c40ad6640 1 -- start start 2024-09-10T20:44:32.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c40ad6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c0d4b90 0x7f0c3c0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c40ad6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c3c161770 0x7f0c3c0b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c40ad6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3c161b50 0x7f0c3c0b9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c40ad6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c0b2f50 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c40ad6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c0b2dd0 con 0x7f0c3c161b50 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c40ad6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0c3c0b30d0 con 0x7f0c3c161770 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c0d4b90 0x7f0c3c0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c0d4b90 0x7f0c3c0cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52646/0 (socket says 172.21.15.28:52646) 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 -- 172.21.15.28:0/797991189 learned_addr learned my addr 172.21.15.28:0/797991189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c33fff640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c3c161770 0x7f0c3c0b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3ad9d640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3c161b50 0x7f0c3c0b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 -- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c3c161770 msgr2=0x7f0c3c0b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c3c161770 0x7f0c3c0b74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 -- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3c161b50 msgr2=0x7f0c3c0b9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3c161b50 0x7f0c3c0b9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 -- 172.21.15.28:0/797991189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3c1f7ee0 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.285+0000 7f0c3a59c640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c0d4b90 0x7f0c3c0cdae0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c000c00 tx=0x7f0c2c032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.286+0000 7f0c32ffd640 1 -- 172.21.15.28:0/797991189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c040070 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.286+0000 7f0c32ffd640 1 -- 172.21.15.28:0/797991189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c2c032e70 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.286+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3c1f81c0 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.286+0000 7f0c32ffd640 1 -- 172.21.15.28:0/797991189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c003d90 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.286+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c3c1f8780 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.287+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c3c0b3870 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.287+0000 7f0c32ffd640 1 -- 172.21.15.28:0/797991189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f0c2c04d030 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.288+0000 7f0c32ffd640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0c1c0785a0 0x7f0c1c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.288+0000 7f0c32ffd640 1 -- 172.21.15.28:0/797991189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f0c2c0bbc60 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.288+0000 7f0c33fff640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0c1c0785a0 0x7f0c1c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.288+0000 7f0c33fff640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0c1c0785a0 0x7f0c1c07aa60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0a1e40 tx=0x7f0c2400b2e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:32.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.292+0000 7f0c32ffd640 1 -- 172.21.15.28:0/797991189 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0c2c083810 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:44:32.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:44:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.482+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f0c3c0b1280 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.482+0000 7f0c32ffd640 1 -- 172.21.15.28:0/797991189 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) ==== 122+0+0 (secure 0 0 0) 0x7f0c2c088240 con 0x7f0c3c0d4b90 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.484+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0c1c0785a0 msgr2=0x7f0c1c07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.484+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0c1c0785a0 0x7f0c1c07aa60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0a1e40 tx=0x7f0c2400b2e0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.484+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c0d4b90 msgr2=0x7f0c3c0cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.484+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c0d4b90 0x7f0c3c0cdae0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c000c00 tx=0x7f0c2c032c80 comp rx=0 tx=0).stop 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 shutdown_connections 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0c1c0785a0 0x7f0c1c07aa60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0c3c161b50 0x7f0c3c0b9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0c3c161770 0x7f0c3c0b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 --2- 172.21.15.28:0/797991189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c3c0d4b90 0x7f0c3c0cdae0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 >> 172.21.15.28:0/797991189 conn(0x7f0c3c0d4070 msgr2=0x7f0c3c0ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 shutdown_connections 2024-09-10T20:44:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.485+0000 7f0c40ad6640 1 -- 172.21.15.28:0/797991189 wait complete. 2024-09-10T20:44:32.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create non_existent_subvolume subvolume_1 2024-09-10T20:44:32.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.605+0000 7fa5814dd640 1 -- 172.21.15.28:0/528752110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa568004800 msgr2=0x7fa568004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.605+0000 7fa5814dd640 1 --2- 172.21.15.28:0/528752110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa568004800 0x7fa568004c80 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fa564009b30 tx=0x7fa568005830 comp rx=0 tx=0).stop 2024-09-10T20:44:32.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.605+0000 7fa57b7fe640 1 -- 172.21.15.28:0/528752110 reap_dead start 2024-09-10T20:44:32.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.605+0000 7fa5814dd640 1 -- 172.21.15.28:0/528752110 shutdown_connections 2024-09-10T20:44:32.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.605+0000 7fa5814dd640 1 -- 172.21.15.28:0/528752110 >> 172.21.15.28:0/528752110 conn(0x7fa57c06d110 msgr2=0x7fa57c0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:32.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.605+0000 7fa5814dd640 1 -- 172.21.15.28:0/528752110 shutdown_connections 2024-09-10T20:44:32.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.605+0000 7fa5814dd640 1 -- 172.21.15.28:0/528752110 wait complete. 2024-09-10T20:44:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 Processor -- start 2024-09-10T20:44:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 -- start start 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa568004800 0x7fa57c1406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c1e1fc0 0x7fa57c155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c155e50 0x7fa57c1df7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c0b30d0 con 0x7fa568004800 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c0b2f50 con 0x7fa57c1e1fc0 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.607+0000 7fa5814dd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa57c0b3250 con 0x7fa57c155e50 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa568004800 0x7fa57c1406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa568004800 0x7fa57c1406e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52670/0 (socket says 172.21.15.28:52670) 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 -- 172.21.15.28:0/1416967787 learned_addr learned my addr 172.21.15.28:0/1416967787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57a7fc640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c1e1fc0 0x7fa57c155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 -- 172.21.15.28:0/1416967787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c155e50 msgr2=0x7fa57c1df7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa57c155e50 0x7fa57c1df7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 -- 172.21.15.28:0/1416967787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c1e1fc0 msgr2=0x7fa57c155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c1e1fc0 0x7fa57c155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 -- 172.21.15.28:0/1416967787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c1e1200 con 0x7fa568004800 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57a7fc640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa57c1e1fc0 0x7fa57c155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa57affd640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa568004800 0x7fa57c1406e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fa574004ea0 tx=0x7fa574004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa5737fe640 1 -- 172.21.15.28:0/1416967787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574015020 con 0x7fa568004800 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa5737fe640 1 -- 172.21.15.28:0/1416967787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa574011960 con 0x7fa568004800 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa5814dd640 1 -- 172.21.15.28:0/1416967787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57c1e14e0 con 0x7fa568004800 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa5737fe640 1 -- 172.21.15.28:0/1416967787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574011d90 con 0x7fa568004800 2024-09-10T20:44:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.608+0000 7fa5814dd640 1 -- 172.21.15.28:0/1416967787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57c1e0520 con 0x7fa568004800 2024-09-10T20:44:32.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.609+0000 7fa5814dd640 1 -- 172.21.15.28:0/1416967787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa57c0b2300 con 0x7fa568004800 2024-09-10T20:44:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.610+0000 7fa5737fe640 1 -- 172.21.15.28:0/1416967787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fa57401a400 con 0x7fa568004800 2024-09-10T20:44:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.611+0000 7fa5737fe640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa55c0785d0 0x7fa55c07aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.611+0000 7fa57a7fc640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa55c0785d0 0x7fa55c07aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.611+0000 7fa5737fe640 1 -- 172.21.15.28:0/1416967787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fa57409b650 con 0x7fa568004800 2024-09-10T20:44:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.611+0000 7fa57a7fc640 1 --2- 172.21.15.28:0/1416967787 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa55c0785d0 0x7fa55c07aa90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa564034000 tx=0x7fa56402d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.615+0000 7fa5737fe640 1 -- 172.21.15.28:0/1416967787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa57401f020 con 0x7fa568004800 2024-09-10T20:44:32.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.829+0000 7fa5814dd640 1 -- 172.21.15.28:0/1416967787 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7fa57c159010 con 0x7fa55c0785d0 2024-09-10T20:44:32.835 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:32.830+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:32.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.831+0000 7fa5737fe640 1 -- 172.21.15.28:0/1416967787 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fa57c159010 con 0x7fa55c0785d0 2024-09-10T20:44:32.835 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:32.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm non_existent_subvolume subvolume_1 2024-09-10T20:44:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 -- 172.21.15.28:0/2541901199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdccc06fa50 msgr2=0x7fdccc15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 --2- 172.21.15.28:0/2541901199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdccc06fa50 0x7fdccc15cbd0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009ca0 tx=0x7fdcc402a980 comp rx=0 tx=0).stop 2024-09-10T20:44:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 -- 172.21.15.28:0/2541901199 shutdown_connections 2024-09-10T20:44:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 --2- 172.21.15.28:0/2541901199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdccc06fa50 0x7fdccc15cbd0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 --2- 172.21.15.28:0/2541901199 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdccc15e7d0 0x7fdccc06f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 -- 172.21.15.28:0/2541901199 >> 172.21.15.28:0/2541901199 conn(0x7fdccc06d110 msgr2=0x7fdccc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 -- 172.21.15.28:0/2541901199 shutdown_connections 2024-09-10T20:44:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.945+0000 7fdcd28b0640 1 -- 172.21.15.28:0/2541901199 wait complete. 2024-09-10T20:44:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.946+0000 7fdcd28b0640 1 Processor -- start 2024-09-10T20:44:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.946+0000 7fdcd28b0640 1 -- start start 2024-09-10T20:44:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd28b0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdccc06fa50 0x7fdccc0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd28b0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdccc1587a0 0x7fdccc1f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd28b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdccc15e7d0 0x7fdccc0abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd28b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc15d4a0 con 0x7fdccc15e7d0 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd28b0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc15d320 con 0x7fdccc1587a0 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd28b0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdccc15d620 con 0x7fdccc06fa50 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdccc15e7d0 0x7fdccc0abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdccc15e7d0 0x7fdccc0abb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52688/0 (socket says 172.21.15.28:52688) 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 -- 172.21.15.28:0/4181420456 learned_addr learned my addr 172.21.15.28:0/4181420456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdccbfff640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdccc06fa50 0x7fdccc0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdccb7fe640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdccc1587a0 0x7fdccc1f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 -- 172.21.15.28:0/4181420456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdccc06fa50 msgr2=0x7fdccc0af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdccc06fa50 0x7fdccc0af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 -- 172.21.15.28:0/4181420456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdccc1587a0 msgr2=0x7fdccc1f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdccc1587a0 0x7fdccc1f5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdcd0e4d640 1 -- 172.21.15.28:0/4181420456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdccc1f25a0 con 0x7fdccc15e7d0 2024-09-10T20:44:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.947+0000 7fdccbfff640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdccc06fa50 0x7fdccc0af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.948+0000 7fdcd0e4d640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdccc15e7d0 0x7fdccc0abb50 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4006010 tx=0x7fdcc4031da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.948+0000 7fdcc8ff9640 1 -- 172.21.15.28:0/4181420456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4048070 con 0x7fdccc15e7d0 2024-09-10T20:44:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.948+0000 7fdcc8ff9640 1 -- 172.21.15.28:0/4181420456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcc4004450 con 0x7fdccc15e7d0 2024-09-10T20:44:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.948+0000 7fdcd28b0640 1 -- 172.21.15.28:0/4181420456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdccc0add70 con 0x7fdccc15e7d0 2024-09-10T20:44:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.948+0000 7fdcc8ff9640 1 -- 172.21.15.28:0/4181420456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4034040 con 0x7fdccc15e7d0 2024-09-10T20:44:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.948+0000 7fdcd28b0640 1 -- 172.21.15.28:0/4181420456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdccc0ac090 con 0x7fdccc15e7d0 2024-09-10T20:44:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.949+0000 7fdcd28b0640 1 -- 172.21.15.28:0/4181420456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdccc15db80 con 0x7fdccc15e7d0 2024-09-10T20:44:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.950+0000 7fdcc8ff9640 1 -- 172.21.15.28:0/4181420456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fdcc4042030 con 0x7fdccc15e7d0 2024-09-10T20:44:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.950+0000 7fdcc8ff9640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdcb80785a0 0x7fdcb807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.950+0000 7fdcc8ff9640 1 -- 172.21.15.28:0/4181420456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fdcc40bd5f0 con 0x7fdccc15e7d0 2024-09-10T20:44:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.950+0000 7fdccbfff640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdcb80785a0 0x7fdcb807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.951+0000 7fdccbfff640 1 --2- 172.21.15.28:0/4181420456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdcb80785a0 0x7fdcb807aa60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdccc154fc0 tx=0x7fdcbc009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:32.954+0000 7fdcc8ff9640 1 -- 172.21.15.28:0/4181420456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdcc4040070 con 0x7fdccc15e7d0 2024-09-10T20:44:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.166+0000 7fdcd28b0640 1 -- 172.21.15.28:0/4181420456 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7fdccc1f2880 con 0x7fdcb80785a0 2024-09-10T20:44:33.172 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:33.167+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:33.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.167+0000 7fdcc8ff9640 1 -- 172.21.15.28:0/4181420456 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fdccc1f2880 con 0x7fdcb80785a0 2024-09-10T20:44:33.172 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:33.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath non_existent_subvolume subvolume_1 2024-09-10T20:44:33.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.281+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/823719931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b480d1840 msgr2=0x7f7b48073f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.281+0000 7f7b4f3b4640 1 --2- 172.21.15.28:0/823719931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b480d1840 0x7f7b48073f50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009860 tx=0x7f7b3802a5a0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.281+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/823719931 shutdown_connections 2024-09-10T20:44:33.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.281+0000 7f7b4f3b4640 1 --2- 172.21.15.28:0/823719931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b480d1840 0x7f7b48073f50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.281+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/823719931 >> 172.21.15.28:0/823719931 conn(0x7f7b480d0d50 msgr2=0x7f7b480d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:33.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.281+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/823719931 shutdown_connections 2024-09-10T20:44:33.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.281+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/823719931 wait complete. 2024-09-10T20:44:33.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.282+0000 7f7b4f3b4640 1 Processor -- start 2024-09-10T20:44:33.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.282+0000 7f7b4f3b4640 1 -- start start 2024-09-10T20:44:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4f3b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b48074490 0x7f7b481f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4f3b4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b480d1840 0x7f7b48144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4f3b4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b48152df0 0x7f7b481447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4f3b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4806ce70 con 0x7f7b48074490 2024-09-10T20:44:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4f3b4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4806ccf0 con 0x7f7b480d1840 2024-09-10T20:44:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4f3b4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7b4806cff0 con 0x7f7b48152df0 2024-09-10T20:44:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4d150640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b48074490 0x7f7b481f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4d150640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b48074490 0x7f7b481f5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52700/0 (socket says 172.21.15.28:52700) 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4d150640 1 -- 172.21.15.28:0/2755760611 learned_addr learned my addr 172.21.15.28:0/2755760611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4c94f640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b480d1840 0x7f7b48144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.283+0000 7f7b4d951640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b48152df0 0x7f7b481447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4d150640 1 -- 172.21.15.28:0/2755760611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b48152df0 msgr2=0x7f7b481447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4d150640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b48152df0 0x7f7b481447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4d150640 1 -- 172.21.15.28:0/2755760611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b480d1840 msgr2=0x7f7b48144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4d150640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b480d1840 0x7f7b48144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4d150640 1 -- 172.21.15.28:0/2755760611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b480cb080 con 0x7f7b48074490 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4d951640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b48152df0 0x7f7b481447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4d150640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b48074490 0x7f7b481f5230 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400c490 tx=0x7f7b4400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b3dffb640 1 -- 172.21.15.28:0/2755760611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44016020 con 0x7f7b48074490 2024-09-10T20:44:33.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b3dffb640 1 -- 172.21.15.28:0/2755760611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b44012a00 con 0x7f7b48074490 2024-09-10T20:44:33.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/2755760611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b480c7bc0 con 0x7f7b48074490 2024-09-10T20:44:33.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b3dffb640 1 -- 172.21.15.28:0/2755760611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44011380 con 0x7f7b48074490 2024-09-10T20:44:33.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.284+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/2755760611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b480c7600 con 0x7f7b48074490 2024-09-10T20:44:33.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.285+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/2755760611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b48073f50 con 0x7f7b48074490 2024-09-10T20:44:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.286+0000 7f7b3dffb640 1 -- 172.21.15.28:0/2755760611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f7b44011520 con 0x7f7b48074490 2024-09-10T20:44:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.286+0000 7f7b3dffb640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b2c0785d0 0x7f7b2c07aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.287+0000 7f7b3dffb640 1 -- 172.21.15.28:0/2755760611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f7b4409a930 con 0x7f7b48074490 2024-09-10T20:44:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.287+0000 7f7b4c94f640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b2c0785d0 0x7f7b2c07aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.287+0000 7f7b4c94f640 1 --2- 172.21.15.28:0/2755760611 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b2c0785d0 0x7f7b2c07aa90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7b38034000 tx=0x7f7b38004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.291+0000 7f7b3dffb640 1 -- 172.21.15.28:0/2755760611 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7b44020030 con 0x7f7b48074490 2024-09-10T20:44:33.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.508+0000 7f7b4f3b4640 1 -- 172.21.15.28:0/2755760611 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f7b4806fc10 con 0x7f7b2c0785d0 2024-09-10T20:44:33.514 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:33.509+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.509+0000 7f7b3dffb640 1 -- 172.21.15.28:0/2755760611 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f7b4806fc10 con 0x7f7b2c0785d0 2024-09-10T20:44:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:33.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info non_existent_subvolume subvolume_1 2024-09-10T20:44:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 -- 172.21.15.28:0/3318822305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2770161b50 msgr2=0x7f27700d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 --2- 172.21.15.28:0/3318822305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2770161b50 0x7f27700d2620 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f277008d410 tx=0x7f275802a4d0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 -- 172.21.15.28:0/3318822305 shutdown_connections 2024-09-10T20:44:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 --2- 172.21.15.28:0/3318822305 >> v2:172.21.15.159:3300/0 conn(0x7f27700d4b90 0x7f27700b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 --2- 172.21.15.28:0/3318822305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2770161b50 0x7f27700d2620 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 --2- 172.21.15.28:0/3318822305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2770161770 0x7f27700d20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 -- 172.21.15.28:0/3318822305 >> 172.21.15.28:0/3318822305 conn(0x7f27700d4070 msgr2=0x7f27700d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 -- 172.21.15.28:0/3318822305 shutdown_connections 2024-09-10T20:44:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.623+0000 7f2775823640 1 -- 172.21.15.28:0/3318822305 wait complete. 2024-09-10T20:44:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.624+0000 7f2775823640 1 Processor -- start 2024-09-10T20:44:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.625+0000 7f2775823640 1 -- start start 2024-09-10T20:44:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.625+0000 7f2775823640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27700d4b90 0x7f277015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.625+0000 7f2775823640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2770161770 0x7f2770156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.625+0000 7f2775823640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2770161b50 0x7f2770157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2775823640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27700b3f30 con 0x7f27700d4b90 2024-09-10T20:44:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2775823640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f27700b3db0 con 0x7f2770161770 2024-09-10T20:44:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2775823640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f27700b40b0 con 0x7f2770161b50 2024-09-10T20:44:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27700d4b90 0x7f277015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27700d4b90 0x7f277015bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52722/0 (socket says 172.21.15.28:52722) 2024-09-10T20:44:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 -- 172.21.15.28:0/359417952 learned_addr learned my addr 172.21.15.28:0/359417952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276f7fe640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2770161b50 0x7f2770157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276e7fc640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2770161770 0x7f2770156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 -- 172.21.15.28:0/359417952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2770161b50 msgr2=0x7f2770157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2770161b50 0x7f2770157190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 -- 172.21.15.28:0/359417952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2770161770 msgr2=0x7f2770156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2770161770 0x7f2770156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 -- 172.21.15.28:0/359417952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277015e200 con 0x7f27700d4b90 2024-09-10T20:44:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276e7fc640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2770161770 0x7f2770156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f276effd640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27700d4b90 0x7f277015bfd0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f276800ea80 tx=0x7f276800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2767fff640 1 -- 172.21.15.28:0/359417952 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276800ce40 con 0x7f27700d4b90 2024-09-10T20:44:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2767fff640 1 -- 172.21.15.28:0/359417952 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2768004c10 con 0x7f27700d4b90 2024-09-10T20:44:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2775823640 1 -- 172.21.15.28:0/359417952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f277015e4e0 con 0x7f27700d4b90 2024-09-10T20:44:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2767fff640 1 -- 172.21.15.28:0/359417952 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768005020 con 0x7f27700d4b90 2024-09-10T20:44:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.626+0000 7f2775823640 1 -- 172.21.15.28:0/359417952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f277015db80 con 0x7f27700d4b90 2024-09-10T20:44:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.627+0000 7f2775823640 1 -- 172.21.15.28:0/359417952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27700b45d0 con 0x7f27700d4b90 2024-09-10T20:44:33.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.628+0000 7f2767fff640 1 -- 172.21.15.28:0/359417952 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f276800f040 con 0x7f27700d4b90 2024-09-10T20:44:33.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.629+0000 7f2767fff640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27500785a0 0x7f275007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.629+0000 7f2767fff640 1 -- 172.21.15.28:0/359417952 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f276809b100 con 0x7f27700d4b90 2024-09-10T20:44:33.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.629+0000 7f276e7fc640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27500785a0 0x7f275007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.629+0000 7f276e7fc640 1 --2- 172.21.15.28:0/359417952 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27500785a0 0x7f275007aa60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2758033000 tx=0x7f2758004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:33.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.633+0000 7f2767fff640 1 -- 172.21.15.28:0/359417952 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2768016030 con 0x7f27700d4b90 2024-09-10T20:44:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.850+0000 7f2775823640 1 -- 172.21.15.28:0/359417952 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f277015e870 con 0x7f27500785a0 2024-09-10T20:44:33.856 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:33.851+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.851+0000 7f2767fff640 1 -- 172.21.15.28:0/359417952 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f277015e870 con 0x7f27500785a0 2024-09-10T20:44:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:33.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize non_existent_subvolume subvolname_1 inf 2024-09-10T20:44:33.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.968+0000 7f40a96fe640 1 -- 172.21.15.28:0/3270416924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40900052e0 msgr2=0x7f409000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.968+0000 7f40a96fe640 1 --2- 172.21.15.28:0/3270416924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40900052e0 0x7f409000b9c0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f409c00c630 tx=0x7f4090005830 comp rx=0 tx=0).stop 2024-09-10T20:44:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.968+0000 7f40a96fe640 1 -- 172.21.15.28:0/3270416924 shutdown_connections 2024-09-10T20:44:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.968+0000 7f40a96fe640 1 --2- 172.21.15.28:0/3270416924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40900052e0 0x7f409000b9c0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.968+0000 7f40a96fe640 1 -- 172.21.15.28:0/3270416924 >> 172.21.15.28:0/3270416924 conn(0x7f40a40d0410 msgr2=0x7f40a40d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.968+0000 7f40a96fe640 1 -- 172.21.15.28:0/3270416924 shutdown_connections 2024-09-10T20:44:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.968+0000 7f40a96fe640 1 -- 172.21.15.28:0/3270416924 wait complete. 2024-09-10T20:44:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.969+0000 7f40a96fe640 1 Processor -- start 2024-09-10T20:44:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a96fe640 1 -- start start 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a96fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40900052e0 0x7f40a40c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a96fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40a40d0810 0x7f40a41f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a96fe640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40a40ad1a0 0x7f40a40ad630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a96fe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f40a40739e0 con 0x7f40900052e0 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a96fe640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f40a40730b0 con 0x7f40a40ad1a0 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a96fe640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f40a406d850 con 0x7f40a40d0810 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40900052e0 0x7f40a40c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40900052e0 0x7f40a40c2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52750/0 (socket says 172.21.15.28:52750) 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.970+0000 7f40a2ffd640 1 -- 172.21.15.28:0/1130149609 learned_addr learned my addr 172.21.15.28:0/1130149609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f409bfff640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40a40d0810 0x7f40a41f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a37fe640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40a40ad1a0 0x7f40a40ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a2ffd640 1 -- 172.21.15.28:0/1130149609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40a40d0810 msgr2=0x7f40a41f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a2ffd640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f40a40d0810 0x7f40a41f4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a2ffd640 1 -- 172.21.15.28:0/1130149609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40a40ad1a0 msgr2=0x7f40a40ad630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a2ffd640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40a40ad1a0 0x7f40a40ad630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a2ffd640 1 -- 172.21.15.28:0/1130149609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a40b60e0 con 0x7f40900052e0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a37fe640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f40a40ad1a0 0x7f40a40ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a2ffd640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40900052e0 0x7f40a40c2ee0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f4094003f80 tx=0x7f409400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f409b7fe640 1 -- 172.21.15.28:0/1130149609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094010a60 con 0x7f40900052e0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f409b7fe640 1 -- 172.21.15.28:0/1130149609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4094015070 con 0x7f40900052e0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a96fe640 1 -- 172.21.15.28:0/1130149609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a40c7080 con 0x7f40900052e0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f409b7fe640 1 -- 172.21.15.28:0/1130149609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094010e80 con 0x7f40900052e0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.971+0000 7f40a96fe640 1 -- 172.21.15.28:0/1130149609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a40c6d00 con 0x7f40900052e0 2024-09-10T20:44:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.972+0000 7f40a96fe640 1 -- 172.21.15.28:0/1130149609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a415e880 con 0x7f40900052e0 2024-09-10T20:44:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.973+0000 7f409b7fe640 1 -- 172.21.15.28:0/1130149609 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f4094022030 con 0x7f40900052e0 2024-09-10T20:44:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.973+0000 7f409b7fe640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40840785a0 0x7f408407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.973+0000 7f409b7fe640 1 -- 172.21.15.28:0/1130149609 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f409409b350 con 0x7f40900052e0 2024-09-10T20:44:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.973+0000 7f409bfff640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40840785a0 0x7f408407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.974+0000 7f409bfff640 1 --2- 172.21.15.28:0/1130149609 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40840785a0 0x7f408407aa60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f408c000c00 tx=0x7f408c006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:33.977+0000 7f409b7fe640 1 -- 172.21.15.28:0/1130149609 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4094062da0 con 0x7f40900052e0 2024-09-10T20:44:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.198+0000 7f40a96fe640 1 -- 172.21.15.28:0/1130149609 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "non_existent_subvolume", "sub_name": "subvolname_1", "new_size": "inf", "target": ["mon-mgr", ""]}) -- 0x7f40a40b5e90 con 0x7f40840785a0 2024-09-10T20:44:34.204 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:34.200+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.200+0000 7f409b7fe640 1 -- 172.21.15.28:0/1130149609 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f40a40b5e90 con 0x7f40840785a0 2024-09-10T20:44:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:34.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume pin non_existent_subvolume subvolname_1 export 1 2024-09-10T20:44:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 -- 172.21.15.28:0/1624870466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc304073d90 msgr2=0x7fc30415f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 --2- 172.21.15.28:0/1624870466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc304073d90 0x7fc30415f3e0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fc3040ce810 tx=0x7fc2fc02a560 comp rx=0 tx=0).stop 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 -- 172.21.15.28:0/1624870466 shutdown_connections 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 --2- 172.21.15.28:0/1624870466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc304073d90 0x7fc30415f3e0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 --2- 172.21.15.28:0/1624870466 >> v2:172.21.15.159:3300/0 conn(0x7fc3040739b0 0x7fc3040ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 -- 172.21.15.28:0/1624870466 >> 172.21.15.28:0/1624870466 conn(0x7fc3040d0d50 msgr2=0x7fc3040d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 -- 172.21.15.28:0/1624870466 shutdown_connections 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.316+0000 7fc30c672640 1 -- 172.21.15.28:0/1624870466 wait complete. 2024-09-10T20:44:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.317+0000 7fc30c672640 1 Processor -- start 2024-09-10T20:44:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30c672640 1 -- start start 2024-09-10T20:44:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30c672640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3040739b0 0x7fc3041f83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30c672640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc304073d90 0x7fc3041f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30c672640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc304074480 0x7fc30414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30c672640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040cf650 con 0x7fc304073d90 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30c672640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040cf4d0 con 0x7fc304074480 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30c672640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc3040cf7d0 con 0x7fc3040739b0 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc309c0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc304073d90 0x7fc3041f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc309c0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc304073d90 0x7fc3041f4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52764/0 (socket says 172.21.15.28:52764) 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc309c0d640 1 -- 172.21.15.28:0/877567639 learned_addr learned my addr 172.21.15.28:0/877567639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.318+0000 7fc30a40e640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3040739b0 0x7fc3041f83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc30ac0f640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc304074480 0x7fc30414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc309c0d640 1 -- 172.21.15.28:0/877567639 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3040739b0 msgr2=0x7fc3041f83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc309c0d640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3040739b0 0x7fc3041f83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc309c0d640 1 -- 172.21.15.28:0/877567639 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc304074480 msgr2=0x7fc30414e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc309c0d640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc304074480 0x7fc30414e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc309c0d640 1 -- 172.21.15.28:0/877567639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3041f8910 con 0x7fc304073d90 2024-09-10T20:44:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc309c0d640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc304073d90 0x7fc3041f4740 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800c400 tx=0x7fc2f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc302ffd640 1 -- 172.21.15.28:0/877567639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8017070 con 0x7fc304073d90 2024-09-10T20:44:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc302ffd640 1 -- 172.21.15.28:0/877567639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f80053c0 con 0x7fc304073d90 2024-09-10T20:44:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc30c672640 1 -- 172.21.15.28:0/877567639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30406fb10 con 0x7fc304073d90 2024-09-10T20:44:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc302ffd640 1 -- 172.21.15.28:0/877567639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8003850 con 0x7fc304073d90 2024-09-10T20:44:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.319+0000 7fc30c672640 1 -- 172.21.15.28:0/877567639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3040700d0 con 0x7fc304073d90 2024-09-10T20:44:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.320+0000 7fc30c672640 1 -- 172.21.15.28:0/877567639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3040ccb90 con 0x7fc304073d90 2024-09-10T20:44:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.321+0000 7fc302ffd640 1 -- 172.21.15.28:0/877567639 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fc2f80039f0 con 0x7fc304073d90 2024-09-10T20:44:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.321+0000 7fc302ffd640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2f00785a0 0x7fc2f007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.321+0000 7fc302ffd640 1 -- 172.21.15.28:0/877567639 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fc2f809a000 con 0x7fc304073d90 2024-09-10T20:44:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.321+0000 7fc30a40e640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2f00785a0 0x7fc2f007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.322+0000 7fc30a40e640 1 --2- 172.21.15.28:0/877567639 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc2f00785a0 0x7fc2f007aa60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4007ec0 tx=0x7fc2f40075b0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.325+0000 7fc302ffd640 1 -- 172.21.15.28:0/877567639 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc2f8061a50 con 0x7fc304073d90 2024-09-10T20:44:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.544+0000 7fc30c672640 1 -- 172.21.15.28:0/877567639 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume pin", "vol_name": "non_existent_subvolume", "sub_name": "subvolname_1", "pin_type": "export", "pin_setting": "1", "target": ["mon-mgr", ""]}) -- 0x7fc304070850 con 0x7fc2f00785a0 2024-09-10T20:44:34.550 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:34.545+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.545+0000 7fc302ffd640 1 -- 172.21.15.28:0/877567639 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fc304070850 con 0x7fc2f00785a0 2024-09-10T20:44:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:34.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls non_existent_subvolume 2024-09-10T20:44:34.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.659+0000 7fe2ac74b640 1 -- 172.21.15.28:0/321943887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe290004800 msgr2=0x7fe290004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.659+0000 7fe2ac74b640 1 --2- 172.21.15.28:0/321943887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe290004800 0x7fe290004c80 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe294006fd0 tx=0x7fe290005830 comp rx=0 tx=0).stop 2024-09-10T20:44:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.659+0000 7fe2ac74b640 1 -- 172.21.15.28:0/321943887 shutdown_connections 2024-09-10T20:44:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.659+0000 7fe2ac74b640 1 --2- 172.21.15.28:0/321943887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe290004800 0x7fe290004c80 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.659+0000 7fe2ac74b640 1 -- 172.21.15.28:0/321943887 >> 172.21.15.28:0/321943887 conn(0x7fe2a406dbf0 msgr2=0x7fe2a406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.659+0000 7fe2ac74b640 1 -- 172.21.15.28:0/321943887 shutdown_connections 2024-09-10T20:44:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.659+0000 7fe2ac74b640 1 -- 172.21.15.28:0/321943887 wait complete. 2024-09-10T20:44:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.660+0000 7fe2ac74b640 1 Processor -- start 2024-09-10T20:44:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2ac74b640 1 -- start start 2024-09-10T20:44:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2ac74b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe290004800 0x7fe2a41f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2ac74b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2a4158010 0x7fe2a41f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2ac74b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2a4152980 0x7fe2a406cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2ac74b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a40d0d00 con 0x7fe2a4158010 2024-09-10T20:44:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2ac74b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a40b3cd0 con 0x7fe2a4152980 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2ac74b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a40d0e80 con 0x7fe290004800 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2a9ce6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2a4158010 0x7fe2a41f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2a9ce6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2a4158010 0x7fe2a41f0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52784/0 (socket says 172.21.15.28:52784) 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.661+0000 7fe2a9ce6640 1 -- 172.21.15.28:0/4100197297 learned_addr learned my addr 172.21.15.28:0/4100197297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2aa4e7640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe290004800 0x7fe2a41f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2aace8640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2a4152980 0x7fe2a406cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2a9ce6640 1 -- 172.21.15.28:0/4100197297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe290004800 msgr2=0x7fe2a41f1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2a9ce6640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe290004800 0x7fe2a41f1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2a9ce6640 1 -- 172.21.15.28:0/4100197297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2a4152980 msgr2=0x7fe2a406cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2a9ce6640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe2a4152980 0x7fe2a406cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2a9ce6640 1 -- 172.21.15.28:0/4100197297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a40ce0d0 con 0x7fe2a4158010 2024-09-10T20:44:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2a9ce6640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2a4158010 0x7fe2a41f0d40 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fe294000c00 tx=0x7fe294001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe29affd640 1 -- 172.21.15.28:0/4100197297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294006580 con 0x7fe2a4158010 2024-09-10T20:44:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe29affd640 1 -- 172.21.15.28:0/4100197297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe294003b10 con 0x7fe2a4158010 2024-09-10T20:44:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe29affd640 1 -- 172.21.15.28:0/4100197297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29402e480 con 0x7fe2a4158010 2024-09-10T20:44:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2ac74b640 1 -- 172.21.15.28:0/4100197297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a41f1280 con 0x7fe2a4158010 2024-09-10T20:44:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.662+0000 7fe2ac74b640 1 -- 172.21.15.28:0/4100197297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a41584a0 con 0x7fe2a4158010 2024-09-10T20:44:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.663+0000 7fe2ac74b640 1 -- 172.21.15.28:0/4100197297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a415bbe0 con 0x7fe2a4158010 2024-09-10T20:44:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.664+0000 7fe29affd640 1 -- 172.21.15.28:0/4100197297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fe29404a070 con 0x7fe2a4158010 2024-09-10T20:44:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.664+0000 7fe29affd640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe2880785a0 0x7fe28807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.664+0000 7fe29affd640 1 -- 172.21.15.28:0/4100197297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fe2940bb480 con 0x7fe2a4158010 2024-09-10T20:44:34.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.664+0000 7fe2aa4e7640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe2880785a0 0x7fe28807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:34.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.665+0000 7fe2aa4e7640 1 --2- 172.21.15.28:0/4100197297 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe2880785a0 0x7fe28807aa60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe2a415ca90 tx=0x7fe2a0006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:34.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.669+0000 7fe29affd640 1 -- 172.21.15.28:0/4100197297 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe2a41f1280 con 0x7fe2a4158010 2024-09-10T20:44:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.881+0000 7fe2ac74b640 1 -- 172.21.15.28:0/4100197297 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "non_existent_subvolume", "target": ["mon-mgr", ""]}) -- 0x7fe2a40705d0 con 0x7fe2880785a0 2024-09-10T20:44:34.887 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:34.882+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.882+0000 7fe29affd640 1 -- 172.21.15.28:0/4100197297 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fe2a40705d0 con 0x7fe2880785a0 2024-09-10T20:44:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:34.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create non_existent_subvolume subvolume_1 snapshot_1 2024-09-10T20:44:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.997+0000 7fa16d657640 1 -- 172.21.15.28:0/3644213276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa168162000 msgr2=0x7fa1680d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.997+0000 7fa16d657640 1 --2- 172.21.15.28:0/3644213276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa168162000 0x7fa1680d4a80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fa158009830 tx=0x7fa15802a640 comp rx=0 tx=0).stop 2024-09-10T20:44:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.997+0000 7fa16d657640 1 -- 172.21.15.28:0/3644213276 shutdown_connections 2024-09-10T20:44:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.997+0000 7fa16d657640 1 --2- 172.21.15.28:0/3644213276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa168162000 0x7fa1680d4a80 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.997+0000 7fa16d657640 1 -- 172.21.15.28:0/3644213276 >> 172.21.15.28:0/3644213276 conn(0x7fa1680d3f90 msgr2=0x7fa1680d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.997+0000 7fa16d657640 1 -- 172.21.15.28:0/3644213276 shutdown_connections 2024-09-10T20:44:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.997+0000 7fa16d657640 1 -- 172.21.15.28:0/3644213276 wait complete. 2024-09-10T20:44:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.999+0000 7fa16d657640 1 Processor -- start 2024-09-10T20:44:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.999+0000 7fa16d657640 1 -- start start 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.999+0000 7fa16d657640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1680b4c20 0x7fa1681f8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.999+0000 7fa16d657640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa168161850 0x7fa1681f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:34.999+0000 7fa16d657640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa168162000 0x7fa1680b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa16d657640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1680d2370 con 0x7fa168161850 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa16d657640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa1680d24f0 con 0x7fa168162000 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa16d657640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa1680d2670 con 0x7fa1680b4c20 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa168161850 0x7fa1681f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa168161850 0x7fa1681f4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52802/0 (socket says 172.21.15.28:52802) 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 -- 172.21.15.28:0/978531916 learned_addr learned my addr 172.21.15.28:0/978531916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa166ffd640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1680b4c20 0x7fa1681f8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1677fe640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa168162000 0x7fa1680b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 -- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1680b4c20 msgr2=0x7fa1681f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1680b4c20 0x7fa1681f8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 -- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa168162000 msgr2=0x7fa1680b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa168162000 0x7fa1680b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 -- 172.21.15.28:0/978531916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1681f5450 con 0x7fa168161850 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa166ffd640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1680b4c20 0x7fa1681f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1677fe640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa168162000 0x7fa1680b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa1667fc640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa168161850 0x7fa1681f4d00 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa15000c450 tx=0x7fa15000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa15f7fe640 1 -- 172.21.15.28:0/978531916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1500052d0 con 0x7fa168161850 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa15f7fe640 1 -- 172.21.15.28:0/978531916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa150005470 con 0x7fa168161850 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa16d657640 1 -- 172.21.15.28:0/978531916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1681f5730 con 0x7fa168161850 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa15f7fe640 1 -- 172.21.15.28:0/978531916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150012870 con 0x7fa168161850 2024-09-10T20:44:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.000+0000 7fa16d657640 1 -- 172.21.15.28:0/978531916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1680cb220 con 0x7fa168161850 2024-09-10T20:44:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.001+0000 7fa16d657640 1 -- 172.21.15.28:0/978531916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1680cb3f0 con 0x7fa168161850 2024-09-10T20:44:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.002+0000 7fa15f7fe640 1 -- 172.21.15.28:0/978531916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fa1500035f0 con 0x7fa168161850 2024-09-10T20:44:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.003+0000 7fa15f7fe640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1480785a0 0x7fa14807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.003+0000 7fa166ffd640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1480785a0 0x7fa14807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.003+0000 7fa15f7fe640 1 -- 172.21.15.28:0/978531916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fa15009a550 con 0x7fa168161850 2024-09-10T20:44:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.003+0000 7fa166ffd640 1 --2- 172.21.15.28:0/978531916 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1480785a0 0x7fa14807aa60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa158034000 tx=0x7fa158004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.007+0000 7fa15f7fe640 1 -- 172.21.15.28:0/978531916 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa1500625e0 con 0x7fa168161850 2024-09-10T20:44:35.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.225+0000 7fa16d657640 1 -- 172.21.15.28:0/978531916 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fa1680b9f10 con 0x7fa1480785a0 2024-09-10T20:44:35.231 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:35.227+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:35.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.227+0000 7fa15f7fe640 1 -- 172.21.15.28:0/978531916 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fa1680b9f10 con 0x7fa1480785a0 2024-09-10T20:44:35.232 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:35.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm non_existent_subvolume subvolume_1 snapshot_1 2024-09-10T20:44:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.344+0000 7f559a3a0640 1 -- 172.21.15.28:0/4204480961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557c004040 msgr2=0x7f557c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.344+0000 7f559a3a0640 1 --2- 172.21.15.28:0/4204480961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557c004040 0x7f557c004800 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f558800bb20 tx=0x7f557c005830 comp rx=0 tx=0).stop 2024-09-10T20:44:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.344+0000 7f559893d640 1 -- 172.21.15.28:0/4204480961 reap_dead start 2024-09-10T20:44:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.344+0000 7f559a3a0640 1 -- 172.21.15.28:0/4204480961 shutdown_connections 2024-09-10T20:44:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.344+0000 7f559a3a0640 1 -- 172.21.15.28:0/4204480961 >> 172.21.15.28:0/4204480961 conn(0x7f55940d3f90 msgr2=0x7f55940d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.344+0000 7f559a3a0640 1 -- 172.21.15.28:0/4204480961 shutdown_connections 2024-09-10T20:44:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.344+0000 7f559a3a0640 1 -- 172.21.15.28:0/4204480961 wait complete. 2024-09-10T20:44:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.345+0000 7f559a3a0640 1 Processor -- start 2024-09-10T20:44:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.345+0000 7f559a3a0640 1 -- start start 2024-09-10T20:44:35.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559a3a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557c004040 0x7f55940b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559a3a0640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55940cd790 0x7f55940cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559a3a0640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55940cc5f0 0x7f55940ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559a3a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55940d4c00 con 0x7f557c004040 2024-09-10T20:44:35.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559a3a0640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f55940d4a80 con 0x7f55940cc5f0 2024-09-10T20:44:35.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559a3a0640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f55940d4d80 con 0x7f55940cd790 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557c004040 0x7f55940b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557c004040 0x7f55940b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52818/0 (socket says 172.21.15.28:52818) 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 -- 172.21.15.28:0/2002468305 learned_addr learned my addr 172.21.15.28:0/2002468305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559893d640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55940cc5f0 0x7f55940ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f55937fe640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55940cd790 0x7f55940cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 -- 172.21.15.28:0/2002468305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55940cd790 msgr2=0x7f55940cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f55940cd790 0x7f55940cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 -- 172.21.15.28:0/2002468305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55940cc5f0 msgr2=0x7f55940ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55940cc5f0 0x7f55940ccaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f5593fff640 1 -- 172.21.15.28:0/2002468305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5594160820 con 0x7f557c004040 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.346+0000 7f559893d640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f55940cc5f0 0x7f55940ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.347+0000 7f5593fff640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f557c004040 0x7f55940b0ac0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f55840066e0 tx=0x7f5584003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:35.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.347+0000 7f5590ff9640 1 -- 172.21.15.28:0/2002468305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55840013b0 con 0x7f557c004040 2024-09-10T20:44:35.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.347+0000 7f5590ff9640 1 -- 172.21.15.28:0/2002468305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5584016020 con 0x7f557c004040 2024-09-10T20:44:35.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.347+0000 7f559a3a0640 1 -- 172.21.15.28:0/2002468305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5594160b00 con 0x7f557c004040 2024-09-10T20:44:35.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.347+0000 7f5590ff9640 1 -- 172.21.15.28:0/2002468305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5584011940 con 0x7f557c004040 2024-09-10T20:44:35.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.347+0000 7f559a3a0640 1 -- 172.21.15.28:0/2002468305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5594145410 con 0x7f557c004040 2024-09-10T20:44:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.348+0000 7f559a3a0640 1 -- 172.21.15.28:0/2002468305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55940d1c60 con 0x7f557c004040 2024-09-10T20:44:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.348+0000 7f5590ff9640 1 -- 172.21.15.28:0/2002468305 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f5584011ae0 con 0x7f557c004040 2024-09-10T20:44:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.349+0000 7f5590ff9640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55800785d0 0x7f558007aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.349+0000 7f5590ff9640 1 -- 172.21.15.28:0/2002468305 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f558409b470 con 0x7f557c004040 2024-09-10T20:44:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.349+0000 7f55937fe640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55800785d0 0x7f558007aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.349+0000 7f55937fe640 1 --2- 172.21.15.28:0/2002468305 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f55800785d0 0x7f558007aa90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5588036000 tx=0x7f55880046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:35.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.353+0000 7f5590ff9640 1 -- 172.21.15.28:0/2002468305 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5584020020 con 0x7f557c004040 2024-09-10T20:44:35.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.571+0000 7f559a3a0640 1 -- 172.21.15.28:0/2002468305 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f55940b9bb0 con 0x7f55800785d0 2024-09-10T20:44:35.577 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:35.573+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.573+0000 7f5590ff9640 1 -- 172.21.15.28:0/2002468305 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f55940b9bb0 con 0x7f55800785d0 2024-09-10T20:44:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:35.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot info non_existent_subvolume subvolume_1 snapshot_1 2024-09-10T20:44:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.689+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/557115723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cec161850 msgr2=0x7f3cec0d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.689+0000 7f3cf0c8c640 1 --2- 172.21.15.28:0/557115723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cec161850 0x7f3cec0d4a80 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009830 tx=0x7f3cd402a640 comp rx=0 tx=0).stop 2024-09-10T20:44:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.689+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/557115723 shutdown_connections 2024-09-10T20:44:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.689+0000 7f3cf0c8c640 1 --2- 172.21.15.28:0/557115723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cec161850 0x7f3cec0d4a80 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.689+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/557115723 >> 172.21.15.28:0/557115723 conn(0x7f3cec0d3f90 msgr2=0x7f3cec0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.689+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/557115723 shutdown_connections 2024-09-10T20:44:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.689+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/557115723 wait complete. 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 Processor -- start 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 -- start start 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cec0b4c20 0x7f3cec1f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cec161850 0x7f3cec1f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cec162000 0x7f3cec159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec0d24f0 con 0x7f3cec0b4c20 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec0d2370 con 0x7f3cec161850 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cf0c8c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3cec0d2670 con 0x7f3cec162000 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cec0b4c20 0x7f3cec1f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cec0b4c20 0x7f3cec1f5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52834/0 (socket says 172.21.15.28:52834) 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.691+0000 7f3cea59c640 1 -- 172.21.15.28:0/3482635577 learned_addr learned my addr 172.21.15.28:0/3482635577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3ce9d9b640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cec161850 0x7f3cec1f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cead9d640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cec162000 0x7f3cec159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cea59c640 1 -- 172.21.15.28:0/3482635577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cec162000 msgr2=0x7f3cec159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cea59c640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cec162000 0x7f3cec159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cea59c640 1 -- 172.21.15.28:0/3482635577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cec161850 msgr2=0x7f3cec1f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cea59c640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cec161850 0x7f3cec1f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cea59c640 1 -- 172.21.15.28:0/3482635577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec1f8d30 con 0x7f3cec0b4c20 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cea59c640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cec0b4c20 0x7f3cec1f5720 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400c730 tx=0x7f3ce400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3ce2ffd640 1 -- 172.21.15.28:0/3482635577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4003b70 con 0x7f3cec0b4c20 2024-09-10T20:44:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3ce2ffd640 1 -- 172.21.15.28:0/3482635577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ce4003d30 con 0x7f3cec0b4c20 2024-09-10T20:44:35.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/3482635577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cec1f9010 con 0x7f3cec0b4c20 2024-09-10T20:44:35.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3ce2ffd640 1 -- 172.21.15.28:0/3482635577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce400aab0 con 0x7f3cec0b4c20 2024-09-10T20:44:35.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.692+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/3482635577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cec1f5200 con 0x7f3cec0b4c20 2024-09-10T20:44:35.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.693+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/3482635577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cec1f53d0 con 0x7f3cec0b4c20 2024-09-10T20:44:35.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.694+0000 7f3ce2ffd640 1 -- 172.21.15.28:0/3482635577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f3ce400ac50 con 0x7f3cec0b4c20 2024-09-10T20:44:35.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.694+0000 7f3ce2ffd640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ccc0785a0 0x7f3ccc07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:35.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.694+0000 7f3ce2ffd640 1 -- 172.21.15.28:0/3482635577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f3ce409a640 con 0x7f3cec0b4c20 2024-09-10T20:44:35.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.695+0000 7f3ce9d9b640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ccc0785a0 0x7f3ccc07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:35.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.695+0000 7f3ce9d9b640 1 --2- 172.21.15.28:0/3482635577 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ccc0785a0 0x7f3ccc07aa60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4034000 tx=0x7f3cd4004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.699+0000 7f3ce2ffd640 1 -- 172.21.15.28:0/3482635577 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3ce4062100 con 0x7f3cec0b4c20 2024-09-10T20:44:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.918+0000 7f3cf0c8c640 1 -- 172.21.15.28:0/3482635577 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot info", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f3cec0d4500 con 0x7f3ccc0785a0 2024-09-10T20:44:35.924 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:35.919+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:35.919+0000 7f3ce2ffd640 1 -- 172.21.15.28:0/3482635577 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f3cec0d4500 con 0x7f3ccc0785a0 2024-09-10T20:44:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:35.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot protect non_existent_subvolume subvolume_1 snapshot_1 2024-09-10T20:44:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 -- 172.21.15.28:0/523392433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe648073e10 msgr2=0x7fe6480add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 --2- 172.21.15.28:0/523392433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe648073e10 0x7fe6480add70 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fe648072aa0 tx=0x7fe64002aa30 comp rx=0 tx=0).stop 2024-09-10T20:44:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 -- 172.21.15.28:0/523392433 shutdown_connections 2024-09-10T20:44:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 --2- 172.21.15.28:0/523392433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe648073e10 0x7fe6480add70 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 --2- 172.21.15.28:0/523392433 >> v2:172.21.15.159:3300/0 conn(0x7fe64815ab00 0x7fe6480b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 -- 172.21.15.28:0/523392433 >> 172.21.15.28:0/523392433 conn(0x7fe6480b2920 msgr2=0x7fe6480b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 -- 172.21.15.28:0/523392433 shutdown_connections 2024-09-10T20:44:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.035+0000 7fe64ef94640 1 -- 172.21.15.28:0/523392433 wait complete. 2024-09-10T20:44:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.036+0000 7fe64ef94640 1 Processor -- start 2024-09-10T20:44:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.036+0000 7fe64ef94640 1 -- start start 2024-09-10T20:44:36.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64ef94640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe648070e50 0x7fe6480c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64ef94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe648073e10 0x7fe6481f87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64ef94640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe64815ab00 0x7fe648154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64ef94640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe648071440 con 0x7fe648073e10 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64ef94640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe6480712c0 con 0x7fe64815ab00 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64ef94640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe6480715c0 con 0x7fe648070e50 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe648073e10 0x7fe6481f87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe648073e10 0x7fe6481f87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52866/0 (socket says 172.21.15.28:52866) 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 -- 172.21.15.28:0/415281958 learned_addr learned my addr 172.21.15.28:0/415281958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64cd30640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe648070e50 0x7fe6480c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe64d531640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe64815ab00 0x7fe648154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 -- 172.21.15.28:0/415281958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe648070e50 msgr2=0x7fe6480c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe648070e50 0x7fe6480c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 -- 172.21.15.28:0/415281958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe64815ab00 msgr2=0x7fe648154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe64815ab00 0x7fe648154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.037+0000 7fe63ffff640 1 -- 172.21.15.28:0/415281958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe648153a30 con 0x7fe648073e10 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.038+0000 7fe63ffff640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe648073e10 0x7fe6481f87f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fe63800c430 tx=0x7fe63800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.038+0000 7fe63d7fa640 1 -- 172.21.15.28:0/415281958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638011790 con 0x7fe648073e10 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.038+0000 7fe63d7fa640 1 -- 172.21.15.28:0/415281958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe638017070 con 0x7fe648073e10 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.038+0000 7fe64ef94640 1 -- 172.21.15.28:0/415281958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe648153d10 con 0x7fe648073e10 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.038+0000 7fe63d7fa640 1 -- 172.21.15.28:0/415281958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638011d00 con 0x7fe648073e10 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.038+0000 7fe64ef94640 1 -- 172.21.15.28:0/415281958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6481542d0 con 0x7fe648073e10 2024-09-10T20:44:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.039+0000 7fe64ef94640 1 -- 172.21.15.28:0/415281958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6480c63f0 con 0x7fe648073e10 2024-09-10T20:44:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.039+0000 7fe63d7fa640 1 -- 172.21.15.28:0/415281958 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fe63801c460 con 0x7fe648073e10 2024-09-10T20:44:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.040+0000 7fe63d7fa640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6340785a0 0x7fe63407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.040+0000 7fe63d7fa640 1 -- 172.21.15.28:0/415281958 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fe63809a600 con 0x7fe648073e10 2024-09-10T20:44:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.040+0000 7fe64cd30640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6340785a0 0x7fe63407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.040+0000 7fe64cd30640 1 --2- 172.21.15.28:0/415281958 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6340785a0 0x7fe63407aa60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe644004630 tx=0x7fe64400b2e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:36.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.044+0000 7fe63d7fa640 1 -- 172.21.15.28:0/415281958 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe6480c63f0 con 0x7fe648073e10 2024-09-10T20:44:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.263+0000 7fe64ef94640 1 -- 172.21.15.28:0/415281958 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot protect", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fe64806da10 con 0x7fe6340785a0 2024-09-10T20:44:36.269 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:36.264+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:36.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.264+0000 7fe63d7fa640 1 -- 172.21.15.28:0/415281958 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fe64806da10 con 0x7fe6340785a0 2024-09-10T20:44:36.269 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:36.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot unprotect non_existent_subvolume subvolume_1 snapshot_1 2024-09-10T20:44:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.383+0000 7f8972710640 1 -- 172.21.15.28:0/87333355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540051c0 msgr2=0x7f895400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.383+0000 7f8972710640 1 --2- 172.21.15.28:0/87333355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540051c0 0x7f895400b9c0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f8964009ed0 tx=0x7f8954005830 comp rx=0 tx=0).stop 2024-09-10T20:44:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.383+0000 7f8972710640 1 -- 172.21.15.28:0/87333355 shutdown_connections 2024-09-10T20:44:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.383+0000 7f8972710640 1 --2- 172.21.15.28:0/87333355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89540051c0 0x7f895400b9c0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.383+0000 7f8972710640 1 -- 172.21.15.28:0/87333355 >> 172.21.15.28:0/87333355 conn(0x7f896c0b2920 msgr2=0x7f896c0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.384+0000 7f8972710640 1 -- 172.21.15.28:0/87333355 shutdown_connections 2024-09-10T20:44:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.384+0000 7f8972710640 1 -- 172.21.15.28:0/87333355 wait complete. 2024-09-10T20:44:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.385+0000 7f8972710640 1 Processor -- start 2024-09-10T20:44:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.385+0000 7f8972710640 1 -- start start 2024-09-10T20:44:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8972710640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540051c0 0x7f896c09d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8972710640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c146750 0x7f896c09dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8972710640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f896c143440 0x7f896c1f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8972710640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f896c15aee0 con 0x7f896c146750 2024-09-10T20:44:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8972710640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f896c0b4630 con 0x7f896c143440 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8972710640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f896c15b060 con 0x7f89540051c0 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c146750 0x7f896c09dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c146750 0x7f896c09dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52886/0 (socket says 172.21.15.28:52886) 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 -- 172.21.15.28:0/664681161 learned_addr learned my addr 172.21.15.28:0/664681161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896bfff640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540051c0 0x7f896c09d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8970cad640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f896c143440 0x7f896c1f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 -- 172.21.15.28:0/664681161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540051c0 msgr2=0x7f896c09d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89540051c0 0x7f896c09d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 -- 172.21.15.28:0/664681161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f896c143440 msgr2=0x7f896c1f5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f896c143440 0x7f896c1f5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 -- 172.21.15.28:0/664681161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c1f66c0 con 0x7f896c146750 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8970cad640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f896c143440 0x7f896c1f5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f896b7fe640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c146750 0x7f896c09dc50 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f8960004f30 tx=0x7f89600016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8968ff9640 1 -- 172.21.15.28:0/664681161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960015020 con 0x7f896c146750 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8968ff9640 1 -- 172.21.15.28:0/664681161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8960011960 con 0x7f896c146750 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8968ff9640 1 -- 172.21.15.28:0/664681161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960011d70 con 0x7f896c146750 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.386+0000 7f8972710640 1 -- 172.21.15.28:0/664681161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c145c70 con 0x7f896c146750 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.387+0000 7f8972710640 1 -- 172.21.15.28:0/664681161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896c146230 con 0x7f896c146750 2024-09-10T20:44:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.387+0000 7f8972710640 1 -- 172.21.15.28:0/664681161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f896c073980 con 0x7f896c146750 2024-09-10T20:44:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.388+0000 7f8968ff9640 1 -- 172.21.15.28:0/664681161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f896001a400 con 0x7f896c146750 2024-09-10T20:44:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.389+0000 7f8968ff9640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f89580785d0 0x7f895807aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.389+0000 7f8968ff9640 1 -- 172.21.15.28:0/664681161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f896009b690 con 0x7f896c146750 2024-09-10T20:44:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.389+0000 7f896bfff640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f89580785d0 0x7f895807aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.389+0000 7f896bfff640 1 --2- 172.21.15.28:0/664681161 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f89580785d0 0x7f895807aa90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f896c0afb90 tx=0x7f895c0067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.393+0000 7f8968ff9640 1 -- 172.21.15.28:0/664681161 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f896001f020 con 0x7f896c146750 2024-09-10T20:44:36.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.611+0000 7f8972710640 1 -- 172.21.15.28:0/664681161 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot unprotect", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f896c0b4e50 con 0x7f89580785d0 2024-09-10T20:44:36.617 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:36.613+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:36.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.613+0000 7f8968ff9640 1 -- 172.21.15.28:0/664681161 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f896c0b4e50 con 0x7f89580785d0 2024-09-10T20:44:36.618 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:36.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls non_existent_subvolume subvolume_1 2024-09-10T20:44:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 -- 172.21.15.28:0/407586363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749015bfc0 msgr2=0x7f74900cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 --2- 172.21.15.28:0/407586363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749015bfc0 0x7f74900cf3c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f7490073c60 tx=0x7f748002a5f0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 -- 172.21.15.28:0/407586363 shutdown_connections 2024-09-10T20:44:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 --2- 172.21.15.28:0/407586363 >> v2:172.21.15.159:3300/0 conn(0x7f74900d0f30 0x7f74900cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 --2- 172.21.15.28:0/407586363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749015bfc0 0x7f74900cf3c0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 -- 172.21.15.28:0/407586363 >> 172.21.15.28:0/407586363 conn(0x7f74900d0410 msgr2=0x7f74900d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 -- 172.21.15.28:0/407586363 shutdown_connections 2024-09-10T20:44:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.727+0000 7f749693d640 1 -- 172.21.15.28:0/407586363 wait complete. 2024-09-10T20:44:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.728+0000 7f749693d640 1 Processor -- start 2024-09-10T20:44:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f749693d640 1 -- start start 2024-09-10T20:44:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f749693d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74900cfe40 0x7f74900c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f749693d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74900d0f30 0x7f74900c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f749693d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f749015bfc0 0x7f74900c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f749693d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74900b1250 con 0x7f74900d0f30 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f749693d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f74900b10d0 con 0x7f749015bfc0 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f749693d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f74900b13d0 con 0x7f74900cfe40 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.729+0000 7f748f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74900d0f30 0x7f74900c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74900d0f30 0x7f74900c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52892/0 (socket says 172.21.15.28:52892) 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 -- 172.21.15.28:0/3443027972 learned_addr learned my addr 172.21.15.28:0/3443027972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f7494eda640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f749015bfc0 0x7f74900c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748ffff640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74900cfe40 0x7f74900c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 -- 172.21.15.28:0/3443027972 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74900cfe40 msgr2=0x7f74900c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74900cfe40 0x7f74900c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 -- 172.21.15.28:0/3443027972 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f749015bfc0 msgr2=0x7f74900c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f749015bfc0 0x7f74900c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 -- 172.21.15.28:0/3443027972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749015c720 con 0x7f74900d0f30 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f7494eda640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f749015bfc0 0x7f74900c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748f7fe640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74900d0f30 0x7f74900c9500 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f748400d770 tx=0x7f748400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748cff9640 1 -- 172.21.15.28:0/3443027972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7484017070 con 0x7f74900d0f30 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748cff9640 1 -- 172.21.15.28:0/3443027972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7484009d20 con 0x7f74900d0f30 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f749693d640 1 -- 172.21.15.28:0/3443027972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749015ca00 con 0x7f74900d0f30 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f748cff9640 1 -- 172.21.15.28:0/3443027972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7484005280 con 0x7f74900d0f30 2024-09-10T20:44:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.730+0000 7f749693d640 1 -- 172.21.15.28:0/3443027972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f749015cfc0 con 0x7f74900d0f30 2024-09-10T20:44:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.731+0000 7f749693d640 1 -- 172.21.15.28:0/3443027972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7490160660 con 0x7f74900d0f30 2024-09-10T20:44:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.732+0000 7f748cff9640 1 -- 172.21.15.28:0/3443027972 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f7484005420 con 0x7f74900d0f30 2024-09-10T20:44:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.733+0000 7f748cff9640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f747c0785a0 0x7f747c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.733+0000 7f748ffff640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f747c0785a0 0x7f747c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.733+0000 7f748cff9640 1 -- 172.21.15.28:0/3443027972 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f748409ab20 con 0x7f74900d0f30 2024-09-10T20:44:36.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.733+0000 7f748ffff640 1 --2- 172.21.15.28:0/3443027972 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f747c0785a0 0x7f747c07aa60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7480033000 tx=0x7f7480004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.737+0000 7f748cff9640 1 -- 172.21.15.28:0/3443027972 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7484062c40 con 0x7f74900d0f30 2024-09-10T20:44:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.951+0000 7f749693d640 1 -- 172.21.15.28:0/3443027972 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f7490076660 con 0x7f747c0785a0 2024-09-10T20:44:36.957 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:36.952+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:36.953+0000 7f748cff9640 1 -- 172.21.15.28:0/3443027972 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f7490076660 con 0x7f747c0785a0 2024-09-10T20:44:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:36.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone non_existent_subvolume subvolume_1 snapshot_1 clone_1 2024-09-10T20:44:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.070+0000 7fc8385f3640 1 -- 172.21.15.28:0/803032831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc818003a50 msgr2=0x7fc818003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.070+0000 7fc8385f3640 1 --2- 172.21.15.28:0/803032831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc818003a50 0x7fc818003e70 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fc82800c2e0 tx=0x7fc818005830 comp rx=0 tx=0).stop 2024-09-10T20:44:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.070+0000 7fc8385f3640 1 -- 172.21.15.28:0/803032831 shutdown_connections 2024-09-10T20:44:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.070+0000 7fc8385f3640 1 --2- 172.21.15.28:0/803032831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc818003a50 0x7fc818003e70 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.070+0000 7fc8385f3640 1 -- 172.21.15.28:0/803032831 >> 172.21.15.28:0/803032831 conn(0x7fc83006d110 msgr2=0x7fc8300b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.070+0000 7fc8385f3640 1 -- 172.21.15.28:0/803032831 shutdown_connections 2024-09-10T20:44:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.070+0000 7fc8385f3640 1 -- 172.21.15.28:0/803032831 wait complete. 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.071+0000 7fc8385f3640 1 Processor -- start 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.071+0000 7fc8385f3640 1 -- start start 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc8385f3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc818003a50 0x7fc8301e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc8385f3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83014afe0 0x7fc830150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc8385f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc830158f50 0x7fc830150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc8385f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8300b2100 con 0x7fc830158f50 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc8385f3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc8300b1f80 con 0x7fc83014afe0 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc8385f3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc8300b2280 con 0x7fc818003a50 2024-09-10T20:44:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc830158f50 0x7fc830150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc830158f50 0x7fc830150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52902/0 (socket says 172.21.15.28:52902) 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 -- 172.21.15.28:0/3151030107 learned_addr learned my addr 172.21.15.28:0/3151030107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc83638f640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc818003a50 0x7fc8301e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc835b8e640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83014afe0 0x7fc830150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 -- 172.21.15.28:0/3151030107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc818003a50 msgr2=0x7fc8301e91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc818003a50 0x7fc8301e91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 -- 172.21.15.28:0/3151030107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83014afe0 msgr2=0x7fc830150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83014afe0 0x7fc830150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc836b90640 1 -- 172.21.15.28:0/3151030107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83014b870 con 0x7fc830158f50 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.072+0000 7fc83638f640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc818003a50 0x7fc8301e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.073+0000 7fc836b90640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc830158f50 0x7fc830150c60 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fc8300b5260 tx=0x7fc8200019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.073+0000 7fc82effd640 1 -- 172.21.15.28:0/3151030107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820011870 con 0x7fc830158f50 2024-09-10T20:44:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.073+0000 7fc82effd640 1 -- 172.21.15.28:0/3151030107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc820017070 con 0x7fc830158f50 2024-09-10T20:44:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.073+0000 7fc8385f3640 1 -- 172.21.15.28:0/3151030107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83013f6b0 con 0x7fc830158f50 2024-09-10T20:44:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.073+0000 7fc82effd640 1 -- 172.21.15.28:0/3151030107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820011da0 con 0x7fc830158f50 2024-09-10T20:44:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.073+0000 7fc8385f3640 1 -- 172.21.15.28:0/3151030107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83013fc30 con 0x7fc830158f50 2024-09-10T20:44:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.074+0000 7fc8385f3640 1 -- 172.21.15.28:0/3151030107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8300ced20 con 0x7fc830158f50 2024-09-10T20:44:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.074+0000 7fc82effd640 1 -- 172.21.15.28:0/3151030107 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7fc82001c460 con 0x7fc830158f50 2024-09-10T20:44:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.075+0000 7fc82effd640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc81c0785d0 0x7fc81c07aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.075+0000 7fc82effd640 1 -- 172.21.15.28:0/3151030107 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fc82009b150 con 0x7fc830158f50 2024-09-10T20:44:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.075+0000 7fc83638f640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc81c0785d0 0x7fc81c07aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.076+0000 7fc83638f640 1 --2- 172.21.15.28:0/3151030107 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc81c0785d0 0x7fc81c07aa90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc828033000 tx=0x7fc8300b3ec0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.079+0000 7fc82effd640 1 -- 172.21.15.28:0/3151030107 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc820063270 con 0x7fc830158f50 2024-09-10T20:44:37.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.296+0000 7fc8385f3640 1 -- 172.21.15.28:0/3151030107 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target_sub_name": "clone_1", "target": ["mon-mgr", ""]}) -- 0x7fc8300cfdc0 con 0x7fc81c0785d0 2024-09-10T20:44:37.306 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.302+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:37.307 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.302+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:37.307 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.302+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:37.307 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.302+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:37.307 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.302+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:44:37.311 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.307+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.307+0000 7fc82effd640 1 -- 172.21.15.28:0/3151030107 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fc8300cfdc0 con 0x7fc81c0785d0 2024-09-10T20:44:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:37.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status non_existent_subvolume clone_1 2024-09-10T20:44:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.421+0000 7f50831ad640 1 -- 172.21.15.28:0/4007758409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5064004d40 msgr2=0x7f50640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.421+0000 7f50831ad640 1 --2- 172.21.15.28:0/4007758409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5064004d40 0x7f50640051c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f507400c610 tx=0x7f5064005830 comp rx=0 tx=0).stop 2024-09-10T20:44:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.421+0000 7f508174a640 1 -- 172.21.15.28:0/4007758409 reap_dead start 2024-09-10T20:44:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.421+0000 7f50831ad640 1 -- 172.21.15.28:0/4007758409 shutdown_connections 2024-09-10T20:44:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.421+0000 7f50831ad640 1 -- 172.21.15.28:0/4007758409 >> 172.21.15.28:0/4007758409 conn(0x7f507c0d0d50 msgr2=0x7f507c0d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.421+0000 7f50831ad640 1 -- 172.21.15.28:0/4007758409 shutdown_connections 2024-09-10T20:44:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.421+0000 7f50831ad640 1 -- 172.21.15.28:0/4007758409 wait complete. 2024-09-10T20:44:37.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.423+0000 7f50831ad640 1 Processor -- start 2024-09-10T20:44:37.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.423+0000 7f50831ad640 1 -- start start 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f507c14f1b0 0x7f507c0ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f507c1f2910 0x7f507c0b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0b3b80 0x7f507c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f507c070a70 con 0x7f507c0b3b80 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f507c0708f0 con 0x7f507c1f2910 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f507c15f000 con 0x7f507c14f1b0 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0b3b80 0x7f507c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0b3b80 0x7f507c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52950/0 (socket says 172.21.15.28:52950) 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 -- 172.21.15.28:0/1249228121 learned_addr learned my addr 172.21.15.28:0/1249228121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f507bfff640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f507c1f2910 0x7f507c0b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f5080f49640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f507c14f1b0 0x7f507c0ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 -- 172.21.15.28:0/1249228121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f507c14f1b0 msgr2=0x7f507c0ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f507c14f1b0 0x7f507c0ca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 -- 172.21.15.28:0/1249228121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f507c1f2910 msgr2=0x7f507c0b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f507c1f2910 0x7f507c0b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 -- 172.21.15.28:0/1249228121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c0c9140 con 0x7f507c0b3b80 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f508174a640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0b3b80 0x7f507c0b0cf0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f5074000f80 tx=0x7f5074004c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50797fa640 1 -- 172.21.15.28:0/1249228121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507402ba10 con 0x7f507c0b3b80 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50797fa640 1 -- 172.21.15.28:0/1249228121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f507402bca0 con 0x7f507c0b3b80 2024-09-10T20:44:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 -- 172.21.15.28:0/1249228121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507c1f2da0 con 0x7f507c0b3b80 2024-09-10T20:44:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50797fa640 1 -- 172.21.15.28:0/1249228121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074006300 con 0x7f507c0b3b80 2024-09-10T20:44:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.424+0000 7f50831ad640 1 -- 172.21.15.28:0/1249228121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f507c1f3360 con 0x7f507c0b3b80 2024-09-10T20:44:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.426+0000 7f50831ad640 1 -- 172.21.15.28:0/1249228121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f507c15f590 con 0x7f507c0b3b80 2024-09-10T20:44:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.426+0000 7f50797fa640 1 -- 172.21.15.28:0/1249228121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f5074048060 con 0x7f507c0b3b80 2024-09-10T20:44:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.427+0000 7f50797fa640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f50680785d0 0x7f506807aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.427+0000 7f50797fa640 1 -- 172.21.15.28:0/1249228121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f50740ba500 con 0x7f507c0b3b80 2024-09-10T20:44:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.427+0000 7f5080f49640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f50680785d0 0x7f506807aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.427+0000 7f5080f49640 1 --2- 172.21.15.28:0/1249228121 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f50680785d0 0x7f506807aa90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f506c009a10 tx=0x7f506c005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:37.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.431+0000 7f50797fa640 1 -- 172.21.15.28:0/1249228121 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5074082590 con 0x7f507c0b3b80 2024-09-10T20:44:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.654+0000 7f50831ad640 1 -- 172.21.15.28:0/1249228121 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "non_existent_subvolume", "clone_name": "clone_1", "target": ["mon-mgr", ""]}) -- 0x7f507c0b6cd0 con 0x7f50680785d0 2024-09-10T20:44:37.660 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.656+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.656+0000 7f50797fa640 1 -- 172.21.15.28:0/1249228121 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f507c0b6cd0 con 0x7f50680785d0 2024-09-10T20:44:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:37.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create non_existent_subvolume group_1 2024-09-10T20:44:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 -- 172.21.15.28:0/3399153827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040d0530 msgr2=0x7f95040b5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 --2- 172.21.15.28:0/3399153827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040d0530 0x7f95040b5870 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f950406da80 tx=0x7f94fc02a5f0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 -- 172.21.15.28:0/3399153827 shutdown_connections 2024-09-10T20:44:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 --2- 172.21.15.28:0/3399153827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040d0530 0x7f95040b5870 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 --2- 172.21.15.28:0/3399153827 >> v2:172.21.15.159:3300/0 conn(0x7f950415c0b0 0x7f95040d28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 -- 172.21.15.28:0/3399153827 >> 172.21.15.28:0/3399153827 conn(0x7f95040d1da0 msgr2=0x7f95040d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 -- 172.21.15.28:0/3399153827 shutdown_connections 2024-09-10T20:44:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.770+0000 7f950baef640 1 -- 172.21.15.28:0/3399153827 wait complete. 2024-09-10T20:44:37.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 Processor -- start 2024-09-10T20:44:37.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 -- start start 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cd3c0 0x7f95040cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95040d0530 0x7f950409f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950415c0b0 0x7f95040ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95040b63b0 con 0x7f95040cd3c0 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f95040b6230 con 0x7f950415c0b0 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.772+0000 7f950baef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f95040b6530 con 0x7f95040d0530 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cd3c0 0x7f95040cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cd3c0 0x7f95040cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52974/0 (socket says 172.21.15.28:52974) 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 -- 172.21.15.28:0/3427325197 learned_addr learned my addr 172.21.15.28:0/3427325197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950a08c640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950415c0b0 0x7f95040ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950908a640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95040d0530 0x7f950409f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 -- 172.21.15.28:0/3427325197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95040d0530 msgr2=0x7f950409f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f95040d0530 0x7f950409f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 -- 172.21.15.28:0/3427325197 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950415c0b0 msgr2=0x7f95040ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f950415c0b0 0x7f95040ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 -- 172.21.15.28:0/3427325197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95041f28d0 con 0x7f95040cd3c0 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950988b640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cd3c0 0x7f95040cbe00 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f94f400eb30 tx=0x7f94f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f95027fc640 1 -- 172.21.15.28:0/3427325197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4010430 con 0x7f95040cd3c0 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f95027fc640 1 -- 172.21.15.28:0/3427325197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94f40105d0 con 0x7f95040cd3c0 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950baef640 1 -- 172.21.15.28:0/3427325197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95041f2bb0 con 0x7f95040cd3c0 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f95027fc640 1 -- 172.21.15.28:0/3427325197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4010a20 con 0x7f95040cd3c0 2024-09-10T20:44:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.773+0000 7f950baef640 1 -- 172.21.15.28:0/3427325197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95041f3170 con 0x7f95040cd3c0 2024-09-10T20:44:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.774+0000 7f950baef640 1 -- 172.21.15.28:0/3427325197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95040b6a50 con 0x7f95040cd3c0 2024-09-10T20:44:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.775+0000 7f95027fc640 1 -- 172.21.15.28:0/3427325197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f94f4010bc0 con 0x7f95040cd3c0 2024-09-10T20:44:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.776+0000 7f95027fc640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94f00785a0 0x7f94f007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.776+0000 7f95027fc640 1 -- 172.21.15.28:0/3427325197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f94f409b7f0 con 0x7f95040cd3c0 2024-09-10T20:44:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.776+0000 7f950908a640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94f00785a0 0x7f94f007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.776+0000 7f950908a640 1 --2- 172.21.15.28:0/3427325197 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f94f00785a0 0x7f94f007aa60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f950409d3b0 tx=0x7f94f8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.780+0000 7f95027fc640 1 -- 172.21.15.28:0/3427325197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f94f4063880 con 0x7f95040cd3c0 2024-09-10T20:44:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.991+0000 7f950baef640 1 -- 172.21.15.28:0/3427325197 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f95040ce590 con 0x7f94f00785a0 2024-09-10T20:44:37.997 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:37.992+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:37.992+0000 7f95027fc640 1 -- 172.21.15.28:0/3427325197 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f95040ce590 con 0x7f94f00785a0 2024-09-10T20:44:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:38.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm non_existent_subvolume group_1 2024-09-10T20:44:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.108+0000 7f10676e4640 1 -- 172.21.15.28:0/2195711762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f104c004d40 msgr2=0x7f104c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.108+0000 7f10676e4640 1 --2- 172.21.15.28:0/2195711762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f104c004d40 0x7f104c0051c0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f1058009f40 tx=0x7f104c005830 comp rx=0 tx=0).stop 2024-09-10T20:44:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.108+0000 7f1065c81640 1 -- 172.21.15.28:0/2195711762 reap_dead start 2024-09-10T20:44:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.108+0000 7f10676e4640 1 -- 172.21.15.28:0/2195711762 shutdown_connections 2024-09-10T20:44:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.108+0000 7f10676e4640 1 -- 172.21.15.28:0/2195711762 >> 172.21.15.28:0/2195711762 conn(0x7f10600d1e60 msgr2=0x7f10600d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.108+0000 7f10676e4640 1 -- 172.21.15.28:0/2195711762 shutdown_connections 2024-09-10T20:44:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.108+0000 7f10676e4640 1 -- 172.21.15.28:0/2195711762 wait complete. 2024-09-10T20:44:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.109+0000 7f10676e4640 1 Processor -- start 2024-09-10T20:44:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.109+0000 7f10676e4640 1 -- start start 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f10676e4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f104c004d40 0x7f10601f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f10676e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10600c8f30 0x7f10600cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f10676e4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10600c9390 0x7f10600cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f10676e4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10600ce930 con 0x7f10600c8f30 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f10676e4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f10600ce7b0 con 0x7f104c004d40 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f10676e4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f10600b27c0 con 0x7f10600c9390 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10600c8f30 0x7f10600cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10600c8f30 0x7f10600cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53000/0 (socket says 172.21.15.28:53000) 2024-09-10T20:44:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 -- 172.21.15.28:0/3441638561 learned_addr learned my addr 172.21.15.28:0/3441638561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1065480640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f104c004d40 0x7f10601f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1065c81640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10600c9390 0x7f10600cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 -- 172.21.15.28:0/3441638561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10600c9390 msgr2=0x7f10600cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f10600c9390 0x7f10600cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 -- 172.21.15.28:0/3441638561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f104c004d40 msgr2=0x7f10601f2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f104c004d40 0x7f10601f2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 -- 172.21.15.28:0/3441638561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10601f2fe0 con 0x7f10600c8f30 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1065480640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f104c004d40 0x7f10601f2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1064c7f640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10600c8f30 0x7f10600cb030 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f1050011ab0 tx=0x7f1050011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1055ffb640 1 -- 172.21.15.28:0/3441638561 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105000fdb0 con 0x7f10600c8f30 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f1055ffb640 1 -- 172.21.15.28:0/3441638561 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1050016070 con 0x7f10600c8f30 2024-09-10T20:44:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.110+0000 7f10676e4640 1 -- 172.21.15.28:0/3441638561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10601f32c0 con 0x7f10600c8f30 2024-09-10T20:44:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.111+0000 7f1055ffb640 1 -- 172.21.15.28:0/3441638561 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050012690 con 0x7f10600c8f30 2024-09-10T20:44:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.111+0000 7f10676e4640 1 -- 172.21.15.28:0/3441638561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106015d6b0 con 0x7f10600c8f30 2024-09-10T20:44:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.112+0000 7f10676e4640 1 -- 172.21.15.28:0/3441638561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10601f37e0 con 0x7f10600c8f30 2024-09-10T20:44:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.112+0000 7f1055ffb640 1 -- 172.21.15.28:0/3441638561 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f1050012830 con 0x7f10600c8f30 2024-09-10T20:44:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.113+0000 7f1055ffb640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10440785a0 0x7f104407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.113+0000 7f1055ffb640 1 -- 172.21.15.28:0/3441638561 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f105009b4e0 con 0x7f10600c8f30 2024-09-10T20:44:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.113+0000 7f1065480640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10440785a0 0x7f104407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.114+0000 7f1065480640 1 --2- 172.21.15.28:0/3441638561 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f10440785a0 0x7f104407aa60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f105c000fb0 tx=0x7f105c001d90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:38.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.117+0000 7f1055ffb640 1 -- 172.21.15.28:0/3441638561 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f10601f37e0 con 0x7f10600c8f30 2024-09-10T20:44:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.327+0000 7f10676e4640 1 -- 172.21.15.28:0/3441638561 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f10600b2280 con 0x7f10440785a0 2024-09-10T20:44:38.333 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:38.328+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.328+0000 7f1055ffb640 1 -- 172.21.15.28:0/3441638561 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f10600b2280 con 0x7f10440785a0 2024-09-10T20:44:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:38.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup getpath non_existent_subvolume group_1 2024-09-10T20:44:38.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 -- 172.21.15.28:0/2220456671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a40b3110 msgr2=0x7f08a40cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 --2- 172.21.15.28:0/2220456671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a40b3110 0x7f08a40cd060 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f089c005fd0 tx=0x7f089c02a8b0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 -- 172.21.15.28:0/2220456671 shutdown_connections 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 --2- 172.21.15.28:0/2220456671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a40b3110 0x7f08a40cd060 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 --2- 172.21.15.28:0/2220456671 >> v2:172.21.15.159:3300/0 conn(0x7f08a40cee60 0x7f08a40ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 -- 172.21.15.28:0/2220456671 >> 172.21.15.28:0/2220456671 conn(0x7f08a40d1f30 msgr2=0x7f08a40d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 -- 172.21.15.28:0/2220456671 shutdown_connections 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.443+0000 7f08a9f05640 1 -- 172.21.15.28:0/2220456671 wait complete. 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.444+0000 7f08a9f05640 1 Processor -- start 2024-09-10T20:44:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.445+0000 7f08a9f05640 1 -- start start 2024-09-10T20:44:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.445+0000 7f08a9f05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b3110 0x7f08a41f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.445+0000 7f08a9f05640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a40cee60 0x7f08a406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.445+0000 7f08a9f05640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a415f580 0x7f08a40c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.445+0000 7f08a9f05640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08a40d1030 con 0x7f08a40b3110 2024-09-10T20:44:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.445+0000 7f08a9f05640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f08a40d0eb0 con 0x7f08a40cee60 2024-09-10T20:44:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.445+0000 7f08a9f05640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f08a40cd5a0 con 0x7f08a415f580 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b3110 0x7f08a41f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b3110 0x7f08a41f5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53010/0 (socket says 172.21.15.28:53010) 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 -- 172.21.15.28:0/3865159714 learned_addr learned my addr 172.21.15.28:0/3865159714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a3fff640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a415f580 0x7f08a40c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a2ffd640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a40cee60 0x7f08a406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 -- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a415f580 msgr2=0x7f08a40c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a415f580 0x7f08a40c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 -- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a40cee60 msgr2=0x7f08a406eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a40cee60 0x7f08a406eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 -- 172.21.15.28:0/3865159714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a40c5440 con 0x7f08a40b3110 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a3fff640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08a415f580 0x7f08a40c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a2ffd640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08a40cee60 0x7f08a406eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a37fe640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b3110 0x7f08a41f5660 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f089400c430 tx=0x7f089400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f088bfff640 1 -- 172.21.15.28:0/3865159714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894011790 con 0x7f08a40b3110 2024-09-10T20:44:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f088bfff640 1 -- 172.21.15.28:0/3865159714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0894017070 con 0x7f08a40b3110 2024-09-10T20:44:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a9f05640 1 -- 172.21.15.28:0/3865159714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a406dd20 con 0x7f08a40b3110 2024-09-10T20:44:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f088bfff640 1 -- 172.21.15.28:0/3865159714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894011ce0 con 0x7f08a40b3110 2024-09-10T20:44:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.446+0000 7f08a9f05640 1 -- 172.21.15.28:0/3865159714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a406e2a0 con 0x7f08a40b3110 2024-09-10T20:44:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.447+0000 7f08a9f05640 1 -- 172.21.15.28:0/3865159714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a40cdb30 con 0x7f08a40b3110 2024-09-10T20:44:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.448+0000 7f088bfff640 1 -- 172.21.15.28:0/3865159714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f089401c460 con 0x7f08a40b3110 2024-09-10T20:44:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.449+0000 7f088bfff640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f08900785a0 0x7f089007aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.449+0000 7f088bfff640 1 -- 172.21.15.28:0/3865159714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f089409a6a0 con 0x7f08a40b3110 2024-09-10T20:44:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.449+0000 7f08a2ffd640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f08900785a0 0x7f089007aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.449+0000 7f08a2ffd640 1 --2- 172.21.15.28:0/3865159714 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f08900785a0 0x7f089007aa60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f08a40c8e30 tx=0x7f0898005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.453+0000 7f088bfff640 1 -- 172.21.15.28:0/3865159714 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f08940628c0 con 0x7f08a40b3110 2024-09-10T20:44:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.670+0000 7f08a9f05640 1 -- 172.21.15.28:0/3865159714 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup getpath", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f08a40cd220 con 0x7f08900785a0 2024-09-10T20:44:38.676 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:38.671+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.671+0000 7f088bfff640 1 -- 172.21.15.28:0/3865159714 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f08a40cd220 con 0x7f08900785a0 2024-09-10T20:44:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:38.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup pin non_existent_subvolume group_1 export 0 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 -- 172.21.15.28:0/2361016732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943c070270 msgr2=0x7f943c0ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 --2- 172.21.15.28:0/2361016732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943c070270 0x7f943c0ca620 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f943c0af790 tx=0x7f943402aac0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 -- 172.21.15.28:0/2361016732 shutdown_connections 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 --2- 172.21.15.28:0/2361016732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943c070270 0x7f943c0ca620 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 --2- 172.21.15.28:0/2361016732 >> v2:172.21.15.159:3300/0 conn(0x7f943c15bd70 0x7f943c06fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 -- 172.21.15.28:0/2361016732 >> 172.21.15.28:0/2361016732 conn(0x7f943c0cfc80 msgr2=0x7f943c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 -- 172.21.15.28:0/2361016732 shutdown_connections 2024-09-10T20:44:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.791+0000 7f944212b640 1 -- 172.21.15.28:0/2361016732 wait complete. 2024-09-10T20:44:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.792+0000 7f944212b640 1 Processor -- start 2024-09-10T20:44:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.792+0000 7f944212b640 1 -- start start 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f944212b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f943c070270 0x7f943c156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f944212b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f943c151cf0 0x7f943c1f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f944212b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943c15bd70 0x7f943c155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f944212b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f943c0caef0 con 0x7f943c15bd70 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f944212b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f943c0cad70 con 0x7f943c070270 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f944212b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f943c0cb070 con 0x7f943c151cf0 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943c15bd70 0x7f943c155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943c15bd70 0x7f943c155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53038/0 (socket says 172.21.15.28:53038) 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 -- 172.21.15.28:0/480839749 learned_addr learned my addr 172.21.15.28:0/480839749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943b7fe640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f943c070270 0x7f943c156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943affd640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f943c151cf0 0x7f943c1f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 -- 172.21.15.28:0/480839749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f943c151cf0 msgr2=0x7f943c1f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f943c151cf0 0x7f943c1f6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 -- 172.21.15.28:0/480839749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f943c070270 msgr2=0x7f943c156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f943c070270 0x7f943c156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.793+0000 7f943bfff640 1 -- 172.21.15.28:0/480839749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943c1f2c30 con 0x7f943c15bd70 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.794+0000 7f943bfff640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943c15bd70 0x7f943c155890 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f943402b0b0 tx=0x7f9434003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.794+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434044070 con 0x7f943c15bd70 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.794+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9434004250 con 0x7f943c15bd70 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.794+0000 7f944212b640 1 -- 172.21.15.28:0/480839749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943c1f2f10 con 0x7f943c15bd70 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.794+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94340046a0 con 0x7f943c15bd70 2024-09-10T20:44:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.794+0000 7f944212b640 1 -- 172.21.15.28:0/480839749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f943c1f34d0 con 0x7f943c15bd70 2024-09-10T20:44:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.795+0000 7f944212b640 1 -- 172.21.15.28:0/480839749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f943c0ca620 con 0x7f943c15bd70 2024-09-10T20:44:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.795+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 101162+0+0 (secure 0 0 0) 0x7f943403f410 con 0x7f943c15bd70 2024-09-10T20:44:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.796+0000 7f9433fff640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f941c0785a0 0x7f941c07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.796+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f94340bb080 con 0x7f943c15bd70 2024-09-10T20:44:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.796+0000 7f943b7fe640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f941c0785a0 0x7f941c07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.796+0000 7f943b7fe640 1 --2- 172.21.15.28:0/480839749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f941c0785a0 0x7f941c07aa60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f942c007fd0 tx=0x7f942c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.800+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9434032030 con 0x7f943c15bd70 2024-09-10T20:44:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:38.839+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f943407ed00 con 0x7f943c15bd70 2024-09-10T20:44:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.014+0000 7f944212b640 1 -- 172.21.15.28:0/480839749 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "non_existent_subvolume", "group_name": "group_1", "pin_type": "export", "pin_setting": "0", "target": ["mon-mgr", ""]}) -- 0x7f943c15d040 con 0x7f941c0785a0 2024-09-10T20:44:39.020 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:39.015+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.016+0000 7f9433fff640 1 -- 172.21.15.28:0/480839749 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f943c15d040 con 0x7f941c0785a0 2024-09-10T20:44:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:39.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup ls non_existent_subvolume 2024-09-10T20:44:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 -- 172.21.15.28:0/3201856685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f80b56e0 msgr2=0x7f37f80d2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 --2- 172.21.15.28:0/3201856685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f80b56e0 0x7f37f80d2a60 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f37e8005fd0 tx=0x7f37e802a530 comp rx=0 tx=0).stop 2024-09-10T20:44:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 -- 172.21.15.28:0/3201856685 shutdown_connections 2024-09-10T20:44:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 --2- 172.21.15.28:0/3201856685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f80b56e0 0x7f37f80d2a60 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 --2- 172.21.15.28:0/3201856685 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37f80d2140 0x7f37f80d2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 -- 172.21.15.28:0/3201856685 >> 172.21.15.28:0/3201856685 conn(0x7f37f80d40f0 msgr2=0x7f37f80d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 -- 172.21.15.28:0/3201856685 shutdown_connections 2024-09-10T20:44:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.129+0000 7f37ffedd640 1 -- 172.21.15.28:0/3201856685 wait complete. 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.130+0000 7f37ffedd640 1 Processor -- start 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.130+0000 7f37ffedd640 1 -- start start 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37ffedd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f80b56e0 0x7f37f80d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37ffedd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37f80d2140 0x7f37f80ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37ffedd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37f8161fd0 0x7f37f80c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37ffedd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37f80b2590 con 0x7f37f80b56e0 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37ffedd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f37f80b2410 con 0x7f37f80d2140 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37ffedd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f37f80b2710 con 0x7f37f8161fd0 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f80b56e0 0x7f37f80d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f80b56e0 0x7f37f80d1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53052/0 (socket says 172.21.15.28:53052) 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 -- 172.21.15.28:0/52575018 learned_addr learned my addr 172.21.15.28:0/52575018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fd478640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37f80d2140 0x7f37f80ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fe47a640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37f8161fd0 0x7f37f80c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 -- 172.21.15.28:0/52575018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37f8161fd0 msgr2=0x7f37f80c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37f8161fd0 0x7f37f80c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 -- 172.21.15.28:0/52575018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37f80d2140 msgr2=0x7f37f80ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37f80d2140 0x7f37f80ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.131+0000 7f37fdc79640 1 -- 172.21.15.28:0/52575018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f81f4790 con 0x7f37f80b56e0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.132+0000 7f37fdc79640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f80b56e0 0x7f37f80d1060 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f37f400d7a0 tx=0x7f37f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.132+0000 7f37ee7fc640 1 -- 172.21.15.28:0/52575018 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4017070 con 0x7f37f80b56e0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.132+0000 7f37ee7fc640 1 -- 172.21.15.28:0/52575018 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37f4009cc0 con 0x7f37f80b56e0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.132+0000 7f37ffedd640 1 -- 172.21.15.28:0/52575018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f81f4a70 con 0x7f37f80b56e0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.132+0000 7f37ee7fc640 1 -- 172.21.15.28:0/52575018 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4005280 con 0x7f37f80b56e0 2024-09-10T20:44:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.132+0000 7f37ffedd640 1 -- 172.21.15.28:0/52575018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f81f5310 con 0x7f37f80b56e0 2024-09-10T20:44:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.133+0000 7f37ffedd640 1 -- 172.21.15.28:0/52575018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f81f5550 con 0x7f37f80b56e0 2024-09-10T20:44:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.134+0000 7f37ee7fc640 1 -- 172.21.15.28:0/52575018 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f37f4005420 con 0x7f37f80b56e0 2024-09-10T20:44:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.134+0000 7f37ee7fc640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37dc078630 0x7f37dc07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.134+0000 7f37ee7fc640 1 -- 172.21.15.28:0/52575018 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f37f409ab30 con 0x7f37f80b56e0 2024-09-10T20:44:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.134+0000 7f37fd478640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37dc078630 0x7f37dc07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.135+0000 7f37fd478640 1 --2- 172.21.15.28:0/52575018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37dc078630 0x7f37dc07aaf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f37e8033000 tx=0x7f37e8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.139+0000 7f37ee7fc640 1 -- 172.21.15.28:0/52575018 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f37f81f5550 con 0x7f37f80b56e0 2024-09-10T20:44:39.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.349+0000 7f37ffedd640 1 -- 172.21.15.28:0/52575018 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup ls", "vol_name": "non_existent_subvolume", "target": ["mon-mgr", ""]}) -- 0x7f37f81f4c40 con 0x7f37dc078630 2024-09-10T20:44:39.355 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:39.350+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:39.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.350+0000 7f37ee7fc640 1 -- 172.21.15.28:0/52575018 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f37f81f4c40 con 0x7f37dc078630 2024-09-10T20:44:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:39.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot create non_existent_subvolume group_1 snapshot_1 2024-09-10T20:44:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.467+0000 7fb4ab88c640 1 -- 172.21.15.28:0/3896140661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb48c0051c0 msgr2=0x7fb48c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.467+0000 7fb4ab88c640 1 --2- 172.21.15.28:0/3896140661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb48c0051c0 0x7fb48c00b9c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fb49c001e40 tx=0x7fb48c005830 comp rx=0 tx=0).stop 2024-09-10T20:44:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.467+0000 7fb4a9e29640 1 -- 172.21.15.28:0/3896140661 reap_dead start 2024-09-10T20:44:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.467+0000 7fb4ab88c640 1 -- 172.21.15.28:0/3896140661 shutdown_connections 2024-09-10T20:44:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.467+0000 7fb4ab88c640 1 -- 172.21.15.28:0/3896140661 >> 172.21.15.28:0/3896140661 conn(0x7fb4a40cfc80 msgr2=0x7fb4a40d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.467+0000 7fb4ab88c640 1 -- 172.21.15.28:0/3896140661 shutdown_connections 2024-09-10T20:44:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.467+0000 7fb4ab88c640 1 -- 172.21.15.28:0/3896140661 wait complete. 2024-09-10T20:44:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.468+0000 7fb4ab88c640 1 Processor -- start 2024-09-10T20:44:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.468+0000 7fb4ab88c640 1 -- start start 2024-09-10T20:44:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4ab88c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb48c0051c0 0x7fb4a40b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4ab88c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4a40b1140 0x7fb4a41f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4ab88c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4a41f5fd0 0x7fb4a41f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4ab88c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a415c320 con 0x7fb48c0051c0 2024-09-10T20:44:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4ab88c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a40cabf0 con 0x7fb4a40b1140 2024-09-10T20:44:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4ab88c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb4a415f850 con 0x7fb4a41f5fd0 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb48c0051c0 0x7fb4a40b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb48c0051c0 0x7fb4a40b0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53074/0 (socket says 172.21.15.28:53074) 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 -- 172.21.15.28:0/392640487 learned_addr learned my addr 172.21.15.28:0/392640487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9e29640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4a41f5fd0 0x7fb4a41f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a8e27640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4a40b1140 0x7fb4a41f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 -- 172.21.15.28:0/392640487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4a41f5fd0 msgr2=0x7fb4a41f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4a41f5fd0 0x7fb4a41f4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 -- 172.21.15.28:0/392640487 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4a40b1140 msgr2=0x7fb4a41f5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb4a40b1140 0x7fb4a41f5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9628640 1 -- 172.21.15.28:0/392640487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a41f3660 con 0x7fb48c0051c0 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.469+0000 7fb4a9e29640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4a41f5fd0 0x7fb4a41f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.470+0000 7fb4a9628640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb48c0051c0 0x7fb4a40b0c00 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fb494011ab0 tx=0x7fb494011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.470+0000 7fb4a1ffb640 1 -- 172.21.15.28:0/392640487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49401e070 con 0x7fb48c0051c0 2024-09-10T20:44:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.470+0000 7fb4a1ffb640 1 -- 172.21.15.28:0/392640487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb49400fd60 con 0x7fb48c0051c0 2024-09-10T20:44:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.470+0000 7fb4a1ffb640 1 -- 172.21.15.28:0/392640487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb494019650 con 0x7fb48c0051c0 2024-09-10T20:44:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.470+0000 7fb4ab88c640 1 -- 172.21.15.28:0/392640487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a41f3900 con 0x7fb48c0051c0 2024-09-10T20:44:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.470+0000 7fb4ab88c640 1 -- 172.21.15.28:0/392640487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a4148d10 con 0x7fb48c0051c0 2024-09-10T20:44:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.471+0000 7fb4ab88c640 1 -- 172.21.15.28:0/392640487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a415fd20 con 0x7fb48c0051c0 2024-09-10T20:44:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.472+0000 7fb4a1ffb640 1 -- 172.21.15.28:0/392640487 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fb4940197f0 con 0x7fb48c0051c0 2024-09-10T20:44:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.472+0000 7fb4a1ffb640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb490078630 0x7fb49007aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.472+0000 7fb4a1ffb640 1 -- 172.21.15.28:0/392640487 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7fb49409b620 con 0x7fb48c0051c0 2024-09-10T20:44:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.472+0000 7fb4a8e27640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb490078630 0x7fb49007aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.473+0000 7fb4a8e27640 1 --2- 172.21.15.28:0/392640487 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb490078630 0x7fb49007aaf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb4a41588a0 tx=0x7fb498001e70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.476+0000 7fb4a1ffb640 1 -- 172.21.15.28:0/392640487 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb4940638f0 con 0x7fb48c0051c0 2024-09-10T20:44:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.687+0000 7fb4ab88c640 1 -- 172.21.15.28:0/392640487 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot create", "vol_name": "non_existent_subvolume", "group_name": "group_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fb4a41f63b0 con 0x7fb490078630 2024-09-10T20:44:39.693 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:39.688+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.689+0000 7fb4a1ffb640 1 -- 172.21.15.28:0/392640487 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fb4a41f63b0 con 0x7fb490078630 2024-09-10T20:44:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:39.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot rm non_existent_subvolume group_1 snapshot_1 2024-09-10T20:44:39.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.806+0000 7f7cd8065640 1 -- 172.21.15.28:0/709218579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc004000 msgr2=0x7f7cbc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.806+0000 7f7cd8065640 1 --2- 172.21.15.28:0/709218579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc004000 0x7f7cbc004800 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000c2e0 tx=0x7f7cbc005830 comp rx=0 tx=0).stop 2024-09-10T20:44:39.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.806+0000 7f7cd6602640 1 -- 172.21.15.28:0/709218579 reap_dead start 2024-09-10T20:44:39.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.806+0000 7f7cd8065640 1 -- 172.21.15.28:0/709218579 shutdown_connections 2024-09-10T20:44:39.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.806+0000 7f7cd8065640 1 -- 172.21.15.28:0/709218579 >> 172.21.15.28:0/709218579 conn(0x7f7cd00d1da0 msgr2=0x7f7cd00d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:39.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.806+0000 7f7cd8065640 1 -- 172.21.15.28:0/709218579 shutdown_connections 2024-09-10T20:44:39.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.806+0000 7f7cd8065640 1 -- 172.21.15.28:0/709218579 wait complete. 2024-09-10T20:44:39.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.807+0000 7f7cd8065640 1 Processor -- start 2024-09-10T20:44:39.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.807+0000 7f7cd8065640 1 -- start start 2024-09-10T20:44:39.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd8065640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cbc004000 0x7f7cd00c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd8065640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd01f8d80 0x7f7cd00c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd8065640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd015d580 0x7f7cd014cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd8065640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0162b10 con 0x7f7cd01f8d80 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd8065640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0162990 con 0x7f7cbc004000 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd8065640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd006e9c0 con 0x7f7cd015d580 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd01f8d80 0x7f7cd00c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd01f8d80 0x7f7cd00c6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53090/0 (socket says 172.21.15.28:53090) 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 -- 172.21.15.28:0/2392236697 learned_addr learned my addr 172.21.15.28:0/2392236697 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5e01640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cbc004000 0x7f7cd00c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd6602640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd015d580 0x7f7cd014cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 -- 172.21.15.28:0/2392236697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd015d580 msgr2=0x7f7cd014cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd015d580 0x7f7cd014cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 -- 172.21.15.28:0/2392236697 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cbc004000 msgr2=0x7f7cd00c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cbc004000 0x7f7cd00c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd5600640 1 -- 172.21.15.28:0/2392236697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd01f9200 con 0x7f7cd01f8d80 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.808+0000 7f7cd6602640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cd015d580 0x7f7cd014cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.809+0000 7f7cd5600640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd01f8d80 0x7f7cd00c6e50 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000c3d0 tx=0x7f7cc0004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.809+0000 7f7cc67fc640 1 -- 172.21.15.28:0/2392236697 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0031040 con 0x7f7cd01f8d80 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.809+0000 7f7cc67fc640 1 -- 172.21.15.28:0/2392236697 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cc0006420 con 0x7f7cd01f8d80 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.809+0000 7f7cc67fc640 1 -- 172.21.15.28:0/2392236697 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0006c90 con 0x7f7cd01f8d80 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.809+0000 7f7cd8065640 1 -- 172.21.15.28:0/2392236697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd00c5f50 con 0x7f7cd01f8d80 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.809+0000 7f7cd8065640 1 -- 172.21.15.28:0/2392236697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd00c6510 con 0x7f7cd01f8d80 2024-09-10T20:44:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.810+0000 7f7cd8065640 1 -- 172.21.15.28:0/2392236697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd006eed0 con 0x7f7cd01f8d80 2024-09-10T20:44:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.810+0000 7f7cc67fc640 1 -- 172.21.15.28:0/2392236697 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f7cc002d670 con 0x7f7cd01f8d80 2024-09-10T20:44:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.811+0000 7f7cc67fc640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cb4078660 0x7f7cb407ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.811+0000 7f7cc67fc640 1 -- 172.21.15.28:0/2392236697 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f7cc00baf50 con 0x7f7cd01f8d80 2024-09-10T20:44:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.811+0000 7f7cd5e01640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cb4078660 0x7f7cb407ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.811+0000 7f7cd5e01640 1 --2- 172.21.15.28:0/2392236697 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cb4078660 0x7f7cb407ab20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009b90 tx=0x7f7ccc005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:39.815+0000 7f7cc67fc640 1 -- 172.21.15.28:0/2392236697 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7cc00830a0 con 0x7f7cd01f8d80 2024-09-10T20:44:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.027+0000 7f7cd8065640 1 -- 172.21.15.28:0/2392236697 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot rm", "vol_name": "non_existent_subvolume", "group_name": "group_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f7cd0070140 con 0x7f7cb4078660 2024-09-10T20:44:40.033 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:40.028+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.028+0000 7f7cc67fc640 1 -- 172.21.15.28:0/2392236697 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f7cd0070140 con 0x7f7cb4078660 2024-09-10T20:44:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:40.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot ls non_existent_subvolume group_1 2024-09-10T20:44:40.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.143+0000 7f35d44f6640 1 -- 172.21.15.28:0/2530077724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0cc840 msgr2=0x7f35cc0ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.143+0000 7f35d44f6640 1 --2- 172.21.15.28:0/2530077724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0cc840 0x7f35cc0ccc70 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f35cc073b20 tx=0x7f35c402aac0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.144+0000 7f35d44f6640 1 -- 172.21.15.28:0/2530077724 shutdown_connections 2024-09-10T20:44:40.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.144+0000 7f35d44f6640 1 --2- 172.21.15.28:0/2530077724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0cc840 0x7f35cc0ccc70 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.144+0000 7f35d44f6640 1 --2- 172.21.15.28:0/2530077724 >> v2:172.21.15.159:3300/0 conn(0x7f35cc0d28c0 0x7f35cc0ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.144+0000 7f35d44f6640 1 --2- 172.21.15.28:0/2530077724 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc0ce7b0 0x7f35cc0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.144+0000 7f35d44f6640 1 -- 172.21.15.28:0/2530077724 >> 172.21.15.28:0/2530077724 conn(0x7f35cc0d1da0 msgr2=0x7f35cc0d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:40.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.144+0000 7f35d44f6640 1 -- 172.21.15.28:0/2530077724 shutdown_connections 2024-09-10T20:44:40.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.144+0000 7f35d44f6640 1 -- 172.21.15.28:0/2530077724 wait complete. 2024-09-10T20:44:40.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.145+0000 7f35d44f6640 1 Processor -- start 2024-09-10T20:44:40.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.145+0000 7f35d44f6640 1 -- start start 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d44f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0cc840 0x7f35cc1f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d44f6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc0ce7b0 0x7f35cc15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d44f6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0d28c0 0x7f35cc1f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d44f6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0cd1b0 con 0x7f35cc0cc840 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d44f6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc15c3c0 con 0x7f35cc0d28c0 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d44f6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc0cd330 con 0x7f35cc0ce7b0 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0cc840 0x7f35cc1f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0cc840 0x7f35cc1f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53116/0 (socket says 172.21.15.28:53116) 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 -- 172.21.15.28:0/3338650711 learned_addr learned my addr 172.21.15.28:0/3338650711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d1a91640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc0ce7b0 0x7f35cc15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2a93640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0d28c0 0x7f35cc1f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 -- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc0ce7b0 msgr2=0x7f35cc15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc0ce7b0 0x7f35cc15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 -- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0d28c0 msgr2=0x7f35cc1f7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0d28c0 0x7f35cc1f7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 -- 172.21.15.28:0/3338650711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc15f4b0 con 0x7f35cc0cc840 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d1a91640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc0ce7b0 0x7f35cc15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2a93640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0d28c0 0x7f35cc1f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d2292640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0cc840 0x7f35cc1f1d80 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f35c800c970 tx=0x7f35c800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35c2ffd640 1 -- 172.21.15.28:0/3338650711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8007bb0 con 0x7f35cc0cc840 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35c2ffd640 1 -- 172.21.15.28:0/3338650711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35c8005020 con 0x7f35cc0cc840 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35d44f6640 1 -- 172.21.15.28:0/3338650711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc06e850 con 0x7f35cc0cc840 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.146+0000 7f35c2ffd640 1 -- 172.21.15.28:0/3338650711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8007e60 con 0x7f35cc0cc840 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.147+0000 7f35d44f6640 1 -- 172.21.15.28:0/3338650711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc06ee10 con 0x7f35cc0cc840 2024-09-10T20:44:40.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.148+0000 7f35d44f6640 1 -- 172.21.15.28:0/3338650711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35cc06d750 con 0x7f35cc0cc840 2024-09-10T20:44:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.148+0000 7f35c2ffd640 1 -- 172.21.15.28:0/3338650711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f35c8005470 con 0x7f35cc0cc840 2024-09-10T20:44:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.149+0000 7f35c2ffd640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b0078630 0x7f35b007aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.149+0000 7f35c2ffd640 1 -- 172.21.15.28:0/3338650711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f35c809aff0 con 0x7f35cc0cc840 2024-09-10T20:44:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.149+0000 7f35d1a91640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b0078630 0x7f35b007aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.149+0000 7f35d1a91640 1 --2- 172.21.15.28:0/3338650711 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b0078630 0x7f35b007aaf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f35cc15b370 tx=0x7f35bc005c10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.153+0000 7f35c2ffd640 1 -- 172.21.15.28:0/3338650711 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35c8014030 con 0x7f35cc0cc840 2024-09-10T20:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.362+0000 7f35d44f6640 1 -- 172.21.15.28:0/3338650711 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot ls", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f35cc06e600 con 0x7f35b0078630 2024-09-10T20:44:40.368 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:44:40.364+0000 7f93ce3fd640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-10T20:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.364+0000 7f35c2ffd640 1 -- 172.21.15.28:0/3338650711 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f35cc06e600 con 0x7f35b0078630 2024-09-10T20:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-10T20:44:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T20:44:40.375 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:44:40.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:40.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:40.470 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:44:40.470 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:40.470 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:40.470 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:40.487 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:40.487 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:40.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:44:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 -- 172.21.15.28:0/3114930048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d800d0140 msgr2=0x7f8d800d05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/3114930048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d800d0140 0x7f8d800d05d0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f8d78005fd0 tx=0x7f8d7802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 -- 172.21.15.28:0/3114930048 shutdown_connections 2024-09-10T20:44:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/3114930048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d800d0140 0x7f8d800d05d0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/3114930048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80155100 0x7f8d8015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 -- 172.21.15.28:0/3114930048 >> 172.21.15.28:0/3114930048 conn(0x7f8d8006cf00 msgr2=0x7f8d8006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 -- 172.21.15.28:0/3114930048 shutdown_connections 2024-09-10T20:44:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.687+0000 7f8d86b6b640 1 -- 172.21.15.28:0/3114930048 wait complete. 2024-09-10T20:44:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.688+0000 7f8d86b6b640 1 Processor -- start 2024-09-10T20:44:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.688+0000 7f8d86b6b640 1 -- start start 2024-09-10T20:44:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d86b6b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d0140 0x7f8d801f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d86b6b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80155100 0x7f8d800b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d86b6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d801554e0 0x7f8d80144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d86b6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80072f60 con 0x7f8d801554e0 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d86b6b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80072de0 con 0x7f8d800d0140 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d86b6b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8d800730e0 con 0x7f8d80155100 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d85108640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d801554e0 0x7f8d80144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d85108640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d801554e0 0x7f8d80144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40324/0 (socket says 172.21.15.28:40324) 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.689+0000 7f8d85108640 1 -- 172.21.15.28:0/2625023405 learned_addr learned my addr 172.21.15.28:0/2625023405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d7ffff640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80155100 0x7f8d800b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d84907640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d0140 0x7f8d801f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d85108640 1 -- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80155100 msgr2=0x7f8d800b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d85108640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80155100 0x7f8d800b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d85108640 1 -- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d0140 msgr2=0x7f8d801f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d85108640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d0140 0x7f8d801f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d85108640 1 -- 172.21.15.28:0/2625023405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d800b1650 con 0x7f8d801554e0 2024-09-10T20:44:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d7ffff640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80155100 0x7f8d800b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d84907640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d0140 0x7f8d801f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d85108640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d801554e0 0x7f8d80144640 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009cd0 tx=0x7f8d780316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d7d7fa640 1 -- 172.21.15.28:0/2625023405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78031da0 con 0x7f8d801554e0 2024-09-10T20:44:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d7d7fa640 1 -- 172.21.15.28:0/2625023405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d78003a60 con 0x7f8d801554e0 2024-09-10T20:44:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d7d7fa640 1 -- 172.21.15.28:0/2625023405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d780330d0 con 0x7f8d801554e0 2024-09-10T20:44:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d800b1930 con 0x7f8d801554e0 2024-09-10T20:44:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.690+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d800ae480 con 0x7f8d801554e0 2024-09-10T20:44:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.691+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d80073600 con 0x7f8d801554e0 2024-09-10T20:44:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.692+0000 7f8d7d7fa640 1 -- 172.21.15.28:0/2625023405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f8d78033270 con 0x7f8d801554e0 2024-09-10T20:44:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.692+0000 7f8d7d7fa640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d64078660 0x7f8d6407ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.692+0000 7f8d7d7fa640 1 -- 172.21.15.28:0/2625023405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f8d780bbdd0 con 0x7f8d801554e0 2024-09-10T20:44:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.692+0000 7f8d84907640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d64078660 0x7f8d6407ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.693+0000 7f8d84907640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d64078660 0x7f8d6407ab20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009b50 tx=0x7f8d74005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.697+0000 7f8d7d7fa640 1 -- 172.21.15.28:0/2625023405 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8d7803f030 con 0x7f8d801554e0 2024-09-10T20:44:40.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.893+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8d800cbdb0 con 0x7f8d801554e0 2024-09-10T20:44:40.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.893+0000 7f8d7d7fa640 1 -- 172.21.15.28:0/2625023405 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v127) ==== 80+0+0 (secure 0 0 0) 0x7f8d78083dd0 con 0x7f8d801554e0 2024-09-10T20:44:40.898 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d64078660 msgr2=0x7f8d6407ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d64078660 0x7f8d6407ab20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009b50 tx=0x7f8d74005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d801554e0 msgr2=0x7f8d80144640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d801554e0 0x7f8d80144640 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009cd0 tx=0x7f8d780316f0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 shutdown_connections 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8d64078660 0x7f8d6407ab20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d801554e0 0x7f8d80144640 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d80155100 0x7f8d800b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 --2- 172.21.15.28:0/2625023405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d800d0140 0x7f8d801f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 >> 172.21.15.28:0/2625023405 conn(0x7f8d8006cf00 msgr2=0x7f8d8009ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 shutdown_connections 2024-09-10T20:44:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:40.897+0000 7f8d86b6b640 1 -- 172.21.15.28:0/2625023405 wait complete. 2024-09-10T20:44:40.911 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:44:40.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:40.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:44:40.941 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:44:40.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:40.941 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:44:40.998 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:40.998 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:40.998 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:41.017 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:41.017 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:41.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:44:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.179+0000 7f5fdc433640 1 -- 172.21.15.28:0/728008640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 msgr2=0x7f5fd40d2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.179+0000 7f5fdc433640 1 --2- 172.21.15.28:0/728008640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd40d2740 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009830 tx=0x7f5fc802a510 comp rx=0 tx=0).stop 2024-09-10T20:44:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.179+0000 7f5fdc433640 1 -- 172.21.15.28:0/728008640 shutdown_connections 2024-09-10T20:44:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.179+0000 7f5fdc433640 1 --2- 172.21.15.28:0/728008640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd40d2740 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.179+0000 7f5fdc433640 1 -- 172.21.15.28:0/728008640 >> 172.21.15.28:0/728008640 conn(0x7f5fd40d4040 msgr2=0x7f5fd40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.179+0000 7f5fdc433640 1 -- 172.21.15.28:0/728008640 shutdown_connections 2024-09-10T20:44:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.179+0000 7f5fdc433640 1 -- 172.21.15.28:0/728008640 wait complete. 2024-09-10T20:44:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.180+0000 7f5fdc433640 1 Processor -- start 2024-09-10T20:44:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fdc433640 1 -- start start 2024-09-10T20:44:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fdc433640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd40b77f0 0x7f5fd41f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fdc433640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd4155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fdc433640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd415e0f0 0x7f5fd4155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fdc433640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd40b8350 con 0x7f5fd40d4c40 2024-09-10T20:44:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fdc433640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd40d2e90 con 0x7f5fd40b77f0 2024-09-10T20:44:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fdc433640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd40b84d0 con 0x7f5fd415e0f0 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fd99ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd4155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fd99ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd4155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40338/0 (socket says 172.21.15.28:40338) 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.181+0000 7f5fd99ce640 1 -- 172.21.15.28:0/1023828533 learned_addr learned my addr 172.21.15.28:0/1023828533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fda1cf640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd40b77f0 0x7f5fd41f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fda9d0640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd415e0f0 0x7f5fd4155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd99ce640 1 -- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd415e0f0 msgr2=0x7f5fd4155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd99ce640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd415e0f0 0x7f5fd4155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd99ce640 1 -- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd40b77f0 msgr2=0x7f5fd41f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd99ce640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd40b77f0 0x7f5fd41f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd99ce640 1 -- 172.21.15.28:0/1023828533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd41f47e0 con 0x7f5fd40d4c40 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fda1cf640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd40b77f0 0x7f5fd41f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fda9d0640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd415e0f0 0x7f5fd4155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd99ce640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd4155150 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009950 tx=0x7f5fc8032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd2ffd640 1 -- 172.21.15.28:0/1023828533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8040070 con 0x7f5fd40d4c40 2024-09-10T20:44:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd2ffd640 1 -- 172.21.15.28:0/1023828533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fc8005310 con 0x7f5fd40d4c40 2024-09-10T20:44:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd40ca580 con 0x7f5fd40d4c40 2024-09-10T20:44:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fd2ffd640 1 -- 172.21.15.28:0/1023828533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc80047a0 con 0x7f5fd40d4c40 2024-09-10T20:44:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.182+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd4160c60 con 0x7f5fd40d4c40 2024-09-10T20:44:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.183+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd40d3040 con 0x7f5fd40d4c40 2024-09-10T20:44:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.184+0000 7f5fd2ffd640 1 -- 172.21.15.28:0/1023828533 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f5fc804c030 con 0x7f5fd40d4c40 2024-09-10T20:44:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.185+0000 7f5fd2ffd640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fc0078630 0x7f5fc007aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.185+0000 7f5fda1cf640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fc0078630 0x7f5fc007aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.185+0000 7f5fd2ffd640 1 -- 172.21.15.28:0/1023828533 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f5fc80bbf00 con 0x7f5fd40d4c40 2024-09-10T20:44:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.185+0000 7f5fda1cf640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fc0078630 0x7f5fc007aaf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4007e90 tx=0x7f5fc40073a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:41.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.189+0000 7f5fd2ffd640 1 -- 172.21.15.28:0/1023828533 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5fd40d3040 con 0x7f5fd40d4c40 2024-09-10T20:44:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.387+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5fd40b8a20 con 0x7f5fd40d4c40 2024-09-10T20:44:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.387+0000 7f5fd2ffd640 1 -- 172.21.15.28:0/1023828533 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v127) ==== 80+0+0 (secure 0 0 0) 0x7f5fd41f47e0 con 0x7f5fd40d4c40 2024-09-10T20:44:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.389+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fc0078630 msgr2=0x7f5fc007aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fc0078630 0x7f5fc007aaf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4007e90 tx=0x7f5fc40073a0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 msgr2=0x7f5fd4155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd4155150 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009950 tx=0x7f5fc8032c10 comp rx=0 tx=0).stop 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 shutdown_connections 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5fc0078630 0x7f5fc007aaf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fd415e0f0 0x7f5fd4155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd40d4c40 0x7f5fd4155150 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 --2- 172.21.15.28:0/1023828533 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fd40b77f0 0x7f5fd41f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 >> 172.21.15.28:0/1023828533 conn(0x7f5fd40d4040 msgr2=0x7f5fd415a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 shutdown_connections 2024-09-10T20:44:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.390+0000 7f5fdc433640 1 -- 172.21.15.28:0/1023828533 wait complete. 2024-09-10T20:44:41.404 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:44:41.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:41.405 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:44:41.438 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:44:41.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:44:41.438 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:44:41.496 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:41.496 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:41.496 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:41.513 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:44:41.513 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:44:41.513 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:41.513 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:41.554 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:44:41.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:41.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 -- 172.21.15.28:0/1045664583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 msgr2=0x7f412c0b7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 --2- 172.21.15.28:0/1045664583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c0b7c40 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f411c005fd0 tx=0x7f411c02a530 comp rx=0 tx=0).stop 2024-09-10T20:44:41.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 -- 172.21.15.28:0/1045664583 shutdown_connections 2024-09-10T20:44:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 --2- 172.21.15.28:0/1045664583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c0b7c40 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 --2- 172.21.15.28:0/1045664583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f412c15e0f0 0x7f412c0d2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 -- 172.21.15.28:0/1045664583 >> 172.21.15.28:0/1045664583 conn(0x7f412c0d4040 msgr2=0x7f412c0d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 -- 172.21.15.28:0/1045664583 shutdown_connections 2024-09-10T20:44:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.655+0000 7f4131775640 1 -- 172.21.15.28:0/1045664583 wait complete. 2024-09-10T20:44:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.656+0000 7f4131775640 1 Processor -- start 2024-09-10T20:44:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f4131775640 1 -- start start 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f4131775640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f4131775640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412c0d4c40 0x7f412c0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f4131775640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f412c15e0f0 0x7f412c0ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f4131775640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0b8300 con 0x7f412c0b77f0 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f4131775640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0b8180 con 0x7f412c0d4c40 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f4131775640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f412c0b8480 con 0x7f412c15e0f0 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f412affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.657+0000 7f412affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40362/0 (socket says 172.21.15.28:40362) 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412affd640 1 -- 172.21.15.28:0/789465357 learned_addr learned my addr 172.21.15.28:0/789465357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412b7fe640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f412c15e0f0 0x7f412c0ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412a7fc640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412c0d4c40 0x7f412c0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412affd640 1 -- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f412c15e0f0 msgr2=0x7f412c0ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412affd640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f412c15e0f0 0x7f412c0ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412affd640 1 -- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412c0d4c40 msgr2=0x7f412c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412affd640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412c0d4c40 0x7f412c0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412affd640 1 -- 172.21.15.28:0/789465357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c1f8e20 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f412affd640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c150530 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f412400cce0 tx=0x7f41240075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f41237fe640 1 -- 172.21.15.28:0/789465357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124004b20 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f41237fe640 1 -- 172.21.15.28:0/789465357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4124004db0 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412c1600b0 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f41237fe640 1 -- 172.21.15.28:0/789465357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124005280 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.658+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412c160670 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.659+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412c0b8a10 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.660+0000 7f41237fe640 1 -- 172.21.15.28:0/789465357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f4124005420 con 0x7f412c0b77f0 2024-09-10T20:44:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.660+0000 7f41237fe640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f410c078630 0x7f410c07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:41.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.661+0000 7f41237fe640 1 -- 172.21.15.28:0/789465357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f412409b290 con 0x7f412c0b77f0 2024-09-10T20:44:41.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.661+0000 7f412a7fc640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f410c078630 0x7f410c07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:41.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.661+0000 7f412a7fc640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f410c078630 0x7f410c07aaf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f412c161890 tx=0x7f4114007e30 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:41.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.665+0000 7f41237fe640 1 -- 172.21.15.28:0/789465357 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4124014030 con 0x7f412c0b77f0 2024-09-10T20:44:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.889+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f412c15b160 con 0x7f412c0b77f0 2024-09-10T20:44:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.889+0000 7f41237fe640 1 -- 172.21.15.28:0/789465357 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5227 (secure 0 0 0) 0x7f4124062bb0 con 0x7f412c0b77f0 2024-09-10T20:44:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:41.894 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"btime":"2024-09-10T20:44:31:807159+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1041363523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1041363523},{"type":"v1","addr":"172.21.15.28:6834","nonce":1041363523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1561732564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1561732564},{"type":"v1","addr":"172.21.15.28:6835","nonce":1561732564}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6521,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/97213244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":97213244},{"type":"v1","addr":"172.21.15.159:6835","nonce":97213244}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:22.133677+0000","modified":"2024-09-10T20:44:25.147825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6964":{"gid":6964,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/3268698079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3268698079},{"type":"v1","addr":"172.21.15.159:6801","nonce":3268698079}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6964,"qdb_cluster":[6964]},"id":10}]} 2024-09-10T20:44:41.894 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 87 2024-09-10T20:44:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f410c078630 msgr2=0x7f410c07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f410c078630 0x7f410c07aaf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f412c161890 tx=0x7f4114007e30 comp rx=0 tx=0).stop 2024-09-10T20:44:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 msgr2=0x7f412c150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c150530 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f412400cce0 tx=0x7f41240075f0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 shutdown_connections 2024-09-10T20:44:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f410c078630 0x7f410c07aaf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f412c15e0f0 0x7f412c0ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f412c0d4c40 0x7f412c0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 --2- 172.21.15.28:0/789465357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c0b77f0 0x7f412c150530 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 >> 172.21.15.28:0/789465357 conn(0x7f412c0d4040 msgr2=0x7f412c0d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 shutdown_connections 2024-09-10T20:44:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:41.892+0000 7f4131775640 1 -- 172.21.15.28:0/789465357 wait complete. 2024-09-10T20:44:41.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 -- 172.21.15.28:0/1020086051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be40d05d0 msgr2=0x7f4be40d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 --2- 172.21.15.28:0/1020086051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be40d05d0 0x7f4be40d2960 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc005fd0 tx=0x7f4bcc02a530 comp rx=0 tx=0).stop 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 -- 172.21.15.28:0/1020086051 shutdown_connections 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 --2- 172.21.15.28:0/1020086051 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be40b5610 0x7f4be40b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 --2- 172.21.15.28:0/1020086051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be40d05d0 0x7f4be40d2960 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 -- 172.21.15.28:0/1020086051 >> 172.21.15.28:0/1020086051 conn(0x7f4be40d1e40 msgr2=0x7f4be40d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 -- 172.21.15.28:0/1020086051 shutdown_connections 2024-09-10T20:44:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.008+0000 7f4be912b640 1 -- 172.21.15.28:0/1020086051 wait complete. 2024-09-10T20:44:42.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.009+0000 7f4be912b640 1 Processor -- start 2024-09-10T20:44:42.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.009+0000 7f4be912b640 1 -- start start 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be912b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be40b5610 0x7f4be41f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be912b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4be40d05d0 0x7f4be4071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be912b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be415bf30 0x7f4be4071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be912b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4be40b5fc0 con 0x7f4be415bf30 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be912b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4be40d0d50 con 0x7f4be40d05d0 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be912b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4be40b6140 con 0x7f4be40b5610 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be415bf30 0x7f4be4071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be415bf30 0x7f4be4071850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40376/0 (socket says 172.21.15.28:40376) 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 -- 172.21.15.28:0/1452688671 learned_addr learned my addr 172.21.15.28:0/1452688671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4bdbfff640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4be40d05d0 0x7f4be4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be2d9d640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be40b5610 0x7f4be41f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 -- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be40b5610 msgr2=0x7f4be41f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be40b5610 0x7f4be41f1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 -- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4be40d05d0 msgr2=0x7f4be4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4be40d05d0 0x7f4be4071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 -- 172.21.15.28:0/1452688671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be4148170 con 0x7f4be415bf30 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4be359e640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be415bf30 0x7f4be4071850 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00b9a0 tx=0x7f4bdc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.010+0000 7f4bdb7fe640 1 -- 172.21.15.28:0/1452688671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc019070 con 0x7f4be415bf30 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.011+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be4148450 con 0x7f4be415bf30 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.011+0000 7f4bdb7fe640 1 -- 172.21.15.28:0/1452688671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4bdc00ccb0 con 0x7f4be415bf30 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.011+0000 7f4bdb7fe640 1 -- 172.21.15.28:0/1452688671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc0146a0 con 0x7f4be415bf30 2024-09-10T20:44:42.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.011+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be4148a10 con 0x7f4be415bf30 2024-09-10T20:44:42.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.012+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be40b6df0 con 0x7f4be415bf30 2024-09-10T20:44:42.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.012+0000 7f4bdb7fe640 1 -- 172.21.15.28:0/1452688671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f4bdc01d3b0 con 0x7f4be415bf30 2024-09-10T20:44:42.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.013+0000 7f4bdb7fe640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4bc4078630 0x7f4bc407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.013+0000 7f4bdb7fe640 1 -- 172.21.15.28:0/1452688671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f4bdc09a970 con 0x7f4be415bf30 2024-09-10T20:44:42.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.013+0000 7f4be2d9d640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4bc4078630 0x7f4bc407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.013+0000 7f4be2d9d640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4bc4078630 0x7f4bc407aaf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400a730 tx=0x7f4bd4008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:42.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.017+0000 7f4bdb7fe640 1 -- 172.21.15.28:0/1452688671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4bdc062ac0 con 0x7f4be415bf30 2024-09-10T20:44:42.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.238+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4be40ccd80 con 0x7f4be415bf30 2024-09-10T20:44:42.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.238+0000 7f4bdb7fe640 1 -- 172.21.15.28:0/1452688671 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5227 (secure 0 0 0) 0x7f4bdc0623c0 con 0x7f4be415bf30 2024-09-10T20:44:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:42.242 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"btime":"2024-09-10T20:44:31:807159+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1041363523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1041363523},{"type":"v1","addr":"172.21.15.28:6834","nonce":1041363523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1561732564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1561732564},{"type":"v1","addr":"172.21.15.28:6835","nonce":1561732564}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6521,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/97213244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":97213244},{"type":"v1","addr":"172.21.15.159:6835","nonce":97213244}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:22.133677+0000","modified":"2024-09-10T20:44:25.147825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6964":{"gid":6964,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/3268698079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3268698079},{"type":"v1","addr":"172.21.15.159:6801","nonce":3268698079}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6964,"qdb_cluster":[6964]},"id":10}]} 2024-09-10T20:44:42.243 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 87 2024-09-10T20:44:42.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.240+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4bc4078630 msgr2=0x7f4bc407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.240+0000 7f4be912b640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4bc4078630 0x7f4bc407aaf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400a730 tx=0x7f4bd4008040 comp rx=0 tx=0).stop 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.240+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be415bf30 msgr2=0x7f4be4071850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.240+0000 7f4be912b640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be415bf30 0x7f4be4071850 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00b9a0 tx=0x7f4bdc00be60 comp rx=0 tx=0).stop 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 shutdown_connections 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4bc4078630 0x7f4bc407aaf0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be415bf30 0x7f4be4071850 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4be40d05d0 0x7f4be4071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 --2- 172.21.15.28:0/1452688671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4be40b5610 0x7f4be41f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 >> 172.21.15.28:0/1452688671 conn(0x7f4be40d1e40 msgr2=0x7f4be4070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 shutdown_connections 2024-09-10T20:44:42.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.241+0000 7f4be912b640 1 -- 172.21.15.28:0/1452688671 wait complete. 2024-09-10T20:44:42.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 -- 172.21.15.28:0/2082953648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f415e0f0 msgr2=0x7f66f40d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 --2- 172.21.15.28:0/2082953648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f415e0f0 0x7f66f40d2030 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f66f408c590 tx=0x7f66ec02a5f0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 -- 172.21.15.28:0/2082953648 shutdown_connections 2024-09-10T20:44:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 --2- 172.21.15.28:0/2082953648 >> v2:172.21.15.159:3300/0 conn(0x7f66f40d4c40 0x7f66f40d2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 --2- 172.21.15.28:0/2082953648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f415e0f0 0x7f66f40d2030 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 -- 172.21.15.28:0/2082953648 >> 172.21.15.28:0/2082953648 conn(0x7f66f40d4040 msgr2=0x7f66f40d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 -- 172.21.15.28:0/2082953648 shutdown_connections 2024-09-10T20:44:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.358+0000 7f66fc124640 1 -- 172.21.15.28:0/2082953648 wait complete. 2024-09-10T20:44:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.359+0000 7f66fc124640 1 Processor -- start 2024-09-10T20:44:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.359+0000 7f66fc124640 1 -- start start 2024-09-10T20:44:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66fc124640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66f40d2ab0 0x7f66f40c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66fc124640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f40d4c40 0x7f66f4146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66fc124640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66f415e0f0 0x7f66f41f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66fc124640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66f4162320 con 0x7f66f40d4c40 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66fc124640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f66f40b8a10 con 0x7f66f40d2ab0 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66fc124640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f66f41624a0 con 0x7f66f415e0f0 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66f96bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f40d4c40 0x7f66f4146ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66f96bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f40d4c40 0x7f66f4146ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40384/0 (socket says 172.21.15.28:40384) 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66f96bf640 1 -- 172.21.15.28:0/1628451041 learned_addr learned my addr 172.21.15.28:0/1628451041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66f9ec0640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66f40d2ab0 0x7f66f40c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.360+0000 7f66fa6c1640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66f415e0f0 0x7f66f41f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f96bf640 1 -- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66f415e0f0 msgr2=0x7f66f41f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f96bf640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66f415e0f0 0x7f66f41f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f96bf640 1 -- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66f40d2ab0 msgr2=0x7f66f40c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f96bf640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66f40d2ab0 0x7f66f40c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f96bf640 1 -- 172.21.15.28:0/1628451041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f4147740 con 0x7f66f40d4c40 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66fa6c1640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66f415e0f0 0x7f66f41f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f96bf640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f40d4c40 0x7f66f4146ec0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f66e800d950 tx=0x7f66e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f27fc640 1 -- 172.21.15.28:0/1628451041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8017070 con 0x7f66f40d4c40 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f27fc640 1 -- 172.21.15.28:0/1628451041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66e8009d20 con 0x7f66f40d4c40 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66f27fc640 1 -- 172.21.15.28:0/1628451041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80106c0 con 0x7f66f40d4c40 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f4147a20 con 0x7f66f40d4c40 2024-09-10T20:44:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.361+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f4147400 con 0x7f66f40d4c40 2024-09-10T20:44:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.362+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f41629c0 con 0x7f66f40d4c40 2024-09-10T20:44:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.363+0000 7f66f27fc640 1 -- 172.21.15.28:0/1628451041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f66e8010860 con 0x7f66f40d4c40 2024-09-10T20:44:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.363+0000 7f66f27fc640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66e0078630 0x7f66e007aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.363+0000 7f66f27fc640 1 -- 172.21.15.28:0/1628451041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f66e809aed0 con 0x7f66f40d4c40 2024-09-10T20:44:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.363+0000 7f66f9ec0640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66e0078630 0x7f66e007aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.364+0000 7f66f9ec0640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66e0078630 0x7f66e007aaf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f66ec033000 tx=0x7f66ec004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.368+0000 7f66f27fc640 1 -- 172.21.15.28:0/1628451041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f66e80628c0 con 0x7f66f40d4c40 2024-09-10T20:44:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.554+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66f40b9fc0 con 0x7f66f40d4c40 2024-09-10T20:44:42.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.555+0000 7f66f27fc640 1 -- 172.21.15.28:0/1628451041 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16783 (secure 0 0 0) 0x7f66e80672f0 con 0x7f66f40d4c40 2024-09-10T20:44:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:42.559 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:23.754781+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:19.761497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:21.093337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66e0078630 msgr2=0x7f66e007aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66e0078630 0x7f66e007aaf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f66ec033000 tx=0x7f66ec004730 comp rx=0 tx=0).stop 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f40d4c40 msgr2=0x7f66f4146ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f40d4c40 0x7f66f4146ec0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f66e800d950 tx=0x7f66e800de10 comp rx=0 tx=0).stop 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 shutdown_connections 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66e0078630 0x7f66e007aaf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66f415e0f0 0x7f66f41f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f40d4c40 0x7f66f4146ec0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 --2- 172.21.15.28:0/1628451041 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66f40d2ab0 0x7f66f40c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 >> 172.21.15.28:0/1628451041 conn(0x7f66f40d4040 msgr2=0x7f66f4160c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 shutdown_connections 2024-09-10T20:44:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.557+0000 7f66fc124640 1 -- 172.21.15.28:0/1628451041 wait complete. 2024-09-10T20:44:42.572 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:44:42.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 -- 172.21.15.28:0/3417926159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c0d2b90 msgr2=0x7f434c161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 --2- 172.21.15.28:0/3417926159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c0d2b90 0x7f434c161b50 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f4334005fd0 tx=0x7f433402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 -- 172.21.15.28:0/3417926159 shutdown_connections 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 --2- 172.21.15.28:0/3417926159 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c162090 0x7f434c0b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 --2- 172.21.15.28:0/3417926159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c0d2b90 0x7f434c161b50 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 -- 172.21.15.28:0/3417926159 >> 172.21.15.28:0/3417926159 conn(0x7f434c0d3fb0 msgr2=0x7f434c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 -- 172.21.15.28:0/3417926159 shutdown_connections 2024-09-10T20:44:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.673+0000 7f43509da640 1 -- 172.21.15.28:0/3417926159 wait complete. 2024-09-10T20:44:42.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.675+0000 7f43509da640 1 Processor -- start 2024-09-10T20:44:42.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.675+0000 7f43509da640 1 -- start start 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d2b90 0x7f434c0a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c15de70 0x7f434c1f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c162090 0x7f434c0b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f434c15e3d0 con 0x7f434c15de70 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f434c15e250 con 0x7f434c162090 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f434c0b7340 con 0x7f434c0d2b90 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c15de70 0x7f434c1f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c15de70 0x7f434c1f54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40398/0 (socket says 172.21.15.28:40398) 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 -- 172.21.15.28:0/1117426397 learned_addr learned my addr 172.21.15.28:0/1117426397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f434a59c640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d2b90 0x7f434c0a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f434ad9d640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c162090 0x7f434c0b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 -- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d2b90 msgr2=0x7f434c0a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d2b90 0x7f434c0a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 -- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c162090 msgr2=0x7f434c0b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c162090 0x7f434c0b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 -- 172.21.15.28:0/1117426397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c1f5a30 con 0x7f434c15de70 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f434a59c640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d2b90 0x7f434c0a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4343fff640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c15de70 0x7f434c1f54f0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4334009850 tx=0x7f43340038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4342ffd640 1 -- 172.21.15.28:0/1117426397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433403f070 con 0x7f434c15de70 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4342ffd640 1 -- 172.21.15.28:0/1117426397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43340047a0 con 0x7f434c15de70 2024-09-10T20:44:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434c0b0e10 con 0x7f434c15de70 2024-09-10T20:44:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f4342ffd640 1 -- 172.21.15.28:0/1117426397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334035030 con 0x7f434c15de70 2024-09-10T20:44:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.676+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434c0b13d0 con 0x7f434c15de70 2024-09-10T20:44:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.677+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f434c0b77a0 con 0x7f434c15de70 2024-09-10T20:44:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.678+0000 7f4342ffd640 1 -- 172.21.15.28:0/1117426397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f4334004b90 con 0x7f434c15de70 2024-09-10T20:44:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.678+0000 7f4342ffd640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f432c078660 0x7f432c07ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.679+0000 7f434a59c640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f432c078660 0x7f432c07ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:42.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.679+0000 7f4342ffd640 1 -- 172.21.15.28:0/1117426397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f43340bba50 con 0x7f434c15de70 2024-09-10T20:44:42.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.679+0000 7f434a59c640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f432c078660 0x7f432c07ab20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f433c000f60 tx=0x7f433c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.683+0000 7f4342ffd640 1 -- 172.21.15.28:0/1117426397 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f434c1f5a30 con 0x7f434c15de70 2024-09-10T20:44:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.905+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f434c0b9f30 con 0x7f434c15de70 2024-09-10T20:44:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.905+0000 7f4342ffd640 1 -- 172.21.15.28:0/1117426397 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5227 (secure 0 0 0) 0x7f4334083a50 con 0x7f434c15de70 2024-09-10T20:44:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:42.910 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"btime":"2024-09-10T20:44:31:807159+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1041363523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1041363523},{"type":"v1","addr":"172.21.15.28:6834","nonce":1041363523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1561732564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1561732564},{"type":"v1","addr":"172.21.15.28:6835","nonce":1561732564}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6521,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/97213244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":97213244},{"type":"v1","addr":"172.21.15.159:6835","nonce":97213244}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:22.133677+0000","modified":"2024-09-10T20:44:25.147825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6964":{"gid":6964,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/3268698079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3268698079},{"type":"v1","addr":"172.21.15.159:6801","nonce":3268698079}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6964,"qdb_cluster":[6964]},"id":10}]} 2024-09-10T20:44:42.910 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 87 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f432c078660 msgr2=0x7f432c07ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f432c078660 0x7f432c07ab20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f433c000f60 tx=0x7f433c009040 comp rx=0 tx=0).stop 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c15de70 msgr2=0x7f434c1f54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c15de70 0x7f434c1f54f0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4334009850 tx=0x7f43340038a0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 shutdown_connections 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f432c078660 0x7f432c07ab20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f434c162090 0x7f434c0b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434c15de70 0x7f434c1f54f0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 --2- 172.21.15.28:0/1117426397 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f434c0d2b90 0x7f434c0a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 >> 172.21.15.28:0/1117426397 conn(0x7f434c0d3fb0 msgr2=0x7f434c0b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:42.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 shutdown_connections 2024-09-10T20:44:42.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:42.908+0000 7f43509da640 1 -- 172.21.15.28:0/1117426397 wait complete. 2024-09-10T20:44:42.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 -- 172.21.15.28:0/3639906271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 msgr2=0x7f01400b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 --2- 172.21.15.28:0/3639906271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f01400b2ba0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f01400d09e0 tx=0x7f013002a540 comp rx=0 tx=0).stop 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 -- 172.21.15.28:0/3639906271 shutdown_connections 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 --2- 172.21.15.28:0/3639906271 >> v2:172.21.15.159:3300/0 conn(0x7f01400b4d10 0x7f01400ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 --2- 172.21.15.28:0/3639906271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f01400b2ba0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 -- 172.21.15.28:0/3639906271 >> 172.21.15.28:0/3639906271 conn(0x7f014006cf00 msgr2=0x7f014006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 -- 172.21.15.28:0/3639906271 shutdown_connections 2024-09-10T20:44:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.024+0000 7f01460b6640 1 -- 172.21.15.28:0/3639906271 wait complete. 2024-09-10T20:44:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.025+0000 7f01460b6640 1 Processor -- start 2024-09-10T20:44:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.025+0000 7f01460b6640 1 -- start start 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f01460b6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01400b4d10 0x7f014014a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f01460b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f014006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f01460b6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f014015df70 0x7f01401578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f01460b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01400d0270 con 0x7f014015a190 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f01460b6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f01400d00f0 con 0x7f014015df70 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f01460b6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f01400d03f0 con 0x7f01400b4d10 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f014006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f014006ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40424/0 (socket says 172.21.15.28:40424) 2024-09-10T20:44:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 -- 172.21.15.28:0/2810694481 learned_addr learned my addr 172.21.15.28:0/2810694481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013f7fe640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01400b4d10 0x7f014014a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013ffff640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f014015df70 0x7f01401578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 -- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01400b4d10 msgr2=0x7f014014a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01400b4d10 0x7f014014a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 -- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f014015df70 msgr2=0x7f01401578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f014015df70 0x7f01401578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 -- 172.21.15.28:0/2810694481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014014aa90 con 0x7f014015a190 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.026+0000 7f013effd640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f014006ff40 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f013400b3f0 tx=0x7f013400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.027+0000 7f0127fff640 1 -- 172.21.15.28:0/2810694481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134015020 con 0x7f014015a190 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.027+0000 7f0127fff640 1 -- 172.21.15.28:0/2810694481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0134011920 con 0x7f014015a190 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.027+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014006f840 con 0x7f014015a190 2024-09-10T20:44:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.027+0000 7f0127fff640 1 -- 172.21.15.28:0/2810694481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134011d50 con 0x7f014015a190 2024-09-10T20:44:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.027+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01401f6480 con 0x7f014015a190 2024-09-10T20:44:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.028+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01400ce030 con 0x7f014015a190 2024-09-10T20:44:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.028+0000 7f0127fff640 1 -- 172.21.15.28:0/2810694481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f013401a400 con 0x7f014015a190 2024-09-10T20:44:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.029+0000 7f0127fff640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f012c078630 0x7f012c07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.029+0000 7f0127fff640 1 -- 172.21.15.28:0/2810694481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f013409ab20 con 0x7f014015a190 2024-09-10T20:44:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.029+0000 7f013f7fe640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f012c078630 0x7f012c07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.029+0000 7f013f7fe640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f012c078630 0x7f012c07aaf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0130033000 tx=0x7f0130004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:43.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.033+0000 7f0127fff640 1 -- 172.21.15.28:0/2810694481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f013401f020 con 0x7f014015a190 2024-09-10T20:44:43.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.253+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f014006d480 con 0x7f014015a190 2024-09-10T20:44:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.254+0000 7f0127fff640 1 -- 172.21.15.28:0/2810694481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5227 (secure 0 0 0) 0x7f01340624d0 con 0x7f014015a190 2024-09-10T20:44:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"btime":"2024-09-10T20:44:31:807159+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1041363523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1041363523},{"type":"v1","addr":"172.21.15.28:6834","nonce":1041363523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1561732564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1561732564},{"type":"v1","addr":"172.21.15.28:6835","nonce":1561732564}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6521,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/97213244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":97213244},{"type":"v1","addr":"172.21.15.159:6835","nonce":97213244}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:22.133677+0000","modified":"2024-09-10T20:44:25.147825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6964":{"gid":6964,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/3268698079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3268698079},{"type":"v1","addr":"172.21.15.159:6801","nonce":3268698079}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6964,"qdb_cluster":[6964]},"id":10}]} 2024-09-10T20:44:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 87 2024-09-10T20:44:43.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f012c078630 msgr2=0x7f012c07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f012c078630 0x7f012c07aaf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0130033000 tx=0x7f0130004730 comp rx=0 tx=0).stop 2024-09-10T20:44:43.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 msgr2=0x7f014006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f014006ff40 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f013400b3f0 tx=0x7f013400b8b0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 shutdown_connections 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f012c078630 0x7f012c07aaf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f014015df70 0x7f01401578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014015a190 0x7f014006ff40 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 --2- 172.21.15.28:0/2810694481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01400b4d10 0x7f014014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 >> 172.21.15.28:0/2810694481 conn(0x7f014006cf00 msgr2=0x7f014015bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 shutdown_connections 2024-09-10T20:44:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.256+0000 7f01460b6640 1 -- 172.21.15.28:0/2810694481 wait complete. 2024-09-10T20:44:43.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:43.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 -- 172.21.15.28:0/101062460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c15ee70 msgr2=0x7f898c0d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 --2- 172.21.15.28:0/101062460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c15ee70 0x7f898c0d0140 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f898c0ced80 tx=0x7f898402aa30 comp rx=0 tx=0).stop 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 -- 172.21.15.28:0/101062460 shutdown_connections 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 --2- 172.21.15.28:0/101062460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c15ee70 0x7f898c0d0140 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 --2- 172.21.15.28:0/101062460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f898c1554e0 0x7f898c15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 --2- 172.21.15.28:0/101062460 >> v2:172.21.15.159:3300/0 conn(0x7f898c155100 0x7f898c15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 -- 172.21.15.28:0/101062460 >> 172.21.15.28:0/101062460 conn(0x7f898c06cf00 msgr2=0x7f898c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 -- 172.21.15.28:0/101062460 shutdown_connections 2024-09-10T20:44:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.372+0000 7f89947c2640 1 -- 172.21.15.28:0/101062460 wait complete. 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.373+0000 7f89947c2640 1 Processor -- start 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.373+0000 7f89947c2640 1 -- start start 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f89947c2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f898c155100 0x7f898c14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f89947c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1554e0 0x7f898c149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f89947c2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f898c15ee70 0x7f898c0b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f89947c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f898c072bd0 con 0x7f898c1554e0 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f89947c2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f898c0d0ab0 con 0x7f898c155100 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f89947c2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f898c072d50 con 0x7f898c15ee70 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f8991d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1554e0 0x7f898c149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f8991d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1554e0 0x7f898c149c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40442/0 (socket says 172.21.15.28:40442) 2024-09-10T20:44:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f8991d5d640 1 -- 172.21.15.28:0/1920961360 learned_addr learned my addr 172.21.15.28:0/1920961360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f899255e640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f898c155100 0x7f898c14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.374+0000 7f8992d5f640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f898c15ee70 0x7f898c0b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8991d5d640 1 -- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f898c15ee70 msgr2=0x7f898c0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8991d5d640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f898c15ee70 0x7f898c0b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8991d5d640 1 -- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f898c155100 msgr2=0x7f898c14a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8991d5d640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f898c155100 0x7f898c14a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8991d5d640 1 -- 172.21.15.28:0/1920961360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c06f450 con 0x7f898c1554e0 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8991d5d640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1554e0 0x7f898c149c50 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f897c00d950 tx=0x7f897c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8982ffd640 1 -- 172.21.15.28:0/1920961360 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c017070 con 0x7f898c1554e0 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8982ffd640 1 -- 172.21.15.28:0/1920961360 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f897c009d20 con 0x7f898c1554e0 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898c06f730 con 0x7f898c1554e0 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f8982ffd640 1 -- 172.21.15.28:0/1920961360 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c0106c0 con 0x7f898c1554e0 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.375+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f898c06fcf0 con 0x7f898c1554e0 2024-09-10T20:44:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.376+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f898c15e930 con 0x7f898c1554e0 2024-09-10T20:44:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.377+0000 7f8982ffd640 1 -- 172.21.15.28:0/1920961360 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f897c010860 con 0x7f898c1554e0 2024-09-10T20:44:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.377+0000 7f8982ffd640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8970078660 0x7f897007ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.377+0000 7f899255e640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8970078660 0x7f897007ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.377+0000 7f8982ffd640 1 -- 172.21.15.28:0/1920961360 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f897c09af00 con 0x7f898c1554e0 2024-09-10T20:44:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.378+0000 7f899255e640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8970078660 0x7f897007ab20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f898c14e2a0 tx=0x7f898800b260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:43.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.382+0000 7f8982ffd640 1 -- 172.21.15.28:0/1920961360 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f897c062860 con 0x7f898c1554e0 2024-09-10T20:44:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.569+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f898c06d380 con 0x7f898c1554e0 2024-09-10T20:44:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.570+0000 7f8982ffd640 1 -- 172.21.15.28:0/1920961360 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16783 (secure 0 0 0) 0x7f897c067290 con 0x7f898c1554e0 2024-09-10T20:44:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:43.574 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:23.754781+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:19.761497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:21.093337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8970078660 msgr2=0x7f897007ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8970078660 0x7f897007ab20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f898c14e2a0 tx=0x7f898800b260 comp rx=0 tx=0).stop 2024-09-10T20:44:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1554e0 msgr2=0x7f898c149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1554e0 0x7f898c149c50 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f897c00d950 tx=0x7f897c00de10 comp rx=0 tx=0).stop 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 shutdown_connections 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8970078660 0x7f897007ab20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f898c15ee70 0x7f898c0b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1554e0 0x7f898c149c50 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 --2- 172.21.15.28:0/1920961360 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f898c155100 0x7f898c14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 >> 172.21.15.28:0/1920961360 conn(0x7f898c06cf00 msgr2=0x7f898c073990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 shutdown_connections 2024-09-10T20:44:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.572+0000 7f89947c2640 1 -- 172.21.15.28:0/1920961360 wait complete. 2024-09-10T20:44:43.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:44:43.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.689+0000 7f4910d17640 1 -- 172.21.15.28:0/50283841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c162090 msgr2=0x7f490c0b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.689+0000 7f4910d17640 1 --2- 172.21.15.28:0/50283841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c162090 0x7f490c0b6e00 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f490c0d4d80 tx=0x7f490402aac0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.690+0000 7f4910d17640 1 -- 172.21.15.28:0/50283841 shutdown_connections 2024-09-10T20:44:43.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.690+0000 7f4910d17640 1 --2- 172.21.15.28:0/50283841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c162090 0x7f490c0b6e00 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.690+0000 7f4910d17640 1 --2- 172.21.15.28:0/50283841 >> v2:172.21.15.159:3300/0 conn(0x7f490c0d2b90 0x7f490c161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.690+0000 7f4910d17640 1 -- 172.21.15.28:0/50283841 >> 172.21.15.28:0/50283841 conn(0x7f490c0d3fb0 msgr2=0x7f490c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:43.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.690+0000 7f4910d17640 1 -- 172.21.15.28:0/50283841 shutdown_connections 2024-09-10T20:44:43.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.690+0000 7f4910d17640 1 -- 172.21.15.28:0/50283841 wait complete. 2024-09-10T20:44:43.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.691+0000 7f4910d17640 1 Processor -- start 2024-09-10T20:44:43.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.691+0000 7f4910d17640 1 -- start start 2024-09-10T20:44:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4910d17640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f490c0d2b90 0x7f490c09dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4910d17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c15de70 0x7f490c09f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4910d17640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f490c162090 0x7f490c0a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4910d17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f490c15e3d0 con 0x7f490c15de70 2024-09-10T20:44:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4910d17640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f490c15e250 con 0x7f490c162090 2024-09-10T20:44:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4910d17640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f490c0b7340 con 0x7f490c0d2b90 2024-09-10T20:44:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c15de70 0x7f490c09f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c15de70 0x7f490c09f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40460/0 (socket says 172.21.15.28:40460) 2024-09-10T20:44:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 -- 172.21.15.28:0/170855788 learned_addr learned my addr 172.21.15.28:0/170855788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f490a59c640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f490c0d2b90 0x7f490c09dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f490ad9d640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f490c162090 0x7f490c0a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 -- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f490c0d2b90 msgr2=0x7f490c09dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f490c0d2b90 0x7f490c09dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 -- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f490c162090 msgr2=0x7f490c0a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f490c162090 0x7f490c0a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 -- 172.21.15.28:0/170855788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490c1f5820 con 0x7f490c15de70 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f490ad9d640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f490c162090 0x7f490c0a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f490a59c640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f490c0d2b90 0x7f490c09dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.692+0000 7f4909d9b640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c15de70 0x7f490c09f8f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f490000e9f0 tx=0x7f490000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.693+0000 7f48faffd640 1 -- 172.21.15.28:0/170855788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490000cd80 con 0x7f490c15de70 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.693+0000 7f48faffd640 1 -- 172.21.15.28:0/170855788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4900010430 con 0x7f490c15de70 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.693+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490c1f4c10 con 0x7f490c15de70 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.693+0000 7f48faffd640 1 -- 172.21.15.28:0/170855788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900010840 con 0x7f490c15de70 2024-09-10T20:44:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.693+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490c1f51d0 con 0x7f490c15de70 2024-09-10T20:44:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.694+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f490c0b7860 con 0x7f490c15de70 2024-09-10T20:44:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.694+0000 7f48faffd640 1 -- 172.21.15.28:0/170855788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f49000109e0 con 0x7f490c15de70 2024-09-10T20:44:43.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.695+0000 7f48faffd640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48ec078660 0x7f48ec07ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:43.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.695+0000 7f48faffd640 1 -- 172.21.15.28:0/170855788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5737+0+0 (secure 0 0 0) 0x7f490009b600 con 0x7f490c15de70 2024-09-10T20:44:43.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.695+0000 7f490a59c640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48ec078660 0x7f48ec07ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:43.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.695+0000 7f490a59c640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48ec078660 0x7f48ec07ab20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f490c0b8c40 tx=0x7f48fc009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:43.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.699+0000 7f48faffd640 1 -- 172.21.15.28:0/170855788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4900063700 con 0x7f490c15de70 2024-09-10T20:44:43.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.922+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f490c0b9f30 con 0x7f490c15de70 2024-09-10T20:44:43.926 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:44:43.923+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:44:43.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.924+0000 7f48faffd640 1 -- 172.21.15.28:0/170855788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) ==== 161+0+0 (secure 0 0 0) 0x7f4900063000 con 0x7f490c15de70 2024-09-10T20:44:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:44:43.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48ec078660 msgr2=0x7f48ec07ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48ec078660 0x7f48ec07ab20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f490c0b8c40 tx=0x7f48fc009040 comp rx=0 tx=0).stop 2024-09-10T20:44:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c15de70 msgr2=0x7f490c09f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c15de70 0x7f490c09f8f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f490000e9f0 tx=0x7f490000eeb0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 shutdown_connections 2024-09-10T20:44:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f48ec078660 0x7f48ec07ab20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f490c162090 0x7f490c0a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c15de70 0x7f490c09f8f0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 --2- 172.21.15.28:0/170855788 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f490c0d2b90 0x7f490c09dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:43.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.926+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 >> 172.21.15.28:0/170855788 conn(0x7f490c0d3fb0 msgr2=0x7f490c0b93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:43.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.927+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 shutdown_connections 2024-09-10T20:44:43.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:43.927+0000 7f4910d17640 1 -- 172.21.15.28:0/170855788 wait complete. 2024-09-10T20:44:43.934 INFO:tasks.ceph.mds.b.smithi159.stderr: -2912> 2024-09-10T20:44:24.738+0000 7f7715542640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:44:43.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:44:43.963 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.045+0000 7f549adad640 1 -- 172.21.15.28:0/996268861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 msgr2=0x7f5494072660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.045+0000 7f549adad640 1 --2- 172.21.15.28:0/996268861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 0x7f5494072660 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f547c005fd0 tx=0x7f547c02a670 comp rx=0 tx=0).stop 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.046+0000 7f549adad640 1 -- 172.21.15.28:0/996268861 shutdown_connections 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.046+0000 7f549adad640 1 --2- 172.21.15.28:0/996268861 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5494072ba0 0x7f54940ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.046+0000 7f549adad640 1 --2- 172.21.15.28:0/996268861 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 0x7f5494072660 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.046+0000 7f549adad640 1 -- 172.21.15.28:0/996268861 >> 172.21.15.28:0/996268861 conn(0x7f54940b1800 msgr2=0x7f54940b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.046+0000 7f549adad640 1 -- 172.21.15.28:0/996268861 shutdown_connections 2024-09-10T20:44:44.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.046+0000 7f549adad640 1 -- 172.21.15.28:0/996268861 wait complete. 2024-09-10T20:44:44.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.047+0000 7f549adad640 1 Processor -- start 2024-09-10T20:44:44.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.047+0000 7f549adad640 1 -- start start 2024-09-10T20:44:44.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549adad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 0x7f549413e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:44.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549adad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5494072ba0 0x7f54941540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:44.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549adad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bbc0 0x7f54941f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:44.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549adad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54940ce880 con 0x7f549415bbc0 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549adad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f54940ce700 con 0x7f5494070020 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549adad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f54940cea00 con 0x7f5494072ba0 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bbc0 0x7f54941f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bbc0 0x7f54941f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40466/0 (socket says 172.21.15.28:40466) 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 -- 172.21.15.28:0/327986367 learned_addr learned my addr 172.21.15.28:0/327986367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f5493fff640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5494072ba0 0x7f54941540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f5498b49640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 0x7f549413e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 -- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5494072ba0 msgr2=0x7f54941540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5494072ba0 0x7f54941540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 -- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 msgr2=0x7f549413e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 0x7f549413e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f549934a640 1 -- 172.21.15.28:0/327986367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54941547b0 con 0x7f549415bbc0 2024-09-10T20:44:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.048+0000 7f5498b49640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 0x7f549413e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.049+0000 7f549934a640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bbc0 0x7f54941f7c70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f548400eb10 tx=0x7f548400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.049+0000 7f54917fa640 1 -- 172.21.15.28:0/327986367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484004b20 con 0x7f549415bbc0 2024-09-10T20:44:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.049+0000 7f54917fa640 1 -- 172.21.15.28:0/327986367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5484004ce0 con 0x7f549415bbc0 2024-09-10T20:44:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.049+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5494154a90 con 0x7f549415bbc0 2024-09-10T20:44:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.049+0000 7f54917fa640 1 -- 172.21.15.28:0/327986367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484005280 con 0x7f549415bbc0 2024-09-10T20:44:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.049+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54941f81b0 con 0x7f549415bbc0 2024-09-10T20:44:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.050+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54940cef20 con 0x7f549415bbc0 2024-09-10T20:44:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.050+0000 7f54917fa640 1 -- 172.21.15.28:0/327986367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f5484005420 con 0x7f549415bbc0 2024-09-10T20:44:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.051+0000 7f54917fa640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.051+0000 7f5498b49640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.051+0000 7f54917fa640 1 -- 172.21.15.28:0/327986367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5823+0+0 (secure 0 0 0) 0x7f548409b700 con 0x7f549415bbc0 2024-09-10T20:44:44.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.052+0000 7f5498b49640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f548c007fd0 tx=0x7f548c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.055+0000 7f54917fa640 1 -- 172.21.15.28:0/327986367 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5484063880 con 0x7f549415bbc0 2024-09-10T20:44:44.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.277+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f54941ea150 con 0x7f549415bbc0 2024-09-10T20:44:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.931+0000 7f54917fa640 1 -- 172.21.15.28:0/327986367 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) ==== 104+0+0 (secure 0 0 0) 0x7f5484063180 con 0x7f549415bbc0 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 msgr2=0x7f547407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f548c007fd0 tx=0x7f548c009040 comp rx=0 tx=0).stop 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bbc0 msgr2=0x7f54941f7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bbc0 0x7f54941f7c70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f548400eb10 tx=0x7f548400c2d0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 shutdown_connections 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bbc0 0x7f54941f7c70 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5494072ba0 0x7f54941540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 --2- 172.21.15.28:0/327986367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5494070020 0x7f549413e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:44.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 >> 172.21.15.28:0/327986367 conn(0x7f54940b1800 msgr2=0x7f54940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:44.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 shutdown_connections 2024-09-10T20:44:44.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:44.934+0000 7f549adad640 1 -- 172.21.15.28:0/327986367 wait complete. 2024-09-10T20:44:44.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:44:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 -- 172.21.15.28:0/3110049785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 msgr2=0x7f958c0cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 --2- 172.21.15.28:0/3110049785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cefc0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f958c0cc7b0 tx=0x7f958002a560 comp rx=0 tx=0).stop 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 -- 172.21.15.28:0/3110049785 shutdown_connections 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 --2- 172.21.15.28:0/3110049785 >> v2:172.21.15.159:3300/0 conn(0x7f958c0c5ef0 0x7f958c0c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 --2- 172.21.15.28:0/3110049785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cefc0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 --2- 172.21.15.28:0/3110049785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f958c15f610 0x7f958c0ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 -- 172.21.15.28:0/3110049785 >> 172.21.15.28:0/3110049785 conn(0x7f958c0d1f10 msgr2=0x7f958c0d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 -- 172.21.15.28:0/3110049785 shutdown_connections 2024-09-10T20:44:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.064+0000 7f95945d3640 1 -- 172.21.15.28:0/3110049785 wait complete. 2024-09-10T20:44:45.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 Processor -- start 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 -- start start 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958c0c5ef0 0x7f958c09f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f958c15f610 0x7f958c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0c6c50 con 0x7f958c0cebc0 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0c6ad0 con 0x7f958c0c5ef0 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.066+0000 7f95945d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f958c0c6dd0 con 0x7f958c15f610 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40498/0 (socket says 172.21.15.28:40498) 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 -- 172.21.15.28:0/1129276337 learned_addr learned my addr 172.21.15.28:0/1129276337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9592b70640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f958c15f610 0x7f958c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f959236f640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958c0c5ef0 0x7f958c09f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 -- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f958c15f610 msgr2=0x7f958c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f958c15f610 0x7f958c0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 -- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958c0c5ef0 msgr2=0x7f958c09f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958c0c5ef0 0x7f958c09f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 -- 172.21.15.28:0/1129276337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c14f150 con 0x7f958c0cebc0 2024-09-10T20:44:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f9591b6e640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cb1b0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f958002ab50 tx=0x7f95800039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f958affd640 1 -- 172.21.15.28:0/1129276337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958003f070 con 0x7f958c0cebc0 2024-09-10T20:44:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f958affd640 1 -- 172.21.15.28:0/1129276337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9580004890 con 0x7f958c0cebc0 2024-09-10T20:44:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c152160 con 0x7f958c0cebc0 2024-09-10T20:44:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f958affd640 1 -- 172.21.15.28:0/1129276337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580035030 con 0x7f958c0cebc0 2024-09-10T20:44:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.067+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c1f6080 con 0x7f958c0cebc0 2024-09-10T20:44:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.068+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958c0cefc0 con 0x7f958c0cebc0 2024-09-10T20:44:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.069+0000 7f958affd640 1 -- 172.21.15.28:0/1129276337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f9580004ce0 con 0x7f958c0cebc0 2024-09-10T20:44:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.069+0000 7f958affd640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9578078630 0x7f957807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.070+0000 7f958affd640 1 -- 172.21.15.28:0/1129276337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5823+0+0 (secure 0 0 0) 0x7f95800bbc00 con 0x7f958c0cebc0 2024-09-10T20:44:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.070+0000 7f959236f640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9578078630 0x7f957807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.070+0000 7f959236f640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9578078630 0x7f957807aaf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f957c005fd0 tx=0x7f957c0095e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.074+0000 7f958affd640 1 -- 172.21.15.28:0/1129276337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f958c14f150 con 0x7f958c0cebc0 2024-09-10T20:44:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.263+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f958c0d1cd0 con 0x7f958c0cebc0 2024-09-10T20:44:45.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.933+0000 7f958affd640 1 -- 172.21.15.28:0/1129276337 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v129) ==== 181+0+0 (secure 0 0 0) 0x7f95800835a0 con 0x7f958c0cebc0 2024-09-10T20:44:45.938 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:44:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9578078630 msgr2=0x7f957807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9578078630 0x7f957807aaf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f957c005fd0 tx=0x7f957c0095e0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 msgr2=0x7f958c0cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cb1b0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f958002ab50 tx=0x7f95800039e0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 shutdown_connections 2024-09-10T20:44:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9578078630 0x7f957807aaf0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f958c15f610 0x7f958c0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c0cebc0 0x7f958c0cb1b0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 --2- 172.21.15.28:0/1129276337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f958c0c5ef0 0x7f958c09f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:45.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 >> 172.21.15.28:0/1129276337 conn(0x7f958c0d1f10 msgr2=0x7f958c071200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:45.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 shutdown_connections 2024-09-10T20:44:45.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:45.937+0000 7f95945d3640 1 -- 172.21.15.28:0/1129276337 wait complete. 2024-09-10T20:44:45.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:44:46.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 -- 172.21.15.28:0/1295778790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 msgr2=0x7f93dc06cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:46.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 --2- 172.21.15.28:0/1295778790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc06cae0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f93dc070d50 tx=0x7f93cc02a560 comp rx=0 tx=0).stop 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 -- 172.21.15.28:0/1295778790 shutdown_connections 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 --2- 172.21.15.28:0/1295778790 >> v2:172.21.15.159:3300/0 conn(0x7f93dc06d020 0x7f93dc0715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 --2- 172.21.15.28:0/1295778790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc06cae0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 -- 172.21.15.28:0/1295778790 >> 172.21.15.28:0/1295778790 conn(0x7f93dc0b2920 msgr2=0x7f93dc0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 -- 172.21.15.28:0/1295778790 shutdown_connections 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.069+0000 7f93e1edd640 1 -- 172.21.15.28:0/1295778790 wait complete. 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.070+0000 7f93e1edd640 1 Processor -- start 2024-09-10T20:44:46.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93e1edd640 1 -- start start 2024-09-10T20:44:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93e1edd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93dc06d020 0x7f93dc15b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93e1edd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93dc071af0 0x7f93dc159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93e1edd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93e1edd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93dc0afe90 con 0x7f93dc0af930 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93e1edd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f93dc0afd10 con 0x7f93dc071af0 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93e1edd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f93dc073e10 con 0x7f93dc06d020 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40508/0 (socket says 172.21.15.28:40508) 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.071+0000 7f93dbfff640 1 -- 172.21.15.28:0/2911633455 learned_addr learned my addr 172.21.15.28:0/2911633455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93db7fe640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93dc06d020 0x7f93dc15b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93daffd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93dc071af0 0x7f93dc159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93dbfff640 1 -- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93dc06d020 msgr2=0x7f93dc15b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93dbfff640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93dc06d020 0x7f93dc15b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93dbfff640 1 -- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93dc071af0 msgr2=0x7f93dc159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93dbfff640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93dc071af0 0x7f93dc159af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93dbfff640 1 -- 172.21.15.28:0/2911633455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93dc15a3f0 con 0x7f93dc0af930 2024-09-10T20:44:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93daffd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93dc071af0 0x7f93dc159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93dbfff640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc156d50 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f93d400c770 tx=0x7f93d400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93c3fff640 1 -- 172.21.15.28:0/2911633455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4017070 con 0x7f93dc0af930 2024-09-10T20:44:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93c3fff640 1 -- 172.21.15.28:0/2911633455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93d4005070 con 0x7f93dc0af930 2024-09-10T20:44:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93c3fff640 1 -- 172.21.15.28:0/2911633455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40054c0 con 0x7f93dc0af930 2024-09-10T20:44:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93dc15a6d0 con 0x7f93dc0af930 2024-09-10T20:44:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.072+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93dc15ac90 con 0x7f93dc0af930 2024-09-10T20:44:46.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.073+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93dc0742c0 con 0x7f93dc0af930 2024-09-10T20:44:46.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.074+0000 7f93c3fff640 1 -- 172.21.15.28:0/2911633455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f93d4005660 con 0x7f93dc0af930 2024-09-10T20:44:46.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.074+0000 7f93c3fff640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c8078630 0x7f93c807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:46.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.074+0000 7f93c3fff640 1 -- 172.21.15.28:0/2911633455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5385+0+0 (secure 0 0 0) 0x7f93d409a2a0 con 0x7f93dc0af930 2024-09-10T20:44:46.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.074+0000 7f93db7fe640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c8078630 0x7f93c807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:46.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.075+0000 7f93db7fe640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c8078630 0x7f93c807aaf0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f93cc033000 tx=0x7f93cc004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:46.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.079+0000 7f93c3fff640 1 -- 172.21.15.28:0/2911633455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f93d4061d60 con 0x7f93dc0af930 2024-09-10T20:44:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.267+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f93dc152bd0 con 0x7f93dc0af930 2024-09-10T20:44:46.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.939+0000 7f93c3fff640 1 -- 172.21.15.28:0/2911633455 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v130) ==== 169+0+0 (secure 0 0 0) 0x7f93d4066790 con 0x7f93dc0af930 2024-09-10T20:44:46.943 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:44:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c8078630 msgr2=0x7f93c807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c8078630 0x7f93c807aaf0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f93cc033000 tx=0x7f93cc004730 comp rx=0 tx=0).stop 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 msgr2=0x7f93dc156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc156d50 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f93d400c770 tx=0x7f93d400cc30 comp rx=0 tx=0).stop 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 shutdown_connections 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f93c8078630 0x7f93c807aaf0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93dc0af930 0x7f93dc156d50 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f93dc071af0 0x7f93dc159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 --2- 172.21.15.28:0/2911633455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f93dc06d020 0x7f93dc15b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 >> 172.21.15.28:0/2911633455 conn(0x7f93dc0b2920 msgr2=0x7f93dc15dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 shutdown_connections 2024-09-10T20:44:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:46.943+0000 7f93e1edd640 1 -- 172.21.15.28:0/2911633455 wait complete. 2024-09-10T20:44:46.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:44:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 -- 172.21.15.28:0/1227431206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840d03f0 msgr2=0x7f92840d0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 --2- 172.21.15.28:0/1227431206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840d03f0 0x7f92840d0d90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f9280005fd0 tx=0x7f928002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 -- 172.21.15.28:0/1227431206 shutdown_connections 2024-09-10T20:44:47.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 --2- 172.21.15.28:0/1227431206 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92840d07d0 0x7f928406e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 --2- 172.21.15.28:0/1227431206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840d03f0 0x7f92840d0d90 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 -- 172.21.15.28:0/1227431206 >> 172.21.15.28:0/1227431206 conn(0x7f92840d1e40 msgr2=0x7f92840d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:47.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 -- 172.21.15.28:0/1227431206 shutdown_connections 2024-09-10T20:44:47.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.070+0000 7f928be8c640 1 -- 172.21.15.28:0/1227431206 wait complete. 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.071+0000 7f928be8c640 1 Processor -- start 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.071+0000 7f928be8c640 1 -- start start 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f928be8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840b21b0 0x7f92840af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f928be8c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92840d03f0 0x7f928409c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f928be8c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92840d07d0 0x7f928409cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f928be8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92840aedb0 con 0x7f92840b21b0 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f928be8c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f92840aec30 con 0x7f92840d03f0 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f928be8c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92840aef30 con 0x7f92840d07d0 2024-09-10T20:44:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f9289c28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840b21b0 0x7f92840af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f9289c28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840b21b0 0x7f92840af670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40522/0 (socket says 172.21.15.28:40522) 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f9289c28640 1 -- 172.21.15.28:0/2418631126 learned_addr learned my addr 172.21.15.28:0/2418631126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f9289427640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92840d03f0 0x7f928409c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.072+0000 7f928a429640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92840d07d0 0x7f928409cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f9289c28640 1 -- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92840d07d0 msgr2=0x7f928409cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f9289c28640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92840d07d0 0x7f928409cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f9289c28640 1 -- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92840d03f0 msgr2=0x7f928409c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f9289c28640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92840d03f0 0x7f928409c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f9289c28640 1 -- 172.21.15.28:0/2418631126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928415ed40 con 0x7f92840b21b0 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f9289c28640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840b21b0 0x7f92840af670 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f9280000c00 tx=0x7f92800038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f927a7fc640 1 -- 172.21.15.28:0/2418631126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928003f070 con 0x7f92840b21b0 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f927a7fc640 1 -- 172.21.15.28:0/2418631126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9280005600 con 0x7f92840b21b0 2024-09-10T20:44:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f928409d3a0 con 0x7f92840b21b0 2024-09-10T20:44:47.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f927a7fc640 1 -- 172.21.15.28:0/2418631126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280035030 con 0x7f92840b21b0 2024-09-10T20:44:47.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.073+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f928415f2c0 con 0x7f92840b21b0 2024-09-10T20:44:47.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.074+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92840d0d90 con 0x7f92840b21b0 2024-09-10T20:44:47.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.075+0000 7f927a7fc640 1 -- 172.21.15.28:0/2418631126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f9280004a00 con 0x7f92840b21b0 2024-09-10T20:44:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.075+0000 7f927a7fc640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078630 0x7f926807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.075+0000 7f927a7fc640 1 -- 172.21.15.28:0/2418631126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7f92800bb680 con 0x7f92840b21b0 2024-09-10T20:44:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.075+0000 7f9289427640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078630 0x7f926807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.076+0000 7f9289427640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078630 0x7f926807aaf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f92840b04d0 tx=0x7f9274009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.080+0000 7f927a7fc640 1 -- 172.21.15.28:0/2418631126 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9280083370 con 0x7f92840b21b0 2024-09-10T20:44:47.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.266+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f92840d2500 con 0x7f92840b21b0 2024-09-10T20:44:47.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.269+0000 7f927a7fc640 1 -- 172.21.15.28:0/2418631126 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v43) ==== 71+0+0 (secure 0 0 0) 0x7f9280087da0 con 0x7f92840b21b0 2024-09-10T20:44:47.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078630 msgr2=0x7f926807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078630 0x7f926807aaf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f92840b04d0 tx=0x7f9274009040 comp rx=0 tx=0).stop 2024-09-10T20:44:47.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840b21b0 msgr2=0x7f92840af670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840b21b0 0x7f92840af670 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f9280000c00 tx=0x7f92800038a0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 shutdown_connections 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9268078630 0x7f926807aaf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92840d07d0 0x7f928409cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92840d03f0 0x7f928409c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 --2- 172.21.15.28:0/2418631126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92840b21b0 0x7f92840af670 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 >> 172.21.15.28:0/2418631126 conn(0x7f92840d1e40 msgr2=0x7f928415d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 shutdown_connections 2024-09-10T20:44:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.271+0000 7f928be8c640 1 -- 172.21.15.28:0/2418631126 wait complete. 2024-09-10T20:44:47.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol' 2024-09-10T20:44:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.398+0000 7f91e873d640 1 -- 172.21.15.28:0/2104402553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91c8004d40 msgr2=0x7f91c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.398+0000 7f91e873d640 1 --2- 172.21.15.28:0/2104402553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91c8004d40 0x7f91c80051c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009f00 tx=0x7f91c8005830 comp rx=0 tx=0).stop 2024-09-10T20:44:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.398+0000 7f91e6cda640 1 -- 172.21.15.28:0/2104402553 reap_dead start 2024-09-10T20:44:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.398+0000 7f91e873d640 1 -- 172.21.15.28:0/2104402553 shutdown_connections 2024-09-10T20:44:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.398+0000 7f91e873d640 1 -- 172.21.15.28:0/2104402553 >> 172.21.15.28:0/2104402553 conn(0x7f91e00d0420 msgr2=0x7f91e00d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.398+0000 7f91e873d640 1 -- 172.21.15.28:0/2104402553 shutdown_connections 2024-09-10T20:44:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.398+0000 7f91e873d640 1 -- 172.21.15.28:0/2104402553 wait complete. 2024-09-10T20:44:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.399+0000 7f91e873d640 1 Processor -- start 2024-09-10T20:44:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.399+0000 7f91e873d640 1 -- start start 2024-09-10T20:44:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e873d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91c8004d40 0x7f91e0154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e873d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91e00b0a70 0x7f91e0147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e873d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e006f990 0x7f91e006fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e873d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91e0073e70 con 0x7f91e006f990 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e873d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f91e0073cf0 con 0x7f91e00b0a70 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e873d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f91e0073ff0 con 0x7f91c8004d40 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e006f990 0x7f91e006fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e006f990 0x7f91e006fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40530/0 (socket says 172.21.15.28:40530) 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 -- 172.21.15.28:0/1237819980 learned_addr learned my addr 172.21.15.28:0/1237819980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e5cd8640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91e00b0a70 0x7f91e0147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e64d9640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91c8004d40 0x7f91e0154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 -- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91c8004d40 msgr2=0x7f91e0154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91c8004d40 0x7f91e0154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 -- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91e00b0a70 msgr2=0x7f91e0147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91e00b0a70 0x7f91e0147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.400+0000 7f91e6cda640 1 -- 172.21.15.28:0/1237819980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e0070810 con 0x7f91e006f990 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.401+0000 7f91e6cda640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e006f990 0x7f91e006fe40 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009f00 tx=0x7f91d0033d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.401+0000 7f91deffd640 1 -- 172.21.15.28:0/1237819980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0043070 con 0x7f91e006f990 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.401+0000 7f91deffd640 1 -- 172.21.15.28:0/1237819980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91d0003c60 con 0x7f91e006f990 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.401+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e00b1090 con 0x7f91e006f990 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.401+0000 7f91deffd640 1 -- 172.21.15.28:0/1237819980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0038040 con 0x7f91e006f990 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.401+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e00b1650 con 0x7f91e006f990 2024-09-10T20:44:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.402+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e00cf3e0 con 0x7f91e006f990 2024-09-10T20:44:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.403+0000 7f91deffd640 1 -- 172.21.15.28:0/1237819980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f91d003b050 con 0x7f91e006f990 2024-09-10T20:44:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.403+0000 7f91deffd640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078630 0x7f91cc07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.403+0000 7f91e64d9640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078630 0x7f91cc07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.403+0000 7f91deffd640 1 -- 172.21.15.28:0/1237819980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7f91d00ba500 con 0x7f91e006f990 2024-09-10T20:44:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.404+0000 7f91e64d9640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078630 0x7f91cc07aaf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009bd0 tx=0x7f91d8005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:47.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.408+0000 7f91deffd640 1 -- 172.21.15.28:0/1237819980 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f91d0003e50 con 0x7f91e006f990 2024-09-10T20:44:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.664+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]} v 0) -- 0x7f91e0068d10 con 0x7f91e006f990 2024-09-10T20:44:47.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.940+0000 7f91deffd640 1 -- 172.21.15.28:0/1237819980 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]}]=0 v366) ==== 151+0+0 (secure 0 0 0) 0x7f91d0082260 con 0x7f91e006f990 2024-09-10T20:44:47.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.942+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078630 msgr2=0x7f91cc07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.942+0000 7f91e873d640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078630 0x7f91cc07aaf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009bd0 tx=0x7f91d8005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.942+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e006f990 msgr2=0x7f91e006fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.942+0000 7f91e873d640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e006f990 0x7f91e006fe40 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009f00 tx=0x7f91d0033d30 comp rx=0 tx=0).stop 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 shutdown_connections 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078630 0x7f91cc07aaf0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e006f990 0x7f91e006fe40 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91e00b0a70 0x7f91e0147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 --2- 172.21.15.28:0/1237819980 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91c8004d40 0x7f91e0154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 >> 172.21.15.28:0/1237819980 conn(0x7f91e00d0420 msgr2=0x7f91e00cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 shutdown_connections 2024-09-10T20:44:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:47.943+0000 7f91e873d640 1 -- 172.21.15.28:0/1237819980 wait complete. 2024-09-10T20:44:47.958 INFO:tasks.cephfs_test_runner:test_subvolume_ops_on_nonexistent_vol (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-10T20:44:47.959 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_upgrade_legacy_to_v1 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- rather than going through a full upgrade cycle, 2024-09-10T20:44:47.959 INFO:tasks.cephfs_test_runner:test_subvolume_upgrade_legacy_to_v1 (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:44:47.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 -- 172.21.15.28:0/4058283791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c161b50 msgr2=0x7f308c161fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 --2- 172.21.15.28:0/4058283791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c161b50 0x7f308c161fc0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f308c08c020 tx=0x7f308402a560 comp rx=0 tx=0).stop 2024-09-10T20:44:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 -- 172.21.15.28:0/4058283791 shutdown_connections 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 --2- 172.21.15.28:0/4058283791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c161b50 0x7f308c161fc0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 --2- 172.21.15.28:0/4058283791 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f308c0cf780 0x7f308c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 --2- 172.21.15.28:0/4058283791 >> v2:172.21.15.159:3300/0 conn(0x7f308c15de70 0x7f308c0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 -- 172.21.15.28:0/4058283791 >> 172.21.15.28:0/4058283791 conn(0x7f308c0d3fb0 msgr2=0x7f308c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 -- 172.21.15.28:0/4058283791 shutdown_connections 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.062+0000 7f3091cd4640 1 -- 172.21.15.28:0/4058283791 wait complete. 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.063+0000 7f3091cd4640 1 Processor -- start 2024-09-10T20:44:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f3091cd4640 1 -- start start 2024-09-10T20:44:48.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f3091cd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c0cf780 0x7f308c161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f3091cd4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f308c15de70 0x7f308c15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f3091cd4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f308c161b50 0x7f308c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f3091cd4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f308c0b6e00 con 0x7f308c0cf780 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f3091cd4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f308c15e460 con 0x7f308c161b50 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f3091cd4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f308c0b6f80 con 0x7f308c15de70 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f308b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c0cf780 0x7f308c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f308b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c0cf780 0x7f308c161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40542/0 (socket says 172.21.15.28:40542) 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.064+0000 7f308b7fe640 1 -- 172.21.15.28:0/264905118 learned_addr learned my addr 172.21.15.28:0/264905118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308bfff640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f308c161b50 0x7f308c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308affd640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f308c15de70 0x7f308c15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308b7fe640 1 -- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f308c15de70 msgr2=0x7f308c15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308b7fe640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f308c15de70 0x7f308c15ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308b7fe640 1 -- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f308c161b50 msgr2=0x7f308c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308b7fe640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f308c161b50 0x7f308c0a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308b7fe640 1 -- 172.21.15.28:0/264905118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c1f4510 con 0x7f308c0cf780 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308affd640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f308c15de70 0x7f308c15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f308b7fe640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c0cf780 0x7f308c161180 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f307c00c8c0 tx=0x7f307c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f3073fff640 1 -- 172.21.15.28:0/264905118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c004220 con 0x7f308c0cf780 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f3073fff640 1 -- 172.21.15.28:0/264905118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f307c0079b0 con 0x7f308c0cf780 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f308c1f47b0 con 0x7f308c0cf780 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f3073fff640 1 -- 172.21.15.28:0/264905118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c007de0 con 0x7f308c0cf780 2024-09-10T20:44:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.065+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f308c15ba50 con 0x7f308c0cf780 2024-09-10T20:44:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.066+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f308c1f4ba0 con 0x7f308c0cf780 2024-09-10T20:44:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.067+0000 7f3073fff640 1 -- 172.21.15.28:0/264905118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f307c005020 con 0x7f308c0cf780 2024-09-10T20:44:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.067+0000 7f3073fff640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3078078630 0x7f307807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.067+0000 7f3073fff640 1 -- 172.21.15.28:0/264905118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7f307c09ac60 con 0x7f308c0cf780 2024-09-10T20:44:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.067+0000 7f308affd640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3078078630 0x7f307807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.068+0000 7f308affd640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3078078630 0x7f307807aaf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f308c1f1ed0 tx=0x7f308000a430 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:48.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.072+0000 7f3073fff640 1 -- 172.21.15.28:0/264905118 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f307c014030 con 0x7f308c0cf780 2024-09-10T20:44:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.256+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f308c0b9f30 con 0x7f308c0cf780 2024-09-10T20:44:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.257+0000 7f3073fff640 1 -- 172.21.15.28:0/264905118 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+13088 (secure 0 0 0) 0x7f307c062910 con 0x7f308c0cf780 2024-09-10T20:44:48.261 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:48.261 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:46.934170+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6801/3268698079":"2024-09-11T20:44:43.923529+0000","172.21.15.159:6800/3268698079":"2024-09-11T20:44:43.923529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3078078630 msgr2=0x7f307807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3078078630 0x7f307807aaf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f308c1f1ed0 tx=0x7f308000a430 comp rx=0 tx=0).stop 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c0cf780 msgr2=0x7f308c161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c0cf780 0x7f308c161180 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f307c00c8c0 tx=0x7f307c00cd80 comp rx=0 tx=0).stop 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 shutdown_connections 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3078078630 0x7f307807aaf0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f308c161b50 0x7f308c0a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f308c15de70 0x7f308c15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 --2- 172.21.15.28:0/264905118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f308c0cf780 0x7f308c161180 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 >> 172.21.15.28:0/264905118 conn(0x7f308c0d3fb0 msgr2=0x7f308c0cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 shutdown_connections 2024-09-10T20:44:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.260+0000 7f3091cd4640 1 -- 172.21.15.28:0/264905118 wait complete. 2024-09-10T20:44:48.275 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:44:48.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:44:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 -- 172.21.15.28:0/1640621787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082415f580 msgr2=0x7f08240ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 --2- 172.21.15.28:0/1640621787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082415f580 0x7f08240ce750 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0824069d50 tx=0x7f082002a560 comp rx=0 tx=0).stop 2024-09-10T20:44:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 -- 172.21.15.28:0/1640621787 shutdown_connections 2024-09-10T20:44:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 --2- 172.21.15.28:0/1640621787 >> v2:172.21.15.159:3300/0 conn(0x7f08240b4d60 0x7f08240b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 --2- 172.21.15.28:0/1640621787 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08240b3110 0x7f08240cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 --2- 172.21.15.28:0/1640621787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082415f580 0x7f08240ce750 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 -- 172.21.15.28:0/1640621787 >> 172.21.15.28:0/1640621787 conn(0x7f08240d1f30 msgr2=0x7f08240d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 -- 172.21.15.28:0/1640621787 shutdown_connections 2024-09-10T20:44:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.381+0000 7f082ab6b640 1 -- 172.21.15.28:0/1640621787 wait complete. 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.382+0000 7f082ab6b640 1 Processor -- start 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.382+0000 7f082ab6b640 1 -- start start 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f082ab6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240b3110 0x7f08241f5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f082ab6b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08240b4d60 0x7f08240c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f082ab6b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f082415f580 0x7f08240c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f082ab6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08240d1030 con 0x7f08240b3110 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f082ab6b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f08240d0eb0 con 0x7f08240b4d60 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f082ab6b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f08240b56d0 con 0x7f082415f580 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240b3110 0x7f08241f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240b3110 0x7f08241f5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40566/0 (socket says 172.21.15.28:40566) 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 -- 172.21.15.28:0/3802192707 learned_addr learned my addr 172.21.15.28:0/3802192707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f081bfff640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08240b4d60 0x7f08240c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0829108640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f082415f580 0x7f08240c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 -- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f082415f580 msgr2=0x7f08240c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f082415f580 0x7f08240c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 -- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08240b4d60 msgr2=0x7f08240c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08240b4d60 0x7f08240c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f0828907640 1 -- 172.21.15.28:0/3802192707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08240c84d0 con 0x7f08240b3110 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.383+0000 7f081bfff640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08240b4d60 0x7f08240c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.384+0000 7f0828907640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240b3110 0x7f08241f5400 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0820009850 tx=0x7f08200039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.384+0000 7f08197fa640 1 -- 172.21.15.28:0/3802192707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082003e030 con 0x7f08240b3110 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.384+0000 7f08197fa640 1 -- 172.21.15.28:0/3802192707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08200047a0 con 0x7f08240b3110 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.384+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08240c87b0 con 0x7f08240b3110 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.384+0000 7f08197fa640 1 -- 172.21.15.28:0/3802192707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0820035040 con 0x7f08240b3110 2024-09-10T20:44:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.384+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082406e000 con 0x7f08240b3110 2024-09-10T20:44:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.385+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f082406e200 con 0x7f08240b3110 2024-09-10T20:44:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.385+0000 7f08197fa640 1 -- 172.21.15.28:0/3802192707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f0820004bf0 con 0x7f08240b3110 2024-09-10T20:44:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.386+0000 7f08197fa640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0810078630 0x7f081007aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.386+0000 7f081bfff640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0810078630 0x7f081007aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.386+0000 7f08197fa640 1 -- 172.21.15.28:0/3802192707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7f08200bb100 con 0x7f08240b3110 2024-09-10T20:44:48.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.387+0000 7f081bfff640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0810078630 0x7f081007aaf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f08240c4f00 tx=0x7f0814008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.390+0000 7f08197fa640 1 -- 172.21.15.28:0/3802192707 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f082406e200 con 0x7f08240b3110 2024-09-10T20:44:48.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.576+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f08240b5c80 con 0x7f08240b3110 2024-09-10T20:44:48.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.577+0000 7f08197fa640 1 -- 172.21.15.28:0/3802192707 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) ==== 93+0+10 (secure 0 0 0) 0x7f0820082df0 con 0x7f08240b3110 2024-09-10T20:44:48.581 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:44:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0810078630 msgr2=0x7f081007aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0810078630 0x7f081007aaf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f08240c4f00 tx=0x7f0814008040 comp rx=0 tx=0).stop 2024-09-10T20:44:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240b3110 msgr2=0x7f08241f5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240b3110 0x7f08241f5400 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0820009850 tx=0x7f08200039e0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 shutdown_connections 2024-09-10T20:44:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0810078630 0x7f081007aaf0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f082415f580 0x7f08240c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f08240b4d60 0x7f08240c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 --2- 172.21.15.28:0/3802192707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240b3110 0x7f08241f5400 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 >> 172.21.15.28:0/3802192707 conn(0x7f08240d1f30 msgr2=0x7f082415d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 shutdown_connections 2024-09-10T20:44:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.579+0000 7f082ab6b640 1 -- 172.21.15.28:0/3802192707 wait complete. 2024-09-10T20:44:48.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1' 2024-09-10T20:44:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 -- 172.21.15.28:0/3023815281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd540ce5f0 msgr2=0x7fcd540ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/3023815281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd540ce5f0 0x7fcd540ceaa0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fcd54073f50 tx=0x7fcd4c02aa30 comp rx=0 tx=0).stop 2024-09-10T20:44:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 -- 172.21.15.28:0/3023815281 shutdown_connections 2024-09-10T20:44:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/3023815281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd540ce5f0 0x7fcd540ceaa0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/3023815281 >> v2:172.21.15.159:3300/0 conn(0x7fcd540cfec0 0x7fcd540d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/3023815281 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd5415bf30 0x7fcd540d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 -- 172.21.15.28:0/3023815281 >> 172.21.15.28:0/3023815281 conn(0x7fcd540d1e40 msgr2=0x7fcd540d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 -- 172.21.15.28:0/3023815281 shutdown_connections 2024-09-10T20:44:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.696+0000 7fcd5c12b640 1 -- 172.21.15.28:0/3023815281 wait complete. 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.697+0000 7fcd5c12b640 1 Processor -- start 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.697+0000 7fcd5c12b640 1 -- start start 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5c12b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd540ce5f0 0x7fcd5406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5c12b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540cfec0 0x7fcd5409c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5c12b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5415bf30 0x7fcd5406f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5c12b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540d0cc0 con 0x7fcd5415bf30 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5c12b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540d0b40 con 0x7fcd540cfec0 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5c12b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcd540cefe0 con 0x7fcd540ce5f0 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5415bf30 0x7fcd5406f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5415bf30 0x7fcd5406f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40576/0 (socket says 172.21.15.28:40576) 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 -- 172.21.15.28:0/219508733 learned_addr learned my addr 172.21.15.28:0/219508733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd59ec7640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd540ce5f0 0x7fcd5406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd596c6640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540cfec0 0x7fcd5409c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 -- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd540ce5f0 msgr2=0x7fcd5406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd540ce5f0 0x7fcd5406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 -- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540cfec0 msgr2=0x7fcd5409c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540cfec0 0x7fcd5409c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 -- 172.21.15.28:0/219508733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd540b26d0 con 0x7fcd5415bf30 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.698+0000 7fcd5a6c8640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5415bf30 0x7fcd5406f4d0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009d60 tx=0x7fcd4c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.699+0000 7fcd4a7fc640 1 -- 172.21.15.28:0/219508733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c047070 con 0x7fcd5415bf30 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.699+0000 7fcd4a7fc640 1 -- 172.21.15.28:0/219508733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd4c004250 con 0x7fcd5415bf30 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.699+0000 7fcd4a7fc640 1 -- 172.21.15.28:0/219508733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c033050 con 0x7fcd5415bf30 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.699+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd541f8060 con 0x7fcd5415bf30 2024-09-10T20:44:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.699+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd541f8620 con 0x7fcd5415bf30 2024-09-10T20:44:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.700+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd540d0300 con 0x7fcd5415bf30 2024-09-10T20:44:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.700+0000 7fcd4a7fc640 1 -- 172.21.15.28:0/219508733 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fcd4c004680 con 0x7fcd5415bf30 2024-09-10T20:44:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.701+0000 7fcd4a7fc640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcd38078630 0x7fcd3807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.701+0000 7fcd4a7fc640 1 -- 172.21.15.28:0/219508733 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7fcd4c0bbe10 con 0x7fcd5415bf30 2024-09-10T20:44:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.701+0000 7fcd59ec7640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcd38078630 0x7fcd3807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.701+0000 7fcd59ec7640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcd38078630 0x7fcd3807aaf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000bad0 tx=0x7fcd5000b2e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.705+0000 7fcd4a7fc640 1 -- 172.21.15.28:0/219508733 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcd4c083a30 con 0x7fcd5415bf30 2024-09-10T20:44:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:48.962+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]} v 0) -- 0x7fcd540b61a0 con 0x7fcd5415bf30 2024-09-10T20:44:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.946+0000 7fcd4a7fc640 1 -- 172.21.15.28:0/219508733 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]}]=0 v368) ==== 152+0+0 (secure 0 0 0) 0x7fcd4c088460 con 0x7fcd5415bf30 2024-09-10T20:44:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.950+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcd38078630 msgr2=0x7fcd3807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.950+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcd38078630 0x7fcd3807aaf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000bad0 tx=0x7fcd5000b2e0 comp rx=0 tx=0).stop 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.950+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5415bf30 msgr2=0x7fcd5406f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.950+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5415bf30 0x7fcd5406f4d0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009d60 tx=0x7fcd4c003f10 comp rx=0 tx=0).stop 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 shutdown_connections 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcd38078630 0x7fcd3807aaf0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5415bf30 0x7fcd5406f4d0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcd540cfec0 0x7fcd5409c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 --2- 172.21.15.28:0/219508733 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcd540ce5f0 0x7fcd5406dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 >> 172.21.15.28:0/219508733 conn(0x7fcd540d1e40 msgr2=0x7fcd540cb910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 shutdown_connections 2024-09-10T20:44:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:49.951+0000 7fcd5c12b640 1 -- 172.21.15.28:0/219508733 wait complete. 2024-09-10T20:44:49.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:44:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 -- 172.21.15.28:0/1964663590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940d2960 msgr2=0x7fb8940ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 --2- 172.21.15.28:0/1964663590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940d2960 0x7fb8940ce850 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fb8940cc420 tx=0x7fb88802a560 comp rx=0 tx=0).stop 2024-09-10T20:44:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 -- 172.21.15.28:0/1964663590 shutdown_connections 2024-09-10T20:44:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 --2- 172.21.15.28:0/1964663590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940d2960 0x7fb8940ce850 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 --2- 172.21.15.28:0/1964663590 >> v2:172.21.15.159:3300/0 conn(0x7fb8940cd9b0 0x7fb8940ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 -- 172.21.15.28:0/1964663590 >> 172.21.15.28:0/1964663590 conn(0x7fb8940d1e40 msgr2=0x7fb8940d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 -- 172.21.15.28:0/1964663590 shutdown_connections 2024-09-10T20:44:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.076+0000 7fb898bc2640 1 -- 172.21.15.28:0/1964663590 wait complete. 2024-09-10T20:44:50.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 Processor -- start 2024-09-10T20:44:50.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 -- start start 2024-09-10T20:44:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940cd9b0 0x7fb8941545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8940d2960 0x7fb8941f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb89415c050 0x7fb8940c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8940b1930 con 0x7fb8940cd9b0 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb8940b17b0 con 0x7fb8940d2960 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb898bc2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb8940b1ab0 con 0x7fb89415c050 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.078+0000 7fb89259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940cd9b0 0x7fb8941545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940cd9b0 0x7fb8941545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40600/0 (socket says 172.21.15.28:40600) 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 -- 172.21.15.28:0/796174074 learned_addr learned my addr 172.21.15.28:0/796174074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb892d9d640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb89415c050 0x7fb8940c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 -- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb89415c050 msgr2=0x7fb8940c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb89415c050 0x7fb8940c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb891d9b640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8940d2960 0x7fb8941f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 -- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8940d2960 msgr2=0x7fb8941f67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8940d2960 0x7fb8941f67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 -- 172.21.15.28:0/796174074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8940c9450 con 0x7fb8940cd9b0 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb892d9d640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb89415c050 0x7fb8940c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb891d9b640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8940d2960 0x7fb8941f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb89259c640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940cd9b0 0x7fb8941545b0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fb88400b410 tx=0x7fb88400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb882ffd640 1 -- 172.21.15.28:0/796174074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884015020 con 0x7fb8940cd9b0 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb882ffd640 1 -- 172.21.15.28:0/796174074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb884011920 con 0x7fb8940cd9b0 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8940cb7c0 con 0x7fb8940cd9b0 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb882ffd640 1 -- 172.21.15.28:0/796174074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884011d70 con 0x7fb8940cd9b0 2024-09-10T20:44:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.079+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89409c110 con 0x7fb8940cd9b0 2024-09-10T20:44:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.080+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8940b1f60 con 0x7fb8940cd9b0 2024-09-10T20:44:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.081+0000 7fb882ffd640 1 -- 172.21.15.28:0/796174074 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fb88401a400 con 0x7fb8940cd9b0 2024-09-10T20:44:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.082+0000 7fb882ffd640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb874078630 0x7fb87407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.082+0000 7fb891d9b640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb874078630 0x7fb87407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.082+0000 7fb882ffd640 1 -- 172.21.15.28:0/796174074 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7fb88409ab70 con 0x7fb8940cd9b0 2024-09-10T20:44:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.082+0000 7fb891d9b640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb874078630 0x7fb87407aaf0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb888033000 tx=0x7fb888004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.086+0000 7fb882ffd640 1 -- 172.21.15.28:0/796174074 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb88401f020 con 0x7fb8940cd9b0 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.270+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fb8940b3320 con 0x7fb8940cd9b0 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.271+0000 7fb882ffd640 1 -- 172.21.15.28:0/796174074 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v43) ==== 59+0+1998 (secure 0 0 0) 0x7fb884062cd0 con 0x7fb8940cd9b0 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:44:50.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:44:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:44:50.277 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:44:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb874078630 msgr2=0x7fb87407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb874078630 0x7fb87407aaf0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb888033000 tx=0x7fb888004730 comp rx=0 tx=0).stop 2024-09-10T20:44:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940cd9b0 msgr2=0x7fb8941545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940cd9b0 0x7fb8941545b0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fb88400b410 tx=0x7fb88400b8d0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 shutdown_connections 2024-09-10T20:44:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb874078630 0x7fb87407aaf0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb89415c050 0x7fb8940c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8940d2960 0x7fb8941f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 --2- 172.21.15.28:0/796174074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940cd9b0 0x7fb8941545b0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 >> 172.21.15.28:0/796174074 conn(0x7fb8940d1e40 msgr2=0x7fb8940af110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.273+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 shutdown_connections 2024-09-10T20:44:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.274+0000 7fb898bc2640 1 -- 172.21.15.28:0/796174074 wait complete. 2024-09-10T20:44:50.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:44:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.393+0000 7f4cd40f7640 1 -- 172.21.15.28:0/1382620655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb80052e0 msgr2=0x7f4cb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.393+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/1382620655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb80052e0 0x7f4cb800b9c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4001d90 tx=0x7f4cb8005830 comp rx=0 tx=0).stop 2024-09-10T20:44:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.393+0000 7f4cd40f7640 1 -- 172.21.15.28:0/1382620655 shutdown_connections 2024-09-10T20:44:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.393+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/1382620655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb80052e0 0x7f4cb800b9c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.393+0000 7f4cd40f7640 1 -- 172.21.15.28:0/1382620655 >> 172.21.15.28:0/1382620655 conn(0x7f4ccc06d110 msgr2=0x7f4ccc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.393+0000 7f4cd40f7640 1 -- 172.21.15.28:0/1382620655 shutdown_connections 2024-09-10T20:44:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.393+0000 7f4cd40f7640 1 -- 172.21.15.28:0/1382620655 wait complete. 2024-09-10T20:44:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 Processor -- start 2024-09-10T20:44:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 -- start start 2024-09-10T20:44:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb80052e0 0x7f4ccc073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc1f48b0 0x7f4ccc150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc150c60 0x7f4ccc15e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc0b1f80 con 0x7f4ccc150c60 2024-09-10T20:44:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc0cf630 con 0x7f4cb80052e0 2024-09-10T20:44:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.395+0000 7f4cd40f7640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4ccc0b2100 con 0x7f4ccc1f48b0 2024-09-10T20:44:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc150c60 0x7f4ccc15e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc150c60 0x7f4ccc15e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38272/0 (socket says 172.21.15.28:38272) 2024-09-10T20:44:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 -- 172.21.15.28:0/489555220 learned_addr learned my addr 172.21.15.28:0/489555220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd1692640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc1f48b0 0x7f4ccc150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd1e93640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb80052e0 0x7f4ccc073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 -- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc1f48b0 msgr2=0x7f4ccc150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc1f48b0 0x7f4ccc150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 -- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb80052e0 msgr2=0x7f4ccc073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb80052e0 0x7f4ccc073740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 -- 172.21.15.28:0/489555220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc159fd0 con 0x7f4ccc150c60 2024-09-10T20:44:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd1e93640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb80052e0 0x7f4ccc073740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd2694640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc150c60 0x7f4ccc15e0f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4001eb0 tx=0x7f4cc4006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cc27fc640 1 -- 172.21.15.28:0/489555220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4003d80 con 0x7f4ccc150c60 2024-09-10T20:44:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cc27fc640 1 -- 172.21.15.28:0/489555220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cc4005a10 con 0x7f4ccc150c60 2024-09-10T20:44:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ccc15a2b0 con 0x7f4ccc150c60 2024-09-10T20:44:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cc27fc640 1 -- 172.21.15.28:0/489555220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4005e60 con 0x7f4ccc150c60 2024-09-10T20:44:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.396+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ccc15a870 con 0x7f4ccc150c60 2024-09-10T20:44:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.397+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ccc14fcf0 con 0x7f4ccc150c60 2024-09-10T20:44:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.398+0000 7f4cc27fc640 1 -- 172.21.15.28:0/489555220 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f4cc4006000 con 0x7f4ccc150c60 2024-09-10T20:44:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.398+0000 7f4cc27fc640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cb0078630 0x7f4cb007aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.399+0000 7f4cd1e93640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cb0078630 0x7f4cb007aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.399+0000 7f4cc27fc640 1 -- 172.21.15.28:0/489555220 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7f4cc40ba330 con 0x7f4ccc150c60 2024-09-10T20:44:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.399+0000 7f4cd1e93640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cb0078630 0x7f4cb007aaf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc074830 tx=0x7f4cc8005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.403+0000 7f4cc27fc640 1 -- 172.21.15.28:0/489555220 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4ccc14fcf0 con 0x7f4ccc150c60 2024-09-10T20:44:50.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.588+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f4ccc1566f0 con 0x7f4ccc150c60 2024-09-10T20:44:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.591+0000 7f4cc27fc640 1 -- 172.21.15.28:0/489555220 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f4ccc1566f0 con 0x7f4ccc150c60 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.593+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cb0078630 msgr2=0x7f4cb007aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.593+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cb0078630 0x7f4cb007aaf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc074830 tx=0x7f4cc8005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.593+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc150c60 msgr2=0x7f4ccc15e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.593+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc150c60 0x7f4ccc15e0f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4001eb0 tx=0x7f4cc4006ca0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 shutdown_connections 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cb0078630 0x7f4cb007aaf0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc150c60 0x7f4ccc15e0f0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ccc1f48b0 0x7f4ccc150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 --2- 172.21.15.28:0/489555220 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cb80052e0 0x7f4ccc073740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 >> 172.21.15.28:0/489555220 conn(0x7f4ccc06d110 msgr2=0x7f4ccc0b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 shutdown_connections 2024-09-10T20:44:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.594+0000 7f4cd40f7640 1 -- 172.21.15.28:0/489555220 wait complete. 2024-09-10T20:44:50.608 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:44:50.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:50.653 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:44:50.708 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:44:50.708 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:50.708 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:50.725 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:44:50.725 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:50.725 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:50.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:44:50.744 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:44:50.744 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:50.780 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:44:50.781 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:44:50.781 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:44:50.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:44:50.802 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:44:50.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:50.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.922+0000 7f98bdf2b640 1 -- 172.21.15.28:0/335307293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a0005210 msgr2=0x7f98a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.922+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/335307293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a0005210 0x7f98a000b9c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f98b000c260 tx=0x7f98a0005830 comp rx=0 tx=0).stop 2024-09-10T20:44:50.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.922+0000 7f98bdf2b640 1 -- 172.21.15.28:0/335307293 shutdown_connections 2024-09-10T20:44:50.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.922+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/335307293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a0005210 0x7f98a000b9c0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.922+0000 7f98bdf2b640 1 -- 172.21.15.28:0/335307293 >> 172.21.15.28:0/335307293 conn(0x7f98b80d3fb0 msgr2=0x7f98b80d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:50.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.922+0000 7f98bdf2b640 1 -- 172.21.15.28:0/335307293 shutdown_connections 2024-09-10T20:44:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.922+0000 7f98bdf2b640 1 -- 172.21.15.28:0/335307293 wait complete. 2024-09-10T20:44:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.923+0000 7f98bdf2b640 1 Processor -- start 2024-09-10T20:44:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.923+0000 7f98bdf2b640 1 -- start start 2024-09-10T20:44:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98bdf2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98a0005210 0x7f98b814ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98bdf2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b80b4b70 0x7f98b80a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98bdf2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98b80a0930 0x7f98b814ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98bdf2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98b80d2d10 con 0x7f98b80b4b70 2024-09-10T20:44:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98bdf2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f98b80d2b90 con 0x7f98b80a0930 2024-09-10T20:44:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98bdf2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f98b80d2e90 con 0x7f98a0005210 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b80b4b70 0x7f98b80a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b80b4b70 0x7f98b80a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38282/0 (socket says 172.21.15.28:38282) 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 -- 172.21.15.28:0/1380822433 learned_addr learned my addr 172.21.15.28:0/1380822433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b77fe640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98a0005210 0x7f98b814ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b7fff640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98b80a0930 0x7f98b814ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 -- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98a0005210 msgr2=0x7f98b814ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98a0005210 0x7f98b814ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 -- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98b80a0930 msgr2=0x7f98b814ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98b80a0930 0x7f98b814ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.924+0000 7f98b6ffd640 1 -- 172.21.15.28:0/1380822433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b80a0de0 con 0x7f98b80b4b70 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.925+0000 7f98b6ffd640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b80b4b70 0x7f98b80a03f0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00d600 tx=0x7f98ac00da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.925+0000 7f989ffff640 1 -- 172.21.15.28:0/1380822433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac014070 con 0x7f98b80b4b70 2024-09-10T20:44:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.925+0000 7f989ffff640 1 -- 172.21.15.28:0/1380822433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98ac00f970 con 0x7f98b80b4b70 2024-09-10T20:44:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.925+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b80cb680 con 0x7f98b80b4b70 2024-09-10T20:44:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.925+0000 7f989ffff640 1 -- 172.21.15.28:0/1380822433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac00fdc0 con 0x7f98b80b4b70 2024-09-10T20:44:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.925+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b80cb940 con 0x7f98b80b4b70 2024-09-10T20:44:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.926+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b80cff20 con 0x7f98b80b4b70 2024-09-10T20:44:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.926+0000 7f989ffff640 1 -- 172.21.15.28:0/1380822433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f98ac021060 con 0x7f98b80b4b70 2024-09-10T20:44:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.927+0000 7f989ffff640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f98a4078630 0x7f98a407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.927+0000 7f989ffff640 1 -- 172.21.15.28:0/1380822433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7f98ac09a740 con 0x7f98b80b4b70 2024-09-10T20:44:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.927+0000 7f98b77fe640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f98a4078630 0x7f98a407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.927+0000 7f98b77fe640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f98a4078630 0x7f98a407aaf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98a8005fd0 tx=0x7f98a8009530 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:50.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:50.931+0000 7f989ffff640 1 -- 172.21.15.28:0/1380822433 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f98ac062a40 con 0x7f98b80b4b70 2024-09-10T20:44:51.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.154+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98b80b9f30 con 0x7f98b80b4b70 2024-09-10T20:44:51.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.154+0000 7f989ffff640 1 -- 172.21.15.28:0/1380822433 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+3908 (secure 0 0 0) 0x7f98ac06b160 con 0x7f98b80b4b70 2024-09-10T20:44:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:51.159 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"btime":"2024-09-10T20:44:44:925388+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1041363523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1041363523},{"type":"v1","addr":"172.21.15.28:6834","nonce":1041363523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1561732564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1561732564},{"type":"v1","addr":"172.21.15.28:6835","nonce":1561732564}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6521,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/97213244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":97213244},{"type":"v1","addr":"172.21.15.159:6835","nonce":97213244}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/424643678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":424643678},{"type":"v1","addr":"172.21.15.159:6801","nonce":424643678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[]} 2024-09-10T20:44:51.159 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 89 2024-09-10T20:44:51.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f98a4078630 msgr2=0x7f98a407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f98a4078630 0x7f98a407aaf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98a8005fd0 tx=0x7f98a8009530 comp rx=0 tx=0).stop 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b80b4b70 msgr2=0x7f98b80a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b80b4b70 0x7f98b80a03f0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00d600 tx=0x7f98ac00da00 comp rx=0 tx=0).stop 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 shutdown_connections 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f98a4078630 0x7f98a407aaf0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f98b80a0930 0x7f98b814ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.156+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b80b4b70 0x7f98b80a03f0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.157+0000 7f98bdf2b640 1 --2- 172.21.15.28:0/1380822433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f98a0005210 0x7f98b814ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.157+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 >> 172.21.15.28:0/1380822433 conn(0x7f98b80d3fb0 msgr2=0x7f98b80b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.157+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 shutdown_connections 2024-09-10T20:44:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.157+0000 7f98bdf2b640 1 -- 172.21.15.28:0/1380822433 wait complete. 2024-09-10T20:44:51.172 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:44:51.172 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:44:51.172 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:44:51.172 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:51.172 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:44:51.172 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:44:51.173 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:44:51.173 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:51.173 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:44:51.173 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:44:51.173 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:44:51.173 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:51.173 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:44:51.174 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:44:51.174 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:44:51.174 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:44:51.181 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:44:51.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:44:51.182 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:44:51.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:44:51.186 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:44:51.186 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:44:51.187 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:44:51.187 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:44:51.224 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:44:51.225 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:51.225 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:44:51.225 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:51.228 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:44:51.229 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:51.229 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:44:51.229 DEBUG:teuthology.parallel:result is None 2024-09-10T20:44:51.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:44:51.302 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:44:51.302 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:44:51.310 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:44:51.310 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:44:51.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.336+0000 7f00d8072640 1 -- 172.21.15.28:0/2145233812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b8003a00 msgr2=0x7f00b8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:51.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.336+0000 7f00d8072640 1 --2- 172.21.15.28:0/2145233812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b8003a00 0x7f00b8003e20 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009b80 tx=0x7f00b8005830 comp rx=0 tx=0).stop 2024-09-10T20:44:51.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.336+0000 7f00d660f640 1 -- 172.21.15.28:0/2145233812 reap_dead start 2024-09-10T20:44:51.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.336+0000 7f00d8072640 1 -- 172.21.15.28:0/2145233812 shutdown_connections 2024-09-10T20:44:51.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.336+0000 7f00d8072640 1 -- 172.21.15.28:0/2145233812 >> 172.21.15.28:0/2145233812 conn(0x7f00d00d3fb0 msgr2=0x7f00d00d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:51.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.337+0000 7f00d8072640 1 -- 172.21.15.28:0/2145233812 shutdown_connections 2024-09-10T20:44:51.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.337+0000 7f00d8072640 1 -- 172.21.15.28:0/2145233812 wait complete. 2024-09-10T20:44:51.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.338+0000 7f00d8072640 1 Processor -- start 2024-09-10T20:44:51.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.338+0000 7f00d8072640 1 -- start start 2024-09-10T20:44:51.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.338+0000 7f00d8072640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b8003a00 0x7f00d0160e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d8072640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d00a03b0 0x7f00d00a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d8072640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00d009e2f0 0x7f00d009e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d8072640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00d00cff50 con 0x7f00d00a03b0 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d8072640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f00d00cfdd0 con 0x7f00b8003a00 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d8072640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f00d015e410 con 0x7f00d009e2f0 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d00a03b0 0x7f00d00a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d00a03b0 0x7f00d00a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38326/0 (socket says 172.21.15.28:38326) 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 -- 172.21.15.28:0/2504626080 learned_addr learned my addr 172.21.15.28:0/2504626080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 -- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00d009e2f0 msgr2=0x7f00d009e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d660f640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00d009e2f0 0x7f00d009e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00d009e2f0 0x7f00d009e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 -- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b8003a00 msgr2=0x7f00d0160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d5e0e640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b8003a00 0x7f00d0160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b8003a00 0x7f00d0160e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 -- 172.21.15.28:0/2504626080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d009f000 con 0x7f00d00a03b0 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d5e0e640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b8003a00 0x7f00d0160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d560d640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d00a03b0 0x7f00d00a0810 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f00c4010cc0 tx=0x7f00c400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00ce7fc640 1 -- 172.21.15.28:0/2504626080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4017070 con 0x7f00d00a03b0 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00ce7fc640 1 -- 172.21.15.28:0/2504626080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00c4015070 con 0x7f00d00a03b0 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00ce7fc640 1 -- 172.21.15.28:0/2504626080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4005020 con 0x7f00d00a03b0 2024-09-10T20:44:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d009f2e0 con 0x7f00d00a03b0 2024-09-10T20:44:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.339+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d009f8a0 con 0x7f00d00a03b0 2024-09-10T20:44:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.340+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d014fc90 con 0x7f00d00a03b0 2024-09-10T20:44:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.341+0000 7f00ce7fc640 1 -- 172.21.15.28:0/2504626080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f00c40051c0 con 0x7f00d00a03b0 2024-09-10T20:44:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.342+0000 7f00ce7fc640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00bc078630 0x7f00bc07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.342+0000 7f00d5e0e640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00bc078630 0x7f00bc07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.342+0000 7f00ce7fc640 1 -- 172.21.15.28:0/2504626080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4971+0+0 (secure 0 0 0) 0x7f00c409b360 con 0x7f00d00a03b0 2024-09-10T20:44:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.342+0000 7f00d5e0e640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00bc078630 0x7f00bc07aaf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f00c0035000 tx=0x7f00c002d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.346+0000 7f00ce7fc640 1 -- 172.21.15.28:0/2504626080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f00d014fc90 con 0x7f00d00a03b0 2024-09-10T20:44:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.544+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f00d0162340 con 0x7f00d00a03b0 2024-09-10T20:44:51.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.961+0000 7f00ce7fc640 1 -- 172.21.15.28:0/2504626080 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v132) ==== 97+0+0 (secure 0 0 0) 0x7f00c4063760 con 0x7f00d00a03b0 2024-09-10T20:44:51.965 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:44:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00bc078630 msgr2=0x7f00bc07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00bc078630 0x7f00bc07aaf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f00c0035000 tx=0x7f00c002d040 comp rx=0 tx=0).stop 2024-09-10T20:44:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d00a03b0 msgr2=0x7f00d00a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d00a03b0 0x7f00d00a0810 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f00c4010cc0 tx=0x7f00c400e720 comp rx=0 tx=0).stop 2024-09-10T20:44:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 shutdown_connections 2024-09-10T20:44:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00bc078630 0x7f00bc07aaf0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00d009e2f0 0x7f00d009e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d00a03b0 0x7f00d00a0810 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 --2- 172.21.15.28:0/2504626080 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00b8003a00 0x7f00d0160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 >> 172.21.15.28:0/2504626080 conn(0x7f00d00d3fb0 msgr2=0x7f00d00b93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 shutdown_connections 2024-09-10T20:44:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:51.964+0000 7f00d8072640 1 -- 172.21.15.28:0/2504626080 wait complete. 2024-09-10T20:44:51.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:44:52.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.086+0000 7f864ea72640 1 -- 172.21.15.28:0/3708216620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 msgr2=0x7f86480cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.086+0000 7f864ea72640 1 --2- 172.21.15.28:0/3708216620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480cf0f0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f8640009830 tx=0x7f864002a510 comp rx=0 tx=0).stop 2024-09-10T20:44:52.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.086+0000 7f864ea72640 1 -- 172.21.15.28:0/3708216620 shutdown_connections 2024-09-10T20:44:52.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.086+0000 7f864ea72640 1 --2- 172.21.15.28:0/3708216620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480cf0f0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.086+0000 7f864ea72640 1 -- 172.21.15.28:0/3708216620 >> 172.21.15.28:0/3708216620 conn(0x7f86480d0420 msgr2=0x7f86480d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.086+0000 7f864ea72640 1 -- 172.21.15.28:0/3708216620 shutdown_connections 2024-09-10T20:44:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.086+0000 7f864ea72640 1 -- 172.21.15.28:0/3708216620 wait complete. 2024-09-10T20:44:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.087+0000 7f864ea72640 1 Processor -- start 2024-09-10T20:44:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.087+0000 7f864ea72640 1 -- start start 2024-09-10T20:44:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864ea72640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86480738b0 0x7f864814a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864ea72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864ea72640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864815eef0 0x7f86480b1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864ea72640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8648073e70 con 0x7f8648151cc0 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864ea72640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8648073cf0 con 0x7f864815eef0 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864ea72640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8648073ff0 con 0x7f86480738b0 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f8647fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f8647fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38348/0 (socket says 172.21.15.28:38348) 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f8647fff640 1 -- 172.21.15.28:0/4230724995 learned_addr learned my addr 172.21.15.28:0/4230724995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864c80e640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86480738b0 0x7f864814a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.088+0000 7f864d00f640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864815eef0 0x7f86480b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f8647fff640 1 -- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86480738b0 msgr2=0x7f864814a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f8647fff640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86480738b0 0x7f864814a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f8647fff640 1 -- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864815eef0 msgr2=0x7f86480b1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f8647fff640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864815eef0 0x7f86480b1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f8647fff640 1 -- 172.21.15.28:0/4230724995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86481f7fa0 con 0x7f8648151cc0 2024-09-10T20:44:52.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f8647fff640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480b1230 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f863800cc50 tx=0x7f863800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:52.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f86457fa640 1 -- 172.21.15.28:0/4230724995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8638016020 con 0x7f8648151cc0 2024-09-10T20:44:52.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f86457fa640 1 -- 172.21.15.28:0/4230724995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8638012420 con 0x7f8648151cc0 2024-09-10T20:44:52.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86481f8280 con 0x7f8648151cc0 2024-09-10T20:44:52.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f86457fa640 1 -- 172.21.15.28:0/4230724995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863800ce60 con 0x7f8648151cc0 2024-09-10T20:44:52.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.089+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86481f8840 con 0x7f8648151cc0 2024-09-10T20:44:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.090+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86480cbb30 con 0x7f8648151cc0 2024-09-10T20:44:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.091+0000 7f86457fa640 1 -- 172.21.15.28:0/4230724995 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f863801a3b0 con 0x7f8648151cc0 2024-09-10T20:44:52.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.091+0000 7f86457fa640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8628078630 0x7f862807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.091+0000 7f86457fa640 1 -- 172.21.15.28:0/4230724995 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4885+0+0 (secure 0 0 0) 0x7f863809ad30 con 0x7f8648151cc0 2024-09-10T20:44:52.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.091+0000 7f864c80e640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8628078630 0x7f862807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.092+0000 7f864c80e640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8628078630 0x7f862807aaf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8640034000 tx=0x7f8640004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:52.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.095+0000 7f86457fa640 1 -- 172.21.15.28:0/4230724995 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8638020030 con 0x7f8648151cc0 2024-09-10T20:44:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.348+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f8648068d10 con 0x7f8648151cc0 2024-09-10T20:44:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.349+0000 7f86457fa640 1 -- 172.21.15.28:0/4230724995 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v34) ==== 80+0+5304 (secure 0 0 0) 0x7f8638062a70 con 0x7f8648151cc0 2024-09-10T20:44:52.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:52.353 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:44:52.353 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:44:52.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.353 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:44:52.354 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:44:52.355 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.356 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:52.357 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:52.357 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.357 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:44:52.358 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:44:52.358 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:44:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:44:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.361 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.372 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:44:52.372 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:44:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8628078630 msgr2=0x7f862807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8628078630 0x7f862807aaf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8640034000 tx=0x7f8640004730 comp rx=0 tx=0).stop 2024-09-10T20:44:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 msgr2=0x7f86480b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480b1230 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f863800cc50 tx=0x7f863800a430 comp rx=0 tx=0).stop 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 shutdown_connections 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8628078630 0x7f862807aaf0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f864815eef0 0x7f86480b1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8648151cc0 0x7f86480b1230 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 --2- 172.21.15.28:0/4230724995 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86480738b0 0x7f864814a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 >> 172.21.15.28:0/4230724995 conn(0x7f86480d0420 msgr2=0x7f86480cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 shutdown_connections 2024-09-10T20:44:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.351+0000 7f864ea72640 1 -- 172.21.15.28:0/4230724995 wait complete. 2024-09-10T20:44:52.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:44:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.476+0000 7f77a0e71640 1 -- 172.21.15.28:0/1678735841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 msgr2=0x7f77880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.476+0000 7f77a0e71640 1 --2- 172.21.15.28:0/1678735841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 0x7f77880051c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f779400c470 tx=0x7f7788005830 comp rx=0 tx=0).stop 2024-09-10T20:44:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.476+0000 7f779ad9d640 1 -- 172.21.15.28:0/1678735841 reap_dead start 2024-09-10T20:44:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.476+0000 7f77a0e71640 1 -- 172.21.15.28:0/1678735841 shutdown_connections 2024-09-10T20:44:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.476+0000 7f77a0e71640 1 -- 172.21.15.28:0/1678735841 >> 172.21.15.28:0/1678735841 conn(0x7f779c06cf00 msgr2=0x7f779c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.476+0000 7f77a0e71640 1 -- 172.21.15.28:0/1678735841 shutdown_connections 2024-09-10T20:44:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.476+0000 7f77a0e71640 1 -- 172.21.15.28:0/1678735841 wait complete. 2024-09-10T20:44:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 Processor -- start 2024-09-10T20:44:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 -- start start 2024-09-10T20:44:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 0x7f779c09cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f779c13a9a0 0x7f779c09b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f779c073e90 0x7f779c074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f779c15ecc0 con 0x7f7788004d40 2024-09-10T20:44:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f779c15eb40 con 0x7f779c13a9a0 2024-09-10T20:44:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.478+0000 7f77a0e71640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f779c0d0ac0 con 0x7f779c073e90 2024-09-10T20:44:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 0x7f779c09cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 0x7f779c09cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38364/0 (socket says 172.21.15.28:38364) 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 -- 172.21.15.28:0/1923336268 learned_addr learned my addr 172.21.15.28:0/1923336268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f7799d9b640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f779c13a9a0 0x7f779c09b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779ad9d640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f779c073e90 0x7f779c074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 -- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f779c073e90 msgr2=0x7f779c074310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f779c073e90 0x7f779c074310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 -- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f779c13a9a0 msgr2=0x7f779c09b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f779c13a9a0 0x7f779c09b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 -- 172.21.15.28:0/1923336268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779c09d080 con 0x7f7788004d40 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779ad9d640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f779c073e90 0x7f779c074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f779a59c640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 0x7f779c09cb40 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f778c009f90 tx=0x7f778c00e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:52.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f7792ffd640 1 -- 172.21.15.28:0/1923336268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c003b80 con 0x7f7788004d40 2024-09-10T20:44:52.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f7792ffd640 1 -- 172.21.15.28:0/1923336268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f778c003d20 con 0x7f7788004d40 2024-09-10T20:44:52.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779c1f1960 con 0x7f7788004d40 2024-09-10T20:44:52.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f7792ffd640 1 -- 172.21.15.28:0/1923336268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c0042b0 con 0x7f7788004d40 2024-09-10T20:44:52.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.479+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f779c1f1f00 con 0x7f7788004d40 2024-09-10T20:44:52.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.480+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f779c15bd50 con 0x7f7788004d40 2024-09-10T20:44:52.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.481+0000 7f7792ffd640 1 -- 172.21.15.28:0/1923336268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f778c004450 con 0x7f7788004d40 2024-09-10T20:44:52.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.481+0000 7f7792ffd640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f777c078660 0x7f777c07ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.481+0000 7f7792ffd640 1 -- 172.21.15.28:0/1923336268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4885+0+0 (secure 0 0 0) 0x7f778c09d7d0 con 0x7f7788004d40 2024-09-10T20:44:52.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.482+0000 7f7799d9b640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f777c078660 0x7f777c07ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.482+0000 7f7799d9b640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f777c078660 0x7f777c07ab20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f779c151f70 tx=0x7f7784005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:52.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.486+0000 7f7792ffd640 1 -- 172.21.15.28:0/1923336268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f779c09d080 con 0x7f7788004d40 2024-09-10T20:44:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.671+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f779c06d3b0 con 0x7f7788004d40 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.672+0000 7f7792ffd640 1 -- 172.21.15.28:0/1923336268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v134) ==== 84+0+65 (secure 0 0 0) 0x7f778c065c20 con 0x7f7788004d40 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:44:52.676 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:44:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f777c078660 msgr2=0x7f777c07ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f777c078660 0x7f777c07ab20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f779c151f70 tx=0x7f7784005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 msgr2=0x7f779c09cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 0x7f779c09cb40 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f778c009f90 tx=0x7f778c00e780 comp rx=0 tx=0).stop 2024-09-10T20:44:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 shutdown_connections 2024-09-10T20:44:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f777c078660 0x7f777c07ab20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f779c073e90 0x7f779c074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f779c13a9a0 0x7f779c09b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 --2- 172.21.15.28:0/1923336268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004d40 0x7f779c09cb40 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 >> 172.21.15.28:0/1923336268 conn(0x7f779c06cf00 msgr2=0x7f779c06ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 shutdown_connections 2024-09-10T20:44:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.674+0000 7f77a0e71640 1 -- 172.21.15.28:0/1923336268 wait complete. 2024-09-10T20:44:52.689 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:44:52.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:44:52.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.795+0000 7f69b499f640 1 -- 172.21.15.28:0/3143835970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 msgr2=0x7f6998004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.795+0000 7f69b499f640 1 --2- 172.21.15.28:0/3143835970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 0x7f6998004c80 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f69a40016c0 tx=0x7f6998005830 comp rx=0 tx=0).stop 2024-09-10T20:44:52.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.795+0000 7f69affff640 1 -- 172.21.15.28:0/3143835970 reap_dead start 2024-09-10T20:44:52.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.795+0000 7f69b499f640 1 -- 172.21.15.28:0/3143835970 shutdown_connections 2024-09-10T20:44:52.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.795+0000 7f69b499f640 1 -- 172.21.15.28:0/3143835970 >> 172.21.15.28:0/3143835970 conn(0x7f69b00d1c90 msgr2=0x7f69b00d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:52.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.795+0000 7f69b499f640 1 -- 172.21.15.28:0/3143835970 shutdown_connections 2024-09-10T20:44:52.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.795+0000 7f69b499f640 1 -- 172.21.15.28:0/3143835970 wait complete. 2024-09-10T20:44:52.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.796+0000 7f69b499f640 1 Processor -- start 2024-09-10T20:44:52.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.796+0000 7f69b499f640 1 -- start start 2024-09-10T20:44:52.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69b499f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 0x7f69b00b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69b499f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69b0147760 0x7f69b01551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69b499f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69b0146ac0 0x7f69b0146f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69b499f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69b0166db0 con 0x7f6998004800 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69b499f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f69b006e780 con 0x7f69b0147760 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69b499f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f69b0166f30 con 0x7f69b0146ac0 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 0x7f69b00b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 0x7f69b00b7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38382/0 (socket says 172.21.15.28:38382) 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69af7fe640 1 -- 172.21.15.28:0/4284964829 learned_addr learned my addr 172.21.15.28:0/4284964829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69aeffd640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69b0147760 0x7f69b01551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.797+0000 7f69affff640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69b0146ac0 0x7f69b0146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69af7fe640 1 -- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69b0146ac0 msgr2=0x7f69b0146f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69af7fe640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69b0146ac0 0x7f69b0146f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69af7fe640 1 -- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69b0147760 msgr2=0x7f69b01551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69af7fe640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69b0147760 0x7f69b01551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69af7fe640 1 -- 172.21.15.28:0/4284964829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b0147480 con 0x7f6998004800 2024-09-10T20:44:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69af7fe640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 0x7f69b00b7390 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f69a0006310 tx=0x7f69a00067d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f6997fff640 1 -- 172.21.15.28:0/4284964829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a000fae0 con 0x7f6998004800 2024-09-10T20:44:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f6997fff640 1 -- 172.21.15.28:0/4284964829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69a0016020 con 0x7f6998004800 2024-09-10T20:44:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b00cbd50 con 0x7f6998004800 2024-09-10T20:44:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f6997fff640 1 -- 172.21.15.28:0/4284964829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a0011430 con 0x7f6998004800 2024-09-10T20:44:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.798+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b00cb4b0 con 0x7f6998004800 2024-09-10T20:44:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.799+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b006dbf0 con 0x7f6998004800 2024-09-10T20:44:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.800+0000 7f6997fff640 1 -- 172.21.15.28:0/4284964829 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f69a00115d0 con 0x7f6998004800 2024-09-10T20:44:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.800+0000 7f6997fff640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f699c078630 0x7f699c07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.800+0000 7f6997fff640 1 -- 172.21.15.28:0/4284964829 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4885+0+0 (secure 0 0 0) 0x7f69a009b4e0 con 0x7f6998004800 2024-09-10T20:44:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.800+0000 7f69aeffd640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f699c078630 0x7f699c07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.801+0000 7f69aeffd640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f699c078630 0x7f699c07aaf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f69a4032000 tx=0x7f69a402c040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:52.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.804+0000 7f6997fff640 1 -- 172.21.15.28:0/4284964829 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f69a0020020 con 0x7f6998004800 2024-09-10T20:44:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:52.995+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f69b00ce560 con 0x7f6998004800 2024-09-10T20:44:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.973+0000 7f6997fff640 1 -- 172.21.15.28:0/4284964829 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v135) ==== 138+0+0 (secure 0 0 0) 0x7f69b0147480 con 0x7f6998004800 2024-09-10T20:44:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:44:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f699c078630 msgr2=0x7f699c07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f699c078630 0x7f699c07aaf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f69a4032000 tx=0x7f69a402c040 comp rx=0 tx=0).stop 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 msgr2=0x7f69b00b7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 0x7f69b00b7390 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f69a0006310 tx=0x7f69a00067d0 comp rx=0 tx=0).stop 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 shutdown_connections 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f699c078630 0x7f699c07aaf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69b0146ac0 0x7f69b0146f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f69b0147760 0x7f69b01551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 --2- 172.21.15.28:0/4284964829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6998004800 0x7f69b00b7390 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.977+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 >> 172.21.15.28:0/4284964829 conn(0x7f69b00d1c90 msgr2=0x7f69b0070980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.978+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 shutdown_connections 2024-09-10T20:44:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:53.979+0000 7f69b499f640 1 -- 172.21.15.28:0/4284964829 wait complete. 2024-09-10T20:44:54.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:44:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.116+0000 7f6ee029b640 1 -- 172.21.15.28:0/2469931826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 msgr2=0x7f6ec00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.116+0000 7f6ee029b640 1 --2- 172.21.15.28:0/2469931826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 0x7f6ec00051c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009b20 tx=0x7f6ec0005830 comp rx=0 tx=0).stop 2024-09-10T20:44:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.116+0000 7f6ede838640 1 -- 172.21.15.28:0/2469931826 reap_dead start 2024-09-10T20:44:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.116+0000 7f6ee029b640 1 -- 172.21.15.28:0/2469931826 shutdown_connections 2024-09-10T20:44:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.116+0000 7f6ee029b640 1 -- 172.21.15.28:0/2469931826 >> 172.21.15.28:0/2469931826 conn(0x7f6ed80d4040 msgr2=0x7f6ed80d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.116+0000 7f6ee029b640 1 -- 172.21.15.28:0/2469931826 shutdown_connections 2024-09-10T20:44:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.116+0000 7f6ee029b640 1 -- 172.21.15.28:0/2469931826 wait complete. 2024-09-10T20:44:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.117+0000 7f6ee029b640 1 Processor -- start 2024-09-10T20:44:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.117+0000 7f6ee029b640 1 -- start start 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ee029b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 0x7f6ed81558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ee029b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ed80b9ac0 0x7f6ed80c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ee029b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ed80b7fa0 0x7f6ed806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ee029b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed8166f50 con 0x7f6ec0004d40 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ee029b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed80b1450 con 0x7f6ed80b9ac0 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ee029b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed81670d0 con 0x7f6ed80b7fa0 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ede037640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 0x7f6ed81558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ede037640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 0x7f6ed81558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38402/0 (socket says 172.21.15.28:38402) 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ede037640 1 -- 172.21.15.28:0/3448906492 learned_addr learned my addr 172.21.15.28:0/3448906492 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6edd836640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ed80b9ac0 0x7f6ed80c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.118+0000 7f6ede838640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ed80b7fa0 0x7f6ed806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ede037640 1 -- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ed80b7fa0 msgr2=0x7f6ed806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ede037640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ed80b7fa0 0x7f6ed806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ede037640 1 -- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ed80b9ac0 msgr2=0x7f6ed80c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ede037640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ed80b9ac0 0x7f6ed80c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ede037640 1 -- 172.21.15.28:0/3448906492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed81f49c0 con 0x7f6ec0004d40 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ede037640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 0x7f6ed81558a0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800ea50 tx=0x7f6ec800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ed6ffd640 1 -- 172.21.15.28:0/3448906492 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec800cdb0 con 0x7f6ec0004d40 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ed6ffd640 1 -- 172.21.15.28:0/3448906492 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ec8016070 con 0x7f6ec0004d40 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed81f4ca0 con 0x7f6ec0004d40 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ed6ffd640 1 -- 172.21.15.28:0/3448906492 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8003ca0 con 0x7f6ec0004d40 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.119+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed81f5260 con 0x7f6ec0004d40 2024-09-10T20:44:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.120+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed80cf540 con 0x7f6ec0004d40 2024-09-10T20:44:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.121+0000 7f6ed6ffd640 1 -- 172.21.15.28:0/3448906492 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f6ec8004030 con 0x7f6ec0004d40 2024-09-10T20:44:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.122+0000 7f6ed6ffd640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6ec4078660 0x7f6ec407ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.122+0000 7f6edd836640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6ec4078660 0x7f6ec407ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.122+0000 7f6ed6ffd640 1 -- 172.21.15.28:0/3448906492 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5255+0+0 (secure 0 0 0) 0x7f6ec809b9c0 con 0x7f6ec0004d40 2024-09-10T20:44:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.122+0000 7f6edd836640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6ec4078660 0x7f6ec407ab20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80b9ec0 tx=0x7f6ecc006b50 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:54.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.126+0000 7f6ed6ffd640 1 -- 172.21.15.28:0/3448906492 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6ec8063760 con 0x7f6ec0004d40 2024-09-10T20:44:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.313+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6ed80d47f0 con 0x7f6ec0004d40 2024-09-10T20:44:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.982+0000 7f6ed6ffd640 1 -- 172.21.15.28:0/3448906492 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v136) ==== 170+0+0 (secure 0 0 0) 0x7f6ec8068190 con 0x7f6ec0004d40 2024-09-10T20:44:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6ec4078660 msgr2=0x7f6ec407ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6ec4078660 0x7f6ec407ab20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80b9ec0 tx=0x7f6ecc006b50 comp rx=0 tx=0).stop 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 msgr2=0x7f6ed81558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 0x7f6ed81558a0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800ea50 tx=0x7f6ec800ef10 comp rx=0 tx=0).stop 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 shutdown_connections 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6ec4078660 0x7f6ec407ab20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ed80b7fa0 0x7f6ed806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ed80b9ac0 0x7f6ed80c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 --2- 172.21.15.28:0/3448906492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec0004d40 0x7f6ed81558a0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 >> 172.21.15.28:0/3448906492 conn(0x7f6ed80d4040 msgr2=0x7f6ed806e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 shutdown_connections 2024-09-10T20:44:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:54.986+0000 7f6ee029b640 1 -- 172.21.15.28:0/3448906492 wait complete. 2024-09-10T20:44:55.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:44:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 -- 172.21.15.28:0/2135540770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc1560c0 msgr2=0x7f68cc06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:55.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 --2- 172.21.15.28:0/2135540770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc1560c0 0x7f68cc06ef70 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f68bc005fd0 tx=0x7f68bc02a530 comp rx=0 tx=0).stop 2024-09-10T20:44:55.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 -- 172.21.15.28:0/2135540770 shutdown_connections 2024-09-10T20:44:55.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 --2- 172.21.15.28:0/2135540770 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68cc0af700 0x7f68cc0d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 --2- 172.21.15.28:0/2135540770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc1560c0 0x7f68cc06ef70 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 -- 172.21.15.28:0/2135540770 >> 172.21.15.28:0/2135540770 conn(0x7f68cc06cf00 msgr2=0x7f68cc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:55.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 -- 172.21.15.28:0/2135540770 shutdown_connections 2024-09-10T20:44:55.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.129+0000 7f68d25da640 1 -- 172.21.15.28:0/2135540770 wait complete. 2024-09-10T20:44:55.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.130+0000 7f68d25da640 1 Processor -- start 2024-09-10T20:44:55.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.131+0000 7f68d25da640 1 -- start start 2024-09-10T20:44:55.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.131+0000 7f68d25da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc0af700 0x7f68cc1f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.131+0000 7f68d25da640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68cc0afb60 0x7f68cc142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.131+0000 7f68d25da640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68cc1560c0 0x7f68cc0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68d25da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc0d1130 con 0x7f68cc0af700 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68d25da640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc0d0fb0 con 0x7f68cc1560c0 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68d25da640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f68cc156610 con 0x7f68cc0afb60 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc0af700 0x7f68cc1f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc0af700 0x7f68cc1f5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38416/0 (socket says 172.21.15.28:38416) 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 -- 172.21.15.28:0/2741077254 learned_addr learned my addr 172.21.15.28:0/2741077254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:55.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cb7fe640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68cc0afb60 0x7f68cc142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68d0b77640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68cc1560c0 0x7f68cc0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 -- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68cc0afb60 msgr2=0x7f68cc142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68cc0afb60 0x7f68cc142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 -- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68cc1560c0 msgr2=0x7f68cc0b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68cc1560c0 0x7f68cc0b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 -- 172.21.15.28:0/2741077254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc1f4df0 con 0x7f68cc0af700 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68cbfff640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc0af700 0x7f68cc1f5860 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f68bc034040 tx=0x7f68bc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68c8ff9640 1 -- 172.21.15.28:0/2741077254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc03e030 con 0x7f68cc0af700 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68c8ff9640 1 -- 172.21.15.28:0/2741077254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68bc0047a0 con 0x7f68cc0af700 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68cc1523b0 con 0x7f68cc0af700 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.132+0000 7f68c8ff9640 1 -- 172.21.15.28:0/2741077254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc035040 con 0x7f68cc0af700 2024-09-10T20:44:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.133+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68cc1f5060 con 0x7f68cc0af700 2024-09-10T20:44:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.134+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68cc06a230 con 0x7f68cc0af700 2024-09-10T20:44:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.134+0000 7f68c8ff9640 1 -- 172.21.15.28:0/2741077254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f68bc004bb0 con 0x7f68cc0af700 2024-09-10T20:44:55.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.135+0000 7f68c8ff9640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68b8078630 0x7f68b807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:55.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.135+0000 7f68c8ff9640 1 -- 172.21.15.28:0/2741077254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5637+0+0 (secure 0 0 0) 0x7f68bc0bb480 con 0x7f68cc0af700 2024-09-10T20:44:55.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.135+0000 7f68cb7fe640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68b8078630 0x7f68b807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:55.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.135+0000 7f68cb7fe640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68b8078630 0x7f68b807aaf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0d01d0 tx=0x7f68c000a2d0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.139+0000 7f68c8ff9640 1 -- 172.21.15.28:0/2741077254 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f68cc06a230 con 0x7f68cc0af700 2024-09-10T20:44:55.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.364+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f68cc073990 con 0x7f68cc0af700 2024-09-10T20:44:55.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.984+0000 7f68c8ff9640 1 -- 172.21.15.28:0/2741077254 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v92) ==== 174+0+0 (secure 0 0 0) 0x7f68cc073990 con 0x7f68cc0af700 2024-09-10T20:44:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 22 and data pool 23 2024-09-10T20:44:55.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.987+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68b8078630 msgr2=0x7f68b807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:55.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.987+0000 7f68d25da640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68b8078630 0x7f68b807aaf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0d01d0 tx=0x7f68c000a2d0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.987+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc0af700 msgr2=0x7f68cc1f5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.987+0000 7f68d25da640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc0af700 0x7f68cc1f5860 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f68bc034040 tx=0x7f68bc0039e0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.988+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 shutdown_connections 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.988+0000 7f68d25da640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f68b8078630 0x7f68b807aaf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.988+0000 7f68d25da640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68cc1560c0 0x7f68cc0b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.988+0000 7f68d25da640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68cc0afb60 0x7f68cc142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.988+0000 7f68d25da640 1 --2- 172.21.15.28:0/2741077254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68cc0af700 0x7f68cc1f5860 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.988+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 >> 172.21.15.28:0/2741077254 conn(0x7f68cc06cf00 msgr2=0x7f68cc15e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:55.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.988+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 shutdown_connections 2024-09-10T20:44:55.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:55.990+0000 7f68d25da640 1 -- 172.21.15.28:0/2741077254 wait complete. 2024-09-10T20:44:56.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.135+0000 7f4f85065640 1 -- 172.21.15.28:0/3149274451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 msgr2=0x7f4f800d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.135+0000 7f4f85065640 1 --2- 172.21.15.28:0/3149274451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f800d0090 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009900 tx=0x7f4f6802a610 comp rx=0 tx=0).stop 2024-09-10T20:44:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.135+0000 7f4f85065640 1 -- 172.21.15.28:0/3149274451 shutdown_connections 2024-09-10T20:44:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.135+0000 7f4f85065640 1 --2- 172.21.15.28:0/3149274451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f800d0090 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.135+0000 7f4f85065640 1 -- 172.21.15.28:0/3149274451 >> 172.21.15.28:0/3149274451 conn(0x7f4f8006cf00 msgr2=0x7f4f8006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.135+0000 7f4f85065640 1 -- 172.21.15.28:0/3149274451 shutdown_connections 2024-09-10T20:44:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.135+0000 7f4f85065640 1 -- 172.21.15.28:0/3149274451 wait complete. 2024-09-10T20:44:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.136+0000 7f4f85065640 1 Processor -- start 2024-09-10T20:44:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.136+0000 7f4f85065640 1 -- start start 2024-09-10T20:44:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f85065640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f801558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f85065640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f800cd3d0 0x7f4f800b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f85065640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f800d05d0 0x7f4f80152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f85065640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f800b3590 con 0x7f4f800cc570 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f85065640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4f800b4b30 con 0x7f4f800d05d0 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f85065640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4f800b3710 con 0x7f4f800cd3d0 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f801558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f801558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38438/0 (socket says 172.21.15.28:38438) 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 -- 172.21.15.28:0/4212558503 learned_addr learned my addr 172.21.15.28:0/4212558503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7f59e640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f800d05d0 0x7f4f80152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 -- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f800cd3d0 msgr2=0x7f4f800b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f77fff640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f800cd3d0 0x7f4f800b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f800cd3d0 0x7f4f800b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 -- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f800d05d0 msgr2=0x7f4f80152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f800d05d0 0x7f4f80152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7ed9d640 1 -- 172.21.15.28:0/4212558503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f801546a0 con 0x7f4f800cc570 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f7f59e640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f800d05d0 0x7f4f80152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.137+0000 7f4f77fff640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f800cd3d0 0x7f4f800b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:44:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.138+0000 7f4f7ed9d640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f801558e0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000c450 tx=0x7f4f7000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.138+0000 7f4f777fe640 1 -- 172.21.15.28:0/4212558503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f700052d0 con 0x7f4f800cc570 2024-09-10T20:44:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.138+0000 7f4f777fe640 1 -- 172.21.15.28:0/4212558503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f70005470 con 0x7f4f800cc570 2024-09-10T20:44:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.138+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f801538e0 con 0x7f4f800cc570 2024-09-10T20:44:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.138+0000 7f4f777fe640 1 -- 172.21.15.28:0/4212558503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70012870 con 0x7f4f800cc570 2024-09-10T20:44:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.138+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f80153ea0 con 0x7f4f800cc570 2024-09-10T20:44:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.139+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f800b3b20 con 0x7f4f800cc570 2024-09-10T20:44:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.139+0000 7f4f777fe640 1 -- 172.21.15.28:0/4212558503 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f4f700035f0 con 0x7f4f800cc570 2024-09-10T20:44:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.140+0000 7f4f777fe640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4f60078630 0x7f4f6007aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.140+0000 7f4f777fe640 1 -- 172.21.15.28:0/4212558503 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5737+0+0 (secure 0 0 0) 0x7f4f7009a440 con 0x7f4f800cc570 2024-09-10T20:44:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.140+0000 7f4f77fff640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4f60078630 0x7f4f6007aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.140+0000 7f4f77fff640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4f60078630 0x7f4f6007aaf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4f68033000 tx=0x7f4f68004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.144+0000 7f4f777fe640 1 -- 172.21.15.28:0/4212558503 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4f800d0a00 con 0x7f4f800cc570 2024-09-10T20:44:56.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.331+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f80069a80 con 0x7f4f800cc570 2024-09-10T20:44:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.332+0000 7f4f777fe640 1 -- 172.21.15.28:0/4212558503 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) ==== 74+0+16796 (secure 0 0 0) 0x7f4f70062610 con 0x7f4f800cc570 2024-09-10T20:44:56.337 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:56.337 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":137,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:55.365854+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:53.000121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:54.318239+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.334+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4f60078630 msgr2=0x7f4f6007aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.334+0000 7f4f85065640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4f60078630 0x7f4f6007aaf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4f68033000 tx=0x7f4f68004730 comp rx=0 tx=0).stop 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.334+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 msgr2=0x7f4f801558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.334+0000 7f4f85065640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f801558e0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000c450 tx=0x7f4f7000c910 comp rx=0 tx=0).stop 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 shutdown_connections 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4f60078630 0x7f4f6007aaf0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f800d05d0 0x7f4f80152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f800cd3d0 0x7f4f800b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 --2- 172.21.15.28:0/4212558503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f800cc570 0x7f4f801558e0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 >> 172.21.15.28:0/4212558503 conn(0x7f4f8006cf00 msgr2=0x7f4f8015a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 shutdown_connections 2024-09-10T20:44:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.335+0000 7f4f85065640 1 -- 172.21.15.28:0/4212558503 wait complete. 2024-09-10T20:44:56.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 -- 172.21.15.28:0/1709454093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0b5a50 msgr2=0x7f35cc0b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 --2- 172.21.15.28:0/1709454093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0b5a50 0x7f35cc0b3c20 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005fd0 tx=0x7f35c002a530 comp rx=0 tx=0).stop 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 -- 172.21.15.28:0/1709454093 shutdown_connections 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 --2- 172.21.15.28:0/1709454093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0b5a50 0x7f35cc0b3c20 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 --2- 172.21.15.28:0/1709454093 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc0747a0 0x7f35cc15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 -- 172.21.15.28:0/1709454093 >> 172.21.15.28:0/1709454093 conn(0x7f35cc06dbf0 msgr2=0x7f35cc06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 -- 172.21.15.28:0/1709454093 shutdown_connections 2024-09-10T20:44:56.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.454+0000 7f35d33d3640 1 -- 172.21.15.28:0/1709454093 wait complete. 2024-09-10T20:44:56.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.455+0000 7f35d33d3640 1 Processor -- start 2024-09-10T20:44:56.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.455+0000 7f35d33d3640 1 -- start start 2024-09-10T20:44:56.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d33d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc06cae0 0x7f35cc1f8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d33d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0747a0 0x7f35cc06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d33d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0b5a50 0x7f35cc15af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d33d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0b06a0 con 0x7f35cc0747a0 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d33d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0b0520 con 0x7f35cc0b5a50 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d33d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc0b42d0 con 0x7f35cc06cae0 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0747a0 0x7f35cc06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0747a0 0x7f35cc06f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38458/0 (socket says 172.21.15.28:38458) 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 -- 172.21.15.28:0/2497158567 learned_addr learned my addr 172.21.15.28:0/2497158567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d116f640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc06cae0 0x7f35cc1f8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d1970640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0b5a50 0x7f35cc15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 -- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc06cae0 msgr2=0x7f35cc1f8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc06cae0 0x7f35cc1f8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 -- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0b5a50 msgr2=0x7f35cc15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0b5a50 0x7f35cc15af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 -- 172.21.15.28:0/2497158567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc06fe50 con 0x7f35cc0747a0 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35d096e640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0747a0 0x7f35cc06f550 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f35c0034040 tx=0x7f35c00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.456+0000 7f35c9ffb640 1 -- 172.21.15.28:0/2497158567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c003e030 con 0x7f35cc0747a0 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.457+0000 7f35c9ffb640 1 -- 172.21.15.28:0/2497158567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35c0003d90 con 0x7f35cc0747a0 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.457+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc070130 con 0x7f35cc0747a0 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.457+0000 7f35c9ffb640 1 -- 172.21.15.28:0/2497158567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c0035040 con 0x7f35cc0747a0 2024-09-10T20:44:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.457+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc1f8490 con 0x7f35cc0747a0 2024-09-10T20:44:56.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.458+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35cc1f8f30 con 0x7f35cc0747a0 2024-09-10T20:44:56.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.458+0000 7f35c9ffb640 1 -- 172.21.15.28:0/2497158567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f35c00049e0 con 0x7f35cc0747a0 2024-09-10T20:44:56.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.459+0000 7f35c9ffb640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b8078630 0x7f35b807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.459+0000 7f35c9ffb640 1 -- 172.21.15.28:0/2497158567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5737+0+0 (secure 0 0 0) 0x7f35c00bb660 con 0x7f35cc0747a0 2024-09-10T20:44:56.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.459+0000 7f35d116f640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b8078630 0x7f35b807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.460+0000 7f35d116f640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b8078630 0x7f35b807aaf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f35c4005fd0 tx=0x7f35c4009680 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.463+0000 7f35c9ffb640 1 -- 172.21.15.28:0/2497158567 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35c0083150 con 0x7f35cc0747a0 2024-09-10T20:44:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.648+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35cc06d2c0 con 0x7f35cc0747a0 2024-09-10T20:44:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.649+0000 7f35c9ffb640 1 -- 172.21.15.28:0/2497158567 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) ==== 74+0+16796 (secure 0 0 0) 0x7f35c0087b80 con 0x7f35cc0747a0 2024-09-10T20:44:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:56.654 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":137,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:55.365854+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:53.000121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:54.318239+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b8078630 msgr2=0x7f35b807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b8078630 0x7f35b807aaf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f35c4005fd0 tx=0x7f35c4009680 comp rx=0 tx=0).stop 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0747a0 msgr2=0x7f35cc06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0747a0 0x7f35cc06f550 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f35c0034040 tx=0x7f35c00038a0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 shutdown_connections 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f35b8078630 0x7f35b807aaf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35cc0b5a50 0x7f35cc15af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc0747a0 0x7f35cc06f550 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 --2- 172.21.15.28:0/2497158567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35cc06cae0 0x7f35cc1f8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 >> 172.21.15.28:0/2497158567 conn(0x7f35cc06dbf0 msgr2=0x7f35cc0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 shutdown_connections 2024-09-10T20:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.652+0000 7f35d33d3640 1 -- 172.21.15.28:0/2497158567 wait complete. 2024-09-10T20:44:56.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.772+0000 7ff2b3552640 1 -- 172.21.15.28:0/965051353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298004800 msgr2=0x7ff298004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.772+0000 7ff2b3552640 1 --2- 172.21.15.28:0/965051353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298004800 0x7ff298004c80 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7ff29c001830 tx=0x7ff298005830 comp rx=0 tx=0).stop 2024-09-10T20:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.773+0000 7ff2b3552640 1 -- 172.21.15.28:0/965051353 shutdown_connections 2024-09-10T20:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.773+0000 7ff2b3552640 1 --2- 172.21.15.28:0/965051353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff298004800 0x7ff298004c80 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.773+0000 7ff2b3552640 1 -- 172.21.15.28:0/965051353 >> 172.21.15.28:0/965051353 conn(0x7ff2ac0d4070 msgr2=0x7ff2ac0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.773+0000 7ff2b3552640 1 -- 172.21.15.28:0/965051353 shutdown_connections 2024-09-10T20:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.773+0000 7ff2b3552640 1 -- 172.21.15.28:0/965051353 wait complete. 2024-09-10T20:44:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.774+0000 7ff2b3552640 1 Processor -- start 2024-09-10T20:44:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.774+0000 7ff2b3552640 1 -- start start 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b3552640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff298004800 0x7ff2ac1f8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b3552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ac06dcf0 0x7ff2ac0c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b3552640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2ac0d0c60 0x7ff2ac149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b3552640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac0d2260 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b3552640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac0d20e0 con 0x7ff2ac0d0c60 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b3552640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff2ac0d23e0 con 0x7ff298004800 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ac06dcf0 0x7ff2ac0c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ac06dcf0 0x7ff2ac0c7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38474/0 (socket says 172.21.15.28:38474) 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 -- 172.21.15.28:0/443797261 learned_addr learned my addr 172.21.15.28:0/443797261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b12ee640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff298004800 0x7ff2ac1f8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b1aef640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2ac0d0c60 0x7ff2ac149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 -- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff298004800 msgr2=0x7ff2ac1f8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff298004800 0x7ff2ac1f8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 -- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2ac0d0c60 msgr2=0x7ff2ac149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2ac0d0c60 0x7ff2ac149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 -- 172.21.15.28:0/443797261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ac1f9230 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2b0aed640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ac06dcf0 0x7ff2ac0c7ba0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7ff29c001830 tx=0x7ff29c006170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.775+0000 7ff2a1ffb640 1 -- 172.21.15.28:0/443797261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c003a00 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.776+0000 7ff2a1ffb640 1 -- 172.21.15.28:0/443797261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff29c003c90 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.776+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ac0c83d0 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.776+0000 7ff2a1ffb640 1 -- 172.21.15.28:0/443797261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c0336e0 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.776+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ac1f84f0 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.777+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2ac0d2940 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.777+0000 7ff2a1ffb640 1 -- 172.21.15.28:0/443797261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7ff29c033880 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.778+0000 7ff2a1ffb640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff290078660 0x7ff29007ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.778+0000 7ff2a1ffb640 1 -- 172.21.15.28:0/443797261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5737+0+0 (secure 0 0 0) 0x7ff29c0baad0 con 0x7ff2ac06dcf0 2024-09-10T20:44:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.778+0000 7ff2b12ee640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff290078660 0x7ff29007ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.778+0000 7ff2b12ee640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff290078660 0x7ff29007ab20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009bf0 tx=0x7ff2a8005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:56.782+0000 7ff2a1ffb640 1 -- 172.21.15.28:0/443797261 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff29c006260 con 0x7ff2ac06dcf0 2024-09-10T20:44:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:57.010+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff2ac0b2320 con 0x7ff2ac06dcf0 2024-09-10T20:44:57.974 INFO:tasks.ceph.mds.b.smithi159.stderr:2024-09-10T20:44:57.970+0000 7f09f8e5c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:44:58.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.006+0000 7ff2a1ffb640 1 -- 172.21.15.28:0/443797261 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v95) ==== 118+0+0 (secure 0 0 0) 0x7ff29c082a80 con 0x7ff2ac06dcf0 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff290078660 msgr2=0x7ff29007ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff290078660 0x7ff29007ab20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009bf0 tx=0x7ff2a8005f90 comp rx=0 tx=0).stop 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ac06dcf0 msgr2=0x7ff2ac0c7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ac06dcf0 0x7ff2ac0c7ba0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7ff29c001830 tx=0x7ff29c006170 comp rx=0 tx=0).stop 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 shutdown_connections 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff290078660 0x7ff29007ab20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2ac0d0c60 0x7ff2ac149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ac06dcf0 0x7ff2ac0c7ba0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 --2- 172.21.15.28:0/443797261 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff298004800 0x7ff2ac1f8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 >> 172.21.15.28:0/443797261 conn(0x7ff2ac0d4070 msgr2=0x7ff2ac160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 shutdown_connections 2024-09-10T20:44:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.009+0000 7ff2b3552640 1 -- 172.21.15.28:0/443797261 wait complete. 2024-09-10T20:44:58.027 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:44:58.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:44:58.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 -- 172.21.15.28:0/2988990304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bfe0 msgr2=0x7f54940cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:58.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 --2- 172.21.15.28:0/2988990304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bfe0 0x7f54940cf140 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f5484009830 tx=0x7f548402a480 comp rx=0 tx=0).stop 2024-09-10T20:44:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 -- 172.21.15.28:0/2988990304 shutdown_connections 2024-09-10T20:44:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 --2- 172.21.15.28:0/2988990304 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54940cecb0 0x7f54940b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 --2- 172.21.15.28:0/2988990304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f549415bfe0 0x7f54940cf140 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 -- 172.21.15.28:0/2988990304 >> 172.21.15.28:0/2988990304 conn(0x7f54940d1e60 msgr2=0x7f54940d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 -- 172.21.15.28:0/2988990304 shutdown_connections 2024-09-10T20:44:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.139+0000 7f549928c640 1 -- 172.21.15.28:0/2988990304 wait complete. 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.140+0000 7f549928c640 1 Processor -- start 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.140+0000 7f549928c640 1 -- start start 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f549928c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54940cecb0 0x7f54940c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f549928c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54940d0440 0x7f54940c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f549928c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f549415bfe0 0x7f54941f8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f549928c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54940b2b50 con 0x7f54940cecb0 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f549928c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f54940b29d0 con 0x7f54940d0440 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f549928c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f54940b2cd0 con 0x7f549415bfe0 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54940cecb0 0x7f54940c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54940cecb0 0x7f54940c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38488/0 (socket says 172.21.15.28:38488) 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 -- 172.21.15.28:0/1389223785 learned_addr learned my addr 172.21.15.28:0/1389223785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f54937fe640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f549415bfe0 0x7f54941f8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f548bfff640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54940d0440 0x7f54940c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 -- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f549415bfe0 msgr2=0x7f54941f8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f549415bfe0 0x7f54941f8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 -- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54940d0440 msgr2=0x7f54940c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54940d0440 0x7f54940c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 -- 172.21.15.28:0/1389223785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54941f93a0 con 0x7f54940cecb0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f54937fe640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f549415bfe0 0x7f54941f8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.141+0000 7f5492ffd640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54940cecb0 0x7f54940c8080 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f5484000c00 tx=0x7f5484033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.142+0000 7f548b7fe640 1 -- 172.21.15.28:0/1389223785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484031d60 con 0x7f54940cecb0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.142+0000 7f548b7fe640 1 -- 172.21.15.28:0/1389223785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5484033e10 con 0x7f54940cecb0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.142+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54940b6480 con 0x7f54940cecb0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.142+0000 7f548b7fe640 1 -- 172.21.15.28:0/1389223785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54840332b0 con 0x7f54940cecb0 2024-09-10T20:44:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.142+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54940b6720 con 0x7f54940cecb0 2024-09-10T20:44:58.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.143+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54940b2420 con 0x7f54940cecb0 2024-09-10T20:44:58.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.144+0000 7f548b7fe640 1 -- 172.21.15.28:0/1389223785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f548404b030 con 0x7f54940cecb0 2024-09-10T20:44:58.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.144+0000 7f548b7fe640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:58.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.144+0000 7f548bfff640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:58.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.144+0000 7f548b7fe640 1 -- 172.21.15.28:0/1389223785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f54840bbac0 con 0x7f54940cecb0 2024-09-10T20:44:58.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.145+0000 7f548bfff640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f549415f7c0 tx=0x7f547c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:58.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.149+0000 7f548b7fe640 1 -- 172.21.15.28:0/1389223785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f54940b2420 con 0x7f54940cecb0 2024-09-10T20:44:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:58.371+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f54940c52e0 con 0x7f54940cecb0 2024-09-10T20:44:59.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.011+0000 7f548b7fe640 1 -- 172.21.15.28:0/1389223785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v96) ==== 122+0+0 (secure 0 0 0) 0x7f54941f93a0 con 0x7f54940cecb0 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.015+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 msgr2=0x7f547407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.015+0000 7f549928c640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f549415f7c0 tx=0x7f547c009040 comp rx=0 tx=0).stop 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.015+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54940cecb0 msgr2=0x7f54940c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.015+0000 7f549928c640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54940cecb0 0x7f54940c8080 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f5484000c00 tx=0x7f5484033c20 comp rx=0 tx=0).stop 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 shutdown_connections 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5474078630 0x7f547407aaf0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f549415bfe0 0x7f54941f8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54940d0440 0x7f54940c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 --2- 172.21.15.28:0/1389223785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54940cecb0 0x7f54940c8080 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 >> 172.21.15.28:0/1389223785 conn(0x7f54940d1e60 msgr2=0x7f54941e7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 shutdown_connections 2024-09-10T20:44:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.016+0000 7f549928c640 1 -- 172.21.15.28:0/1389223785 wait complete. 2024-09-10T20:44:59.034 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:44:59.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:44:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 -- 172.21.15.28:0/3958435041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140b2a20 msgr2=0x7f13140d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 --2- 172.21.15.28:0/3958435041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140b2a20 0x7f13140d0000 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f1314069850 tx=0x7f12fc02a4a0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 -- 172.21.15.28:0/3958435041 shutdown_connections 2024-09-10T20:44:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 --2- 172.21.15.28:0/3958435041 >> v2:172.21.15.159:3300/0 conn(0x7f1314160520 0x7f13140b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 --2- 172.21.15.28:0/3958435041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140b2a20 0x7f13140d0000 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 -- 172.21.15.28:0/3958435041 >> 172.21.15.28:0/3958435041 conn(0x7f13140d1bd0 msgr2=0x7f13140d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 -- 172.21.15.28:0/3958435041 shutdown_connections 2024-09-10T20:44:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.145+0000 7f13191a1640 1 -- 172.21.15.28:0/3958435041 wait complete. 2024-09-10T20:44:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.146+0000 7f13191a1640 1 Processor -- start 2024-09-10T20:44:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f13191a1640 1 -- start start 2024-09-10T20:44:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f13191a1640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13140b2a20 0x7f13140c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f13191a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140cfc20 0x7f131406e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f13191a1640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1314160520 0x7f13140b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f13191a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1314160ad0 con 0x7f13140cfc20 2024-09-10T20:44:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f13191a1640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1314160950 con 0x7f1314160520 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f13191a1640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1314160c50 con 0x7f13140b2a20 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f130bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140cfc20 0x7f131406e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f130bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140cfc20 0x7f131406e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38510/0 (socket says 172.21.15.28:38510) 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.147+0000 7f130bfff640 1 -- 172.21.15.28:0/3168935749 learned_addr learned my addr 172.21.15.28:0/3168935749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f13189a0640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1314160520 0x7f13140b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f1313fff640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13140b2a20 0x7f13140c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f130bfff640 1 -- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13140b2a20 msgr2=0x7f13140c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f130bfff640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13140b2a20 0x7f13140c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f130bfff640 1 -- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1314160520 msgr2=0x7f13140b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f130bfff640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1314160520 0x7f13140b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f130bfff640 1 -- 172.21.15.28:0/3168935749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13140c5020 con 0x7f13140cfc20 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f130bfff640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140cfc20 0x7f131406e5f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009920 tx=0x7f12fc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f13117fa640 1 -- 172.21.15.28:0/3168935749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc040070 con 0x7f13140cfc20 2024-09-10T20:44:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f13117fa640 1 -- 172.21.15.28:0/3168935749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12fc005310 con 0x7f13140cfc20 2024-09-10T20:44:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131406ef80 con 0x7f13140cfc20 2024-09-10T20:44:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f13117fa640 1 -- 172.21.15.28:0/3168935749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0047a0 con 0x7f13140cfc20 2024-09-10T20:44:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.148+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131406f540 con 0x7f13140cfc20 2024-09-10T20:44:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.149+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13140d0000 con 0x7f13140cfc20 2024-09-10T20:44:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.150+0000 7f13117fa640 1 -- 172.21.15.28:0/3168935749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f12fc004940 con 0x7f13140cfc20 2024-09-10T20:44:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.151+0000 7f13117fa640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12f4078660 0x7f12f407ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.151+0000 7f1313fff640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12f4078660 0x7f12f407ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.151+0000 7f13117fa640 1 -- 172.21.15.28:0/3168935749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f12fc0bbef0 con 0x7f13140cfc20 2024-09-10T20:44:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.151+0000 7f1313fff640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12f4078660 0x7f12f407ab20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f13040016b0 tx=0x7f1304009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.155+0000 7f13117fa640 1 -- 172.21.15.28:0/3168935749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f12fc083e20 con 0x7f13140cfc20 2024-09-10T20:44:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.378+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13140ce0e0 con 0x7f13140cfc20 2024-09-10T20:44:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.379+0000 7f13117fa640 1 -- 172.21.15.28:0/3168935749 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5227 (secure 0 0 0) 0x7f12fc083720 con 0x7f13140cfc20 2024-09-10T20:44:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:59.383 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":96,"btime":"2024-09-10T20:44:59:007364+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4242811460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4242811460},{"type":"v1","addr":"172.21.15.28:6834","nonce":4242811460}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/914514353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":914514353},{"type":"v1","addr":"172.21.15.159:6835","nonce":914514353}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8334,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2842589826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2842589826},{"type":"v1","addr":"172.21.15.28:6835","nonce":2842589826}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:55.365818+0000","modified":"2024-09-10T20:44:58.372881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8346},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8346":{"gid":8346,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/103941194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":103941194},{"type":"v1","addr":"172.21.15.159:6801","nonce":103941194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8346,"qdb_cluster":[8346]},"id":11}]} 2024-09-10T20:44:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 96 2024-09-10T20:44:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12f4078660 msgr2=0x7f12f407ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12f4078660 0x7f12f407ab20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f13040016b0 tx=0x7f1304009040 comp rx=0 tx=0).stop 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140cfc20 msgr2=0x7f131406e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140cfc20 0x7f131406e5f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009920 tx=0x7f12fc032c10 comp rx=0 tx=0).stop 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 shutdown_connections 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12f4078660 0x7f12f407ab20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1314160520 0x7f13140b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13140cfc20 0x7f131406e5f0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 --2- 172.21.15.28:0/3168935749 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13140b2a20 0x7f13140c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.381+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 >> 172.21.15.28:0/3168935749 conn(0x7f13140d1bd0 msgr2=0x7f13140ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.382+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 shutdown_connections 2024-09-10T20:44:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.382+0000 7f13191a1640 1 -- 172.21.15.28:0/3168935749 wait complete. 2024-09-10T20:44:59.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:44:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.497+0000 7fe25e3da640 1 -- 172.21.15.28:0/3944056916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe25815bf30 msgr2=0x7fe2580d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.497+0000 7fe25e3da640 1 --2- 172.21.15.28:0/3944056916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe25815bf30 0x7fe2580d2960 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fe2580d1a20 tx=0x7fe24802a560 comp rx=0 tx=0).stop 2024-09-10T20:44:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.497+0000 7fe25e3da640 1 -- 172.21.15.28:0/3944056916 shutdown_connections 2024-09-10T20:44:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.497+0000 7fe25e3da640 1 --2- 172.21.15.28:0/3944056916 >> v2:172.21.15.159:3300/0 conn(0x7fe2580cfec0 0x7fe2580d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.497+0000 7fe25e3da640 1 --2- 172.21.15.28:0/3944056916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe25815bf30 0x7fe2580d2960 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.497+0000 7fe25e3da640 1 -- 172.21.15.28:0/3944056916 >> 172.21.15.28:0/3944056916 conn(0x7fe2580d1e40 msgr2=0x7fe2580d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.498+0000 7fe25e3da640 1 -- 172.21.15.28:0/3944056916 shutdown_connections 2024-09-10T20:44:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.498+0000 7fe25e3da640 1 -- 172.21.15.28:0/3944056916 wait complete. 2024-09-10T20:44:59.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.499+0000 7fe25e3da640 1 Processor -- start 2024-09-10T20:44:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.499+0000 7fe25e3da640 1 -- start start 2024-09-10T20:44:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe25e3da640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ce5f0 0x7fe25814a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe25e3da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2580cfec0 0x7fe25815e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe25e3da640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25815bf30 0x7fe25815ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe25e3da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2580d0cc0 con 0x7fe2580cfec0 2024-09-10T20:44:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe25e3da640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe2580d0b40 con 0x7fe25815bf30 2024-09-10T20:44:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe25e3da640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe2580b6620 con 0x7fe2580ce5f0 2024-09-10T20:44:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2580cfec0 0x7fe25815e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2580cfec0 0x7fe25815e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38520/0 (socket says 172.21.15.28:38520) 2024-09-10T20:44:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 -- 172.21.15.28:0/2702647811 learned_addr learned my addr 172.21.15.28:0/2702647811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe25c977640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25815bf30 0x7fe25815ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe257fff640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ce5f0 0x7fe25814a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 -- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ce5f0 msgr2=0x7fe25814a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ce5f0 0x7fe25814a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 -- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25815bf30 msgr2=0x7fe25815ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25815bf30 0x7fe25815ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 -- 172.21.15.28:0/2702647811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2581f6520 con 0x7fe2580cfec0 2024-09-10T20:44:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe2577fe640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2580cfec0 0x7fe25815e7c0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00b4d0 tx=0x7fe24c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe254ff9640 1 -- 172.21.15.28:0/2702647811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c014070 con 0x7fe2580cfec0 2024-09-10T20:44:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe254ff9640 1 -- 172.21.15.28:0/2702647811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe24c010960 con 0x7fe2580cfec0 2024-09-10T20:44:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.500+0000 7fe254ff9640 1 -- 172.21.15.28:0/2702647811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c010db0 con 0x7fe2580cfec0 2024-09-10T20:44:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.501+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2581f6800 con 0x7fe2580cfec0 2024-09-10T20:44:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.501+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2581532a0 con 0x7fe2580cfec0 2024-09-10T20:44:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.501+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2580d0330 con 0x7fe2580cfec0 2024-09-10T20:44:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.502+0000 7fe254ff9640 1 -- 172.21.15.28:0/2702647811 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fe24c00f3c0 con 0x7fe2580cfec0 2024-09-10T20:44:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.503+0000 7fe254ff9640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe244078630 0x7fe24407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.503+0000 7fe254ff9640 1 -- 172.21.15.28:0/2702647811 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7fe24c09ab70 con 0x7fe2580cfec0 2024-09-10T20:44:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.503+0000 7fe257fff640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe244078630 0x7fe24407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.503+0000 7fe257fff640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe244078630 0x7fe24407aaf0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe248033000 tx=0x7fe248004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.507+0000 7fe254ff9640 1 -- 172.21.15.28:0/2702647811 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe24c062b70 con 0x7fe2580cfec0 2024-09-10T20:44:59.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.692+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2580b70b0 con 0x7fe2580cfec0 2024-09-10T20:44:59.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.693+0000 7fe254ff9640 1 -- 172.21.15.28:0/2702647811 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+16783 (secure 0 0 0) 0x7fe24c062470 con 0x7fe2580cfec0 2024-09-10T20:44:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:44:59.697 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:44:56.996330+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:53.000121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:54.318239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:44:59.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe244078630 msgr2=0x7fe24407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe244078630 0x7fe24407aaf0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe248033000 tx=0x7fe248004730 comp rx=0 tx=0).stop 2024-09-10T20:44:59.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2580cfec0 msgr2=0x7fe25815e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2580cfec0 0x7fe25815e7c0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00b4d0 tx=0x7fe24c00b990 comp rx=0 tx=0).stop 2024-09-10T20:44:59.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 shutdown_connections 2024-09-10T20:44:59.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe244078630 0x7fe24407aaf0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe25815bf30 0x7fe25815ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2580cfec0 0x7fe25815e7c0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 --2- 172.21.15.28:0/2702647811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2580ce5f0 0x7fe25814a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 >> 172.21.15.28:0/2702647811 conn(0x7fe2580d1e40 msgr2=0x7fe2580c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:59.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 shutdown_connections 2024-09-10T20:44:59.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.695+0000 7fe25e3da640 1 -- 172.21.15.28:0/2702647811 wait complete. 2024-09-10T20:44:59.711 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:44:59.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:44:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.813+0000 7f972668e640 1 -- 172.21.15.28:0/4011984178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080041e0 msgr2=0x7f9708004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.813+0000 7f972668e640 1 --2- 172.21.15.28:0/4011984178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080041e0 0x7f9708004800 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f9714009b20 tx=0x7f9708005830 comp rx=0 tx=0).stop 2024-09-10T20:44:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.813+0000 7f9724c2b640 1 -- 172.21.15.28:0/4011984178 reap_dead start 2024-09-10T20:44:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.813+0000 7f972668e640 1 -- 172.21.15.28:0/4011984178 shutdown_connections 2024-09-10T20:44:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.813+0000 7f972668e640 1 -- 172.21.15.28:0/4011984178 >> 172.21.15.28:0/4011984178 conn(0x7f97200d1e10 msgr2=0x7f97200d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:44:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.813+0000 7f972668e640 1 -- 172.21.15.28:0/4011984178 shutdown_connections 2024-09-10T20:44:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.813+0000 7f972668e640 1 -- 172.21.15.28:0/4011984178 wait complete. 2024-09-10T20:44:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.814+0000 7f972668e640 1 Processor -- start 2024-09-10T20:44:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.814+0000 7f972668e640 1 -- start start 2024-09-10T20:44:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f972668e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97080041e0 0x7f97201f5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f972668e640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f972015fc50 0x7f97200cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f972668e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97201600d0 0x7f97200cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f972668e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9720162500 con 0x7f97201600d0 2024-09-10T20:44:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f972668e640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9720162380 con 0x7f972015fc50 2024-09-10T20:44:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f972668e640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9720162b60 con 0x7f97080041e0 2024-09-10T20:44:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f9724c2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97201600d0 0x7f97200cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f9724c2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97201600d0 0x7f97200cba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38544/0 (socket says 172.21.15.28:38544) 2024-09-10T20:44:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f9724c2b640 1 -- 172.21.15.28:0/416510837 learned_addr learned my addr 172.21.15.28:0/416510837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:44:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f971ffff640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97080041e0 0x7f97201f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f971f7fe640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f972015fc50 0x7f97200cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.815+0000 7f9724c2b640 1 -- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97080041e0 msgr2=0x7f97201f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f9724c2b640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97080041e0 0x7f97201f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f9724c2b640 1 -- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f972015fc50 msgr2=0x7f97200cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f9724c2b640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f972015fc50 0x7f97200cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f9724c2b640 1 -- 172.21.15.28:0/416510837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9720160610 con 0x7f97201600d0 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f9724c2b640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97201600d0 0x7f97200cba50 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f97200d1de0 tx=0x7f9718006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f971cff9640 1 -- 172.21.15.28:0/416510837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718011900 con 0x7f97201600d0 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f971cff9640 1 -- 172.21.15.28:0/416510837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9718011aa0 con 0x7f97201600d0 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97201608f0 con 0x7f97201600d0 2024-09-10T20:44:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f971cff9640 1 -- 172.21.15.28:0/416510837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718010420 con 0x7f97201600d0 2024-09-10T20:44:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.816+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9720160eb0 con 0x7f97201600d0 2024-09-10T20:44:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.817+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97200d2b60 con 0x7f97201600d0 2024-09-10T20:44:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.818+0000 7f971cff9640 1 -- 172.21.15.28:0/416510837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f97180105c0 con 0x7f97201600d0 2024-09-10T20:44:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.818+0000 7f971cff9640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f970c078660 0x7f970c07ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:44:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.818+0000 7f971cff9640 1 -- 172.21.15.28:0/416510837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f971809afa0 con 0x7f97201600d0 2024-09-10T20:44:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.818+0000 7f971ffff640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f970c078660 0x7f970c07ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:44:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.818+0000 7f971ffff640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f970c078660 0x7f970c07ab20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9720161f10 tx=0x7f9710000fa0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:44:59.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:44:59.822+0000 7f971cff9640 1 -- 172.21.15.28:0/416510837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f97200d2b60 con 0x7f97201600d0 2024-09-10T20:45:00.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.003+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97200d05f0 con 0x7f970c078660 2024-09-10T20:45:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.007+0000 7f971cff9640 1 -- 172.21.15.28:0/416510837 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240215 (secure 0 0 0) 0x7f97200d05f0 con 0x7f970c078660 2024-09-10T20:45:00.014 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:45:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f970c078660 msgr2=0x7f970c07ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f970c078660 0x7f970c07ab20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9720161f10 tx=0x7f9710000fa0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97201600d0 msgr2=0x7f97200cba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97201600d0 0x7f97200cba50 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f97200d1de0 tx=0x7f9718006430 comp rx=0 tx=0).stop 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 shutdown_connections 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f970c078660 0x7f970c07ab20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97201600d0 0x7f97200cba50 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f972015fc50 0x7f97200cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 --2- 172.21.15.28:0/416510837 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97080041e0 0x7f97201f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 >> 172.21.15.28:0/416510837 conn(0x7f97200d1e10 msgr2=0x7f97200b56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 shutdown_connections 2024-09-10T20:45:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.010+0000 7f972668e640 1 -- 172.21.15.28:0/416510837 wait complete. 2024-09-10T20:45:00.026 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-10T20:44:58.325580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":280492,"kb_used_data":14316,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754694228,"statfs":{"total":773094113280,"available":772806889472,"internally_reserved":0,"allocated":14659584,"data_stored":7104239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.351089"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982379+0000","last_change":"2024-09-10T20:44:53.982379+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982379+0000","last_undegraded":"2024-09-10T20:44:53.982379+0000","last_fullsized":"2024-09-10T20:44:53.982379+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.981620+0000","last_change":"2024-09-10T20:44:53.981620+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.981620+0000","last_undegraded":"2024-09-10T20:44:53.981620+0000","last_fullsized":"2024-09-10T20:44:53.981620+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982252+0000","last_change":"2024-09-10T20:44:53.982252+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982252+0000","last_undegraded":"2024-09-10T20:44:53.982252+0000","last_fullsized":"2024-09-10T20:44:53.982252+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.979348+0000","last_change":"2024-09-10T20:44:53.979348+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.979348+0000","last_undegraded":"2024-09-10T20:44:53.979348+0000","last_fullsized":"2024-09-10T20:44:53.979348+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982840+0000","last_change":"2024-09-10T20:44:53.982840+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982840+0000","last_undegraded":"2024-09-10T20:44:53.982840+0000","last_fullsized":"2024-09-10T20:44:53.982840+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.980296+0000","last_change":"2024-09-10T20:44:53.980296+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.980296+0000","last_undegraded":"2024-09-10T20:44:53.980296+0000","last_fullsized":"2024-09-10T20:44:53.980296+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.984653+0000","last_change":"2024-09-10T20:44:53.984653+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.984653+0000","last_undegraded":"2024-09-10T20:44:53.984653+0000","last_fullsized":"2024-09-10T20:44:53.984653+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978823+0000","last_change":"2024-09-10T20:44:53.978823+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978823+0000","last_undegraded":"2024-09-10T20:44:53.978823+0000","last_fullsized":"2024-09-10T20:44:53.978823+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982738+0000","last_change":"2024-09-10T20:44:53.982738+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982738+0000","last_undegraded":"2024-09-10T20:44:53.982738+0000","last_fullsized":"2024-09-10T20:44:53.982738+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978825+0000","last_change":"2024-09-10T20:44:53.978825+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978825+0000","last_undegraded":"2024-09-10T20:44:53.978825+0000","last_fullsized":"2024-09-10T20:44:53.978825+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.980876+0000","last_change":"2024-09-10T20:44:53.980876+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.980876+0000","last_undegraded":"2024-09-10T20:44:53.980876+0000","last_fullsized":"2024-09-10T20:44:53.980876+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.980064+0000","last_change":"2024-09-10T20:44:53.980064+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.980064+0000","last_undegraded":"2024-09-10T20:44:53.980064+0000","last_fullsized":"2024-09-10T20:44:53.980064+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982336+0000","last_change":"2024-09-10T20:44:53.982336+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982336+0000","last_undegraded":"2024-09-10T20:44:53.982336+0000","last_fullsized":"2024-09-10T20:44:53.982336+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.983642+0000","last_change":"2024-09-10T20:44:53.983642+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.983642+0000","last_undegraded":"2024-09-10T20:44:53.983642+0000","last_fullsized":"2024-09-10T20:44:53.983642+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982736+0000","last_change":"2024-09-10T20:44:53.982736+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982736+0000","last_undegraded":"2024-09-10T20:44:53.982736+0000","last_fullsized":"2024-09-10T20:44:53.982736+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.981399+0000","last_change":"2024-09-10T20:44:53.981399+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.981399+0000","last_undegraded":"2024-09-10T20:44:53.981399+0000","last_fullsized":"2024-09-10T20:44:53.981399+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.981154+0000","last_change":"2024-09-10T20:44:53.981154+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.981154+0000","last_undegraded":"2024-09-10T20:44:53.981154+0000","last_fullsized":"2024-09-10T20:44:53.981154+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982404+0000","last_change":"2024-09-10T20:44:53.982404+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982404+0000","last_undegraded":"2024-09-10T20:44:53.982404+0000","last_fullsized":"2024-09-10T20:44:53.982404+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978951+0000","last_change":"2024-09-10T20:44:53.978951+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978951+0000","last_undegraded":"2024-09-10T20:44:53.978951+0000","last_fullsized":"2024-09-10T20:44:53.978951+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978803+0000","last_change":"2024-09-10T20:44:53.978803+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978803+0000","last_undegraded":"2024-09-10T20:44:53.978803+0000","last_fullsized":"2024-09-10T20:44:53.978803+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978970+0000","last_change":"2024-09-10T20:44:53.978970+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978970+0000","last_undegraded":"2024-09-10T20:44:53.978970+0000","last_fullsized":"2024-09-10T20:44:53.978970+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.980279+0000","last_change":"2024-09-10T20:44:53.980279+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.980279+0000","last_undegraded":"2024-09-10T20:44:53.980279+0000","last_fullsized":"2024-09-10T20:44:53.980279+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978582+0000","last_change":"2024-09-10T20:44:53.978582+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978582+0000","last_undegraded":"2024-09-10T20:44:53.978582+0000","last_fullsized":"2024-09-10T20:44:53.978582+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982717+0000","last_change":"2024-09-10T20:44:53.982717+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982717+0000","last_undegraded":"2024-09-10T20:44:53.982717+0000","last_fullsized":"2024-09-10T20:44:53.982717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978707+0000","last_change":"2024-09-10T20:44:53.978707+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978707+0000","last_undegraded":"2024-09-10T20:44:53.978707+0000","last_fullsized":"2024-09-10T20:44:53.978707+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.981825+0000","last_change":"2024-09-10T20:44:53.981825+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.981825+0000","last_undegraded":"2024-09-10T20:44:53.981825+0000","last_fullsized":"2024-09-10T20:44:53.981825+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.978811+0000","last_change":"2024-09-10T20:44:53.978811+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.978811+0000","last_undegraded":"2024-09-10T20:44:53.978811+0000","last_fullsized":"2024-09-10T20:44:53.978811+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.980980+0000","last_change":"2024-09-10T20:44:53.980980+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.980980+0000","last_undegraded":"2024-09-10T20:44:53.980980+0000","last_fullsized":"2024-09-10T20:44:53.980980+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982557+0000","last_change":"2024-09-10T20:44:53.982557+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982557+0000","last_undegraded":"2024-09-10T20:44:53.982557+0000","last_fullsized":"2024-09-10T20:44:53.982557+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982579+0000","last_change":"2024-09-10T20:44:53.982579+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982579+0000","last_undegraded":"2024-09-10T20:44:53.982579+0000","last_fullsized":"2024-09-10T20:44:53.982579+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.981286+0000","last_change":"2024-09-10T20:44:53.981286+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.981286+0000","last_undegraded":"2024-09-10T20:44:53.981286+0000","last_fullsized":"2024-09-10T20:44:53.981286+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T20:44:54.974267+0000","last_change":"2024-09-10T20:44:54.974267+0000","last_active":"2024-09-10T20:44:54.974267+0000","last_peered":"2024-09-10T20:44:54.974267+0000","last_clean":"2024-09-10T20:44:54.974267+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:54.974267+0000","last_undegraded":"2024-09-10T20:44:54.974267+0000","last_fullsized":"2024-09-10T20:44:54.974267+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-10T20:44:53.982006+0000","last_change":"2024-09-10T20:44:53.982006+0000","last_active":"2024-09-10T20:44:53.972007+0000","last_peered":"2024-09-10T20:44:53.972007+0000","last_clean":"2024-09-10T20:44:53.972007+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T20:44:53.982006+0000","last_undegraded":"2024-09-10T20:44:53.982006+0000","last_fullsized":"2024-09-10T20:44:53.982006+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":348,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-10T20:44:53.983014+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:44:53.983014+0000","last_peered":"2024-09-10T20:44:53.983014+0000","last_clean":"2024-09-10T20:44:53.983014+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:44:53.983014+0000","last_undegraded":"2024-09-10T20:44:53.983014+0000","last_fullsized":"2024-09-10T20:44:53.983014+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542230,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1691648,"data_stored":744091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1691648,"data_stored":744091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542229,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33436,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338404,"statfs":{"total":96636764160,"available":96602525696,"internally_reserved":0,"allocated":2285568,"data_stored":1334459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542229,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1679360,"data_stored":738246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509527,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36992,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334848,"statfs":{"total":96636764160,"available":96598884352,"internally_reserved":0,"allocated":1679360,"data_stored":738246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509526,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1679360,"data_stored":738246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509526,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37568,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334272,"statfs":{"total":96636764160,"available":96598294528,"internally_reserved":0,"allocated":2273280,"data_stored":1328614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509526,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36996,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334844,"statfs":{"total":96636764160,"available":96598880256,"internally_reserved":0,"allocated":1679360,"data_stored":738246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:45:00.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:45:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/2290259903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64151cc0 msgr2=0x7f1d640cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/2290259903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64151cc0 0x7f1d640cf0f0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f1d60005fd0 tx=0x7f1d6002a520 comp rx=0 tx=0).stop 2024-09-10T20:45:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/2290259903 shutdown_connections 2024-09-10T20:45:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/2290259903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d6415eef0 0x7f1d640cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/2290259903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d64151cc0 0x7f1d640cf0f0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/2290259903 >> 172.21.15.28:0/2290259903 conn(0x7f1d640d0420 msgr2=0x7f1d640d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/2290259903 shutdown_connections 2024-09-10T20:45:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.167+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/2290259903 wait complete. 2024-09-10T20:45:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.168+0000 7f1d6b4c6640 1 Processor -- start 2024-09-10T20:45:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.168+0000 7f1d6b4c6640 1 -- start start 2024-09-10T20:45:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d6b4c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d640738b0 0x7f1d640b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d6b4c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d64151cc0 0x7f1d64153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d6b4c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6415eef0 0x7f1d6406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d6b4c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d64073e70 con 0x7f1d6415eef0 2024-09-10T20:45:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d6b4c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1d64073cf0 con 0x7f1d64151cc0 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d6b4c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1d64073ff0 con 0x7f1d640738b0 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6415eef0 0x7f1d6406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6415eef0 0x7f1d6406ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38570/0 (socket says 172.21.15.28:38570) 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 -- 172.21.15.28:0/57826100 learned_addr learned my addr 172.21.15.28:0/57826100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69262640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d640738b0 0x7f1d640b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d68a61640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d64151cc0 0x7f1d64153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 -- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d640738b0 msgr2=0x7f1d640b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d640738b0 0x7f1d640b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 -- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d64151cc0 msgr2=0x7f1d64153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d64151cc0 0x7f1d64153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.169+0000 7f1d69a63640 1 -- 172.21.15.28:0/57826100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d64154540 con 0x7f1d6415eef0 2024-09-10T20:45:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.170+0000 7f1d69a63640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6415eef0 0x7f1d6406ff80 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00b9a0 tx=0x7f1d5c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.170+0000 7f1d59ffb640 1 -- 172.21.15.28:0/57826100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c019070 con 0x7f1d6415eef0 2024-09-10T20:45:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.170+0000 7f1d59ffb640 1 -- 172.21.15.28:0/57826100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d5c00ccb0 con 0x7f1d6415eef0 2024-09-10T20:45:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.170+0000 7f1d59ffb640 1 -- 172.21.15.28:0/57826100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c0146a0 con 0x7f1d6415eef0 2024-09-10T20:45:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.170+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d641f29b0 con 0x7f1d6415eef0 2024-09-10T20:45:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.170+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d641f2f70 con 0x7f1d6415eef0 2024-09-10T20:45:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.171+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d640cba10 con 0x7f1d6415eef0 2024-09-10T20:45:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.172+0000 7f1d59ffb640 1 -- 172.21.15.28:0/57826100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f1d5c01d3b0 con 0x7f1d6415eef0 2024-09-10T20:45:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.172+0000 7f1d59ffb640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d48078660 0x7f1d4807ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.172+0000 7f1d59ffb640 1 -- 172.21.15.28:0/57826100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f1d5c09a860 con 0x7f1d6415eef0 2024-09-10T20:45:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.172+0000 7f1d69262640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d48078660 0x7f1d4807ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.173+0000 7f1d69262640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d48078660 0x7f1d4807ab20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d60033000 tx=0x7f1d60004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.176+0000 7f1d59ffb640 1 -- 172.21.15.28:0/57826100 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1d5c062350 con 0x7f1d6415eef0 2024-09-10T20:45:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.357+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1d6406cf40 con 0x7f1d48078660 2024-09-10T20:45:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.361+0000 7f1d59ffb640 1 -- 172.21.15.28:0/57826100 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252481 (secure 0 0 0) 0x7f1d6406cf40 con 0x7f1d48078660 2024-09-10T20:45:00.368 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:45:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d48078660 msgr2=0x7f1d4807ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d48078660 0x7f1d4807ab20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d60033000 tx=0x7f1d60004730 comp rx=0 tx=0).stop 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6415eef0 msgr2=0x7f1d6406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6415eef0 0x7f1d6406ff80 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00b9a0 tx=0x7f1d5c00be60 comp rx=0 tx=0).stop 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 shutdown_connections 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1d48078660 0x7f1d4807ab20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6415eef0 0x7f1d6406ff80 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d64151cc0 0x7f1d64153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 --2- 172.21.15.28:0/57826100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d640738b0 0x7f1d640b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 >> 172.21.15.28:0/57826100 conn(0x7f1d640d0420 msgr2=0x7f1d640cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 shutdown_connections 2024-09-10T20:45:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.364+0000 7f1d6b4c6640 1 -- 172.21.15.28:0/57826100 wait complete. 2024-09-10T20:45:00.379 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-10T20:45:00.325914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":280892,"kb_used_data":14728,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693828,"statfs":{"total":773094113280,"available":772806479872,"internally_reserved":0,"allocated":15081472,"data_stored":7293756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000991"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489784+0000","last_change":"2024-09-10T20:44:56.002922+0000","last_active":"2024-09-10T20:44:57.489784+0000","last_peered":"2024-09-10T20:44:57.489784+0000","last_clean":"2024-09-10T20:44:57.489784+0000","last_became_active":"2024-09-10T20:44:56.002573+0000","last_became_peered":"2024-09-10T20:44:56.002573+0000","last_unstale":"2024-09-10T20:44:57.489784+0000","last_undegraded":"2024-09-10T20:44:57.489784+0000","last_fullsized":"2024-09-10T20:44:57.489784+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:52.260828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490260+0000","last_change":"2024-09-10T20:44:56.003046+0000","last_active":"2024-09-10T20:44:57.490260+0000","last_peered":"2024-09-10T20:44:57.490260+0000","last_clean":"2024-09-10T20:44:57.490260+0000","last_became_active":"2024-09-10T20:44:56.002743+0000","last_became_peered":"2024-09-10T20:44:56.002743+0000","last_unstale":"2024-09-10T20:44:57.490260+0000","last_undegraded":"2024-09-10T20:44:57.490260+0000","last_fullsized":"2024-09-10T20:44:57.490260+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:43:29.501950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490382+0000","last_change":"2024-09-10T20:44:56.008625+0000","last_active":"2024-09-10T20:44:57.490382+0000","last_peered":"2024-09-10T20:44:57.490382+0000","last_clean":"2024-09-10T20:44:57.490382+0000","last_became_active":"2024-09-10T20:44:56.008377+0000","last_became_peered":"2024-09-10T20:44:56.008377+0000","last_unstale":"2024-09-10T20:44:57.490382+0000","last_undegraded":"2024-09-10T20:44:57.490382+0000","last_fullsized":"2024-09-10T20:44:57.490382+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:42:22.428060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.005243+0000","last_change":"2024-09-10T20:44:56.013864+0000","last_active":"2024-09-10T20:44:57.005243+0000","last_peered":"2024-09-10T20:44:57.005243+0000","last_clean":"2024-09-10T20:44:57.005243+0000","last_became_active":"2024-09-10T20:44:56.013613+0000","last_became_peered":"2024-09-10T20:44:56.013613+0000","last_unstale":"2024-09-10T20:44:57.005243+0000","last_undegraded":"2024-09-10T20:44:57.005243+0000","last_fullsized":"2024-09-10T20:44:57.005243+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:25:03.913304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294476+0000","last_change":"2024-09-10T20:44:55.999940+0000","last_active":"2024-09-10T20:44:57.294476+0000","last_peered":"2024-09-10T20:44:57.294476+0000","last_clean":"2024-09-10T20:44:57.294476+0000","last_became_active":"2024-09-10T20:44:55.999601+0000","last_became_peered":"2024-09-10T20:44:55.999601+0000","last_unstale":"2024-09-10T20:44:57.294476+0000","last_undegraded":"2024-09-10T20:44:57.294476+0000","last_fullsized":"2024-09-10T20:44:57.294476+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:10.403324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490284+0000","last_change":"2024-09-10T20:44:56.010212+0000","last_active":"2024-09-10T20:44:57.490284+0000","last_peered":"2024-09-10T20:44:57.490284+0000","last_clean":"2024-09-10T20:44:57.490284+0000","last_became_active":"2024-09-10T20:44:56.009945+0000","last_became_peered":"2024-09-10T20:44:56.009945+0000","last_unstale":"2024-09-10T20:44:57.490284+0000","last_undegraded":"2024-09-10T20:44:57.490284+0000","last_fullsized":"2024-09-10T20:44:57.490284+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:51:22.912534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490529+0000","last_change":"2024-09-10T20:44:56.014399+0000","last_active":"2024-09-10T20:44:57.490529+0000","last_peered":"2024-09-10T20:44:57.490529+0000","last_clean":"2024-09-10T20:44:57.490529+0000","last_became_active":"2024-09-10T20:44:56.013373+0000","last_became_peered":"2024-09-10T20:44:56.013373+0000","last_unstale":"2024-09-10T20:44:57.490529+0000","last_undegraded":"2024-09-10T20:44:57.490529+0000","last_fullsized":"2024-09-10T20:44:57.490529+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.540751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489570+0000","last_change":"2024-09-10T20:44:55.998604+0000","last_active":"2024-09-10T20:44:57.489570+0000","last_peered":"2024-09-10T20:44:57.489570+0000","last_clean":"2024-09-10T20:44:57.489570+0000","last_became_active":"2024-09-10T20:44:55.998325+0000","last_became_peered":"2024-09-10T20:44:55.998325+0000","last_unstale":"2024-09-10T20:44:57.489570+0000","last_undegraded":"2024-09-10T20:44:57.489570+0000","last_fullsized":"2024-09-10T20:44:57.489570+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:26:31.259778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490362+0000","last_change":"2024-09-10T20:44:56.012912+0000","last_active":"2024-09-10T20:44:57.490362+0000","last_peered":"2024-09-10T20:44:57.490362+0000","last_clean":"2024-09-10T20:44:57.490362+0000","last_became_active":"2024-09-10T20:44:56.012611+0000","last_became_peered":"2024-09-10T20:44:56.012611+0000","last_unstale":"2024-09-10T20:44:57.490362+0000","last_undegraded":"2024-09-10T20:44:57.490362+0000","last_fullsized":"2024-09-10T20:44:57.490362+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.540751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490224+0000","last_change":"2024-09-10T20:44:56.008557+0000","last_active":"2024-09-10T20:44:57.490224+0000","last_peered":"2024-09-10T20:44:57.490224+0000","last_clean":"2024-09-10T20:44:57.490224+0000","last_became_active":"2024-09-10T20:44:56.008215+0000","last_became_peered":"2024-09-10T20:44:56.008215+0000","last_unstale":"2024-09-10T20:44:57.490224+0000","last_undegraded":"2024-09-10T20:44:57.490224+0000","last_fullsized":"2024-09-10T20:44:57.490224+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:04:23.804609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294750+0000","last_change":"2024-09-10T20:44:56.010585+0000","last_active":"2024-09-10T20:44:57.294750+0000","last_peered":"2024-09-10T20:44:57.294750+0000","last_clean":"2024-09-10T20:44:57.294750+0000","last_became_active":"2024-09-10T20:44:56.010023+0000","last_became_peered":"2024-09-10T20:44:56.010023+0000","last_unstale":"2024-09-10T20:44:57.294750+0000","last_undegraded":"2024-09-10T20:44:57.294750+0000","last_fullsized":"2024-09-10T20:44:57.294750+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:07.453580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.002696+0000","last_change":"2024-09-10T20:44:56.012133+0000","last_active":"2024-09-10T20:44:57.002696+0000","last_peered":"2024-09-10T20:44:57.002696+0000","last_clean":"2024-09-10T20:44:57.002696+0000","last_became_active":"2024-09-10T20:44:56.011879+0000","last_became_peered":"2024-09-10T20:44:56.011879+0000","last_unstale":"2024-09-10T20:44:57.002696+0000","last_undegraded":"2024-09-10T20:44:57.002696+0000","last_fullsized":"2024-09-10T20:44:57.002696+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:42.896434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.002848+0000","last_change":"2024-09-10T20:44:56.015288+0000","last_active":"2024-09-10T20:44:57.002848+0000","last_peered":"2024-09-10T20:44:57.002848+0000","last_clean":"2024-09-10T20:44:57.002848+0000","last_became_active":"2024-09-10T20:44:56.014764+0000","last_became_peered":"2024-09-10T20:44:56.014764+0000","last_unstale":"2024-09-10T20:44:57.002848+0000","last_undegraded":"2024-09-10T20:44:57.002848+0000","last_fullsized":"2024-09-10T20:44:57.002848+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:19.566997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489952+0000","last_change":"2024-09-10T20:44:56.016614+0000","last_active":"2024-09-10T20:44:57.489952+0000","last_peered":"2024-09-10T20:44:57.489952+0000","last_clean":"2024-09-10T20:44:57.489952+0000","last_became_active":"2024-09-10T20:44:56.016330+0000","last_became_peered":"2024-09-10T20:44:56.016330+0000","last_unstale":"2024-09-10T20:44:57.489952+0000","last_undegraded":"2024-09-10T20:44:57.489952+0000","last_fullsized":"2024-09-10T20:44:57.489952+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:28.975034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489508+0000","last_change":"2024-09-10T20:44:56.017259+0000","last_active":"2024-09-10T20:44:57.489508+0000","last_peered":"2024-09-10T20:44:57.489508+0000","last_clean":"2024-09-10T20:44:57.489508+0000","last_became_active":"2024-09-10T20:44:56.016952+0000","last_became_peered":"2024-09-10T20:44:56.016952+0000","last_unstale":"2024-09-10T20:44:57.489508+0000","last_undegraded":"2024-09-10T20:44:57.489508+0000","last_fullsized":"2024-09-10T20:44:57.489508+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:42:22.146081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489600+0000","last_change":"2024-09-10T20:44:56.002528+0000","last_active":"2024-09-10T20:44:57.489600+0000","last_peered":"2024-09-10T20:44:57.489600+0000","last_clean":"2024-09-10T20:44:57.489600+0000","last_became_active":"2024-09-10T20:44:56.002233+0000","last_became_peered":"2024-09-10T20:44:56.002233+0000","last_unstale":"2024-09-10T20:44:57.489600+0000","last_undegraded":"2024-09-10T20:44:57.489600+0000","last_fullsized":"2024-09-10T20:44:57.489600+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:24.322200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490118+0000","last_change":"2024-09-10T20:44:56.006676+0000","last_active":"2024-09-10T20:44:57.490118+0000","last_peered":"2024-09-10T20:44:57.490118+0000","last_clean":"2024-09-10T20:44:57.490118+0000","last_became_active":"2024-09-10T20:44:56.006407+0000","last_became_peered":"2024-09-10T20:44:56.006407+0000","last_unstale":"2024-09-10T20:44:57.490118+0000","last_undegraded":"2024-09-10T20:44:57.490118+0000","last_fullsized":"2024-09-10T20:44:57.490118+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:12:19.098368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004596+0000","last_change":"2024-09-10T20:44:56.009164+0000","last_active":"2024-09-10T20:44:57.004596+0000","last_peered":"2024-09-10T20:44:57.004596+0000","last_clean":"2024-09-10T20:44:57.004596+0000","last_became_active":"2024-09-10T20:44:56.008739+0000","last_became_peered":"2024-09-10T20:44:56.008739+0000","last_unstale":"2024-09-10T20:44:57.004596+0000","last_undegraded":"2024-09-10T20:44:57.004596+0000","last_fullsized":"2024-09-10T20:44:57.004596+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:13.077630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490264+0000","last_change":"2024-09-10T20:44:56.014040+0000","last_active":"2024-09-10T20:44:57.490264+0000","last_peered":"2024-09-10T20:44:57.490264+0000","last_clean":"2024-09-10T20:44:57.490264+0000","last_became_active":"2024-09-10T20:44:56.013104+0000","last_became_peered":"2024-09-10T20:44:56.013104+0000","last_unstale":"2024-09-10T20:44:57.490264+0000","last_undegraded":"2024-09-10T20:44:57.490264+0000","last_fullsized":"2024-09-10T20:44:57.490264+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:06.345433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.002712+0000","last_change":"2024-09-10T20:44:55.999545+0000","last_active":"2024-09-10T20:44:57.002712+0000","last_peered":"2024-09-10T20:44:57.002712+0000","last_clean":"2024-09-10T20:44:57.002712+0000","last_became_active":"2024-09-10T20:44:55.999187+0000","last_became_peered":"2024-09-10T20:44:55.999187+0000","last_unstale":"2024-09-10T20:44:57.002712+0000","last_undegraded":"2024-09-10T20:44:57.002712+0000","last_fullsized":"2024-09-10T20:44:57.002712+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:19.372546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490126+0000","last_change":"2024-09-10T20:44:56.005952+0000","last_active":"2024-09-10T20:44:57.490126+0000","last_peered":"2024-09-10T20:44:57.490126+0000","last_clean":"2024-09-10T20:44:57.490126+0000","last_became_active":"2024-09-10T20:44:56.005611+0000","last_became_peered":"2024-09-10T20:44:56.005611+0000","last_unstale":"2024-09-10T20:44:57.490126+0000","last_undegraded":"2024-09-10T20:44:57.490126+0000","last_fullsized":"2024-09-10T20:44:57.490126+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:42.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294552+0000","last_change":"2024-09-10T20:44:56.011692+0000","last_active":"2024-09-10T20:44:57.294552+0000","last_peered":"2024-09-10T20:44:57.294552+0000","last_clean":"2024-09-10T20:44:57.294552+0000","last_became_active":"2024-09-10T20:44:56.011373+0000","last_became_peered":"2024-09-10T20:44:56.011373+0000","last_unstale":"2024-09-10T20:44:57.294552+0000","last_undegraded":"2024-09-10T20:44:57.294552+0000","last_fullsized":"2024-09-10T20:44:57.294552+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:38.896533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.003919+0000","last_change":"2024-09-10T20:44:56.004282+0000","last_active":"2024-09-10T20:44:57.003919+0000","last_peered":"2024-09-10T20:44:57.003919+0000","last_clean":"2024-09-10T20:44:57.003919+0000","last_became_active":"2024-09-10T20:44:56.003958+0000","last_became_peered":"2024-09-10T20:44:56.003958+0000","last_unstale":"2024-09-10T20:44:57.003919+0000","last_undegraded":"2024-09-10T20:44:57.003919+0000","last_fullsized":"2024-09-10T20:44:57.003919+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:35:23.271084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294422+0000","last_change":"2024-09-10T20:44:56.004222+0000","last_active":"2024-09-10T20:44:57.294422+0000","last_peered":"2024-09-10T20:44:57.294422+0000","last_clean":"2024-09-10T20:44:57.294422+0000","last_became_active":"2024-09-10T20:44:56.003945+0000","last_became_peered":"2024-09-10T20:44:56.003945+0000","last_unstale":"2024-09-10T20:44:57.294422+0000","last_undegraded":"2024-09-10T20:44:57.294422+0000","last_fullsized":"2024-09-10T20:44:57.294422+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:22.422125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294687+0000","last_change":"2024-09-10T20:44:56.000862+0000","last_active":"2024-09-10T20:44:57.294687+0000","last_peered":"2024-09-10T20:44:57.294687+0000","last_clean":"2024-09-10T20:44:57.294687+0000","last_became_active":"2024-09-10T20:44:56.000578+0000","last_became_peered":"2024-09-10T20:44:56.000578+0000","last_unstale":"2024-09-10T20:44:57.294687+0000","last_undegraded":"2024-09-10T20:44:57.294687+0000","last_fullsized":"2024-09-10T20:44:57.294687+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:45:08.421051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004535+0000","last_change":"2024-09-10T20:44:56.010453+0000","last_active":"2024-09-10T20:44:57.004535+0000","last_peered":"2024-09-10T20:44:57.004535+0000","last_clean":"2024-09-10T20:44:57.004535+0000","last_became_active":"2024-09-10T20:44:56.010077+0000","last_became_peered":"2024-09-10T20:44:56.010077+0000","last_unstale":"2024-09-10T20:44:57.004535+0000","last_undegraded":"2024-09-10T20:44:57.004535+0000","last_fullsized":"2024-09-10T20:44:57.004535+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:56.302047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294909+0000","last_change":"2024-09-10T20:44:56.010297+0000","last_active":"2024-09-10T20:44:57.294909+0000","last_peered":"2024-09-10T20:44:57.294909+0000","last_clean":"2024-09-10T20:44:57.294909+0000","last_became_active":"2024-09-10T20:44:56.009717+0000","last_became_peered":"2024-09-10T20:44:56.009717+0000","last_unstale":"2024-09-10T20:44:57.294909+0000","last_undegraded":"2024-09-10T20:44:57.294909+0000","last_fullsized":"2024-09-10T20:44:57.294909+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:32:51.873875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489868+0000","last_change":"2024-09-10T20:44:56.012993+0000","last_active":"2024-09-10T20:44:57.489868+0000","last_peered":"2024-09-10T20:44:57.489868+0000","last_clean":"2024-09-10T20:44:57.489868+0000","last_became_active":"2024-09-10T20:44:56.012667+0000","last_became_peered":"2024-09-10T20:44:56.012667+0000","last_unstale":"2024-09-10T20:44:57.489868+0000","last_undegraded":"2024-09-10T20:44:57.489868+0000","last_fullsized":"2024-09-10T20:44:57.489868+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:28.848274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489832+0000","last_change":"2024-09-10T20:44:56.006900+0000","last_active":"2024-09-10T20:44:57.489832+0000","last_peered":"2024-09-10T20:44:57.489832+0000","last_clean":"2024-09-10T20:44:57.489832+0000","last_became_active":"2024-09-10T20:44:56.006611+0000","last_became_peered":"2024-09-10T20:44:56.006611+0000","last_unstale":"2024-09-10T20:44:57.489832+0000","last_undegraded":"2024-09-10T20:44:57.489832+0000","last_fullsized":"2024-09-10T20:44:57.489832+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:33:22.807792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489506+0000","last_change":"2024-09-10T20:44:56.016596+0000","last_active":"2024-09-10T20:44:57.489506+0000","last_peered":"2024-09-10T20:44:57.489506+0000","last_clean":"2024-09-10T20:44:57.489506+0000","last_became_active":"2024-09-10T20:44:56.015993+0000","last_became_peered":"2024-09-10T20:44:56.015993+0000","last_unstale":"2024-09-10T20:44:57.489506+0000","last_undegraded":"2024-09-10T20:44:57.489506+0000","last_fullsized":"2024-09-10T20:44:57.489506+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:17.478569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490368+0000","last_change":"2024-09-10T20:44:56.010316+0000","last_active":"2024-09-10T20:44:57.490368+0000","last_peered":"2024-09-10T20:44:57.490368+0000","last_clean":"2024-09-10T20:44:57.490368+0000","last_became_active":"2024-09-10T20:44:56.010068+0000","last_became_peered":"2024-09-10T20:44:56.010068+0000","last_unstale":"2024-09-10T20:44:57.490368+0000","last_undegraded":"2024-09-10T20:44:57.490368+0000","last_fullsized":"2024-09-10T20:44:57.490368+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:25:52.667779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489616+0000","last_change":"2024-09-10T20:44:56.008200+0000","last_active":"2024-09-10T20:44:57.489616+0000","last_peered":"2024-09-10T20:44:57.489616+0000","last_clean":"2024-09-10T20:44:57.489616+0000","last_became_active":"2024-09-10T20:44:56.007929+0000","last_became_peered":"2024-09-10T20:44:56.007929+0000","last_unstale":"2024-09-10T20:44:57.489616+0000","last_undegraded":"2024-09-10T20:44:57.489616+0000","last_fullsized":"2024-09-10T20:44:57.489616+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:16:27.105624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004596+0000","last_change":"2024-09-10T20:44:56.013071+0000","last_active":"2024-09-10T20:44:57.004596+0000","last_peered":"2024-09-10T20:44:57.004596+0000","last_clean":"2024-09-10T20:44:57.004596+0000","last_became_active":"2024-09-10T20:44:56.012815+0000","last_became_peered":"2024-09-10T20:44:56.012815+0000","last_unstale":"2024-09-10T20:44:57.004596+0000","last_undegraded":"2024-09-10T20:44:57.004596+0000","last_fullsized":"2024-09-10T20:44:57.004596+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:46.664254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1f","version":"137'4","reported_seq":19,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489859+0000","last_change":"2024-09-10T20:44:55.006177+0000","last_active":"2024-09-10T20:44:57.489859+0000","last_peered":"2024-09-10T20:44:57.489859+0000","last_clean":"2024-09-10T20:44:57.489859+0000","last_became_active":"2024-09-10T20:44:55.005197+0000","last_became_peered":"2024-09-10T20:44:55.005197+0000","last_unstale":"2024-09-10T20:44:57.489859+0000","last_undegraded":"2024-09-10T20:44:57.489859+0000","last_fullsized":"2024-09-10T20:44:57.489859+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:20:35.259433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489819+0000","last_change":"2024-09-10T20:44:56.013105+0000","last_active":"2024-09-10T20:44:57.489819+0000","last_peered":"2024-09-10T20:44:57.489819+0000","last_clean":"2024-09-10T20:44:57.489819+0000","last_became_active":"2024-09-10T20:44:56.012838+0000","last_became_peered":"2024-09-10T20:44:56.012838+0000","last_unstale":"2024-09-10T20:44:57.489819+0000","last_undegraded":"2024-09-10T20:44:57.489819+0000","last_fullsized":"2024-09-10T20:44:57.489819+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:12.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"137'1","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490102+0000","last_change":"2024-09-10T20:44:55.001784+0000","last_active":"2024-09-10T20:44:57.490102+0000","last_peered":"2024-09-10T20:44:57.490102+0000","last_clean":"2024-09-10T20:44:57.490102+0000","last_became_active":"2024-09-10T20:44:55.001377+0000","last_became_peered":"2024-09-10T20:44:55.001377+0000","last_unstale":"2024-09-10T20:44:57.490102+0000","last_undegraded":"2024-09-10T20:44:57.490102+0000","last_fullsized":"2024-09-10T20:44:57.490102+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:51.337186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489936+0000","last_change":"2024-09-10T20:44:56.005728+0000","last_active":"2024-09-10T20:44:57.489936+0000","last_peered":"2024-09-10T20:44:57.489936+0000","last_clean":"2024-09-10T20:44:57.489936+0000","last_became_active":"2024-09-10T20:44:56.005406+0000","last_became_peered":"2024-09-10T20:44:56.005406+0000","last_unstale":"2024-09-10T20:44:57.489936+0000","last_undegraded":"2024-09-10T20:44:57.489936+0000","last_fullsized":"2024-09-10T20:44:57.489936+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:18:56.287787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1d","version":"137'2","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489702+0000","last_change":"2024-09-10T20:44:55.000071+0000","last_active":"2024-09-10T20:44:57.489702+0000","last_peered":"2024-09-10T20:44:57.489702+0000","last_clean":"2024-09-10T20:44:57.489702+0000","last_became_active":"2024-09-10T20:44:54.999757+0000","last_became_peered":"2024-09-10T20:44:54.999757+0000","last_unstale":"2024-09-10T20:44:57.489702+0000","last_undegraded":"2024-09-10T20:44:57.489702+0000","last_fullsized":"2024-09-10T20:44:57.489702+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:46:40.030761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490179+0000","last_change":"2024-09-10T20:44:56.013109+0000","last_active":"2024-09-10T20:44:57.490179+0000","last_peered":"2024-09-10T20:44:57.490179+0000","last_clean":"2024-09-10T20:44:57.490179+0000","last_became_active":"2024-09-10T20:44:56.012820+0000","last_became_peered":"2024-09-10T20:44:56.012820+0000","last_unstale":"2024-09-10T20:44:57.490179+0000","last_undegraded":"2024-09-10T20:44:57.490179+0000","last_fullsized":"2024-09-10T20:44:57.490179+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:06.345433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1c","version":"137'2","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.005249+0000","last_change":"2024-09-10T20:44:55.002770+0000","last_active":"2024-09-10T20:44:57.005249+0000","last_peered":"2024-09-10T20:44:57.005249+0000","last_clean":"2024-09-10T20:44:57.005249+0000","last_became_active":"2024-09-10T20:44:55.002484+0000","last_became_peered":"2024-09-10T20:44:55.002484+0000","last_unstale":"2024-09-10T20:44:57.005249+0000","last_undegraded":"2024-09-10T20:44:57.005249+0000","last_fullsized":"2024-09-10T20:44:57.005249+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:37:46.026217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.005180+0000","last_change":"2024-09-10T20:44:56.011503+0000","last_active":"2024-09-10T20:44:57.005180+0000","last_peered":"2024-09-10T20:44:57.005180+0000","last_clean":"2024-09-10T20:44:57.005180+0000","last_became_active":"2024-09-10T20:44:56.011233+0000","last_became_peered":"2024-09-10T20:44:56.011233+0000","last_unstale":"2024-09-10T20:44:57.005180+0000","last_undegraded":"2024-09-10T20:44:57.005180+0000","last_fullsized":"2024-09-10T20:44:57.005180+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:34.156298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"137'1","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489804+0000","last_change":"2024-09-10T20:44:55.007422+0000","last_active":"2024-09-10T20:44:57.489804+0000","last_peered":"2024-09-10T20:44:57.489804+0000","last_clean":"2024-09-10T20:44:57.489804+0000","last_became_active":"2024-09-10T20:44:55.007102+0000","last_became_peered":"2024-09-10T20:44:55.007102+0000","last_unstale":"2024-09-10T20:44:57.489804+0000","last_undegraded":"2024-09-10T20:44:57.489804+0000","last_fullsized":"2024-09-10T20:44:57.489804+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:58:31.692483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490364+0000","last_change":"2024-09-10T20:44:56.007226+0000","last_active":"2024-09-10T20:44:57.490364+0000","last_peered":"2024-09-10T20:44:57.490364+0000","last_clean":"2024-09-10T20:44:57.490364+0000","last_became_active":"2024-09-10T20:44:56.006875+0000","last_became_peered":"2024-09-10T20:44:56.006875+0000","last_unstale":"2024-09-10T20:44:57.490364+0000","last_undegraded":"2024-09-10T20:44:57.490364+0000","last_fullsized":"2024-09-10T20:44:57.490364+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:42.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1a","version":"137'1","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.005160+0000","last_change":"2024-09-10T20:44:55.000100+0000","last_active":"2024-09-10T20:44:57.005160+0000","last_peered":"2024-09-10T20:44:57.005160+0000","last_clean":"2024-09-10T20:44:57.005160+0000","last_became_active":"2024-09-10T20:44:54.999779+0000","last_became_peered":"2024-09-10T20:44:54.999779+0000","last_unstale":"2024-09-10T20:44:57.005160+0000","last_undegraded":"2024-09-10T20:44:57.005160+0000","last_fullsized":"2024-09-10T20:44:57.005160+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:37:02.259063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.003583+0000","last_change":"2024-09-10T20:44:56.001509+0000","last_active":"2024-09-10T20:44:57.003583+0000","last_peered":"2024-09-10T20:44:57.003583+0000","last_clean":"2024-09-10T20:44:57.003583+0000","last_became_active":"2024-09-10T20:44:56.001246+0000","last_became_peered":"2024-09-10T20:44:56.001246+0000","last_unstale":"2024-09-10T20:44:57.003583+0000","last_undegraded":"2024-09-10T20:44:57.003583+0000","last_fullsized":"2024-09-10T20:44:57.003583+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:00.191464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489486+0000","last_change":"2024-09-10T20:44:55.005610+0000","last_active":"2024-09-10T20:44:57.489486+0000","last_peered":"2024-09-10T20:44:57.489486+0000","last_clean":"2024-09-10T20:44:57.489486+0000","last_became_active":"2024-09-10T20:44:55.005244+0000","last_became_peered":"2024-09-10T20:44:55.005244+0000","last_unstale":"2024-09-10T20:44:57.489486+0000","last_undegraded":"2024-09-10T20:44:57.489486+0000","last_fullsized":"2024-09-10T20:44:57.489486+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:34:29.085469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.003969+0000","last_change":"2024-09-10T20:44:56.012668+0000","last_active":"2024-09-10T20:44:57.003969+0000","last_peered":"2024-09-10T20:44:57.003969+0000","last_clean":"2024-09-10T20:44:57.003969+0000","last_became_active":"2024-09-10T20:44:56.012364+0000","last_became_peered":"2024-09-10T20:44:56.012364+0000","last_unstale":"2024-09-10T20:44:57.003969+0000","last_undegraded":"2024-09-10T20:44:57.003969+0000","last_fullsized":"2024-09-10T20:44:57.003969+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:25.627682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.003346+0000","last_change":"2024-09-10T20:44:54.993481+0000","last_active":"2024-09-10T20:44:57.003346+0000","last_peered":"2024-09-10T20:44:57.003346+0000","last_clean":"2024-09-10T20:44:57.003346+0000","last_became_active":"2024-09-10T20:44:54.993219+0000","last_became_peered":"2024-09-10T20:44:54.993219+0000","last_unstale":"2024-09-10T20:44:57.003346+0000","last_undegraded":"2024-09-10T20:44:57.003346+0000","last_fullsized":"2024-09-10T20:44:57.003346+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:16.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489838+0000","last_change":"2024-09-10T20:44:56.007958+0000","last_active":"2024-09-10T20:44:57.489838+0000","last_peered":"2024-09-10T20:44:57.489838+0000","last_clean":"2024-09-10T20:44:57.489838+0000","last_became_active":"2024-09-10T20:44:56.007696+0000","last_became_peered":"2024-09-10T20:44:56.007696+0000","last_unstale":"2024-09-10T20:44:57.489838+0000","last_undegraded":"2024-09-10T20:44:57.489838+0000","last_fullsized":"2024-09-10T20:44:57.489838+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:42:22.146081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490033+0000","last_change":"2024-09-10T20:44:55.004218+0000","last_active":"2024-09-10T20:44:57.490033+0000","last_peered":"2024-09-10T20:44:57.490033+0000","last_clean":"2024-09-10T20:44:57.490033+0000","last_became_active":"2024-09-10T20:44:55.003870+0000","last_became_peered":"2024-09-10T20:44:55.003870+0000","last_unstale":"2024-09-10T20:44:57.490033+0000","last_undegraded":"2024-09-10T20:44:57.490033+0000","last_fullsized":"2024-09-10T20:44:57.490033+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:35:22.840525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490096+0000","last_change":"2024-09-10T20:44:56.008695+0000","last_active":"2024-09-10T20:44:57.490096+0000","last_peered":"2024-09-10T20:44:57.490096+0000","last_clean":"2024-09-10T20:44:57.490096+0000","last_became_active":"2024-09-10T20:44:56.008400+0000","last_became_peered":"2024-09-10T20:44:56.008400+0000","last_unstale":"2024-09-10T20:44:57.490096+0000","last_undegraded":"2024-09-10T20:44:57.490096+0000","last_fullsized":"2024-09-10T20:44:57.490096+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:20:18.199887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.15","version":"137'3","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004405+0000","last_change":"2024-09-10T20:44:54.992717+0000","last_active":"2024-09-10T20:44:57.004405+0000","last_peered":"2024-09-10T20:44:57.004405+0000","last_clean":"2024-09-10T20:44:57.004405+0000","last_became_active":"2024-09-10T20:44:54.992410+0000","last_became_peered":"2024-09-10T20:44:54.992410+0000","last_unstale":"2024-09-10T20:44:57.004405+0000","last_undegraded":"2024-09-10T20:44:57.004405+0000","last_fullsized":"2024-09-10T20:44:57.004405+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:01.135690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490080+0000","last_change":"2024-09-10T20:44:56.005404+0000","last_active":"2024-09-10T20:44:57.490080+0000","last_peered":"2024-09-10T20:44:57.490080+0000","last_clean":"2024-09-10T20:44:57.490080+0000","last_became_active":"2024-09-10T20:44:56.005141+0000","last_became_peered":"2024-09-10T20:44:56.005141+0000","last_unstale":"2024-09-10T20:44:57.490080+0000","last_undegraded":"2024-09-10T20:44:57.490080+0000","last_fullsized":"2024-09-10T20:44:57.490080+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:42:19.198028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.14","version":"137'6","reported_seq":21,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294531+0000","last_change":"2024-09-10T20:44:54.996387+0000","last_active":"2024-09-10T20:44:57.294531+0000","last_peered":"2024-09-10T20:44:57.294531+0000","last_clean":"2024-09-10T20:44:57.294531+0000","last_became_active":"2024-09-10T20:44:54.996096+0000","last_became_peered":"2024-09-10T20:44:54.996096+0000","last_unstale":"2024-09-10T20:44:57.294531+0000","last_undegraded":"2024-09-10T20:44:57.294531+0000","last_fullsized":"2024-09-10T20:44:57.294531+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:58:31.603366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489900+0000","last_change":"2024-09-10T20:44:56.005121+0000","last_active":"2024-09-10T20:44:57.489900+0000","last_peered":"2024-09-10T20:44:57.489900+0000","last_clean":"2024-09-10T20:44:57.489900+0000","last_became_active":"2024-09-10T20:44:56.004846+0000","last_became_peered":"2024-09-10T20:44:56.004846+0000","last_unstale":"2024-09-10T20:44:57.489900+0000","last_undegraded":"2024-09-10T20:44:57.489900+0000","last_fullsized":"2024-09-10T20:44:57.489900+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:26:31.259778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"137'2","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004274+0000","last_change":"2024-09-10T20:44:54.993806+0000","last_active":"2024-09-10T20:44:57.004274+0000","last_peered":"2024-09-10T20:44:57.004274+0000","last_clean":"2024-09-10T20:44:57.004274+0000","last_became_active":"2024-09-10T20:44:54.993519+0000","last_became_peered":"2024-09-10T20:44:54.993519+0000","last_unstale":"2024-09-10T20:44:57.004274+0000","last_undegraded":"2024-09-10T20:44:57.004274+0000","last_fullsized":"2024-09-10T20:44:57.004274+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:54.622231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004830+0000","last_change":"2024-09-10T20:44:56.010446+0000","last_active":"2024-09-10T20:44:57.004830+0000","last_peered":"2024-09-10T20:44:57.004830+0000","last_clean":"2024-09-10T20:44:57.004830+0000","last_became_active":"2024-09-10T20:44:56.009967+0000","last_became_peered":"2024-09-10T20:44:56.009967+0000","last_unstale":"2024-09-10T20:44:57.004830+0000","last_undegraded":"2024-09-10T20:44:57.004830+0000","last_fullsized":"2024-09-10T20:44:57.004830+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:49.800668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490394+0000","last_change":"2024-09-10T20:44:54.998296+0000","last_active":"2024-09-10T20:44:57.490394+0000","last_peered":"2024-09-10T20:44:57.490394+0000","last_clean":"2024-09-10T20:44:57.490394+0000","last_became_active":"2024-09-10T20:44:54.997998+0000","last_became_peered":"2024-09-10T20:44:54.997998+0000","last_unstale":"2024-09-10T20:44:57.490394+0000","last_undegraded":"2024-09-10T20:44:57.490394+0000","last_fullsized":"2024-09-10T20:44:57.490394+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:08:23.134372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489680+0000","last_change":"2024-09-10T20:44:56.014647+0000","last_active":"2024-09-10T20:44:57.489680+0000","last_peered":"2024-09-10T20:44:57.489680+0000","last_clean":"2024-09-10T20:44:57.489680+0000","last_became_active":"2024-09-10T20:44:56.014374+0000","last_became_peered":"2024-09-10T20:44:56.014374+0000","last_unstale":"2024-09-10T20:44:57.489680+0000","last_undegraded":"2024-09-10T20:44:57.489680+0000","last_fullsized":"2024-09-10T20:44:57.489680+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:52:28.800857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490424+0000","last_change":"2024-09-10T20:44:55.004433+0000","last_active":"2024-09-10T20:44:57.490424+0000","last_peered":"2024-09-10T20:44:57.490424+0000","last_clean":"2024-09-10T20:44:57.490424+0000","last_became_active":"2024-09-10T20:44:55.004131+0000","last_became_peered":"2024-09-10T20:44:55.004131+0000","last_unstale":"2024-09-10T20:44:57.490424+0000","last_undegraded":"2024-09-10T20:44:57.490424+0000","last_fullsized":"2024-09-10T20:44:57.490424+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:42:33.535648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489718+0000","last_change":"2024-09-10T20:44:56.005508+0000","last_active":"2024-09-10T20:44:57.489718+0000","last_peered":"2024-09-10T20:44:57.489718+0000","last_clean":"2024-09-10T20:44:57.489718+0000","last_became_active":"2024-09-10T20:44:56.005233+0000","last_became_peered":"2024-09-10T20:44:56.005233+0000","last_unstale":"2024-09-10T20:44:57.489718+0000","last_undegraded":"2024-09-10T20:44:57.489718+0000","last_fullsized":"2024-09-10T20:44:57.489718+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:20:18.199887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"137'8","reported_seq":23,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294372+0000","last_change":"2024-09-10T20:44:54.994640+0000","last_active":"2024-09-10T20:44:57.294372+0000","last_peered":"2024-09-10T20:44:57.294372+0000","last_clean":"2024-09-10T20:44:57.294372+0000","last_became_active":"2024-09-10T20:44:54.994359+0000","last_became_peered":"2024-09-10T20:44:54.994359+0000","last_unstale":"2024-09-10T20:44:57.294372+0000","last_undegraded":"2024-09-10T20:44:57.294372+0000","last_fullsized":"2024-09-10T20:44:57.294372+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:50:29.690155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490658+0000","last_change":"2024-09-10T20:44:56.008250+0000","last_active":"2024-09-10T20:44:57.490658+0000","last_peered":"2024-09-10T20:44:57.490658+0000","last_clean":"2024-09-10T20:44:57.490658+0000","last_became_active":"2024-09-10T20:44:56.007960+0000","last_became_peered":"2024-09-10T20:44:56.007960+0000","last_unstale":"2024-09-10T20:44:57.490658+0000","last_undegraded":"2024-09-10T20:44:57.490658+0000","last_fullsized":"2024-09-10T20:44:57.490658+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:52.260828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"137'1","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294697+0000","last_change":"2024-09-10T20:44:55.001849+0000","last_active":"2024-09-10T20:44:57.294697+0000","last_peered":"2024-09-10T20:44:57.294697+0000","last_clean":"2024-09-10T20:44:57.294697+0000","last_became_active":"2024-09-10T20:44:55.001535+0000","last_became_peered":"2024-09-10T20:44:55.001535+0000","last_unstale":"2024-09-10T20:44:57.294697+0000","last_undegraded":"2024-09-10T20:44:57.294697+0000","last_fullsized":"2024-09-10T20:44:57.294697+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:56:12.188453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004868+0000","last_change":"2024-09-10T20:44:56.005513+0000","last_active":"2024-09-10T20:44:57.004868+0000","last_peered":"2024-09-10T20:44:57.004868+0000","last_clean":"2024-09-10T20:44:57.004868+0000","last_became_active":"2024-09-10T20:44:56.005254+0000","last_became_peered":"2024-09-10T20:44:56.005254+0000","last_unstale":"2024-09-10T20:44:57.004868+0000","last_undegraded":"2024-09-10T20:44:57.004868+0000","last_fullsized":"2024-09-10T20:44:57.004868+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:53.541549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"137'2","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490227+0000","last_change":"2024-09-10T20:44:55.004183+0000","last_active":"2024-09-10T20:44:57.490227+0000","last_peered":"2024-09-10T20:44:57.490227+0000","last_clean":"2024-09-10T20:44:57.490227+0000","last_became_active":"2024-09-10T20:44:55.003934+0000","last_became_peered":"2024-09-10T20:44:55.003934+0000","last_unstale":"2024-09-10T20:44:57.490227+0000","last_undegraded":"2024-09-10T20:44:57.490227+0000","last_fullsized":"2024-09-10T20:44:57.490227+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:15.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004855+0000","last_change":"2024-09-10T20:44:56.014640+0000","last_active":"2024-09-10T20:44:57.004855+0000","last_peered":"2024-09-10T20:44:57.004855+0000","last_clean":"2024-09-10T20:44:57.004855+0000","last_became_active":"2024-09-10T20:44:56.014193+0000","last_became_peered":"2024-09-10T20:44:56.014193+0000","last_unstale":"2024-09-10T20:44:57.004855+0000","last_undegraded":"2024-09-10T20:44:57.004855+0000","last_fullsized":"2024-09-10T20:44:57.004855+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:41:22.521728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.d","version":"137'1","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294839+0000","last_change":"2024-09-10T20:44:55.004538+0000","last_active":"2024-09-10T20:44:57.294839+0000","last_peered":"2024-09-10T20:44:57.294839+0000","last_clean":"2024-09-10T20:44:57.294839+0000","last_became_active":"2024-09-10T20:44:55.004210+0000","last_became_peered":"2024-09-10T20:44:55.004210+0000","last_unstale":"2024-09-10T20:44:57.294839+0000","last_undegraded":"2024-09-10T20:44:57.294839+0000","last_fullsized":"2024-09-10T20:44:57.294839+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:34:14.482972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004337+0000","last_change":"2024-09-10T20:44:56.008151+0000","last_active":"2024-09-10T20:44:57.004337+0000","last_peered":"2024-09-10T20:44:57.004337+0000","last_clean":"2024-09-10T20:44:57.004337+0000","last_became_active":"2024-09-10T20:44:56.007881+0000","last_became_peered":"2024-09-10T20:44:56.007881+0000","last_unstale":"2024-09-10T20:44:57.004337+0000","last_undegraded":"2024-09-10T20:44:57.004337+0000","last_fullsized":"2024-09-10T20:44:57.004337+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:19.839593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.002996+0000","last_change":"2024-09-10T20:44:54.997850+0000","last_active":"2024-09-10T20:44:57.002996+0000","last_peered":"2024-09-10T20:44:57.002996+0000","last_clean":"2024-09-10T20:44:57.002996+0000","last_became_active":"2024-09-10T20:44:54.997555+0000","last_became_peered":"2024-09-10T20:44:54.997555+0000","last_unstale":"2024-09-10T20:44:57.002996+0000","last_undegraded":"2024-09-10T20:44:57.002996+0000","last_fullsized":"2024-09-10T20:44:57.002996+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:00:32.987168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489377+0000","last_change":"2024-09-10T20:44:56.009413+0000","last_active":"2024-09-10T20:44:57.489377+0000","last_peered":"2024-09-10T20:44:57.489377+0000","last_clean":"2024-09-10T20:44:57.489377+0000","last_became_active":"2024-09-10T20:44:56.009169+0000","last_became_peered":"2024-09-10T20:44:56.009169+0000","last_unstale":"2024-09-10T20:44:57.489377+0000","last_undegraded":"2024-09-10T20:44:57.489377+0000","last_fullsized":"2024-09-10T20:44:57.489377+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:14:24.540751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.b","version":"137'2","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004174+0000","last_change":"2024-09-10T20:44:55.001865+0000","last_active":"2024-09-10T20:44:57.004174+0000","last_peered":"2024-09-10T20:44:57.004174+0000","last_clean":"2024-09-10T20:44:57.004174+0000","last_became_active":"2024-09-10T20:44:55.001593+0000","last_became_peered":"2024-09-10T20:44:55.001593+0000","last_unstale":"2024-09-10T20:44:57.004174+0000","last_undegraded":"2024-09-10T20:44:57.004174+0000","last_fullsized":"2024-09-10T20:44:57.004174+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:40:00.707442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004114+0000","last_change":"2024-09-10T20:44:56.002051+0000","last_active":"2024-09-10T20:44:57.004114+0000","last_peered":"2024-09-10T20:44:57.004114+0000","last_clean":"2024-09-10T20:44:57.004114+0000","last_became_active":"2024-09-10T20:44:56.001770+0000","last_became_peered":"2024-09-10T20:44:56.001770+0000","last_unstale":"2024-09-10T20:44:57.004114+0000","last_undegraded":"2024-09-10T20:44:57.004114+0000","last_fullsized":"2024-09-10T20:44:57.004114+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:41:58.849863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.002515+0000","last_change":"2024-09-10T20:44:55.000052+0000","last_active":"2024-09-10T20:44:57.002515+0000","last_peered":"2024-09-10T20:44:57.002515+0000","last_clean":"2024-09-10T20:44:57.002515+0000","last_became_active":"2024-09-10T20:44:54.999727+0000","last_became_peered":"2024-09-10T20:44:54.999727+0000","last_unstale":"2024-09-10T20:44:57.002515+0000","last_undegraded":"2024-09-10T20:44:57.002515+0000","last_fullsized":"2024-09-10T20:44:57.002515+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:11.433766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489297+0000","last_change":"2024-09-10T20:44:55.997033+0000","last_active":"2024-09-10T20:44:57.489297+0000","last_peered":"2024-09-10T20:44:57.489297+0000","last_clean":"2024-09-10T20:44:57.489297+0000","last_became_active":"2024-09-10T20:44:55.996570+0000","last_became_peered":"2024-09-10T20:44:55.996570+0000","last_unstale":"2024-09-10T20:44:57.489297+0000","last_undegraded":"2024-09-10T20:44:57.489297+0000","last_fullsized":"2024-09-10T20:44:57.489297+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:37:26.350607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004048+0000","last_change":"2024-09-10T20:44:55.000012+0000","last_active":"2024-09-10T20:44:57.004048+0000","last_peered":"2024-09-10T20:44:57.004048+0000","last_clean":"2024-09-10T20:44:57.004048+0000","last_became_active":"2024-09-10T20:44:54.999747+0000","last_became_peered":"2024-09-10T20:44:54.999747+0000","last_unstale":"2024-09-10T20:44:57.004048+0000","last_undegraded":"2024-09-10T20:44:57.004048+0000","last_fullsized":"2024-09-10T20:44:57.004048+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:53:04.217324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490365+0000","last_change":"2024-09-10T20:44:56.015518+0000","last_active":"2024-09-10T20:44:57.490365+0000","last_peered":"2024-09-10T20:44:57.490365+0000","last_clean":"2024-09-10T20:44:57.490365+0000","last_became_active":"2024-09-10T20:44:56.015217+0000","last_became_peered":"2024-09-10T20:44:56.015217+0000","last_unstale":"2024-09-10T20:44:57.490365+0000","last_undegraded":"2024-09-10T20:44:57.490365+0000","last_fullsized":"2024-09-10T20:44:57.490365+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:04:23.804609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.002531+0000","last_change":"2024-09-10T20:44:55.002762+0000","last_active":"2024-09-10T20:44:57.002531+0000","last_peered":"2024-09-10T20:44:57.002531+0000","last_clean":"2024-09-10T20:44:57.002531+0000","last_became_active":"2024-09-10T20:44:55.002467+0000","last_became_peered":"2024-09-10T20:44:55.002467+0000","last_unstale":"2024-09-10T20:44:57.002531+0000","last_undegraded":"2024-09-10T20:44:57.002531+0000","last_fullsized":"2024-09-10T20:44:57.002531+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:14.486934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004733+0000","last_change":"2024-09-10T20:44:56.006308+0000","last_active":"2024-09-10T20:44:57.004733+0000","last_peered":"2024-09-10T20:44:57.004733+0000","last_clean":"2024-09-10T20:44:57.004733+0000","last_became_active":"2024-09-10T20:44:56.005980+0000","last_became_peered":"2024-09-10T20:44:56.005980+0000","last_unstale":"2024-09-10T20:44:57.004733+0000","last_undegraded":"2024-09-10T20:44:57.004733+0000","last_fullsized":"2024-09-10T20:44:57.004733+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:57.639676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"137'2","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490192+0000","last_change":"2024-09-10T20:44:54.998331+0000","last_active":"2024-09-10T20:44:57.490192+0000","last_peered":"2024-09-10T20:44:57.490192+0000","last_clean":"2024-09-10T20:44:57.490192+0000","last_became_active":"2024-09-10T20:44:54.998013+0000","last_became_peered":"2024-09-10T20:44:54.998013+0000","last_unstale":"2024-09-10T20:44:57.490192+0000","last_undegraded":"2024-09-10T20:44:57.490192+0000","last_fullsized":"2024-09-10T20:44:57.490192+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:19:11.702974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004740+0000","last_change":"2024-09-10T20:44:56.003530+0000","last_active":"2024-09-10T20:44:57.004740+0000","last_peered":"2024-09-10T20:44:57.004740+0000","last_clean":"2024-09-10T20:44:57.004740+0000","last_became_active":"2024-09-10T20:44:56.002130+0000","last_became_peered":"2024-09-10T20:44:56.002130+0000","last_unstale":"2024-09-10T20:44:57.004740+0000","last_undegraded":"2024-09-10T20:44:57.004740+0000","last_fullsized":"2024-09-10T20:44:57.004740+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:21.364687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.6","version":"138'1","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.003090+0000","last_change":"2024-09-10T20:44:55.000518+0000","last_active":"2024-09-10T20:44:57.003090+0000","last_peered":"2024-09-10T20:44:57.003090+0000","last_clean":"2024-09-10T20:44:57.003090+0000","last_became_active":"2024-09-10T20:44:55.000257+0000","last_became_peered":"2024-09-10T20:44:55.000257+0000","last_unstale":"2024-09-10T20:44:57.003090+0000","last_undegraded":"2024-09-10T20:44:57.003090+0000","last_fullsized":"2024-09-10T20:44:57.003090+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:29:19.995308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.005096+0000","last_change":"2024-09-10T20:44:55.997515+0000","last_active":"2024-09-10T20:44:57.005096+0000","last_peered":"2024-09-10T20:44:57.005096+0000","last_clean":"2024-09-10T20:44:57.005096+0000","last_became_active":"2024-09-10T20:44:55.997223+0000","last_became_peered":"2024-09-10T20:44:55.997223+0000","last_unstale":"2024-09-10T20:44:57.005096+0000","last_undegraded":"2024-09-10T20:44:57.005096+0000","last_fullsized":"2024-09-10T20:44:57.005096+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:52:03.317677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490210+0000","last_change":"2024-09-10T20:44:55.006327+0000","last_active":"2024-09-10T20:44:57.490210+0000","last_peered":"2024-09-10T20:44:57.490210+0000","last_clean":"2024-09-10T20:44:57.490210+0000","last_became_active":"2024-09-10T20:44:55.006024+0000","last_became_peered":"2024-09-10T20:44:55.006024+0000","last_unstale":"2024-09-10T20:44:57.490210+0000","last_undegraded":"2024-09-10T20:44:57.490210+0000","last_fullsized":"2024-09-10T20:44:57.490210+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:14:51.347600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.004323+0000","last_change":"2024-09-10T20:44:56.003786+0000","last_active":"2024-09-10T20:44:57.004323+0000","last_peered":"2024-09-10T20:44:57.004323+0000","last_clean":"2024-09-10T20:44:57.004323+0000","last_became_active":"2024-09-10T20:44:56.003476+0000","last_became_peered":"2024-09-10T20:44:56.003476+0000","last_unstale":"2024-09-10T20:44:57.004323+0000","last_undegraded":"2024-09-10T20:44:57.004323+0000","last_fullsized":"2024-09-10T20:44:57.004323+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:23.391608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.005329+0000","last_change":"2024-09-10T20:44:55.001854+0000","last_active":"2024-09-10T20:44:57.005329+0000","last_peered":"2024-09-10T20:44:57.005329+0000","last_clean":"2024-09-10T20:44:57.005329+0000","last_became_active":"2024-09-10T20:44:55.001567+0000","last_became_peered":"2024-09-10T20:44:55.001567+0000","last_unstale":"2024-09-10T20:44:57.005329+0000","last_undegraded":"2024-09-10T20:44:57.005329+0000","last_fullsized":"2024-09-10T20:44:57.005329+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:07.977973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490323+0000","last_change":"2024-09-10T20:44:56.006070+0000","last_active":"2024-09-10T20:44:57.490323+0000","last_peered":"2024-09-10T20:44:57.490323+0000","last_clean":"2024-09-10T20:44:57.490323+0000","last_became_active":"2024-09-10T20:44:56.005598+0000","last_became_peered":"2024-09-10T20:44:56.005598+0000","last_unstale":"2024-09-10T20:44:57.490323+0000","last_undegraded":"2024-09-10T20:44:57.490323+0000","last_fullsized":"2024-09-10T20:44:57.490323+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:03:04.181831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294527+0000","last_change":"2024-09-10T20:44:54.991914+0000","last_active":"2024-09-10T20:44:57.294527+0000","last_peered":"2024-09-10T20:44:57.294527+0000","last_clean":"2024-09-10T20:44:57.294527+0000","last_became_active":"2024-09-10T20:44:54.991617+0000","last_became_peered":"2024-09-10T20:44:54.991617+0000","last_unstale":"2024-09-10T20:44:57.294527+0000","last_undegraded":"2024-09-10T20:44:57.294527+0000","last_fullsized":"2024-09-10T20:44:57.294527+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:29:05.636749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489601+0000","last_change":"2024-09-10T20:44:56.012199+0000","last_active":"2024-09-10T20:44:57.489601+0000","last_peered":"2024-09-10T20:44:57.489601+0000","last_clean":"2024-09-10T20:44:57.489601+0000","last_became_active":"2024-09-10T20:44:56.011881+0000","last_became_peered":"2024-09-10T20:44:56.011881+0000","last_unstale":"2024-09-10T20:44:57.489601+0000","last_undegraded":"2024-09-10T20:44:57.489601+0000","last_fullsized":"2024-09-10T20:44:57.489601+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:48.188534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489971+0000","last_change":"2024-09-10T20:44:55.000704+0000","last_active":"2024-09-10T20:44:57.489971+0000","last_peered":"2024-09-10T20:44:57.489971+0000","last_clean":"2024-09-10T20:44:57.489971+0000","last_became_active":"2024-09-10T20:44:55.000261+0000","last_became_peered":"2024-09-10T20:44:55.000261+0000","last_unstale":"2024-09-10T20:44:57.489971+0000","last_undegraded":"2024-09-10T20:44:57.489971+0000","last_fullsized":"2024-09-10T20:44:57.489971+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:59:12.152209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490244+0000","last_change":"2024-09-10T20:44:56.014150+0000","last_active":"2024-09-10T20:44:57.490244+0000","last_peered":"2024-09-10T20:44:57.490244+0000","last_clean":"2024-09-10T20:44:57.490244+0000","last_became_active":"2024-09-10T20:44:56.013245+0000","last_became_peered":"2024-09-10T20:44:56.013245+0000","last_unstale":"2024-09-10T20:44:57.490244+0000","last_undegraded":"2024-09-10T20:44:57.490244+0000","last_fullsized":"2024-09-10T20:44:57.490244+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:28.975034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489582+0000","last_change":"2024-09-10T20:44:55.006369+0000","last_active":"2024-09-10T20:44:57.489582+0000","last_peered":"2024-09-10T20:44:57.489582+0000","last_clean":"2024-09-10T20:44:57.489582+0000","last_became_active":"2024-09-10T20:44:55.006067+0000","last_became_peered":"2024-09-10T20:44:55.006067+0000","last_unstale":"2024-09-10T20:44:57.489582+0000","last_undegraded":"2024-09-10T20:44:57.489582+0000","last_fullsized":"2024-09-10T20:44:57.489582+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:13:04.641503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489293+0000","last_change":"2024-09-10T20:44:56.006748+0000","last_active":"2024-09-10T20:44:57.489293+0000","last_peered":"2024-09-10T20:44:57.489293+0000","last_clean":"2024-09-10T20:44:57.489293+0000","last_became_active":"2024-09-10T20:44:56.006443+0000","last_became_peered":"2024-09-10T20:44:56.006443+0000","last_unstale":"2024-09-10T20:44:57.489293+0000","last_undegraded":"2024-09-10T20:44:57.489293+0000","last_fullsized":"2024-09-10T20:44:57.489293+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:06.345433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.489379+0000","last_change":"2024-09-10T20:44:55.006657+0000","last_active":"2024-09-10T20:44:57.489379+0000","last_peered":"2024-09-10T20:44:57.489379+0000","last_clean":"2024-09-10T20:44:57.489379+0000","last_became_active":"2024-09-10T20:44:55.005513+0000","last_became_peered":"2024-09-10T20:44:55.005513+0000","last_unstale":"2024-09-10T20:44:57.489379+0000","last_undegraded":"2024-09-10T20:44:57.489379+0000","last_fullsized":"2024-09-10T20:44:57.489379+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:39:37.910050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.490622+0000","last_change":"2024-09-10T20:44:56.007469+0000","last_active":"2024-09-10T20:44:57.490622+0000","last_peered":"2024-09-10T20:44:57.490622+0000","last_clean":"2024-09-10T20:44:57.490622+0000","last_became_active":"2024-09-10T20:44:56.007184+0000","last_became_peered":"2024-09-10T20:44:56.007184+0000","last_unstale":"2024-09-10T20:44:57.490622+0000","last_undegraded":"2024-09-10T20:44:57.490622+0000","last_fullsized":"2024-09-10T20:44:57.490622+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:54.974267+0000","last_clean_scrub_stamp":"2024-09-10T20:44:54.974267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:48.188534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294635+0000","last_change":"2024-09-10T20:44:54.998266+0000","last_active":"2024-09-10T20:44:57.294635+0000","last_peered":"2024-09-10T20:44:57.294635+0000","last_clean":"2024-09-10T20:44:57.294635+0000","last_became_active":"2024-09-10T20:44:54.997713+0000","last_became_peered":"2024-09-10T20:44:54.997713+0000","last_unstale":"2024-09-10T20:44:57.294635+0000","last_undegraded":"2024-09-10T20:44:57.294635+0000","last_fullsized":"2024-09-10T20:44:57.294635+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:44:53.972007+0000","last_clean_scrub_stamp":"2024-09-10T20:44:53.972007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:21.372886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":354,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-10T20:44:57.294510+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:44:57.294510+0000","last_peered":"2024-09-10T20:44:57.294510+0000","last_clean":"2024-09-10T20:44:57.294510+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:44:57.294510+0000","last_undegraded":"2024-09-10T20:44:57.294510+0000","last_fullsized":"2024-09-10T20:44:57.294510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542231,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1728512,"data_stored":763525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542232,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1732608,"data_stored":763559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542230,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2330624,"data_stored":1354565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542230,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1732608,"data_stored":763559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509528,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37052,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334788,"statfs":{"total":96636764160,"available":96598822912,"internally_reserved":0,"allocated":1744896,"data_stored":765159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509527,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1732608,"data_stored":764107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509527,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37640,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334200,"statfs":{"total":96636764160,"available":96598220800,"internally_reserved":0,"allocated":2347008,"data_stored":1355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509527,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1732608,"data_stored":763595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:45:00.384 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:45:00.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:45:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 -- 172.21.15.28:0/1942069041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 msgr2=0x7f2f10070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 --2- 172.21.15.28:0/1942069041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10070430 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009830 tx=0x7f2f0402a510 comp rx=0 tx=0).stop 2024-09-10T20:45:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 -- 172.21.15.28:0/1942069041 shutdown_connections 2024-09-10T20:45:00.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 --2- 172.21.15.28:0/1942069041 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f100cffc0 0x7f2f100d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 --2- 172.21.15.28:0/1942069041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10070430 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 -- 172.21.15.28:0/1942069041 >> 172.21.15.28:0/1942069041 conn(0x7f2f1006cf00 msgr2=0x7f2f1006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 -- 172.21.15.28:0/1942069041 shutdown_connections 2024-09-10T20:45:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.529+0000 7f2f15f52640 1 -- 172.21.15.28:0/1942069041 wait complete. 2024-09-10T20:45:00.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.530+0000 7f2f15f52640 1 Processor -- start 2024-09-10T20:45:00.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.530+0000 7f2f15f52640 1 -- start start 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f15f52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f100cf500 0x7f2f10154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f15f52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f100cffc0 0x7f2f100ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f15f52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f15f52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f100ae7d0 con 0x7f2f1015f720 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f15f52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2f100ae650 con 0x7f2f100cffc0 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f15f52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2f100ae950 con 0x7f2f100cf500 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38234/0 (socket says 172.21.15.28:38234) 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 -- 172.21.15.28:0/2015614311 learned_addr learned my addr 172.21.15.28:0/2015614311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:00.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0effd640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f100cffc0 0x7f2f100ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0f7fe640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f100cf500 0x7f2f10154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 -- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f100cf500 msgr2=0x7f2f10154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f100cf500 0x7f2f10154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 -- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f100cffc0 msgr2=0x7f2f100ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f100cffc0 0x7f2f100ce470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0ffff640 1 -- 172.21.15.28:0/2015614311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f101f4830 con 0x7f2f1015f720 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0effd640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f100cffc0 0x7f2f100ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.531+0000 7f2f0f7fe640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f100cf500 0x7f2f10154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.532+0000 7f2f0ffff640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10152f90 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800b970 tx=0x7f2f0800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.532+0000 7f2ef7fff640 1 -- 172.21.15.28:0/2015614311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08019070 con 0x7f2f1015f720 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.532+0000 7f2ef7fff640 1 -- 172.21.15.28:0/2015614311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f0800cc50 con 0x7f2f1015f720 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.532+0000 7f2ef7fff640 1 -- 172.21.15.28:0/2015614311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080146a0 con 0x7f2f1015f720 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.532+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f10153b00 con 0x7f2f1015f720 2024-09-10T20:45:00.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.532+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f101540c0 con 0x7f2f1015f720 2024-09-10T20:45:00.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.533+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f100aeea0 con 0x7f2f1015f720 2024-09-10T20:45:00.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.533+0000 7f2ef7fff640 1 -- 172.21.15.28:0/2015614311 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f2f0801d3b0 con 0x7f2f1015f720 2024-09-10T20:45:00.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.534+0000 7f2ef7fff640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2efc078630 0x7f2efc07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.534+0000 7f2ef7fff640 1 -- 172.21.15.28:0/2015614311 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f2f0809aa00 con 0x7f2f1015f720 2024-09-10T20:45:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.534+0000 7f2f0f7fe640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2efc078630 0x7f2efc07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.534+0000 7f2f0f7fe640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2efc078630 0x7f2efc07aaf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2f00001740 tx=0x7f2f00009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:00.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.538+0000 7f2ef7fff640 1 -- 172.21.15.28:0/2015614311 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2f08062ac0 con 0x7f2f1015f720 2024-09-10T20:45:00.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.798+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f2f1015ee40 con 0x7f2f1015f720 2024-09-10T20:45:00.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.800+0000 7f2ef7fff640 1 -- 172.21.15.28:0/2015614311 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v35) ==== 190+0+147 (secure 0 0 0) 0x7f2f080623c0 con 0x7f2f1015f720 2024-09-10T20:45:00.804 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:45:00.804 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:45:00.804 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:45:00.804 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:45:00.804 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:45:00.805 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:45:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2efc078630 msgr2=0x7f2efc07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2efc078630 0x7f2efc07aaf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2f00001740 tx=0x7f2f00009040 comp rx=0 tx=0).stop 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 msgr2=0x7f2f10152f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10152f90 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800b970 tx=0x7f2f0800be30 comp rx=0 tx=0).stop 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 shutdown_connections 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2efc078630 0x7f2efc07aaf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1015f720 0x7f2f10152f90 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f100cffc0 0x7f2f100ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 --2- 172.21.15.28:0/2015614311 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f100cf500 0x7f2f10154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 >> 172.21.15.28:0/2015614311 conn(0x7f2f1006cf00 msgr2=0x7f2f10075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.802+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 shutdown_connections 2024-09-10T20:45:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.803+0000 7f2f15f52640 1 -- 172.21.15.28:0/2015614311 wait complete. 2024-09-10T20:45:00.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.919+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/1785045279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b0003a50 msgr2=0x7fd6b0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.919+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/1785045279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b0003a50 0x7fd6b0003e70 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80016c0 tx=0x7fd6b0005830 comp rx=0 tx=0).stop 2024-09-10T20:45:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.919+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/1785045279 shutdown_connections 2024-09-10T20:45:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.919+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/1785045279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b0003a50 0x7fd6b0003e70 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.919+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/1785045279 >> 172.21.15.28:0/1785045279 conn(0x7fd6c806cf00 msgr2=0x7fd6c806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.919+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/1785045279 shutdown_connections 2024-09-10T20:45:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.919+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/1785045279 wait complete. 2024-09-10T20:45:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.920+0000 7fd6cdcfe640 1 Processor -- start 2024-09-10T20:45:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.920+0000 7fd6cdcfe640 1 -- start start 2024-09-10T20:45:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6cdcfe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6b0003a50 0x7fd6c8145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6cdcfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c80b09d0 0x7fd6c80b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6cdcfe640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6c809ccb0 0x7fd6c809d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6cdcfe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c815c0e0 con 0x7fd6c80b09d0 2024-09-10T20:45:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6cdcfe640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c815bf60 con 0x7fd6c809ccb0 2024-09-10T20:45:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6cdcfe640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c815c260 con 0x7fd6b0003a50 2024-09-10T20:45:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c80b09d0 0x7fd6c80b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c80b09d0 0x7fd6c80b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38240/0 (socket says 172.21.15.28:38240) 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 -- 172.21.15.28:0/3469184165 learned_addr learned my addr 172.21.15.28:0/3469184165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c77fe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6b0003a50 0x7fd6c8145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c7fff640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6c809ccb0 0x7fd6c809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 -- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6b0003a50 msgr2=0x7fd6c8145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6b0003a50 0x7fd6c8145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 -- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6c809ccb0 msgr2=0x7fd6c809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6c809ccb0 0x7fd6c809d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.921+0000 7fd6c6ffd640 1 -- 172.21.15.28:0/3469184165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c80b1390 con 0x7fd6c80b09d0 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.922+0000 7fd6c6ffd640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c80b09d0 0x7fd6c80b0e50 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00f770 tx=0x7fd6bc00fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.922+0000 7fd6affff640 1 -- 172.21.15.28:0/3469184165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00baa0 con 0x7fd6c80b09d0 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.922+0000 7fd6affff640 1 -- 172.21.15.28:0/3469184165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6bc00bd30 con 0x7fd6c80b09d0 2024-09-10T20:45:00.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.922+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c8147df0 con 0x7fd6c80b09d0 2024-09-10T20:45:00.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.922+0000 7fd6affff640 1 -- 172.21.15.28:0/3469184165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0116a0 con 0x7fd6c80b09d0 2024-09-10T20:45:00.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.922+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c81483b0 con 0x7fd6c80b09d0 2024-09-10T20:45:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.923+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c80732e0 con 0x7fd6c80b09d0 2024-09-10T20:45:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.924+0000 7fd6affff640 1 -- 172.21.15.28:0/3469184165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fd6bc023030 con 0x7fd6c80b09d0 2024-09-10T20:45:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.924+0000 7fd6affff640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6b4078630 0x7fd6b407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.924+0000 7fd6c77fe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6b4078630 0x7fd6b407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.924+0000 7fd6affff640 1 -- 172.21.15.28:0/3469184165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7fd6bc09b0d0 con 0x7fd6c80b09d0 2024-09-10T20:45:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.925+0000 7fd6c77fe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6b4078630 0x7fd6b407aaf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8032000 tx=0x7fd6b802c040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:00.929+0000 7fd6affff640 1 -- 172.21.15.28:0/3469184165 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd6c80732e0 con 0x7fd6c80b09d0 2024-09-10T20:45:01.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.151+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6c80d0300 con 0x7fd6c80b09d0 2024-09-10T20:45:01.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.151+0000 7fd6affff640 1 -- 172.21.15.28:0/3469184165 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5227 (secure 0 0 0) 0x7fd6bc062ac0 con 0x7fd6c80b09d0 2024-09-10T20:45:01.155 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":96,"btime":"2024-09-10T20:44:59:007364+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4242811460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4242811460},{"type":"v1","addr":"172.21.15.28:6834","nonce":4242811460}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/914514353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":914514353},{"type":"v1","addr":"172.21.15.159:6835","nonce":914514353}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8334,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2842589826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2842589826},{"type":"v1","addr":"172.21.15.28:6835","nonce":2842589826}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:55.365818+0000","modified":"2024-09-10T20:44:58.372881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8346},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8346":{"gid":8346,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/103941194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":103941194},{"type":"v1","addr":"172.21.15.159:6801","nonce":103941194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8346,"qdb_cluster":[8346]},"id":11}]} 2024-09-10T20:45:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 96 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6b4078630 msgr2=0x7fd6b407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6b4078630 0x7fd6b407aaf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8032000 tx=0x7fd6b802c040 comp rx=0 tx=0).stop 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c80b09d0 msgr2=0x7fd6c80b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c80b09d0 0x7fd6c80b0e50 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00f770 tx=0x7fd6bc00fb70 comp rx=0 tx=0).stop 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 shutdown_connections 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6b4078630 0x7fd6b407aaf0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6c809ccb0 0x7fd6c809d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c80b09d0 0x7fd6c80b0e50 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 --2- 172.21.15.28:0/3469184165 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6b0003a50 0x7fd6c8145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 >> 172.21.15.28:0/3469184165 conn(0x7fd6c806cf00 msgr2=0x7fd6c8072bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 shutdown_connections 2024-09-10T20:45:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.154+0000 7fd6cdcfe640 1 -- 172.21.15.28:0/3469184165 wait complete. 2024-09-10T20:45:01.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:44:55.365818+0000', 'modified': '2024-09-10T20:44:58.372881+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8346}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8346': {'gid': 8346, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.159:6801/103941194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 103941194}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 103941194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8346, 'qdb_cluster': [8346]} 2024-09-10T20:45:01.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:45:01.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-10T20:45:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.677+0000 7fec7d55d640 1 -- 172.21.15.28:0/1833346697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec60003a50 msgr2=0x7fec60003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.677+0000 7fec7d55d640 1 --2- 172.21.15.28:0/1833346697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec60003a50 0x7fec60003e70 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fec6800baf0 tx=0x7fec60005830 comp rx=0 tx=0).stop 2024-09-10T20:45:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.677+0000 7fec7d55d640 1 -- 172.21.15.28:0/1833346697 shutdown_connections 2024-09-10T20:45:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.677+0000 7fec7d55d640 1 --2- 172.21.15.28:0/1833346697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec60003a50 0x7fec60003e70 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.677+0000 7fec7d55d640 1 -- 172.21.15.28:0/1833346697 >> 172.21.15.28:0/1833346697 conn(0x7fec7806cf00 msgr2=0x7fec7806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.677+0000 7fec7d55d640 1 -- 172.21.15.28:0/1833346697 shutdown_connections 2024-09-10T20:45:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.677+0000 7fec7d55d640 1 -- 172.21.15.28:0/1833346697 wait complete. 2024-09-10T20:45:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.679+0000 7fec7d55d640 1 Processor -- start 2024-09-10T20:45:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.679+0000 7fec7d55d640 1 -- start start 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.679+0000 7fec7d55d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec60003a50 0x7fec781f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.679+0000 7fec7d55d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec78151aa0 0x7fec78152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec7d55d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec78152750 0x7fec7815d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec7d55d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec780cc570 con 0x7fec78152750 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec7d55d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fec780b4a90 con 0x7fec78151aa0 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec7d55d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fec780cc6f0 con 0x7fec60003a50 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec78152750 0x7fec7815d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec78152750 0x7fec7815d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38256/0 (socket says 172.21.15.28:38256) 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 -- 172.21.15.28:0/91894144 learned_addr learned my addr 172.21.15.28:0/91894144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec76ffd640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec60003a50 0x7fec781f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec767fc640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec78151aa0 0x7fec78152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 -- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec60003a50 msgr2=0x7fec781f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec60003a50 0x7fec781f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 -- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec78151aa0 msgr2=0x7fec78152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec78151aa0 0x7fec78152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 -- 172.21.15.28:0/91894144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec781f8640 con 0x7fec78152750 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec777fe640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec78152750 0x7fec7815d3e0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fec780d0bb0 tx=0x7fec70001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec5f7fe640 1 -- 172.21.15.28:0/91894144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70011870 con 0x7fec78152750 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec5f7fe640 1 -- 172.21.15.28:0/91894144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec70017070 con 0x7fec78152750 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec5f7fe640 1 -- 172.21.15.28:0/91894144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70011da0 con 0x7fec78152750 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec780746f0 con 0x7fec78152750 2024-09-10T20:45:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.680+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec780b25e0 con 0x7fec78152750 2024-09-10T20:45:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.682+0000 7fec5f7fe640 1 -- 172.21.15.28:0/91894144 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fec7001c460 con 0x7fec78152750 2024-09-10T20:45:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.683+0000 7fec5f7fe640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec64078630 0x7fec6407aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.683+0000 7fec5f7fe640 1 -- 172.21.15.28:0/91894144 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7fec7009b0c0 con 0x7fec78152750 2024-09-10T20:45:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.683+0000 7fec76ffd640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec64078630 0x7fec6407aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.683+0000 7fec80409740 1 Processor -- start 2024-09-10T20:45:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.683+0000 7fec76ffd640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec64078630 0x7fec6407aaf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec68036000 tx=0x7fec6803d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.683+0000 7fec80409740 1 -- start start 2024-09-10T20:45:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec80409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2e61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f29b5f0 0x555c7f29ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec80409740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c7f212820 con 0x555c7f2c61b0 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x555c7f201c50 con 0x555c7f2e5df0 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x555c7f205a60 con 0x555c7f29b5f0 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38268/0 (socket says 172.21.15.28:38268) 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.684+0000 7fec76ffd640 1 -- 172.21.15.28:0/3419769717 learned_addr learned my addr 172.21.15.28:0/3419769717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec777fe640 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f29b5f0 0x555c7f29ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec76ffd640 1 -- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f29b5f0 msgr2=0x555c7f29ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec767fc640 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2e61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec76ffd640 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f29b5f0 0x555c7f29ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec76ffd640 1 -- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 msgr2=0x555c7f2e61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec76ffd640 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2e61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec76ffd640 1 -- 172.21.15.28:0/3419769717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c7ef63850 con 0x555c7f2c61b0 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec777fe640 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f29b5f0 0x555c7f29ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec767fc640 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2e61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec76ffd640 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29db20 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fec68034040 tx=0x7fec60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146a2bb6bb0f9e7e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec5cff9640 1 -- 172.21.15.28:0/3419769717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68032ce0 con 0x555c7f2c61b0 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec5cff9640 1 -- 172.21.15.28:0/3419769717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec68037040 con 0x555c7f2c61b0 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.685+0000 7fec5cff9640 1 -- 172.21.15.28:0/3419769717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68045650 con 0x555c7f2c61b0 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 -- 172.21.15.28:0/3419769717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 msgr2=0x555c7f29db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29db20 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fec68034040 tx=0x7fec60005830 comp rx=0 tx=0).stop 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 -- 172.21.15.28:0/3419769717 shutdown_connections 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f29b5f0 0x555c7f29ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2e61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 --2- 172.21.15.28:0/3419769717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29db20 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 -- 172.21.15.28:0/3419769717 >> 172.21.15.28:0/3419769717 conn(0x555c7f29c6a0 msgr2=0x555c7f2a00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 -- 172.21.15.28:0/3419769717 shutdown_connections 2024-09-10T20:45:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 -- 172.21.15.28:0/3419769717 wait complete. 2024-09-10T20:45:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 -- start start 2024-09-10T20:45:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.686+0000 7fec80409740 1 Processor -- start 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f2be2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2bf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec80409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2bf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec80409740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c7ef72620 con 0x555c7f2beb60 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x555c7f1a75f0 con 0x555c7f2e5df0 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x555c7f29a850 con 0x555c7f2c61b0 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec80409740 10 client.0 _ll_register_callbacks cb 0x555c7f1ab0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2bf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38278/0 (socket says 172.21.15.28:38278) 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.687+0000 7fec777fe640 1 -- 172.21.15.28:0/3869250191 learned_addr learned my addr 172.21.15.28:0/3869250191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec76ffd640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f2be2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec767fc640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2bf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec777fe640 1 -- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 msgr2=0x555c7f2be2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec777fe640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f2be2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec777fe640 1 -- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 msgr2=0x555c7f2bf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec777fe640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2bf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec777fe640 1 -- 172.21.15.28:0/3869250191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c7efda6b0 con 0x555c7f2beb60 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec76ffd640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f2be2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec767fc640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2bf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec777fe640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2bf930 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fec70098e50 tx=0x7fec7005f0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6f8de303da8a69 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec55ffb640 1 -- 172.21.15.28:0/3869250191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70062010 con 0x555c7f2beb60 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec55ffb640 1 -- 172.21.15.28:0/3869250191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7fec7005e500 con 0x555c7f2beb60 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec68037040 con 0x555c7f2beb60 2024-09-10T20:45:01.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec55ffb640 1 -- 172.21.15.28:0/3869250191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fec70062bb0 con 0x555c7f2beb60 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec80409740 10 client.8472 fetch_fsmap learned FSMap version 96 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x555c7f2c8db0 con 0x555c7f2beb60 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec55ffb640 1 -- 172.21.15.28:0/3869250191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec68037040 con 0x555c7f2beb60 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec55ffb640 1 -- 172.21.15.28:0/3869250191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 96) ==== 3502+0+0 (secure 0 0 0) 0x7fec7005e700 con 0x555c7f2beb60 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec80409740 10 client.8472 fetch_fsmap finished waiting for FSMap version 96 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.688+0000 7fec80409740 10 client.8472 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.689+0000 7fec80409740 20 client.8472 populate_metadata read hostname 'smithi028' 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.689+0000 7fec80409740 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c7f10 0x555c7f2c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.689+0000 7fec80409740 4 client.8472 mds_command: new command op to 8346 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.689+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 --> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c7f214770 con 0x555c7f2c7f10 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.689+0000 7fec76ffd640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c7f10 0x555c7f2c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.689+0000 7fec55ffb640 10 client.8472 ms_handle_connect on v2:172.21.15.159:6800/103941194 2024-09-10T20:45:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.689+0000 7fec76ffd640 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c7f10 0x555c7f2c82f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f7c4b0a5457137 server_cookie=6f8900ded90e181d in_seq=0 out_seq=0 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec55ffb640 1 -- 172.21.15.28:0/3869250191 <== mds.0 v2:172.21.15.159:6800/103941194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555c7f214770 con 0x555c7f2c7f10 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec55ffb640 10 client.8472 handle_command_reply: tid=0 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 client.8472 shutdown 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 msgr2=0x555c7f2bf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2bf930 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fec70098e50 tx=0x7fec7005f0d0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 shutdown_connections 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c7f10 0x555c7f2c82f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2bf930 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f2bf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec55ffb640 0 client.8472 ms_handle_reset on v2:172.21.15.159:6800/103941194 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 --2- 172.21.15.28:0/3869250191 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f2be2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 >> 172.21.15.28:0/3869250191 conn(0x555c7f29b5f0 msgr2=0x555c7f2a0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 shutdown_connections 2024-09-10T20:45:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.690+0000 7fec80409740 1 -- 172.21.15.28:0/3869250191 wait complete. 2024-09-10T20:45:01.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.691+0000 7fec80409740 20 client.8472 trim_cache size 0 max 16384 2024-09-10T20:45:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.709+0000 7fec80409740 1 Processor -- start 2024-09-10T20:45:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.709+0000 7fec80409740 1 -- start start 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2beb60 0x555c7f29b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec80409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2e5df0 0x555c7f2dcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec80409740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec78200890 con 0x555c7f2c61b0 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fec7809d850 con 0x555c7f2beb60 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fec780b1610 con 0x555c7f2e5df0 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38284/0 (socket says 172.21.15.28:38284) 2024-09-10T20:45:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 -- 172.21.15.28:0/4106182679 learned_addr learned my addr 172.21.15.28:0/4106182679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec76ffd640 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2beb60 0x555c7f29b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 -- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2e5df0 msgr2=0x555c7f2dcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec777fe640 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2e5df0 0x555c7f2dcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2e5df0 0x555c7f2dcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 -- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2beb60 msgr2=0x555c7f29b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2beb60 0x555c7f29b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 -- 172.21.15.28:0/4106182679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c7f2bfd00 con 0x555c7f2c61b0 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec76ffd640 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2beb60 0x555c7f29b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec777fe640 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2e5df0 0x555c7f2dcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.710+0000 7fec767fc640 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29bf10 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fec78077490 tx=0x7fec6c0011c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f5fb6986c46bb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec57fff640 1 -- 172.21.15.28:0/4106182679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c013cd0 con 0x555c7f2c61b0 2024-09-10T20:45:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec57fff640 1 -- 172.21.15.28:0/4106182679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec6c0103f0 con 0x555c7f2c61b0 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec57fff640 1 -- 172.21.15.28:0/4106182679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c010840 con 0x555c7f2c61b0 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 -- 172.21.15.28:0/4106182679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 msgr2=0x555c7f29bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29bf10 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fec78077490 tx=0x7fec6c0011c0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 -- 172.21.15.28:0/4106182679 shutdown_connections 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2e5df0 0x555c7f2dcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2c61b0 0x555c7f29bf10 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 --2- 172.21.15.28:0/4106182679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2beb60 0x555c7f29b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 -- 172.21.15.28:0/4106182679 >> 172.21.15.28:0/4106182679 conn(0x555c7f29b5f0 msgr2=0x555c7f2e61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 -- 172.21.15.28:0/4106182679 shutdown_connections 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.711+0000 7fec80409740 1 -- 172.21.15.28:0/4106182679 wait complete. 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.712+0000 7fec80409740 1 -- start start 2024-09-10T20:45:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.712+0000 7fec80409740 1 Processor -- start 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2f4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f369ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f3651b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c7f2c1c60 con 0x555c7f2beb60 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x555c7f2be880 con 0x555c7f2e5df0 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x555c7f2f8570 con 0x555c7f2c61b0 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 10 client.0 _ll_register_callbacks cb 0x7fec70004b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2f4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38288/0 (socket says 172.21.15.28:38288) 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 -- 172.21.15.28:0/2389327073 learned_addr learned my addr 172.21.15.28:0/2389327073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec777fe640 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f3651b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec767fc640 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f369ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 -- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 msgr2=0x555c7f369ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f369ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 -- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 msgr2=0x555c7f3651b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f3651b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 -- 172.21.15.28:0/2389327073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c7f2dd780 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec76ffd640 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2f4590 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fec68005ad0 tx=0x7fec68032ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dcbffe3c0ec323 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec55ffb640 1 -- 172.21.15.28:0/2389327073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6803b980 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec55ffb640 1 -- 172.21.15.28:0/2389327073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7fec6803bb20 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec6c0103f0 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec55ffb640 1 -- 172.21.15.28:0/2389327073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fec6803bd20 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 10 client.8478 fetch_fsmap learned FSMap version 96 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.713+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x555c7f3656f0 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec55ffb640 1 -- 172.21.15.28:0/2389327073 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec6c0103f0 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec55ffb640 1 -- 172.21.15.28:0/2389327073 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 96) ==== 3502+0+0 (secure 0 0 0) 0x7fec68061450 con 0x555c7f2beb60 2024-09-10T20:45:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec80409740 10 client.8478 fetch_fsmap finished waiting for FSMap version 96 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec80409740 10 client.8478 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec80409740 20 client.8478 populate_metadata read hostname 'smithi028' 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec80409740 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c4d60 0x555c7f2c5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec80409740 4 client.8478 mds_command: new command op to 8346 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 --> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] -- command(tid 0: {"prefix": "status"}) -- 0x555c7f210220 con 0x555c7f2c4d60 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec767fc640 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c4d60 0x555c7f2c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec767fc640 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c4d60 0x555c7f2c5140 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66c9d020fc0d503 server_cookie=dd328e44cf6de254 in_seq=0 out_seq=0 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.714+0000 7fec55ffb640 10 client.8478 ms_handle_connect on v2:172.21.15.159:6800/103941194 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec55ffb640 1 -- 172.21.15.28:0/2389327073 <== mds.0 v2:172.21.15.159:6800/103941194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x555c7f210220 con 0x555c7f2c4d60 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec55ffb640 10 client.8478 handle_command_reply: tid=0 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 client.8478 shutdown 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 msgr2=0x555c7f2f4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2f4590 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fec68005ad0 tx=0x7fec68032ba0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 shutdown_connections 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:6800/103941194,v1:172.21.15.159:6801/103941194] conn(0x555c7f2c4d60 0x555c7f2c5140 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x555c7f2e5df0 0x555c7f3651b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x555c7f2c61b0 0x555c7f369ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec55ffb640 0 client.8478 ms_handle_reset on v2:172.21.15.159:6800/103941194 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 --2- 172.21.15.28:0/2389327073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c7f2beb60 0x555c7f2f4590 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 >> 172.21.15.28:0/2389327073 conn(0x555c7f29b5f0 msgr2=0x555c7f2bf2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 shutdown_connections 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 1 -- 172.21.15.28:0/2389327073 wait complete. 2024-09-10T20:45:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec80409740 20 client.8478 trim_cache size 0 max 16384 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8346, 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.730216854, 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 96, 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 138, 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 138, 2024-09-10T20:45:01.730 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.40837702 2024-09-10T20:45:01.731 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec64078630 msgr2=0x7fec6407aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec64078630 0x7fec6407aaf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec68036000 tx=0x7fec6803d040 comp rx=0 tx=0).stop 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec78152750 msgr2=0x7fec7815d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec78152750 0x7fec7815d3e0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fec780d0bb0 tx=0x7fec70001a70 comp rx=0 tx=0).stop 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 shutdown_connections 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fec64078630 0x7fec6407aaf0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec78152750 0x7fec7815d3e0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec78151aa0 0x7fec78152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.715+0000 7fec7d55d640 1 --2- 172.21.15.28:0/91894144 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec60003a50 0x7fec781f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.716+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 >> 172.21.15.28:0/91894144 conn(0x7fec7806cf00 msgr2=0x7fec780acfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.716+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 shutdown_connections 2024-09-10T20:45:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.716+0000 7fec7d55d640 1 -- 172.21.15.28:0/91894144 wait complete. 2024-09-10T20:45:01.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:45:01.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.835+0000 7fb921018640 1 -- 172.21.15.28:0/3989656805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb908003a50 msgr2=0x7fb908003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.835+0000 7fb921018640 1 --2- 172.21.15.28:0/3989656805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb908003a50 0x7fb908003e70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00c310 tx=0x7fb908005830 comp rx=0 tx=0).stop 2024-09-10T20:45:01.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.835+0000 7fb921018640 1 -- 172.21.15.28:0/3989656805 shutdown_connections 2024-09-10T20:45:01.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.835+0000 7fb921018640 1 --2- 172.21.15.28:0/3989656805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb908003a50 0x7fb908003e70 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.835+0000 7fb921018640 1 -- 172.21.15.28:0/3989656805 >> 172.21.15.28:0/3989656805 conn(0x7fb91c06dbf0 msgr2=0x7fb91c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:01.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.835+0000 7fb921018640 1 -- 172.21.15.28:0/3989656805 shutdown_connections 2024-09-10T20:45:01.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.835+0000 7fb921018640 1 -- 172.21.15.28:0/3989656805 wait complete. 2024-09-10T20:45:01.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.836+0000 7fb921018640 1 Processor -- start 2024-09-10T20:45:01.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.836+0000 7fb921018640 1 -- start start 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb921018640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb908003a50 0x7fb91c1442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb921018640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c0cd320 0x7fb91c146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb921018640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb91c0cd760 0x7fb91c146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb921018640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c15e6e0 con 0x7fb91c0cd320 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb921018640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c0b54a0 con 0x7fb908003a50 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb921018640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb91c15e860 con 0x7fb91c0cd760 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c0cd320 0x7fb91c146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c0cd320 0x7fb91c146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38300/0 (socket says 172.21.15.28:38300) 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 -- 172.21.15.28:0/1169926061 learned_addr learned my addr 172.21.15.28:0/1169926061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb91ad9d640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb908003a50 0x7fb91c1442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb91b59e640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb91c0cd760 0x7fb91c146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 -- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb91c0cd760 msgr2=0x7fb91c146fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb91c0cd760 0x7fb91c146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 -- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb908003a50 msgr2=0x7fb91c1442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb908003a50 0x7fb91c1442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb913fff640 1 -- 172.21.15.28:0/1169926061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91c0afac0 con 0x7fb91c0cd320 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.837+0000 7fb91ad9d640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb908003a50 0x7fb91c1442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.838+0000 7fb913fff640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c0cd320 0x7fb91c146aa0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb904010ad0 tx=0x7fb904010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.838+0000 7fb9137fe640 1 -- 172.21.15.28:0/1169926061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90401a470 con 0x7fb91c0cd320 2024-09-10T20:45:01.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.838+0000 7fb9137fe640 1 -- 172.21.15.28:0/1169926061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb904015070 con 0x7fb91c0cd320 2024-09-10T20:45:01.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.838+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91c0afda0 con 0x7fb91c0cd320 2024-09-10T20:45:01.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.838+0000 7fb9137fe640 1 -- 172.21.15.28:0/1169926061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90401a870 con 0x7fb91c0cd320 2024-09-10T20:45:01.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.838+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb91c144a10 con 0x7fb91c0cd320 2024-09-10T20:45:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.839+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91c0d1220 con 0x7fb91c0cd320 2024-09-10T20:45:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.839+0000 7fb9137fe640 1 -- 172.21.15.28:0/1169926061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fb904022030 con 0x7fb91c0cd320 2024-09-10T20:45:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.840+0000 7fb9137fe640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8fc078630 0x7fb8fc07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.840+0000 7fb9137fe640 1 -- 172.21.15.28:0/1169926061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7fb90409b640 con 0x7fb91c0cd320 2024-09-10T20:45:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.840+0000 7fb91ad9d640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8fc078630 0x7fb8fc07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.841+0000 7fb91ad9d640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8fc078630 0x7fb8fc07aaf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb90c033000 tx=0x7fb91c15f420 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:01.844+0000 7fb9137fe640 1 -- 172.21.15.28:0/1169926061 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb904063660 con 0x7fb91c0cd320 2024-09-10T20:45:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.067+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb91c0d0a60 con 0x7fb91c0cd320 2024-09-10T20:45:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.068+0000 7fb9137fe640 1 -- 172.21.15.28:0/1169926061 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v96) ==== 53+0+73 (secure 0 0 0) 0x7fb904062f60 con 0x7fb91c0cd320 2024-09-10T20:45:02.072 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8fc078630 msgr2=0x7fb8fc07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8fc078630 0x7fb8fc07aaf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb90c033000 tx=0x7fb91c15f420 comp rx=0 tx=0).stop 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c0cd320 msgr2=0x7fb91c146aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c0cd320 0x7fb91c146aa0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb904010ad0 tx=0x7fb904010f90 comp rx=0 tx=0).stop 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 shutdown_connections 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb8fc078630 0x7fb8fc07aaf0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb91c0cd760 0x7fb91c146fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c0cd320 0x7fb91c146aa0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 --2- 172.21.15.28:0/1169926061 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb908003a50 0x7fb91c1442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 >> 172.21.15.28:0/1169926061 conn(0x7fb91c06dbf0 msgr2=0x7fb91c15db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 shutdown_connections 2024-09-10T20:45:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.070+0000 7fb921018640 1 -- 172.21.15.28:0/1169926061 wait complete. 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:45:02.085 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:45:02.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:02.086 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:45:02.102 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:45:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:45:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:45:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42329sec preferred_lft 42329sec 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:02.104 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:02.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:02.105 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:45:02.167 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:45:02.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:02.169 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:45:02.225 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:45:02.226 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:45:02.226 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:45:02.226 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:45:02.226 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:45:02.227 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:45:02.227 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:45:02.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:45:02.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:45:02.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:45:02.227 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:45:02.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:45:02.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:02.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:02.356 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:45:02.373 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:45:02.373 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:45:02.373 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:45:02.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:02.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:45:02.440 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:02.440 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:02.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:45:02.484 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:45:02.497 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:45:02.497 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:02.497 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:02.584 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:45:02.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:02.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:02.670 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:02.670 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:45:02.670 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:45:02.678 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:02.678 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:45:02.679 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:45:02.679 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:02.679 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:45:02.684 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:45:02.685 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:45:02.685 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:45:02.685 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:45:02.685 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:45:02.685 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:45:02.687 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:45:02.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:02.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:02.713 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:45:02.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:45:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:02.818 INFO:teuthology.orchestra.run.smithi028.stdout:client8487 2024-09-10T20:45:02.818 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:02.818 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:02.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 -- 172.21.15.28:0/747603128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 msgr2=0x7faa640d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/747603128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640d0300 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7faa50009830 tx=0x7faa5002a510 comp rx=0 tx=0).stop 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 -- 172.21.15.28:0/747603128 shutdown_connections 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/747603128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640d0300 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/747603128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa6415bf30 0x7faa640d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 -- 172.21.15.28:0/747603128 >> 172.21.15.28:0/747603128 conn(0x7faa640d1e40 msgr2=0x7faa640d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 -- 172.21.15.28:0/747603128 shutdown_connections 2024-09-10T20:45:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.976+0000 7faa6ac8c640 1 -- 172.21.15.28:0/747603128 wait complete. 2024-09-10T20:45:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.977+0000 7faa6ac8c640 1 Processor -- start 2024-09-10T20:45:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.977+0000 7faa6ac8c640 1 -- start start 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa6ac8c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa640ce5f0 0x7faa641e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa6ac8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa6ac8c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa6415bf30 0x7faa64148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa6ac8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa640d0cc0 con 0x7faa640cfec0 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa6ac8c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faa640d0b40 con 0x7faa640ce5f0 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa6ac8c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faa640cf1b0 con 0x7faa6415bf30 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38314/0 (socket says 172.21.15.28:38314) 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 -- 172.21.15.28:0/1778073847 learned_addr learned my addr 172.21.15.28:0/1778073847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa68a28640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa640ce5f0 0x7faa641e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa69229640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa6415bf30 0x7faa64148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 -- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa6415bf30 msgr2=0x7faa64148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa6415bf30 0x7faa64148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 -- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa640ce5f0 msgr2=0x7faa641e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa640ce5f0 0x7faa641e56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 -- 172.21.15.28:0/1778073847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa6414efd0 con 0x7faa640cfec0 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa63fff640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640c8790 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7faa50009950 tx=0x7faa50032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.978+0000 7faa617fa640 1 -- 172.21.15.28:0/1778073847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa50040070 con 0x7faa640cfec0 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.979+0000 7faa617fa640 1 -- 172.21.15.28:0/1778073847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa50003af0 con 0x7faa640cfec0 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.979+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa641e5ca0 con 0x7faa640cfec0 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.979+0000 7faa617fa640 1 -- 172.21.15.28:0/1778073847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa50045460 con 0x7faa640cfec0 2024-09-10T20:45:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.979+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6406eac0 con 0x7faa640cfec0 2024-09-10T20:45:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.980+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa6406e600 con 0x7faa640cfec0 2024-09-10T20:45:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.980+0000 7faa617fa640 1 -- 172.21.15.28:0/1778073847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7faa5004d030 con 0x7faa640cfec0 2024-09-10T20:45:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.981+0000 7faa617fa640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa48078660 0x7faa4807ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.981+0000 7faa617fa640 1 -- 172.21.15.28:0/1778073847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7faa500bc1a0 con 0x7faa640cfec0 2024-09-10T20:45:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.981+0000 7faa68a28640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa48078660 0x7faa4807ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.981+0000 7faa68a28640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa48078660 0x7faa4807ab20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faa641f7f00 tx=0x7faa58008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:02.985+0000 7faa617fa640 1 -- 172.21.15.28:0/1778073847 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faa6406e600 con 0x7faa640cfec0 2024-09-10T20:45:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.210+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faa640d0300 con 0x7faa640cfec0 2024-09-10T20:45:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.210+0000 7faa617fa640 1 -- 172.21.15.28:0/1778073847 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v96) ==== 53+0+73 (secure 0 0 0) 0x7faa50084150 con 0x7faa640cfec0 2024-09-10T20:45:03.214 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:45:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa48078660 msgr2=0x7faa4807ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa48078660 0x7faa4807ab20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faa641f7f00 tx=0x7faa58008040 comp rx=0 tx=0).stop 2024-09-10T20:45:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 msgr2=0x7faa640c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640c8790 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7faa50009950 tx=0x7faa50032d50 comp rx=0 tx=0).stop 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 shutdown_connections 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa48078660 0x7faa4807ab20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa6415bf30 0x7faa64148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa640cfec0 0x7faa640c8790 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 --2- 172.21.15.28:0/1778073847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa640ce5f0 0x7faa641e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 >> 172.21.15.28:0/1778073847 conn(0x7faa640d1e40 msgr2=0x7faa640b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 shutdown_connections 2024-09-10T20:45:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:03.212+0000 7faa6ac8c640 1 -- 172.21.15.28:0/1778073847 wait complete. 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:45:03.227 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:45:03.228 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:45:03.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:03.228 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:45:03.247 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:45:03.247 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:45:03.247 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:03.248 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42346sec preferred_lft 42346sec 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:45:03.249 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:45:03.250 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:03.250 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:03.250 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:45:03.250 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:45:03.250 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:03.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:03.250 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:45:03.316 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:45:03.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:03.318 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:45:03.378 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:45:03.379 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:45:03.379 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:45:03.379 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:45:03.379 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:45:03.379 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:45:03.379 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:45:03.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:45:03.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:45:03.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:45:03.380 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:45:03.473 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:45:03.507 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:03.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:03.513 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:45:03.529 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:45:03.529 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:45:03.529 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:45:03.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:03.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:45:03.595 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:03.595 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:03.651 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:45:03.652 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:03.652 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:03.706 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:45:03.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:03.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:03.790 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:03.790 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:45:03.790 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:45:03.799 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:03.799 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:45:03.799 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:45:03.799 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:03.799 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:03.803 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:03.804 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:45:03.804 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:45:03.804 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:03.804 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:03.804 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:45:03.804 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:45:03.804 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:45:03.805 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:45:03.805 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:45:03.805 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:45:03.805 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:45:03.805 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:45:03.806 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:45:03.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:03.806 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:03.871 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:45:03.961 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:45:03.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:03.968 INFO:teuthology.orchestra.run.smithi159.stdout:client6893 2024-09-10T20:45:03.968 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:03.969 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:03.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:45:04.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.106+0000 7f66ecdc2640 1 -- 172.21.15.28:0/1438230199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d0003a00 msgr2=0x7f66d0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:04.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.106+0000 7f66ecdc2640 1 --2- 172.21.15.28:0/1438230199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d0003a00 0x7f66d0003e20 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009f20 tx=0x7f66d0005830 comp rx=0 tx=0).stop 2024-09-10T20:45:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.106+0000 7f66e6d9d640 1 -- 172.21.15.28:0/1438230199 reap_dead start 2024-09-10T20:45:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.106+0000 7f66ecdc2640 1 -- 172.21.15.28:0/1438230199 shutdown_connections 2024-09-10T20:45:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.106+0000 7f66ecdc2640 1 -- 172.21.15.28:0/1438230199 >> 172.21.15.28:0/1438230199 conn(0x7f66e80d3f90 msgr2=0x7f66e80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.106+0000 7f66ecdc2640 1 -- 172.21.15.28:0/1438230199 shutdown_connections 2024-09-10T20:45:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.106+0000 7f66ecdc2640 1 -- 172.21.15.28:0/1438230199 wait complete. 2024-09-10T20:45:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 Processor -- start 2024-09-10T20:45:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 -- start start 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66d0003a00 0x7f66e81542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e81f4e00 0x7f66e8154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66e81580f0 0x7f66e8158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66e80d1c60 con 0x7f66e81f4e00 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f66e81626e0 con 0x7f66d0003a00 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.108+0000 7f66ecdc2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f66e80d1de0 con 0x7f66e81580f0 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e81f4e00 0x7f66e8154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e81f4e00 0x7f66e8154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38328/0 (socket says 172.21.15.28:38328) 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 -- 172.21.15.28:0/799701063 learned_addr learned my addr 172.21.15.28:0/799701063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e659c640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66d0003a00 0x7f66e81542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e6d9d640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66e81580f0 0x7f66e8158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 -- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66e81580f0 msgr2=0x7f66e8158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66e81580f0 0x7f66e8158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 -- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66d0003a00 msgr2=0x7f66e81542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66d0003a00 0x7f66e81542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 -- 172.21.15.28:0/799701063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e8155a80 con 0x7f66e81f4e00 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e659c640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66d0003a00 0x7f66e81542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e6d9d640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66e81580f0 0x7f66e8158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66e5d9b640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e81f4e00 0x7f66e8154b70 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0066e0 tx=0x7f66dc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66d6ffd640 1 -- 172.21.15.28:0/799701063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc015070 con 0x7f66e81f4e00 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66d6ffd640 1 -- 172.21.15.28:0/799701063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66dc011900 con 0x7f66e81f4e00 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66d6ffd640 1 -- 172.21.15.28:0/799701063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc011d50 con 0x7f66e81f4e00 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e81554c0 con 0x7f66e81f4e00 2024-09-10T20:45:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.109+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e8155fc0 con 0x7f66e81f4e00 2024-09-10T20:45:04.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.110+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e815d310 con 0x7f66e81f4e00 2024-09-10T20:45:04.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.111+0000 7f66d6ffd640 1 -- 172.21.15.28:0/799701063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f66dc010420 con 0x7f66e81f4e00 2024-09-10T20:45:04.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.111+0000 7f66d6ffd640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66c8078630 0x7f66c807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.112+0000 7f66d6ffd640 1 -- 172.21.15.28:0/799701063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f66dc09b800 con 0x7f66e81f4e00 2024-09-10T20:45:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.112+0000 7f66e659c640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66c8078630 0x7f66c807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.112+0000 7f66e659c640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66c8078630 0x7f66c807aaf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f66d8036000 tx=0x7f66d802d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:04.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.116+0000 7f66d6ffd640 1 -- 172.21.15.28:0/799701063 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f66dc0637c0 con 0x7f66e81f4e00 2024-09-10T20:45:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:04.339+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f66e815cd50 con 0x7f66e81f4e00 2024-09-10T20:45:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.028+0000 7f66d6ffd640 1 -- 172.21.15.28:0/799701063 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v97) ==== 136+0+0 (secure 0 0 0) 0x7f66dc0630c0 con 0x7f66e81f4e00 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66c8078630 msgr2=0x7f66c807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66c8078630 0x7f66c807aaf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f66d8036000 tx=0x7f66d802d040 comp rx=0 tx=0).stop 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e81f4e00 msgr2=0x7f66e8154b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e81f4e00 0x7f66e8154b70 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0066e0 tx=0x7f66dc0039c0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 shutdown_connections 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f66c8078630 0x7f66c807aaf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f66e81580f0 0x7f66e8158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e81f4e00 0x7f66e8154b70 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 --2- 172.21.15.28:0/799701063 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f66d0003a00 0x7f66e81542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 >> 172.21.15.28:0/799701063 conn(0x7f66e80d3f90 msgr2=0x7f66e815ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 shutdown_connections 2024-09-10T20:45:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.031+0000 7f66ecdc2640 1 -- 172.21.15.28:0/799701063 wait complete. 2024-09-10T20:45:05.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:45:05.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.153+0000 7efe55d5d640 1 -- 172.21.15.28:0/1477969758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe38004800 msgr2=0x7efe38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.153+0000 7efe55d5d640 1 --2- 172.21.15.28:0/1477969758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe38004800 0x7efe38004c80 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7efe44006f70 tx=0x7efe38005830 comp rx=0 tx=0).stop 2024-09-10T20:45:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.153+0000 7efe55d5d640 1 -- 172.21.15.28:0/1477969758 shutdown_connections 2024-09-10T20:45:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.153+0000 7efe55d5d640 1 --2- 172.21.15.28:0/1477969758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe38004800 0x7efe38004c80 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.153+0000 7efe55d5d640 1 -- 172.21.15.28:0/1477969758 >> 172.21.15.28:0/1477969758 conn(0x7efe500d3fb0 msgr2=0x7efe500d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.153+0000 7efe55d5d640 1 -- 172.21.15.28:0/1477969758 shutdown_connections 2024-09-10T20:45:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.153+0000 7efe55d5d640 1 -- 172.21.15.28:0/1477969758 wait complete. 2024-09-10T20:45:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.154+0000 7efe55d5d640 1 Processor -- start 2024-09-10T20:45:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.154+0000 7efe55d5d640 1 -- start start 2024-09-10T20:45:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe55d5d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe38004800 0x7efe5015f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe55d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe50160d20 0x7efe5014fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe55d5d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe500b46f0 0x7efe501f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe55d5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe500d2d10 con 0x7efe50160d20 2024-09-10T20:45:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe55d5d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe500d2b90 con 0x7efe38004800 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe55d5d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe500d2e90 con 0x7efe500b46f0 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe50160d20 0x7efe5014fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe50160d20 0x7efe5014fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38338/0 (socket says 172.21.15.28:38338) 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe4effd640 1 -- 172.21.15.28:0/3552954481 learned_addr learned my addr 172.21.15.28:0/3552954481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.155+0000 7efe4f7fe640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe38004800 0x7efe5015f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4ffff640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe500b46f0 0x7efe501f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4effd640 1 -- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe500b46f0 msgr2=0x7efe501f48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4effd640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe500b46f0 0x7efe501f48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4effd640 1 -- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe38004800 msgr2=0x7efe5015f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4effd640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe38004800 0x7efe5015f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4effd640 1 -- 172.21.15.28:0/3552954481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5015fc80 con 0x7efe50160d20 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4ffff640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe500b46f0 0x7efe501f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe4effd640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe50160d20 0x7efe5014fb10 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7efe44006f70 tx=0x7efe44001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe37fff640 1 -- 172.21.15.28:0/3552954481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44032030 con 0x7efe50160d20 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe37fff640 1 -- 172.21.15.28:0/3552954481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe44003af0 con 0x7efe50160d20 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe37fff640 1 -- 172.21.15.28:0/3552954481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44037450 con 0x7efe50160d20 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe5015ff60 con 0x7efe50160d20 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.156+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe50160520 con 0x7efe50160d20 2024-09-10T20:45:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.157+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe50161b50 con 0x7efe50160d20 2024-09-10T20:45:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.158+0000 7efe37fff640 1 -- 172.21.15.28:0/3552954481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7efe44039440 con 0x7efe50160d20 2024-09-10T20:45:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.158+0000 7efe37fff640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe3c078630 0x7efe3c07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.158+0000 7efe37fff640 1 -- 172.21.15.28:0/3552954481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7efe440bab30 con 0x7efe50160d20 2024-09-10T20:45:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.159+0000 7efe4f7fe640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe3c078630 0x7efe3c07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.159+0000 7efe4f7fe640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe3c078630 0x7efe3c07aaf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efe500d1410 tx=0x7efe40009500 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:05.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.163+0000 7efe37fff640 1 -- 172.21.15.28:0/3552954481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efe4403f070 con 0x7efe50160d20 2024-09-10T20:45:05.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.373+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7efe500b4f70 con 0x7efe3c078630 2024-09-10T20:45:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.375+0000 7efe37fff640 1 -- 172.21.15.28:0/3552954481 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7efe500b4f70 con 0x7efe3c078630 2024-09-10T20:45:05.379 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:45:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:45:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:05.379 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:45:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe3c078630 msgr2=0x7efe3c07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe3c078630 0x7efe3c07aaf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efe500d1410 tx=0x7efe40009500 comp rx=0 tx=0).stop 2024-09-10T20:45:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe50160d20 msgr2=0x7efe5014fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe50160d20 0x7efe5014fb10 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7efe44006f70 tx=0x7efe44001d40 comp rx=0 tx=0).stop 2024-09-10T20:45:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 shutdown_connections 2024-09-10T20:45:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe3c078630 0x7efe3c07aaf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe500b46f0 0x7efe501f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe50160d20 0x7efe5014fb10 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 --2- 172.21.15.28:0/3552954481 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe38004800 0x7efe5015f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 >> 172.21.15.28:0/3552954481 conn(0x7efe500d3fb0 msgr2=0x7efe500ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 shutdown_connections 2024-09-10T20:45:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.377+0000 7efe55d5d640 1 -- 172.21.15.28:0/3552954481 wait complete. 2024-09-10T20:45:05.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:45:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 -- 172.21.15.28:0/26196412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 msgr2=0x7fa3000b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 --2- 172.21.15.28:0/26196412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000b1830 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009830 tx=0x7fa2f402a480 comp rx=0 tx=0).stop 2024-09-10T20:45:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 -- 172.21.15.28:0/26196412 shutdown_connections 2024-09-10T20:45:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 --2- 172.21.15.28:0/26196412 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa300161b50 0x7fa3000b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 --2- 172.21.15.28:0/26196412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000b1830 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 -- 172.21.15.28:0/26196412 >> 172.21.15.28:0/26196412 conn(0x7fa3000d4070 msgr2=0x7fa3000d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 -- 172.21.15.28:0/26196412 shutdown_connections 2024-09-10T20:45:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.496+0000 7fa306cbd640 1 -- 172.21.15.28:0/26196412 wait complete. 2024-09-10T20:45:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.497+0000 7fa306cbd640 1 Processor -- start 2024-09-10T20:45:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.497+0000 7fa306cbd640 1 -- start start 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa306cbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa306cbd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa300161770 0x7fa3001f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa306cbd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa300161b50 0x7fa3000ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa306cbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3000b2dd0 con 0x7fa3000d27f0 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa306cbd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa3000d4e60 con 0x7fa300161b50 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa306cbd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3000b2f50 con 0x7fa300161770 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa304a59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa304a59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38358/0 (socket says 172.21.15.28:38358) 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa304a59640 1 -- 172.21.15.28:0/832181728 learned_addr learned my addr 172.21.15.28:0/832181728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa2fffff640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa300161770 0x7fa3001f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.498+0000 7fa30525a640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa300161b50 0x7fa3000ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa304a59640 1 -- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa300161770 msgr2=0x7fa3001f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa304a59640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa300161770 0x7fa3001f5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa304a59640 1 -- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa300161b50 msgr2=0x7fa3000ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa304a59640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa300161b50 0x7fa3000ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa304a59640 1 -- 172.21.15.28:0/832181728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3001f63e0 con 0x7fa3000d27f0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa30525a640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa300161b50 0x7fa3000ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa304a59640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000cd380 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000c450 tx=0x7fa2f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa2fd7fa640 1 -- 172.21.15.28:0/832181728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0016020 con 0x7fa3000d27f0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa2fd7fa640 1 -- 172.21.15.28:0/832181728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2f0012920 con 0x7fa3000d27f0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa2fd7fa640 1 -- 172.21.15.28:0/832181728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0012d70 con 0x7fa3000d27f0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3001f66c0 con 0x7fa3000d27f0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.499+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3001f6c80 con 0x7fa3000d27f0 2024-09-10T20:45:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.500+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3000d20e0 con 0x7fa3000d27f0 2024-09-10T20:45:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.501+0000 7fa2fd7fa640 1 -- 172.21.15.28:0/832181728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7fa2f001b400 con 0x7fa3000d27f0 2024-09-10T20:45:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.501+0000 7fa2fd7fa640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa2ec078630 0x7fa2ec07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.502+0000 7fa2fffff640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa2ec078630 0x7fa2ec07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.502+0000 7fa2fd7fa640 1 -- 172.21.15.28:0/832181728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7fa2f009adc0 con 0x7fa3000d27f0 2024-09-10T20:45:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.502+0000 7fa2fffff640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa2ec078630 0x7fa2ec07aaf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4034000 tx=0x7fa2f4004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.506+0000 7fa2fd7fa640 1 -- 172.21.15.28:0/832181728 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa2f0020020 con 0x7fa3000d27f0 2024-09-10T20:45:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.690+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fa300160e90 con 0x7fa3000d27f0 2024-09-10T20:45:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.690+0000 7fa2fd7fa640 1 -- 172.21.15.28:0/832181728 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7fa2f0062d30 con 0x7fa3000d27f0 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa2ec078630 msgr2=0x7fa2ec07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa2ec078630 0x7fa2ec07aaf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4034000 tx=0x7fa2f4004730 comp rx=0 tx=0).stop 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 msgr2=0x7fa3000cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000cd380 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000c450 tx=0x7fa2f000c910 comp rx=0 tx=0).stop 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 shutdown_connections 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa2ec078630 0x7fa2ec07aaf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa300161b50 0x7fa3000ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa300161770 0x7fa3001f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 --2- 172.21.15.28:0/832181728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3000d27f0 0x7fa3000cd380 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 >> 172.21.15.28:0/832181728 conn(0x7fa3000d4070 msgr2=0x7fa3000ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 shutdown_connections 2024-09-10T20:45:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.692+0000 7fa306cbd640 1 -- 172.21.15.28:0/832181728 wait complete. 2024-09-10T20:45:05.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:05.707 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvol49700) 2024-09-10T20:45:05.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:05.740 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp80420/subvol59601) 2024-09-10T20:45:05.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol49700 2024-09-10T20:45:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.954+0000 7ff59cf52640 1 -- 172.21.15.28:0/1254402810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598073ea0 msgr2=0x7ff5980706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.954+0000 7ff59cf52640 1 --2- 172.21.15.28:0/1254402810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598073ea0 0x7ff5980706e0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff598071b10 tx=0x7ff58002a480 comp rx=0 tx=0).stop 2024-09-10T20:45:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.954+0000 7ff59cf52640 1 -- 172.21.15.28:0/1254402810 shutdown_connections 2024-09-10T20:45:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.954+0000 7ff59cf52640 1 --2- 172.21.15.28:0/1254402810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598073ea0 0x7ff5980706e0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.954+0000 7ff59cf52640 1 -- 172.21.15.28:0/1254402810 >> 172.21.15.28:0/1254402810 conn(0x7ff5980d0d50 msgr2=0x7ff5980d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.954+0000 7ff59cf52640 1 -- 172.21.15.28:0/1254402810 shutdown_connections 2024-09-10T20:45:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.954+0000 7ff59cf52640 1 -- 172.21.15.28:0/1254402810 wait complete. 2024-09-10T20:45:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.955+0000 7ff59cf52640 1 Processor -- start 2024-09-10T20:45:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff59cf52640 1 -- start start 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff59cf52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff598073ea0 0x7ff5981f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff59cf52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598151db0 0x7ff5981f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff59cf52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59815bd80 0x7ff5981f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff59cf52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5980ccc40 con 0x7ff598151db0 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff59cf52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff5980ccac0 con 0x7ff598073ea0 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff59cf52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff5980ccdc0 con 0x7ff59815bd80 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff595d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598151db0 0x7ff5981f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff595d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598151db0 0x7ff5981f5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38384/0 (socket says 172.21.15.28:38384) 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.956+0000 7ff595d9b640 1 -- 172.21.15.28:0/487523629 learned_addr learned my addr 172.21.15.28:0/487523629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff59659c640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff598073ea0 0x7ff5981f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff596d9d640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59815bd80 0x7ff5981f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff595d9b640 1 -- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59815bd80 msgr2=0x7ff5981f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff595d9b640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59815bd80 0x7ff5981f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff595d9b640 1 -- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff598073ea0 msgr2=0x7ff5981f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff595d9b640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff598073ea0 0x7ff5981f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff595d9b640 1 -- 172.21.15.28:0/487523629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5981f4c00 con 0x7ff598151db0 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff595d9b640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598151db0 0x7ff5981f5690 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7ff580009920 tx=0x7ff580032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff58effd640 1 -- 172.21.15.28:0/487523629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580040070 con 0x7ff598151db0 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff58effd640 1 -- 172.21.15.28:0/487523629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff580005310 con 0x7ff598151db0 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff58effd640 1 -- 172.21.15.28:0/487523629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580003800 con 0x7ff598151db0 2024-09-10T20:45:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5981f5bd0 con 0x7ff598151db0 2024-09-10T20:45:05.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.957+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5981f8f30 con 0x7ff598151db0 2024-09-10T20:45:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.958+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5980706e0 con 0x7ff598151db0 2024-09-10T20:45:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.959+0000 7ff58effd640 1 -- 172.21.15.28:0/487523629 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7ff58004c030 con 0x7ff598151db0 2024-09-10T20:45:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.959+0000 7ff58effd640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff578078660 0x7ff57807ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.959+0000 7ff58effd640 1 -- 172.21.15.28:0/487523629 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7ff5800bbc00 con 0x7ff598151db0 2024-09-10T20:45:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.960+0000 7ff59659c640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff578078660 0x7ff57807ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.960+0000 7ff59659c640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff578078660 0x7ff57807ab20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff59815af20 tx=0x7ff590007e90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:05.964+0000 7ff58effd640 1 -- 172.21.15.28:0/487523629 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff5800835f0 con 0x7ff598151db0 2024-09-10T20:45:06.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.180+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol49700", "target": ["mon-mgr", ""]}) -- 0x7ff59815f820 con 0x7ff578078660 2024-09-10T20:45:06.189 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:06.185+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:06.189 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:06.185+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:06.189 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:06.185+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:06.189 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:06.185+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:06.190 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:06.185+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:06.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.213+0000 7ff58effd640 1 -- 172.21.15.28:0/487523629 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7ff59815f820 con 0x7ff578078660 2024-09-10T20:45:06.217 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol49700 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff578078660 msgr2=0x7ff57807ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff578078660 0x7ff57807ab20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff59815af20 tx=0x7ff590007e90 comp rx=0 tx=0).stop 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598151db0 msgr2=0x7ff5981f5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598151db0 0x7ff5981f5690 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7ff580009920 tx=0x7ff580032d50 comp rx=0 tx=0).stop 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 shutdown_connections 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff578078660 0x7ff57807ab20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff59815bd80 0x7ff5981f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff598151db0 0x7ff5981f5690 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 --2- 172.21.15.28:0/487523629 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff598073ea0 0x7ff5981f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 >> 172.21.15.28:0/487523629 conn(0x7ff5980d0d50 msgr2=0x7ff5980cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:06.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 shutdown_connections 2024-09-10T20:45:06.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.216+0000 7ff59cf52640 1 -- 172.21.15.28:0/487523629 wait complete. 2024-09-10T20:45:06.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol59601 subvol_grp80420 2024-09-10T20:45:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 -- 172.21.15.28:0/1652733762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 msgr2=0x7f19f80add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 --2- 172.21.15.28:0/1652733762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 0x7f19f80add70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f19f0005fd0 tx=0x7f19f002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 -- 172.21.15.28:0/1652733762 shutdown_connections 2024-09-10T20:45:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 --2- 172.21.15.28:0/1652733762 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 0x7f19f80add70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 --2- 172.21.15.28:0/1652733762 >> v2:172.21.15.159:3300/0 conn(0x7f19f80b5a80 0x7f19f8073700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 -- 172.21.15.28:0/1652733762 >> 172.21.15.28:0/1652733762 conn(0x7f19f80b2920 msgr2=0x7f19f80b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 -- 172.21.15.28:0/1652733762 shutdown_connections 2024-09-10T20:45:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.336+0000 7f19ff86a640 1 -- 172.21.15.28:0/1652733762 wait complete. 2024-09-10T20:45:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.337+0000 7f19ff86a640 1 Processor -- start 2024-09-10T20:45:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19ff86a640 1 -- start start 2024-09-10T20:45:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19ff86a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 0x7f19f81f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19ff86a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f80b5a80 0x7f19f81577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19ff86a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19f815ab00 0x7f19f81f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19ff86a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19f815aee0 con 0x7f19f815ab00 2024-09-10T20:45:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19ff86a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f19f80b4630 con 0x7f19f80b5a80 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19ff86a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f19f815b060 con 0x7f19f8073c40 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19fde07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19f815ab00 0x7f19f81f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19fde07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19f815ab00 0x7f19f81f5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38418/0 (socket says 172.21.15.28:38418) 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.338+0000 7f19fde07640 1 -- 172.21.15.28:0/2539541462 learned_addr learned my addr 172.21.15.28:0/2539541462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fd606640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 0x7f19f81f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fce05640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f80b5a80 0x7f19f81577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fde07640 1 -- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 msgr2=0x7f19f81f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fde07640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 0x7f19f81f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fde07640 1 -- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f80b5a80 msgr2=0x7f19f81577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fde07640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f80b5a80 0x7f19f81577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fde07640 1 -- 172.21.15.28:0/2539541462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f81f66a0 con 0x7f19f815ab00 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19fde07640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19f815ab00 0x7f19f81f5c00 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f19f0009cd0 tx=0x7f19f00316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19edffb640 1 -- 172.21.15.28:0/2539541462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f0040070 con 0x7f19f815ab00 2024-09-10T20:45:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19edffb640 1 -- 172.21.15.28:0/2539541462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19f0004040 con 0x7f19f815ab00 2024-09-10T20:45:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19edffb640 1 -- 172.21.15.28:0/2539541462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f0004470 con 0x7f19f815ab00 2024-09-10T20:45:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f81f6980 con 0x7f19f815ab00 2024-09-10T20:45:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.339+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f81f2590 con 0x7f19f815ab00 2024-09-10T20:45:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.340+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f80ae510 con 0x7f19f815ab00 2024-09-10T20:45:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.341+0000 7f19edffb640 1 -- 172.21.15.28:0/2539541462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f19f004d030 con 0x7f19f815ab00 2024-09-10T20:45:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.342+0000 7f19edffb640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f19dc078630 0x7f19dc07aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.342+0000 7f19edffb640 1 -- 172.21.15.28:0/2539541462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f19f00bbda0 con 0x7f19f815ab00 2024-09-10T20:45:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.342+0000 7f19fd606640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f19dc078630 0x7f19dc07aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.342+0000 7f19fd606640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f19dc078630 0x7f19dc07aaf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f19f4007fd0 tx=0x7f19f4001390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.346+0000 7f19edffb640 1 -- 172.21.15.28:0/2539541462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f19f0083f70 con 0x7f19f815ab00 2024-09-10T20:45:06.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.560+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol59601", "group_name": "subvol_grp80420", "target": ["mon-mgr", ""]}) -- 0x7f19f80b4e50 con 0x7f19dc078630 2024-09-10T20:45:06.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.578+0000 7f19edffb640 1 -- 172.21.15.28:0/2539541462 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f19f80b4e50 con 0x7f19dc078630 2024-09-10T20:45:06.583 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp80420/subvol59601 2024-09-10T20:45:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f19dc078630 msgr2=0x7f19dc07aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f19dc078630 0x7f19dc07aaf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f19f4007fd0 tx=0x7f19f4001390 comp rx=0 tx=0).stop 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19f815ab00 msgr2=0x7f19f81f5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19f815ab00 0x7f19f81f5c00 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f19f0009cd0 tx=0x7f19f00316f0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 shutdown_connections 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f19dc078630 0x7f19dc07aaf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19f815ab00 0x7f19f81f5c00 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19f80b5a80 0x7f19f81577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 --2- 172.21.15.28:0/2539541462 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19f8073c40 0x7f19f81f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 >> 172.21.15.28:0/2539541462 conn(0x7f19f80b2920 msgr2=0x7f19f815d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 shutdown_connections 2024-09-10T20:45:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.580+0000 7f19ff86a640 1 -- 172.21.15.28:0/2539541462 wait complete. 2024-09-10T20:45:06.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol49700 2024-09-10T20:45:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 -- 172.21.15.28:0/2349877901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680b3110 msgr2=0x7f30680cec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 --2- 172.21.15.28:0/2349877901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680b3110 0x7f30680cec90 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f3068073bc0 tx=0x7f305002a5f0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 -- 172.21.15.28:0/2349877901 shutdown_connections 2024-09-10T20:45:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 --2- 172.21.15.28:0/2349877901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680b3110 0x7f30680cec90 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 --2- 172.21.15.28:0/2349877901 >> v2:172.21.15.159:3300/0 conn(0x7f306815f580 0x7f30680ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 -- 172.21.15.28:0/2349877901 >> 172.21.15.28:0/2349877901 conn(0x7f30680d1f30 msgr2=0x7f30680d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 -- 172.21.15.28:0/2349877901 shutdown_connections 2024-09-10T20:45:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.700+0000 7f306d3e2640 1 -- 172.21.15.28:0/2349877901 wait complete. 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.701+0000 7f306d3e2640 1 Processor -- start 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.701+0000 7f306d3e2640 1 -- start start 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f306d3e2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30680b3110 0x7f30680c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f306d3e2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30680b4d60 0x7f30680c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f306d3e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306815f580 0x7f306815e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f306d3e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30680b2d00 con 0x7f306815f580 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f306d3e2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f30680d10c0 con 0x7f30680b4d60 2024-09-10T20:45:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f306d3e2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f30680cdb30 con 0x7f30680b3110 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306815f580 0x7f306815e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306815f580 0x7f306815e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38426/0 (socket says 172.21.15.28:38426) 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 -- 172.21.15.28:0/365101352 learned_addr learned my addr 172.21.15.28:0/365101352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f3066ffd640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30680b3110 0x7f30680c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f305ffff640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30680b4d60 0x7f30680c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 -- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30680b3110 msgr2=0x7f30680c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30680b3110 0x7f30680c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 -- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30680b4d60 msgr2=0x7f30680c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30680b4d60 0x7f30680c5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.702+0000 7f30677fe640 1 -- 172.21.15.28:0/365101352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30681f8720 con 0x7f306815f580 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.703+0000 7f30677fe640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306815f580 0x7f306815e210 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f306000c7a0 tx=0x7f306000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.703+0000 7f305f7fe640 1 -- 172.21.15.28:0/365101352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060003b70 con 0x7f306815f580 2024-09-10T20:45:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.703+0000 7f305f7fe640 1 -- 172.21.15.28:0/365101352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3060003d10 con 0x7f306815f580 2024-09-10T20:45:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.703+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30681f8a00 con 0x7f306815f580 2024-09-10T20:45:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.703+0000 7f305f7fe640 1 -- 172.21.15.28:0/365101352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306000aab0 con 0x7f306815f580 2024-09-10T20:45:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.703+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30681f8fc0 con 0x7f306815f580 2024-09-10T20:45:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.704+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f306806ad20 con 0x7f306815f580 2024-09-10T20:45:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.705+0000 7f305f7fe640 1 -- 172.21.15.28:0/365101352 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 101213+0+0 (secure 0 0 0) 0x7f306000ac50 con 0x7f306815f580 2024-09-10T20:45:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.705+0000 7f305f7fe640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3048078630 0x7f304807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.705+0000 7f305f7fe640 1 -- 172.21.15.28:0/365101352 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5737+0+0 (secure 0 0 0) 0x7f306009a6c0 con 0x7f306815f580 2024-09-10T20:45:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.705+0000 7f3066ffd640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3048078630 0x7f304807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.706+0000 7f3066ffd640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3048078630 0x7f304807aaf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f306806cfd0 tx=0x7f305800b2e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:06.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.709+0000 7f305f7fe640 1 -- 172.21.15.28:0/365101352 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f306806ad20 con 0x7f306815f580 2024-09-10T20:45:06.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.919+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol49700", "target": ["mon-mgr", ""]}) -- 0x7f30680ce750 con 0x7f3048078630 2024-09-10T20:45:06.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.925+0000 7f305f7fe640 1 -- 172.21.15.28:0/365101352 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f30680ce750 con 0x7f3048078630 2024-09-10T20:45:06.930 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol49700 2024-09-10T20:45:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3048078630 msgr2=0x7f304807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3048078630 0x7f304807aaf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f306806cfd0 tx=0x7f305800b2e0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306815f580 msgr2=0x7f306815e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306815f580 0x7f306815e210 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f306000c7a0 tx=0x7f306000cc60 comp rx=0 tx=0).stop 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 shutdown_connections 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3048078630 0x7f304807aaf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306815f580 0x7f306815e210 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f30680b4d60 0x7f30680c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 --2- 172.21.15.28:0/365101352 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f30680b3110 0x7f30680c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 >> 172.21.15.28:0/365101352 conn(0x7f30680d1f30 msgr2=0x7f30680c5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 shutdown_connections 2024-09-10T20:45:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:06.927+0000 7f306d3e2640 1 -- 172.21.15.28:0/365101352 wait complete. 2024-09-10T20:45:06.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:06.943 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/7a666752a15ac0f5bd1a3f83e65a33a0.meta) 2024-09-10T20:45:11.319 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:45:11.319 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:45:11.320 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:45:11.320 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol49700 2024-09-10T20:45:11.320 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:45:11.320 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:11.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol59601 subvol_grp80420 2024-09-10T20:45:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.438+0000 7f7b855b4640 1 -- 172.21.15.28:0/3079233938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 msgr2=0x7f7b6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.438+0000 7f7b855b4640 1 --2- 172.21.15.28:0/3079233938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 0x7f7b6c0051c0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f7b70005fd0 tx=0x7f7b6c005830 comp rx=0 tx=0).stop 2024-09-10T20:45:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.438+0000 7f7b7f7fe640 1 -- 172.21.15.28:0/3079233938 reap_dead start 2024-09-10T20:45:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.438+0000 7f7b855b4640 1 -- 172.21.15.28:0/3079233938 shutdown_connections 2024-09-10T20:45:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.438+0000 7f7b855b4640 1 -- 172.21.15.28:0/3079233938 >> 172.21.15.28:0/3079233938 conn(0x7f7b800cfc80 msgr2=0x7f7b800d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.438+0000 7f7b855b4640 1 -- 172.21.15.28:0/3079233938 shutdown_connections 2024-09-10T20:45:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.438+0000 7f7b855b4640 1 -- 172.21.15.28:0/3079233938 wait complete. 2024-09-10T20:45:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.439+0000 7f7b855b4640 1 Processor -- start 2024-09-10T20:45:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.439+0000 7f7b855b4640 1 -- start start 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b855b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 0x7f7b8014a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b855b4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b800b37c0 0x7f7b800b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b855b4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b801f2cf0 0x7f7b801f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b855b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80074510 con 0x7f7b6c004d40 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b855b4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80074390 con 0x7f7b800b37c0 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b855b4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8015c350 con 0x7f7b801f2cf0 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 0x7f7b8014a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 0x7f7b8014a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38486/0 (socket says 172.21.15.28:38486) 2024-09-10T20:45:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 -- 172.21.15.28:0/2845760128 learned_addr learned my addr 172.21.15.28:0/2845760128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7f7fe640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b801f2cf0 0x7f7b801f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7e7fc640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b800b37c0 0x7f7b800b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 -- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b801f2cf0 msgr2=0x7f7b801f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b801f2cf0 0x7f7b801f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 -- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b800b37c0 msgr2=0x7f7b800b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b800b37c0 0x7f7b800b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.440+0000 7f7b7effd640 1 -- 172.21.15.28:0/2845760128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b801f61c0 con 0x7f7b6c004d40 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.441+0000 7f7b7effd640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 0x7f7b8014a7e0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800e650 tx=0x7f7b7800eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.441+0000 7f7b777fe640 1 -- 172.21.15.28:0/2845760128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78015020 con 0x7f7b6c004d40 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.441+0000 7f7b777fe640 1 -- 172.21.15.28:0/2845760128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b78004320 con 0x7f7b6c004d40 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.441+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b801f64a0 con 0x7f7b6c004d40 2024-09-10T20:45:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.441+0000 7f7b777fe640 1 -- 172.21.15.28:0/2845760128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7800ca00 con 0x7f7b6c004d40 2024-09-10T20:45:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.441+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b801f6a60 con 0x7f7b6c004d40 2024-09-10T20:45:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.442+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8015f850 con 0x7f7b6c004d40 2024-09-10T20:45:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.443+0000 7f7b777fe640 1 -- 172.21.15.28:0/2845760128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 101264+0+0 (secure 0 0 0) 0x7f7b7800cba0 con 0x7f7b6c004d40 2024-09-10T20:45:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.443+0000 7f7b777fe640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b600786f0 0x7f7b6007abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.443+0000 7f7b777fe640 1 -- 172.21.15.28:0/2845760128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f7b7809bac0 con 0x7f7b6c004d40 2024-09-10T20:45:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.443+0000 7f7b7e7fc640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b600786f0 0x7f7b6007abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.444+0000 7f7b7e7fc640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b600786f0 0x7f7b6007abb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7b80070510 tx=0x7f7b680069e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.448+0000 7f7b777fe640 1 -- 172.21.15.28:0/2845760128 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7b7801f020 con 0x7f7b6c004d40 2024-09-10T20:45:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.659+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol59601", "group_name": "subvol_grp80420", "target": ["mon-mgr", ""]}) -- 0x7f7b801f6c30 con 0x7f7b600786f0 2024-09-10T20:45:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.663+0000 7f7b777fe640 1 -- 172.21.15.28:0/2845760128 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7b801f6c30 con 0x7f7b600786f0 2024-09-10T20:45:11.668 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp80420/subvol59601 2024-09-10T20:45:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b600786f0 msgr2=0x7f7b6007abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b600786f0 0x7f7b6007abb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7b80070510 tx=0x7f7b680069e0 comp rx=0 tx=0).stop 2024-09-10T20:45:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 msgr2=0x7f7b8014a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 0x7f7b8014a7e0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800e650 tx=0x7f7b7800eb10 comp rx=0 tx=0).stop 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 shutdown_connections 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7b600786f0 0x7f7b6007abb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b801f2cf0 0x7f7b801f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b800b37c0 0x7f7b800b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 --2- 172.21.15.28:0/2845760128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b6c004d40 0x7f7b8014a7e0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 >> 172.21.15.28:0/2845760128 conn(0x7f7b800cfc80 msgr2=0x7f7b8015e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 shutdown_connections 2024-09-10T20:45:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:11.665+0000 7f7b855b4640 1 -- 172.21.15.28:0/2845760128 wait complete. 2024-09-10T20:45:11.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:11.682 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/61410e9e104285618e0fbc5f3c6ef8c0.meta) 2024-09-10T20:45:16.317 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:45:16.317 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:45:16.317 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:45:16.317 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/subvol_grp80420/subvol59601 2024-09-10T20:45:16.317 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:45:16.317 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:16.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol49700 2024-09-10T20:45:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.443+0000 7f373e02a640 1 -- 172.21.15.28:0/747568413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3720003c80 msgr2=0x7f3720004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.443+0000 7f373e02a640 1 --2- 172.21.15.28:0/747568413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3720003c80 0x7f3720004080 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3728001830 tx=0x7f3720005830 comp rx=0 tx=0).stop 2024-09-10T20:45:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.443+0000 7f373e02a640 1 -- 172.21.15.28:0/747568413 shutdown_connections 2024-09-10T20:45:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.443+0000 7f373e02a640 1 --2- 172.21.15.28:0/747568413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3720003c80 0x7f3720004080 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.443+0000 7f373e02a640 1 -- 172.21.15.28:0/747568413 >> 172.21.15.28:0/747568413 conn(0x7f373806cf00 msgr2=0x7f373806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.443+0000 7f373e02a640 1 -- 172.21.15.28:0/747568413 shutdown_connections 2024-09-10T20:45:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.443+0000 7f373e02a640 1 -- 172.21.15.28:0/747568413 wait complete. 2024-09-10T20:45:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.444+0000 7f373e02a640 1 Processor -- start 2024-09-10T20:45:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.444+0000 7f373e02a640 1 -- start start 2024-09-10T20:45:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f373e02a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3720003c80 0x7f37380b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f373e02a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37381e59b0 0x7f37381f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f373e02a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37380b1650 0x7f37380b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f373e02a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f373806f300 con 0x7f37380b1650 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f373e02a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f373806f180 con 0x7f3720003c80 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f373e02a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f373806f480 con 0x7f37381e59b0 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f3737fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37380b1650 0x7f37380b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f3737fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37380b1650 0x7f37380b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38506/0 (socket says 172.21.15.28:38506) 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f3737fff640 1 -- 172.21.15.28:0/4271817761 learned_addr learned my addr 172.21.15.28:0/4271817761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f37377fe640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3720003c80 0x7f37380b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.445+0000 7f3736ffd640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37381e59b0 0x7f37381f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f3737fff640 1 -- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37381e59b0 msgr2=0x7f37381f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f3737fff640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37381e59b0 0x7f37381f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f3737fff640 1 -- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3720003c80 msgr2=0x7f37380b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f3737fff640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3720003c80 0x7f37380b2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f3737fff640 1 -- 172.21.15.28:0/4271817761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37381f2fc0 con 0x7f37380b1650 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f37377fe640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3720003c80 0x7f37380b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f3736ffd640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37381e59b0 0x7f37381f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f3737fff640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37380b1650 0x7f37380b1ae0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f3738072bb0 tx=0x7f3730006420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f371ffff640 1 -- 172.21.15.28:0/4271817761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3730011870 con 0x7f37380b1650 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f371ffff640 1 -- 172.21.15.28:0/4271817761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3730017070 con 0x7f37380b1650 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37381f32a0 con 0x7f37380b1650 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f371ffff640 1 -- 172.21.15.28:0/4271817761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3730011de0 con 0x7f37380b1650 2024-09-10T20:45:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.446+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37381f3b60 con 0x7f37380b1650 2024-09-10T20:45:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.447+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37380d0b50 con 0x7f37380b1650 2024-09-10T20:45:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.448+0000 7f371ffff640 1 -- 172.21.15.28:0/4271817761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 101264+0+0 (secure 0 0 0) 0x7f373001c460 con 0x7f37380b1650 2024-09-10T20:45:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.448+0000 7f371ffff640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37240786f0 0x7f372407abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.449+0000 7f371ffff640 1 -- 172.21.15.28:0/4271817761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f373009b670 con 0x7f37380b1650 2024-09-10T20:45:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.449+0000 7f37377fe640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37240786f0 0x7f372407abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.449+0000 7f37377fe640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37240786f0 0x7f372407abb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3728032000 tx=0x7f3728035040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:16.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.453+0000 7f371ffff640 1 -- 172.21.15.28:0/4271817761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3730063290 con 0x7f37380b1650 2024-09-10T20:45:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.669+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol49700", "target": ["mon-mgr", ""]}) -- 0x7f37381f2d60 con 0x7f37240786f0 2024-09-10T20:45:16.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.684+0000 7f371ffff640 1 -- 172.21.15.28:0/4271817761 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3728006230 con 0x7f37240786f0 2024-09-10T20:45:16.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37240786f0 msgr2=0x7f372407abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37240786f0 0x7f372407abb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3728032000 tx=0x7f3728035040 comp rx=0 tx=0).stop 2024-09-10T20:45:16.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37380b1650 msgr2=0x7f37380b1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37380b1650 0x7f37380b1ae0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f3738072bb0 tx=0x7f3730006420 comp rx=0 tx=0).stop 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 shutdown_connections 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f37240786f0 0x7f372407abb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37380b1650 0x7f37380b1ae0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f37381e59b0 0x7f37381f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 --2- 172.21.15.28:0/4271817761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3720003c80 0x7f37380b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 >> 172.21.15.28:0/4271817761 conn(0x7f373806cf00 msgr2=0x7f37380b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 shutdown_connections 2024-09-10T20:45:16.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.686+0000 7f373e02a640 1 -- 172.21.15.28:0/4271817761 wait complete. 2024-09-10T20:45:16.692 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.687+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.692 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.687+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.692 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.687+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.692 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.687+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.692 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.687+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol59601 subvol_grp80420 2024-09-10T20:45:16.706 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.701+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.706 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.701+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.706 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.701+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.706 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.701+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.706 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:16.701+0000 7f93d0c02640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.809+0000 7f00ed033640 1 -- 172.21.15.28:0/4125501866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e80b4c20 msgr2=0x7f00e80d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.809+0000 7f00ed033640 1 --2- 172.21.15.28:0/4125501866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e80b4c20 0x7f00e80d4a80 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f00d8005fd0 tx=0x7f00d802a940 comp rx=0 tx=0).stop 2024-09-10T20:45:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.809+0000 7f00ed033640 1 -- 172.21.15.28:0/4125501866 shutdown_connections 2024-09-10T20:45:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.809+0000 7f00ed033640 1 --2- 172.21.15.28:0/4125501866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e80b4c20 0x7f00e80d4a80 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.809+0000 7f00ed033640 1 -- 172.21.15.28:0/4125501866 >> 172.21.15.28:0/4125501866 conn(0x7f00e80d3f90 msgr2=0x7f00e80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.809+0000 7f00ed033640 1 -- 172.21.15.28:0/4125501866 shutdown_connections 2024-09-10T20:45:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.809+0000 7f00ed033640 1 -- 172.21.15.28:0/4125501866 wait complete. 2024-09-10T20:45:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.810+0000 7f00ed033640 1 Processor -- start 2024-09-10T20:45:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00ed033640 1 -- start start 2024-09-10T20:45:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00ed033640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00e80b4c20 0x7f00e81e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00ed033640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00e8161850 0x7f00e815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00ed033640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e8162000 0x7f00e81ea460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00ed033640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00e80d24f0 con 0x7f00e8162000 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00ed033640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f00e80d2370 con 0x7f00e8161850 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00ed033640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f00e80d2670 con 0x7f00e80b4c20 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00e759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e8162000 0x7f00e81ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.811+0000 7f00e759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e8162000 0x7f00e81ea460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38560/0 (socket says 172.21.15.28:38560) 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e759e640 1 -- 172.21.15.28:0/2879906491 learned_addr learned my addr 172.21.15.28:0/2879906491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e6d9d640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00e80b4c20 0x7f00e81e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e659c640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00e8161850 0x7f00e815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e759e640 1 -- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00e80b4c20 msgr2=0x7f00e81e7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e759e640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00e80b4c20 0x7f00e81e7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e759e640 1 -- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00e8161850 msgr2=0x7f00e815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e759e640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00e8161850 0x7f00e815f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e759e640 1 -- 172.21.15.28:0/2879906491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e815fc50 con 0x7f00e8162000 2024-09-10T20:45:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00e759e640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e8162000 0x7f00e81ea460 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009cd0 tx=0x7f00d8031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00df7fe640 1 -- 172.21.15.28:0/2879906491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8004240 con 0x7f00e8162000 2024-09-10T20:45:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00df7fe640 1 -- 172.21.15.28:0/2879906491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00d80043e0 con 0x7f00e8162000 2024-09-10T20:45:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e815ff30 con 0x7f00e8162000 2024-09-10T20:45:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00df7fe640 1 -- 172.21.15.28:0/2879906491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8004830 con 0x7f00e8162000 2024-09-10T20:45:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.812+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e81604f0 con 0x7f00e8162000 2024-09-10T20:45:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.813+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e80d4a80 con 0x7f00e8162000 2024-09-10T20:45:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.814+0000 7f00df7fe640 1 -- 172.21.15.28:0/2879906491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 101264+0+0 (secure 0 0 0) 0x7f00d802f070 con 0x7f00e8162000 2024-09-10T20:45:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.814+0000 7f00df7fe640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00c80786c0 0x7f00c807ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.815+0000 7f00df7fe640 1 -- 172.21.15.28:0/2879906491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f00d80bc650 con 0x7f00e8162000 2024-09-10T20:45:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.815+0000 7f00e6d9d640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00c80786c0 0x7f00c807ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.815+0000 7f00e6d9d640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00c80786c0 0x7f00c807ab80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f00e8161170 tx=0x7f00e0007e90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:16.819+0000 7f00df7fe640 1 -- 172.21.15.28:0/2879906491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f00d8083ae0 con 0x7f00e8162000 2024-09-10T20:45:17.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.032+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol59601", "group_name": "subvol_grp80420", "target": ["mon-mgr", ""]}) -- 0x7f00e81e81e0 con 0x7f00c80786c0 2024-09-10T20:45:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.048+0000 7f00df7fe640 1 -- 172.21.15.28:0/2879906491 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f00e81e81e0 con 0x7f00c80786c0 2024-09-10T20:45:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00c80786c0 msgr2=0x7f00c807ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00c80786c0 0x7f00c807ab80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f00e8161170 tx=0x7f00e0007e90 comp rx=0 tx=0).stop 2024-09-10T20:45:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e8162000 msgr2=0x7f00e81ea460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e8162000 0x7f00e81ea460 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009cd0 tx=0x7f00d8031e20 comp rx=0 tx=0).stop 2024-09-10T20:45:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 shutdown_connections 2024-09-10T20:45:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f00c80786c0 0x7f00c807ab80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e8162000 0x7f00e81ea460 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f00e8161850 0x7f00e815f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 --2- 172.21.15.28:0/2879906491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f00e80b4c20 0x7f00e81e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 >> 172.21.15.28:0/2879906491 conn(0x7f00e80d3f90 msgr2=0x7f00e80b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 shutdown_connections 2024-09-10T20:45:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.051+0000 7f00ed033640 1 -- 172.21.15.28:0/2879906491 wait complete. 2024-09-10T20:45:17.066 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-10T20:45:17.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:45:17.119 INFO:teuthology.orchestra.run.smithi028.stdout:2 2024-09-10T20:45:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:17.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp80420 2024-09-10T20:45:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 -- 172.21.15.28:0/1453264366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 msgr2=0x7f42b00ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 --2- 172.21.15.28:0/1453264366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b00ced20 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f42b00b3f00 tx=0x7f42a802a4d0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 -- 172.21.15.28:0/1453264366 shutdown_connections 2024-09-10T20:45:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 --2- 172.21.15.28:0/1453264366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b00ced20 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 --2- 172.21.15.28:0/1453264366 >> v2:172.21.15.159:3300/0 conn(0x7f42b00b3580 0x7f42b006e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 --2- 172.21.15.28:0/1453264366 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42b0152e00 0x7f42b00b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 -- 172.21.15.28:0/1453264366 >> 172.21.15.28:0/1453264366 conn(0x7f42b006d110 msgr2=0x7f42b00b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 -- 172.21.15.28:0/1453264366 shutdown_connections 2024-09-10T20:45:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.229+0000 7f42b5433640 1 -- 172.21.15.28:0/1453264366 wait complete. 2024-09-10T20:45:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.230+0000 7f42b5433640 1 Processor -- start 2024-09-10T20:45:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.230+0000 7f42b5433640 1 -- start start 2024-09-10T20:45:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42b5433640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b0159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42b5433640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42b0152e00 0x7f42b01f8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42b5433640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42b0073b60 0x7f42b0073ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42b5433640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42b00cf5f0 con 0x7f42b006e940 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42b5433640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f42b00cf470 con 0x7f42b0073b60 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42b5433640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f42b015f820 con 0x7f42b0152e00 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b0159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b0159e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38580/0 (socket says 172.21.15.28:38580) 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.231+0000 7f42aeffd640 1 -- 172.21.15.28:0/369175771 learned_addr learned my addr 172.21.15.28:0/369175771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42ae7fc640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42b0152e00 0x7f42b01f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42af7fe640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42b0073b60 0x7f42b0073ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42aeffd640 1 -- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42b0152e00 msgr2=0x7f42b01f8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42aeffd640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42b0152e00 0x7f42b01f8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42aeffd640 1 -- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42b0073b60 msgr2=0x7f42b0073ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42aeffd640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42b0073b60 0x7f42b0073ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42aeffd640 1 -- 172.21.15.28:0/369175771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b00ad9d0 con 0x7f42b006e940 2024-09-10T20:45:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42af7fe640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42b0073b60 0x7f42b0073ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42aeffd640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b0159e30 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f42a000ba80 tx=0x7f42a000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42a77fe640 1 -- 172.21.15.28:0/369175771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a0019070 con 0x7f42b006e940 2024-09-10T20:45:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42a77fe640 1 -- 172.21.15.28:0/369175771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42a000cd10 con 0x7f42b006e940 2024-09-10T20:45:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42a77fe640 1 -- 172.21.15.28:0/369175771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a00146a0 con 0x7f42b006e940 2024-09-10T20:45:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b00adcb0 con 0x7f42b006e940 2024-09-10T20:45:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.232+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b00ae270 con 0x7f42b006e940 2024-09-10T20:45:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.233+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42b006e400 con 0x7f42b006e940 2024-09-10T20:45:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.234+0000 7f42a77fe640 1 -- 172.21.15.28:0/369175771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 101264+0+0 (secure 0 0 0) 0x7f42a001d3b0 con 0x7f42b006e940 2024-09-10T20:45:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.234+0000 7f42a77fe640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42900786c0 0x7f429007ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.234+0000 7f42a77fe640 1 -- 172.21.15.28:0/369175771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f42a009b010 con 0x7f42b006e940 2024-09-10T20:45:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.234+0000 7f42ae7fc640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42900786c0 0x7f429007ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.235+0000 7f42ae7fc640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42900786c0 0x7f429007ab80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f42b00af5a0 tx=0x7f4298006f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.238+0000 7f42a77fe640 1 -- 172.21.15.28:0/369175771 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f42a00624e0 con 0x7f42b006e940 2024-09-10T20:45:17.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.448+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp80420", "target": ["mon-mgr", ""]}) -- 0x7f42b0074690 con 0x7f42900786c0 2024-09-10T20:45:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.453+0000 7f42a77fe640 1 -- 172.21.15.28:0/369175771 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f42980095d0 con 0x7f42900786c0 2024-09-10T20:45:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42900786c0 msgr2=0x7f429007ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42900786c0 0x7f429007ab80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f42b00af5a0 tx=0x7f4298006f90 comp rx=0 tx=0).stop 2024-09-10T20:45:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 msgr2=0x7f42b0159e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b0159e30 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f42a000ba80 tx=0x7f42a000bf40 comp rx=0 tx=0).stop 2024-09-10T20:45:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 shutdown_connections 2024-09-10T20:45:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42900786c0 0x7f429007ab80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42b0073b60 0x7f42b0073ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42b0152e00 0x7f42b01f8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 --2- 172.21.15.28:0/369175771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b006e940 0x7f42b0159e30 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 >> 172.21.15.28:0/369175771 conn(0x7f42b006d110 msgr2=0x7f42b015d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 shutdown_connections 2024-09-10T20:45:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.455+0000 7f42b5433640 1 -- 172.21.15.28:0/369175771 wait complete. 2024-09-10T20:45:17.470 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:45:17.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:17.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:17.569 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:45:17.569 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:17.569 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:17.569 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:17.586 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:17.586 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:17.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:45:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 -- 172.21.15.28:0/2324600878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c15ee90 msgr2=0x7fab5c15bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 --2- 172.21.15.28:0/2324600878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c15ee90 0x7fab5c15bd80 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fab5c15b1a0 tx=0x7fab4402a4d0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 -- 172.21.15.28:0/2324600878 shutdown_connections 2024-09-10T20:45:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 --2- 172.21.15.28:0/2324600878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c15ee90 0x7fab5c15bd80 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 --2- 172.21.15.28:0/2324600878 >> v2:172.21.15.159:3300/0 conn(0x7fab5c0d0070 0x7fab5c0d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 -- 172.21.15.28:0/2324600878 >> 172.21.15.28:0/2324600878 conn(0x7fab5c06cf00 msgr2=0x7fab5c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 -- 172.21.15.28:0/2324600878 shutdown_connections 2024-09-10T20:45:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.788+0000 7fab61356640 1 -- 172.21.15.28:0/2324600878 wait complete. 2024-09-10T20:45:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.789+0000 7fab61356640 1 Processor -- start 2024-09-10T20:45:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab61356640 1 -- start start 2024-09-10T20:45:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab61356640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c0b3520 0x7fab5c1f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab61356640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab5c0d0070 0x7fab5c152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab61356640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab5c15ee90 0x7fab5c1485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab61356640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c072a50 con 0x7fab5c0b3520 2024-09-10T20:45:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab61356640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c0728d0 con 0x7fab5c15ee90 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab61356640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fab5c072bd0 con 0x7fab5c0d0070 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab5affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c0b3520 0x7fab5c1f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab5affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c0b3520 0x7fab5c1f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38600/0 (socket says 172.21.15.28:38600) 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.790+0000 7fab5affd640 1 -- 172.21.15.28:0/2280557405 learned_addr learned my addr 172.21.15.28:0/2280557405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab527fc640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab5c0d0070 0x7fab5c152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5b7fe640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab5c15ee90 0x7fab5c1485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5affd640 1 -- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab5c0d0070 msgr2=0x7fab5c152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5affd640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab5c0d0070 0x7fab5c152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5affd640 1 -- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab5c15ee90 msgr2=0x7fab5c1485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5affd640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab5c15ee90 0x7fab5c1485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5affd640 1 -- 172.21.15.28:0/2280557405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c09de50 con 0x7fab5c0b3520 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5b7fe640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab5c15ee90 0x7fab5c1485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab5affd640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c0b3520 0x7fab5c1f2710 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00c430 tx=0x7fab4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab53fff640 1 -- 172.21.15.28:0/2280557405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c011790 con 0x7fab5c0b3520 2024-09-10T20:45:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab53fff640 1 -- 172.21.15.28:0/2280557405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab4c017070 con 0x7fab5c0b3520 2024-09-10T20:45:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab53fff640 1 -- 172.21.15.28:0/2280557405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c011d00 con 0x7fab5c0b3520 2024-09-10T20:45:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c153160 con 0x7fab5c0b3520 2024-09-10T20:45:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.791+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c09aad0 con 0x7fab5c0b3520 2024-09-10T20:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.792+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5c0b4430 con 0x7fab5c0b3520 2024-09-10T20:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.793+0000 7fab53fff640 1 -- 172.21.15.28:0/2280557405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 101264+0+0 (secure 0 0 0) 0x7fab4c01c460 con 0x7fab5c0b3520 2024-09-10T20:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.793+0000 7fab53fff640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fab3c0786c0 0x7fab3c07ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.793+0000 7fab53fff640 1 -- 172.21.15.28:0/2280557405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7fab4c09ae90 con 0x7fab5c0b3520 2024-09-10T20:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.793+0000 7fab527fc640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fab3c0786c0 0x7fab3c07ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.794+0000 7fab527fc640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fab3c0786c0 0x7fab3c07ab80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fab44033000 tx=0x7fab44004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:17.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.798+0000 7fab53fff640 1 -- 172.21.15.28:0/2280557405 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fab5c0b4430 con 0x7fab5c0b3520 2024-09-10T20:45:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.999+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fab5c15aad0 con 0x7fab5c0b3520 2024-09-10T20:45:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:17.999+0000 7fab53fff640 1 -- 172.21.15.28:0/2280557405 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v140) ==== 80+0+0 (secure 0 0 0) 0x7fab4c0a0020 con 0x7fab5c0b3520 2024-09-10T20:45:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fab3c0786c0 msgr2=0x7fab3c07ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fab3c0786c0 0x7fab3c07ab80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fab44033000 tx=0x7fab44004730 comp rx=0 tx=0).stop 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c0b3520 msgr2=0x7fab5c1f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c0b3520 0x7fab5c1f2710 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00c430 tx=0x7fab4c00c8f0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 shutdown_connections 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fab3c0786c0 0x7fab3c07ab80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fab5c15ee90 0x7fab5c1485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fab5c0d0070 0x7fab5c152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 --2- 172.21.15.28:0/2280557405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5c0b3520 0x7fab5c1f2710 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 >> 172.21.15.28:0/2280557405 conn(0x7fab5c06cf00 msgr2=0x7fab5c15cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 shutdown_connections 2024-09-10T20:45:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.002+0000 7fab61356640 1 -- 172.21.15.28:0/2280557405 wait complete. 2024-09-10T20:45:18.017 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:45:18.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:18.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:18.048 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:45:18.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:18.049 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:45:18.104 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:18.105 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:18.105 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:18.124 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:18.124 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:18.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:45:18.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.285+0000 7fe0a8d53640 1 -- 172.21.15.28:0/3859257774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 msgr2=0x7fe090003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.285+0000 7fe0a8d53640 1 --2- 172.21.15.28:0/3859257774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 0x7fe090003e70 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe094009b50 tx=0x7fe090005830 comp rx=0 tx=0).stop 2024-09-10T20:45:18.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.285+0000 7fe0a2d9d640 1 -- 172.21.15.28:0/3859257774 reap_dead start 2024-09-10T20:45:18.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.285+0000 7fe0a8d53640 1 -- 172.21.15.28:0/3859257774 shutdown_connections 2024-09-10T20:45:18.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.285+0000 7fe0a8d53640 1 -- 172.21.15.28:0/3859257774 >> 172.21.15.28:0/3859257774 conn(0x7fe0a40d1e10 msgr2=0x7fe0a40d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:18.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.285+0000 7fe0a8d53640 1 -- 172.21.15.28:0/3859257774 shutdown_connections 2024-09-10T20:45:18.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.285+0000 7fe0a8d53640 1 -- 172.21.15.28:0/3859257774 wait complete. 2024-09-10T20:45:18.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.287+0000 7fe0a8d53640 1 Processor -- start 2024-09-10T20:45:18.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.287+0000 7fe0a8d53640 1 -- start start 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a8d53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 0x7fe0a40b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a8d53640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0a4148e60 0x7fe0a414b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a8d53640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0a41495f0 0x7fe0a4149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a8d53640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a40b0d80 con 0x7fe090003a50 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a8d53640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a40b0c00 con 0x7fe0a41495f0 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a8d53640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a40b0f00 con 0x7fe0a4148e60 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 0x7fe0a40b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 0x7fe0a40b4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38630/0 (socket says 172.21.15.28:38630) 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 -- 172.21.15.28:0/973410984 learned_addr learned my addr 172.21.15.28:0/973410984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe099d9b640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0a4148e60 0x7fe0a414b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a2d9d640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0a41495f0 0x7fe0a4149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 -- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0a4148e60 msgr2=0x7fe0a414b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0a4148e60 0x7fe0a414b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 -- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0a41495f0 msgr2=0x7fe0a4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0a41495f0 0x7fe0a4149a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 -- 172.21.15.28:0/973410984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a414bbc0 con 0x7fe090003a50 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a2d9d640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0a41495f0 0x7fe0a4149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.288+0000 7fe0a259c640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 0x7fe0a40b4030 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fe094009a70 tx=0x7fe094033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.289+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094038040 con 0x7fe090003a50 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.289+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe09402b7e0 con 0x7fe090003a50 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.289+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a41492e0 con 0x7fe090003a50 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.289+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094009720 con 0x7fe090003a50 2024-09-10T20:45:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.289+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a414c140 con 0x7fe090003a50 2024-09-10T20:45:18.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.290+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a4071260 con 0x7fe090003a50 2024-09-10T20:45:18.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.290+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 101264+0+0 (secure 0 0 0) 0x7fe09403b030 con 0x7fe090003a50 2024-09-10T20:45:18.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.291+0000 7fe09b7fe640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe0840786f0 0x7fe08407abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.291+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7fe0940bb650 con 0x7fe090003a50 2024-09-10T20:45:18.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.291+0000 7fe099d9b640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe0840786f0 0x7fe08407abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.291+0000 7fe099d9b640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe0840786f0 0x7fe08407abb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cfdd0 tx=0x7fe08c005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.295+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe0940831e0 con 0x7fe090003a50 2024-09-10T20:45:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.398+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fe09407ed50 con 0x7fe090003a50 2024-09-10T20:45:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.497+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe0a40b45c0 con 0x7fe090003a50 2024-09-10T20:45:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.498+0000 7fe09b7fe640 1 -- 172.21.15.28:0/973410984 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v140) ==== 80+0+0 (secure 0 0 0) 0x7fe0940b47e0 con 0x7fe090003a50 2024-09-10T20:45:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:45:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe0840786f0 msgr2=0x7fe08407abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe0840786f0 0x7fe08407abb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cfdd0 tx=0x7fe08c005f90 comp rx=0 tx=0).stop 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 msgr2=0x7fe0a40b4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 0x7fe0a40b4030 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fe094009a70 tx=0x7fe094033670 comp rx=0 tx=0).stop 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 shutdown_connections 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe0840786f0 0x7fe08407abb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe0a41495f0 0x7fe0a4149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe0a4148e60 0x7fe0a414b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 --2- 172.21.15.28:0/973410984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe090003a50 0x7fe0a40b4030 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 >> 172.21.15.28:0/973410984 conn(0x7fe0a40d1e10 msgr2=0x7fe0a40ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 shutdown_connections 2024-09-10T20:45:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.500+0000 7fe0a8d53640 1 -- 172.21.15.28:0/973410984 wait complete. 2024-09-10T20:45:18.514 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:45:18.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:18.515 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:18.547 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:45:18.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:18.547 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:45:18.605 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:18.605 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:18.605 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:18.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:45:18.622 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:18.622 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:18.622 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:18.663 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:45:18.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.764+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2269464737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 msgr2=0x7f1c380d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.764+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2269464737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c380d2b90 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009830 tx=0x7f1c2802a510 comp rx=0 tx=0).stop 2024-09-10T20:45:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.764+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2269464737 shutdown_connections 2024-09-10T20:45:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.764+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2269464737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c380d2b90 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.764+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2269464737 >> 172.21.15.28:0/2269464737 conn(0x7f1c380d3fb0 msgr2=0x7f1c380d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.764+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2269464737 shutdown_connections 2024-09-10T20:45:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.764+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2269464737 wait complete. 2024-09-10T20:45:18.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 Processor -- start 2024-09-10T20:45:18.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 -- start start 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380cf780 0x7f1c380a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c3806d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38162260 0x7f1c3806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c380cff80 con 0x7f1c3815de70 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1c380cfe00 con 0x7f1c380cf780 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.766+0000 7f1c3dc9b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3815e470 con 0x7f1c38162260 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c3806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c3806d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38660/0 (socket says 172.21.15.28:38660) 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 -- 172.21.15.28:0/2695302706 learned_addr learned my addr 172.21.15.28:0/2695302706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c377fe640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380cf780 0x7f1c380a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c37fff640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38162260 0x7f1c3806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 -- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38162260 msgr2=0x7f1c3806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38162260 0x7f1c3806da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 -- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380cf780 msgr2=0x7f1c380a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380cf780 0x7f1c380a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 -- 172.21.15.28:0/2695302706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c381f4660 con 0x7f1c3815de70 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c3806d520 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00c500 tx=0x7f1c2c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c1ffff640 1 -- 172.21.15.28:0/2695302706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c016020 con 0x7f1c3815de70 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c1ffff640 1 -- 172.21.15.28:0/2695302706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c2c012920 con 0x7f1c3815de70 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c1ffff640 1 -- 172.21.15.28:0/2695302706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c012d90 con 0x7f1c3815de70 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c381f4940 con 0x7f1c3815de70 2024-09-10T20:45:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.767+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c381f4f00 con 0x7f1c3815de70 2024-09-10T20:45:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.768+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c380b7830 con 0x7f1c3815de70 2024-09-10T20:45:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.769+0000 7f1c1ffff640 1 -- 172.21.15.28:0/2695302706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f1c2c01a3b0 con 0x7f1c3815de70 2024-09-10T20:45:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.769+0000 7f1c1ffff640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c240787e0 0x7f1c2407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.770+0000 7f1c1ffff640 1 -- 172.21.15.28:0/2695302706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f1c2c098690 con 0x7f1c3815de70 2024-09-10T20:45:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.770+0000 7f1c377fe640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c240787e0 0x7f1c2407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.770+0000 7f1c377fe640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c240787e0 0x7f1c2407aca0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1c28034000 tx=0x7f1c28004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.774+0000 7f1c1ffff640 1 -- 172.21.15.28:0/2695302706 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c2c020030 con 0x7f1c3815de70 2024-09-10T20:45:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.995+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c38074dc0 con 0x7f1c3815de70 2024-09-10T20:45:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.996+0000 7f1c1ffff640 1 -- 172.21.15.28:0/2695302706 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5227 (secure 0 0 0) 0x7f1c2c062bc0 con 0x7f1c3815de70 2024-09-10T20:45:19.000 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:19.000 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":97,"btime":"2024-09-10T20:45:05:024454+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4242811460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4242811460},{"type":"v1","addr":"172.21.15.28:6834","nonce":4242811460}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/914514353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":914514353},{"type":"v1","addr":"172.21.15.159:6835","nonce":914514353}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8334,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2842589826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2842589826},{"type":"v1","addr":"172.21.15.28:6835","nonce":2842589826}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:55.365818+0000","modified":"2024-09-10T20:44:58.372881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8346},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8346":{"gid":8346,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/103941194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":103941194},{"type":"v1","addr":"172.21.15.159:6801","nonce":103941194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8346,"qdb_cluster":[8346]},"id":11}]} 2024-09-10T20:45:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 97 2024-09-10T20:45:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.998+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c240787e0 msgr2=0x7f1c2407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.998+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c240787e0 0x7f1c2407aca0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1c28034000 tx=0x7f1c28004730 comp rx=0 tx=0).stop 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.998+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 msgr2=0x7f1c3806d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.998+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c3806d520 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00c500 tx=0x7f1c2c00c9c0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 shutdown_connections 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1c240787e0 0x7f1c2407aca0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c38162260 0x7f1c3806da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c3815de70 0x7f1c3806d520 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 --2- 172.21.15.28:0/2695302706 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c380cf780 0x7f1c380a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 >> 172.21.15.28:0/2695302706 conn(0x7f1c380d3fb0 msgr2=0x7f1c380d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 shutdown_connections 2024-09-10T20:45:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:18.999+0000 7f1c3dc9b640 1 -- 172.21.15.28:0/2695302706 wait complete. 2024-09-10T20:45:19.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.116+0000 7f5631a3c640 1 -- 172.21.15.28:0/2902996640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 msgr2=0x7f561800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.116+0000 7f5631a3c640 1 --2- 172.21.15.28:0/2902996640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f561800b9c0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f561c0016c0 tx=0x7f5618005830 comp rx=0 tx=0).stop 2024-09-10T20:45:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.116+0000 7f5631a3c640 1 -- 172.21.15.28:0/2902996640 shutdown_connections 2024-09-10T20:45:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.116+0000 7f5631a3c640 1 --2- 172.21.15.28:0/2902996640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f561800b9c0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.116+0000 7f5631a3c640 1 -- 172.21.15.28:0/2902996640 >> 172.21.15.28:0/2902996640 conn(0x7f562c0d3fb0 msgr2=0x7f562c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.116+0000 7f5631a3c640 1 -- 172.21.15.28:0/2902996640 shutdown_connections 2024-09-10T20:45:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.116+0000 7f5631a3c640 1 -- 172.21.15.28:0/2902996640 wait complete. 2024-09-10T20:45:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.117+0000 7f5631a3c640 1 Processor -- start 2024-09-10T20:45:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.117+0000 7f5631a3c640 1 -- start start 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f5631a3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f562c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f5631a3c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f562c160550 0x7f562c1609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f5631a3c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f562c0a0360 0x7f562c0a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f5631a3c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f562c0d2d10 con 0x7f56180051c0 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f5631a3c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f562c0d2b90 con 0x7f562c0a0360 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f5631a3c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f562c0d2e90 con 0x7f562c160550 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f562c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f562c0b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38682/0 (socket says 172.21.15.28:38682) 2024-09-10T20:45:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 -- 172.21.15.28:0/925640660 learned_addr learned my addr 172.21.15.28:0/925640660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 -- 172.21.15.28:0/925640660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f562c160550 msgr2=0x7f562c1609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f562c160550 0x7f562c1609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 -- 172.21.15.28:0/925640660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f562c0a0360 msgr2=0x7f562c0a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562b7fe640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f562c0a0360 0x7f562c0a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f562c0a0360 0x7f562c0a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.118+0000 7f562affd640 1 -- 172.21.15.28:0/925640660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562c15f970 con 0x7f56180051c0 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.119+0000 7f562affd640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f562c0b46c0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f5624010b80 tx=0x7f562400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.119+0000 7f5623fff640 1 -- 172.21.15.28:0/925640660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624017070 con 0x7f56180051c0 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.119+0000 7f5623fff640 1 -- 172.21.15.28:0/925640660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5624003c80 con 0x7f56180051c0 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.119+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562c09dc90 con 0x7f56180051c0 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.119+0000 7f5623fff640 1 -- 172.21.15.28:0/925640660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624004280 con 0x7f56180051c0 2024-09-10T20:45:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.119+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f562c09e250 con 0x7f56180051c0 2024-09-10T20:45:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.120+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f562c0cff20 con 0x7f56180051c0 2024-09-10T20:45:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.121+0000 7f5623fff640 1 -- 172.21.15.28:0/925640660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f5624004420 con 0x7f56180051c0 2024-09-10T20:45:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.121+0000 7f5623fff640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f560c0787e0 0x7f560c07aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.121+0000 7f5623fff640 1 -- 172.21.15.28:0/925640660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f56240990c0 con 0x7f56180051c0 2024-09-10T20:45:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.121+0000 7f562a7fc640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f560c0787e0 0x7f560c07aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.122+0000 7f562a7fc640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f560c0787e0 0x7f560c07aca0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5614001c20 tx=0x7f5614001bb0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.125+0000 7f5623fff640 1 -- 172.21.15.28:0/925640660 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5624063670 con 0x7f56180051c0 2024-09-10T20:45:19.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.348+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f562c0b9f30 con 0x7f56180051c0 2024-09-10T20:45:19.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.348+0000 7f5623fff640 1 -- 172.21.15.28:0/925640660 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5227 (secure 0 0 0) 0x7f5624062f70 con 0x7f56180051c0 2024-09-10T20:45:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:19.353 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":97,"btime":"2024-09-10T20:45:05:024454+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4242811460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4242811460},{"type":"v1","addr":"172.21.15.28:6834","nonce":4242811460}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/914514353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":914514353},{"type":"v1","addr":"172.21.15.159:6835","nonce":914514353}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8334,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2842589826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2842589826},{"type":"v1","addr":"172.21.15.28:6835","nonce":2842589826}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:55.365818+0000","modified":"2024-09-10T20:44:58.372881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8346},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8346":{"gid":8346,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/103941194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":103941194},{"type":"v1","addr":"172.21.15.159:6801","nonce":103941194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8346,"qdb_cluster":[8346]},"id":11}]} 2024-09-10T20:45:19.353 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 97 2024-09-10T20:45:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f560c0787e0 msgr2=0x7f560c07aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f560c0787e0 0x7f560c07aca0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5614001c20 tx=0x7f5614001bb0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 msgr2=0x7f562c0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f562c0b46c0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f5624010b80 tx=0x7f562400e720 comp rx=0 tx=0).stop 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 shutdown_connections 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f560c0787e0 0x7f560c07aca0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f562c0a0360 0x7f562c0a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f562c160550 0x7f562c1609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 --2- 172.21.15.28:0/925640660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56180051c0 0x7f562c0b46c0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 >> 172.21.15.28:0/925640660 conn(0x7f562c0d3fb0 msgr2=0x7f562c0b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 shutdown_connections 2024-09-10T20:45:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.350+0000 7f5631a3c640 1 -- 172.21.15.28:0/925640660 wait complete. 2024-09-10T20:45:19.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:19.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/3302120937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 msgr2=0x7f5e28162410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/3302120937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e28162410 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f5e2808c900 tx=0x7f5e2002a9a0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/3302120937 shutdown_connections 2024-09-10T20:45:19.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/3302120937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e28162410 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/3302120937 >> v2:172.21.15.159:3300/0 conn(0x7f5e2815df70 0x7f5e280d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/3302120937 >> 172.21.15.28:0/3302120937 conn(0x7f5e2809d070 msgr2=0x7f5e280d3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/3302120937 shutdown_connections 2024-09-10T20:45:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.467+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/3302120937 wait complete. 2024-09-10T20:45:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.468+0000 7f5e2d9a1640 1 Processor -- start 2024-09-10T20:45:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.468+0000 7f5e2d9a1640 1 -- start start 2024-09-10T20:45:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d9a1640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e280cfb30 0x7f5e280cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d9a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e280c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d9a1640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e2815df70 0x7f5e2815a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d9a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e280b7250 con 0x7f5e280d0fa0 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d9a1640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5e280b70d0 con 0x7f5e2815df70 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d9a1640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5e280b73d0 con 0x7f5e280cfb30 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e280c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e280c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38696/0 (socket says 172.21.15.28:38696) 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 -- 172.21.15.28:0/1039398067 learned_addr learned my addr 172.21.15.28:0/1039398067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d1a0640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e2815df70 0x7f5e2815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2c99f640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e280cfb30 0x7f5e280cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 -- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e280cfb30 msgr2=0x7f5e280cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e280cfb30 0x7f5e280cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 -- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e2815df70 msgr2=0x7f5e2815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e2815df70 0x7f5e2815a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e27fff640 1 -- 172.21.15.28:0/1039398067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e281f8fd0 con 0x7f5e280d0fa0 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.469+0000 7f5e2d1a0640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e2815df70 0x7f5e2815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.470+0000 7f5e27fff640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e280c8f00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00c400 tx=0x7f5e1c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.470+0000 7f5e257fa640 1 -- 172.21.15.28:0/1039398067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c017070 con 0x7f5e280d0fa0 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.470+0000 7f5e257fa640 1 -- 172.21.15.28:0/1039398067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e1c0035f0 con 0x7f5e280d0fa0 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.470+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2815a8c0 con 0x7f5e280d0fa0 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.470+0000 7f5e257fa640 1 -- 172.21.15.28:0/1039398067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c003a80 con 0x7f5e280d0fa0 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.470+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e281f84f0 con 0x7f5e280d0fa0 2024-09-10T20:45:19.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.471+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e280b78e0 con 0x7f5e280d0fa0 2024-09-10T20:45:19.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.472+0000 7f5e257fa640 1 -- 172.21.15.28:0/1039398067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f5e1c003c20 con 0x7f5e280d0fa0 2024-09-10T20:45:19.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.472+0000 7f5e257fa640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5e140787e0 0x7f5e1407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.472+0000 7f5e257fa640 1 -- 172.21.15.28:0/1039398067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f5e1c097820 con 0x7f5e280d0fa0 2024-09-10T20:45:19.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.472+0000 7f5e2c99f640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5e140787e0 0x7f5e1407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.473+0000 7f5e2c99f640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5e140787e0 0x7f5e1407aca0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5e280c8a30 tx=0x7f5e18008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:19.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.476+0000 7f5e257fa640 1 -- 172.21.15.28:0/1039398067 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5e1c061700 con 0x7f5e280d0fa0 2024-09-10T20:45:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.660+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e280d0670 con 0x7f5e280d0fa0 2024-09-10T20:45:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.661+0000 7f5e257fa640 1 -- 172.21.15.28:0/1039398067 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) ==== 74+0+16880 (secure 0 0 0) 0x7f5e1c066130 con 0x7f5e280d0fa0 2024-09-10T20:45:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:19.665 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":140,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:11.050898+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:53.000121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:54.318239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"23.6","mappings":[{"from":5,"to":3}]},{"pgid":"23.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.663+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5e140787e0 msgr2=0x7f5e1407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.663+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5e140787e0 0x7f5e1407aca0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5e280c8a30 tx=0x7f5e18008040 comp rx=0 tx=0).stop 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.663+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 msgr2=0x7f5e280c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.663+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e280c8f00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00c400 tx=0x7f5e1c00c8c0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 shutdown_connections 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5e140787e0 0x7f5e1407aca0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5e2815df70 0x7f5e2815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e280d0fa0 0x7f5e280c8f00 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 --2- 172.21.15.28:0/1039398067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5e280cfb30 0x7f5e280cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 >> 172.21.15.28:0/1039398067 conn(0x7f5e2809d070 msgr2=0x7f5e280b7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 shutdown_connections 2024-09-10T20:45:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.664+0000 7f5e2d9a1640 1 -- 172.21.15.28:0/1039398067 wait complete. 2024-09-10T20:45:19.679 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:45:19.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:19.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 -- 172.21.15.28:0/445887688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 msgr2=0x7f97400d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 --2- 172.21.15.28:0/445887688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f97400d2b90 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f9738009830 tx=0x7f973802a510 comp rx=0 tx=0).stop 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 -- 172.21.15.28:0/445887688 shutdown_connections 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 --2- 172.21.15.28:0/445887688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9740162260 0x7f97400cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 --2- 172.21.15.28:0/445887688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f97400d2b90 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 -- 172.21.15.28:0/445887688 >> 172.21.15.28:0/445887688 conn(0x7f97400d3fb0 msgr2=0x7f97400d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 -- 172.21.15.28:0/445887688 shutdown_connections 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.780+0000 7f9747d2b640 1 -- 172.21.15.28:0/445887688 wait complete. 2024-09-10T20:45:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 Processor -- start 2024-09-10T20:45:19.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 -- start start 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97400cf780 0x7f97400a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f974014ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9740162260 0x7f974014f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97400b7190 con 0x7f974015de70 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f97400b7010 con 0x7f9740162260 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f9747d2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f97400b7310 con 0x7f97400cf780 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f97452c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f974014ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f97452c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f974014ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38722/0 (socket says 172.21.15.28:38722) 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.782+0000 7f97452c6640 1 -- 172.21.15.28:0/3383497811 learned_addr learned my addr 172.21.15.28:0/3383497811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f97452c6640 1 -- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97400cf780 msgr2=0x7f97400a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f9745ac7640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97400cf780 0x7f97400a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f97452c6640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97400cf780 0x7f97400a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f97462c8640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9740162260 0x7f974014f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f97452c6640 1 -- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9740162260 msgr2=0x7f974014f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f97452c6640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9740162260 0x7f974014f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f97452c6640 1 -- 172.21.15.28:0/3383497811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f974014fac0 con 0x7f974015de70 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f97452c6640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f974014ee80 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f973400c500 tx=0x7f973400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f973e7fc640 1 -- 172.21.15.28:0/3383497811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9734016020 con 0x7f974015de70 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f973e7fc640 1 -- 172.21.15.28:0/3383497811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9734012920 con 0x7f974015de70 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f973e7fc640 1 -- 172.21.15.28:0/3383497811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9734012d90 con 0x7f974015de70 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f974014fda0 con 0x7f974015de70 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.783+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97400cc450 con 0x7f974015de70 2024-09-10T20:45:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.784+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97400b7830 con 0x7f974015de70 2024-09-10T20:45:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.785+0000 7f973e7fc640 1 -- 172.21.15.28:0/3383497811 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f973401a3b0 con 0x7f974015de70 2024-09-10T20:45:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.785+0000 7f973e7fc640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f972c0787e0 0x7f972c07aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.785+0000 7f973e7fc640 1 -- 172.21.15.28:0/3383497811 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f97340985f0 con 0x7f974015de70 2024-09-10T20:45:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.785+0000 7f9745ac7640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f972c0787e0 0x7f972c07aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.786+0000 7f9745ac7640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f972c0787e0 0x7f972c07aca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9738034000 tx=0x7f9738004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:19.790+0000 7f973e7fc640 1 -- 172.21.15.28:0/3383497811 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9734020030 con 0x7f974015de70 2024-09-10T20:45:20.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.014+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97400b9790 con 0x7f974015de70 2024-09-10T20:45:20.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.015+0000 7f973e7fc640 1 -- 172.21.15.28:0/3383497811 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5227 (secure 0 0 0) 0x7f9734062b20 con 0x7f974015de70 2024-09-10T20:45:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":97,"btime":"2024-09-10T20:45:05:024454+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4242811460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4242811460},{"type":"v1","addr":"172.21.15.28:6834","nonce":4242811460}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/914514353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":914514353},{"type":"v1","addr":"172.21.15.159:6835","nonce":914514353}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8334,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2842589826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2842589826},{"type":"v1","addr":"172.21.15.28:6835","nonce":2842589826}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:55.365818+0000","modified":"2024-09-10T20:44:58.372881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8346},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8346":{"gid":8346,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/103941194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":103941194},{"type":"v1","addr":"172.21.15.159:6801","nonce":103941194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8346,"qdb_cluster":[8346]},"id":11}]} 2024-09-10T20:45:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 97 2024-09-10T20:45:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f972c0787e0 msgr2=0x7f972c07aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f972c0787e0 0x7f972c07aca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9738034000 tx=0x7f9738004730 comp rx=0 tx=0).stop 2024-09-10T20:45:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 msgr2=0x7f974014ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f974014ee80 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f973400c500 tx=0x7f973400c9c0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 shutdown_connections 2024-09-10T20:45:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f972c0787e0 0x7f972c07aca0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9740162260 0x7f974014f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974015de70 0x7f974014ee80 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 --2- 172.21.15.28:0/3383497811 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f97400cf780 0x7f97400a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 >> 172.21.15.28:0/3383497811 conn(0x7f97400d3fb0 msgr2=0x7f97400b8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 shutdown_connections 2024-09-10T20:45:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.017+0000 7f9747d2b640 1 -- 172.21.15.28:0/3383497811 wait complete. 2024-09-10T20:45:20.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.133+0000 7fe6bbc85640 1 -- 172.21.15.28:0/2844219560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 msgr2=0x7fe6a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.133+0000 7fe6bbc85640 1 --2- 172.21.15.28:0/2844219560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 0x7fe6a00051c0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00c730 tx=0x7fe6a0005830 comp rx=0 tx=0).stop 2024-09-10T20:45:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.133+0000 7fe6ba222640 1 -- 172.21.15.28:0/2844219560 reap_dead start 2024-09-10T20:45:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.133+0000 7fe6bbc85640 1 -- 172.21.15.28:0/2844219560 shutdown_connections 2024-09-10T20:45:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.133+0000 7fe6bbc85640 1 -- 172.21.15.28:0/2844219560 >> 172.21.15.28:0/2844219560 conn(0x7fe6b406cf00 msgr2=0x7fe6b406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.133+0000 7fe6bbc85640 1 -- 172.21.15.28:0/2844219560 shutdown_connections 2024-09-10T20:45:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.133+0000 7fe6bbc85640 1 -- 172.21.15.28:0/2844219560 wait complete. 2024-09-10T20:45:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.134+0000 7fe6bbc85640 1 Processor -- start 2024-09-10T20:45:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.134+0000 7fe6bbc85640 1 -- start start 2024-09-10T20:45:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6bbc85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 0x7fe6b4140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6bbc85640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b41f60f0 0x7fe6b40b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6bbc85640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b41414c0 0x7fe6b41e7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6bbc85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b415ecc0 con 0x7fe6a0004d40 2024-09-10T20:45:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6bbc85640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b415eb40 con 0x7fe6b41414c0 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6bbc85640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b4072bd0 con 0x7fe6b41f60f0 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6b9a21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 0x7fe6b4140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6b9a21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 0x7fe6b4140d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38746/0 (socket says 172.21.15.28:38746) 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6b9a21640 1 -- 172.21.15.28:0/1488497817 learned_addr learned my addr 172.21.15.28:0/1488497817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6ba222640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b41414c0 0x7fe6b41e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.135+0000 7fe6b9220640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b41f60f0 0x7fe6b40b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6b9a21640 1 -- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b41f60f0 msgr2=0x7fe6b40b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6b9a21640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b41f60f0 0x7fe6b40b17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6b9a21640 1 -- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b41414c0 msgr2=0x7fe6b41e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6b9a21640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b41414c0 0x7fe6b41e7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6b9a21640 1 -- 172.21.15.28:0/1488497817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b41f6660 con 0x7fe6a0004d40 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6b9220640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b41f60f0 0x7fe6b40b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6b9a21640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 0x7fe6b4140d50 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0011ae0 tx=0x7fe6b0009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6aa7fc640 1 -- 172.21.15.28:0/1488497817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b000fc30 con 0x7fe6a0004d40 2024-09-10T20:45:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6aa7fc640 1 -- 172.21.15.28:0/1488497817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6b000fdd0 con 0x7fe6a0004d40 2024-09-10T20:45:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6aa7fc640 1 -- 172.21.15.28:0/1488497817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0004280 con 0x7fe6a0004d40 2024-09-10T20:45:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b41f6940 con 0x7fe6a0004d40 2024-09-10T20:45:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.136+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b41f6f00 con 0x7fe6a0004d40 2024-09-10T20:45:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.137+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b415bd50 con 0x7fe6a0004d40 2024-09-10T20:45:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.138+0000 7fe6aa7fc640 1 -- 172.21.15.28:0/1488497817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fe6b00035f0 con 0x7fe6a0004d40 2024-09-10T20:45:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.138+0000 7fe6aa7fc640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe698078810 0x7fe69807acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.138+0000 7fe6aa7fc640 1 -- 172.21.15.28:0/1488497817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7fe6b0098f10 con 0x7fe6a0004d40 2024-09-10T20:45:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.138+0000 7fe6b9220640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe698078810 0x7fe69807acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.139+0000 7fe6b9220640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe698078810 0x7fe69807acd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4154b10 tx=0x7fe6a400b2e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:20.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.142+0000 7fe6aa7fc640 1 -- 172.21.15.28:0/1488497817 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe6b0062e70 con 0x7fe6a0004d40 2024-09-10T20:45:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.366+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6b415da00 con 0x7fe6a0004d40 2024-09-10T20:45:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.366+0000 7fe6aa7fc640 1 -- 172.21.15.28:0/1488497817 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5227 (secure 0 0 0) 0x7fe6b00678a0 con 0x7fe6a0004d40 2024-09-10T20:45:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:20.371 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":97,"btime":"2024-09-10T20:45:05:024454+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4242811460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4242811460},{"type":"v1","addr":"172.21.15.28:6834","nonce":4242811460}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/914514353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":914514353},{"type":"v1","addr":"172.21.15.159:6835","nonce":914514353}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8334,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2842589826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2842589826},{"type":"v1","addr":"172.21.15.28:6835","nonce":2842589826}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:44:55.365818+0000","modified":"2024-09-10T20:44:58.372881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8346},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8346":{"gid":8346,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6801/103941194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":103941194},{"type":"v1","addr":"172.21.15.159:6801","nonce":103941194}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8346,"qdb_cluster":[8346]},"id":11}]} 2024-09-10T20:45:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 97 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe698078810 msgr2=0x7fe69807acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe698078810 0x7fe69807acd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4154b10 tx=0x7fe6a400b2e0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 msgr2=0x7fe6b4140d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 0x7fe6b4140d50 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0011ae0 tx=0x7fe6b0009f10 comp rx=0 tx=0).stop 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 shutdown_connections 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe698078810 0x7fe69807acd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6b41414c0 0x7fe6b41e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6b41f60f0 0x7fe6b40b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 --2- 172.21.15.28:0/1488497817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a0004d40 0x7fe6b4140d50 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 >> 172.21.15.28:0/1488497817 conn(0x7fe6b406cf00 msgr2=0x7fe6b415c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:20.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 shutdown_connections 2024-09-10T20:45:20.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.369+0000 7fe6bbc85640 1 -- 172.21.15.28:0/1488497817 wait complete. 2024-09-10T20:45:20.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.486+0000 7fc897b89640 1 -- 172.21.15.28:0/3254250863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc87c004d90 msgr2=0x7fc87c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.486+0000 7fc897b89640 1 --2- 172.21.15.28:0/3254250863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc87c004d90 0x7fc87c005210 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fc888009ed0 tx=0x7fc87c005830 comp rx=0 tx=0).stop 2024-09-10T20:45:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.486+0000 7fc896126640 1 -- 172.21.15.28:0/3254250863 reap_dead start 2024-09-10T20:45:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.486+0000 7fc897b89640 1 -- 172.21.15.28:0/3254250863 shutdown_connections 2024-09-10T20:45:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.487+0000 7fc897b89640 1 -- 172.21.15.28:0/3254250863 >> 172.21.15.28:0/3254250863 conn(0x7fc8900d1e40 msgr2=0x7fc8900d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.487+0000 7fc897b89640 1 -- 172.21.15.28:0/3254250863 shutdown_connections 2024-09-10T20:45:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.487+0000 7fc897b89640 1 -- 172.21.15.28:0/3254250863 wait complete. 2024-09-10T20:45:20.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.488+0000 7fc897b89640 1 Processor -- start 2024-09-10T20:45:20.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.488+0000 7fc897b89640 1 -- start start 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc87c004d90 0x7fc89015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc89015b720 0x7fc8901f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc890160860 0x7fc89015d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc890162cc0 con 0x7fc89015b720 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc890162b40 con 0x7fc87c004d90 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc890162e40 con 0x7fc890160860 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc89015b720 0x7fc8901f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc89015b720 0x7fc8901f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51568/0 (socket says 172.21.15.28:51568) 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 -- 172.21.15.28:0/1708832962 learned_addr learned my addr 172.21.15.28:0/1708832962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895925640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc87c004d90 0x7fc89015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc896126640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc890160860 0x7fc89015d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 -- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc890160860 msgr2=0x7fc89015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc890160860 0x7fc89015d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 -- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc87c004d90 msgr2=0x7fc89015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc87c004d90 0x7fc89015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 -- 172.21.15.28:0/1708832962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc890159d30 con 0x7fc89015b720 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc896126640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc890160860 0x7fc89015d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc895124640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc89015b720 0x7fc8901f7ca0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fc880003e90 tx=0x7fc880001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc8867fc640 1 -- 172.21.15.28:0/1708832962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880015070 con 0x7fc89015b720 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc8867fc640 1 -- 172.21.15.28:0/1708832962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc880011900 con 0x7fc89015b720 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc8867fc640 1 -- 172.21.15.28:0/1708832962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880011d50 con 0x7fc89015b720 2024-09-10T20:45:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc890161420 con 0x7fc89015b720 2024-09-10T20:45:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.489+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc890159fd0 con 0x7fc89015b720 2024-09-10T20:45:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.490+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc89006f810 con 0x7fc89015b720 2024-09-10T20:45:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.491+0000 7fc8867fc640 1 -- 172.21.15.28:0/1708832962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fc8800103c0 con 0x7fc89015b720 2024-09-10T20:45:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.492+0000 7fc8867fc640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8740787e0 0x7fc87407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.492+0000 7fc895925640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8740787e0 0x7fc87407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.492+0000 7fc8867fc640 1 -- 172.21.15.28:0/1708832962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7fc88009c190 con 0x7fc89015b720 2024-09-10T20:45:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.492+0000 7fc895925640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8740787e0 0x7fc87407aca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0019b0 tx=0x7fc88c001cb0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.496+0000 7fc8867fc640 1 -- 172.21.15.28:0/1708832962 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc89006f810 con 0x7fc89015b720 2024-09-10T20:45:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.682+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc890162110 con 0x7fc89015b720 2024-09-10T20:45:20.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.683+0000 7fc8867fc640 1 -- 172.21.15.28:0/1708832962 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) ==== 74+0+16880 (secure 0 0 0) 0x7fc880063c00 con 0x7fc89015b720 2024-09-10T20:45:20.687 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:20.687 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":140,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:11.050898+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:44:53.000121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-10T20:44:54.318239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"23.6","mappings":[{"from":5,"to":3}]},{"pgid":"23.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8740787e0 msgr2=0x7fc87407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8740787e0 0x7fc87407aca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0019b0 tx=0x7fc88c001cb0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc89015b720 msgr2=0x7fc8901f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc89015b720 0x7fc8901f7ca0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fc880003e90 tx=0x7fc880001800 comp rx=0 tx=0).stop 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 shutdown_connections 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc8740787e0 0x7fc87407aca0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc890160860 0x7fc89015d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc89015b720 0x7fc8901f7ca0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 --2- 172.21.15.28:0/1708832962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc87c004d90 0x7fc89015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.685+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 >> 172.21.15.28:0/1708832962 conn(0x7fc8900d1e40 msgr2=0x7fc8900cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:20.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.686+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 shutdown_connections 2024-09-10T20:45:20.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.686+0000 7fc897b89640 1 -- 172.21.15.28:0/1708832962 wait complete. 2024-09-10T20:45:20.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:45:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.804+0000 7f9aafacd640 1 -- 172.21.15.28:0/3552131035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a90004d40 msgr2=0x7f9a900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.804+0000 7f9aafacd640 1 --2- 172.21.15.28:0/3552131035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a90004d40 0x7f9a900051c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800c780 tx=0x7f9a90005830 comp rx=0 tx=0).stop 2024-09-10T20:45:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.804+0000 7f9aae06a640 1 -- 172.21.15.28:0/3552131035 reap_dead start 2024-09-10T20:45:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.804+0000 7f9aafacd640 1 -- 172.21.15.28:0/3552131035 shutdown_connections 2024-09-10T20:45:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.804+0000 7f9aafacd640 1 -- 172.21.15.28:0/3552131035 >> 172.21.15.28:0/3552131035 conn(0x7f9aa80d1e40 msgr2=0x7f9aa80d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.804+0000 7f9aafacd640 1 -- 172.21.15.28:0/3552131035 shutdown_connections 2024-09-10T20:45:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.804+0000 7f9aafacd640 1 -- 172.21.15.28:0/3552131035 wait complete. 2024-09-10T20:45:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.805+0000 7f9aafacd640 1 Processor -- start 2024-09-10T20:45:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.805+0000 7f9aafacd640 1 -- start start 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aafacd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa81616a0 0x7f9aa80b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aafacd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa81f84c0 0x7f9aa80b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aafacd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80c6240 0x7f9aa815b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aafacd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8162290 con 0x7f9aa80c6240 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aafacd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8162110 con 0x7f9aa81f84c0 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aafacd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa8162410 con 0x7f9aa81616a0 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80c6240 0x7f9aa815b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aad869640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa81616a0 0x7f9aa80b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aad068640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa81f84c0 0x7f9aa80b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80c6240 0x7f9aa815b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51592/0 (socket says 172.21.15.28:51592) 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 -- 172.21.15.28:0/194275755 learned_addr learned my addr 172.21.15.28:0/194275755 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 -- 172.21.15.28:0/194275755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa81616a0 msgr2=0x7f9aa80b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa81616a0 0x7f9aa80b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 -- 172.21.15.28:0/194275755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa81f84c0 msgr2=0x7f9aa80b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa81f84c0 0x7f9aa80b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.806+0000 7f9aae06a640 1 -- 172.21.15.28:0/194275755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa81f89c0 con 0x7f9aa80c6240 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.807+0000 7f9aae06a640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80c6240 0x7f9aa815b640 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800c8a0 tx=0x7f9a98001200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.807+0000 7f9aa67fc640 1 -- 172.21.15.28:0/194275755 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800c3a0 con 0x7f9aa80c6240 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.807+0000 7f9aa67fc640 1 -- 172.21.15.28:0/194275755 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a98001a40 con 0x7f9aa80c6240 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.807+0000 7f9aa67fc640 1 -- 172.21.15.28:0/194275755 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9802bbe0 con 0x7f9aa80c6240 2024-09-10T20:45:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.807+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa81f8ca0 con 0x7f9aa80c6240 2024-09-10T20:45:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.807+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa81f9260 con 0x7f9aa80c6240 2024-09-10T20:45:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.808+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa80d0d90 con 0x7f9aa80c6240 2024-09-10T20:45:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.809+0000 7f9aa67fc640 1 -- 172.21.15.28:0/194275755 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f9a98048070 con 0x7f9aa80c6240 2024-09-10T20:45:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.809+0000 7f9aa67fc640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9a94078810 0x7f9a9407acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.809+0000 7f9aa67fc640 1 -- 172.21.15.28:0/194275755 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5795+0+0 (secure 0 0 0) 0x7f9a980b8270 con 0x7f9aa80c6240 2024-09-10T20:45:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.809+0000 7f9aad869640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9a94078810 0x7f9a9407acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.810+0000 7f9aad869640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9a94078810 0x7f9a9407acd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa0006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:20.813+0000 7f9aa67fc640 1 -- 172.21.15.28:0/194275755 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9a98045060 con 0x7f9aa80c6240 2024-09-10T20:45:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.034+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9aa80b4090 con 0x7f9aa80c6240 2024-09-10T20:45:21.038 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:45:21.035+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:45:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.036+0000 7f9aa67fc640 1 -- 172.21.15.28:0/194275755 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v98) ==== 161+0+0 (secure 0 0 0) 0x7f9aa80b3750 con 0x7f9aa80c6240 2024-09-10T20:45:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:45:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9a94078810 msgr2=0x7f9a9407acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9a94078810 0x7f9a9407acd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa0006b70 comp rx=0 tx=0).stop 2024-09-10T20:45:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80c6240 msgr2=0x7f9aa815b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80c6240 0x7f9aa815b640 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800c8a0 tx=0x7f9a98001200 comp rx=0 tx=0).stop 2024-09-10T20:45:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 shutdown_connections 2024-09-10T20:45:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9a94078810 0x7f9a9407acd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa80c6240 0x7f9aa815b640 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9aa81f84c0 0x7f9aa80b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 --2- 172.21.15.28:0/194275755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9aa81616a0 0x7f9aa80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 >> 172.21.15.28:0/194275755 conn(0x7f9aa80d1e40 msgr2=0x7f9aa815cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 shutdown_connections 2024-09-10T20:45:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.039+0000 7f9aafacd640 1 -- 172.21.15.28:0/194275755 wait complete. 2024-09-10T20:45:21.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:45:21.080 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:45:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.158+0000 7f7a6bf10640 1 -- 172.21.15.28:0/726357201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c004800 msgr2=0x7f7a4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.158+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/726357201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c004800 0x7f7a4c004c80 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7a58006fd0 tx=0x7f7a4c005830 comp rx=0 tx=0).stop 2024-09-10T20:45:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.158+0000 7f7a6bf10640 1 -- 172.21.15.28:0/726357201 shutdown_connections 2024-09-10T20:45:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.158+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/726357201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4c004800 0x7f7a4c004c80 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.158+0000 7f7a6bf10640 1 -- 172.21.15.28:0/726357201 >> 172.21.15.28:0/726357201 conn(0x7f7a640d40f0 msgr2=0x7f7a640d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.158+0000 7f7a6bf10640 1 -- 172.21.15.28:0/726357201 shutdown_connections 2024-09-10T20:45:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.158+0000 7f7a6bf10640 1 -- 172.21.15.28:0/726357201 wait complete. 2024-09-10T20:45:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.159+0000 7f7a6bf10640 1 Processor -- start 2024-09-10T20:45:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.159+0000 7f7a6bf10640 1 -- start start 2024-09-10T20:45:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6bf10640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4c004800 0x7f7a640ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6bf10640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a640b6f20 0x7f7a640ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6bf10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a641f3a20 0x7f7a641f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6bf10640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a64162360 con 0x7f7a641f3a20 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6bf10640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7a641621e0 con 0x7f7a4c004800 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6bf10640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7a641624e0 con 0x7f7a640b6f20 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6a4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a641f3a20 0x7f7a641f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6a4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a641f3a20 0x7f7a641f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51610/0 (socket says 172.21.15.28:51610) 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a6a4ad640 1 -- 172.21.15.28:0/1053912200 learned_addr learned my addr 172.21.15.28:0/1053912200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a69cac640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4c004800 0x7f7a640ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.160+0000 7f7a694ab640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a640b6f20 0x7f7a640ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6a4ad640 1 -- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a640b6f20 msgr2=0x7f7a640ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6a4ad640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a640b6f20 0x7f7a640ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6a4ad640 1 -- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4c004800 msgr2=0x7f7a640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6a4ad640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4c004800 0x7f7a640ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6a4ad640 1 -- 172.21.15.28:0/1053912200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a64151540 con 0x7f7a641f3a20 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a694ab640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a640b6f20 0x7f7a640ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6a4ad640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a641f3a20 0x7f7a641f3e80 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7a640d4090 tx=0x7f7a5c011c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a627fc640 1 -- 172.21.15.28:0/1053912200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c004440 con 0x7f7a641f3a20 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a627fc640 1 -- 172.21.15.28:0/1053912200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a5c00f8c0 con 0x7f7a641f3a20 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a640b7430 con 0x7f7a641f3a20 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a627fc640 1 -- 172.21.15.28:0/1053912200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c00fd10 con 0x7f7a641f3a20 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.161+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a640b9b00 con 0x7f7a641f3a20 2024-09-10T20:45:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.162+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a640b9ed0 con 0x7f7a641f3a20 2024-09-10T20:45:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.163+0000 7f7a627fc640 1 -- 172.21.15.28:0/1053912200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f7a5c018030 con 0x7f7a641f3a20 2024-09-10T20:45:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.163+0000 7f7a627fc640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7a500787e0 0x7f7a5007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.163+0000 7f7a627fc640 1 -- 172.21.15.28:0/1053912200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5881+0+0 (secure 0 0 0) 0x7f7a5c098b00 con 0x7f7a641f3a20 2024-09-10T20:45:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.163+0000 7f7a69cac640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7a500787e0 0x7f7a5007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.164+0000 7f7a69cac640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7a500787e0 0x7f7a5007aca0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7a54006180 tx=0x7f7a54006ef0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.167+0000 7f7a627fc640 1 -- 172.21.15.28:0/1053912200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7a5c0630b0 con 0x7f7a641f3a20 2024-09-10T20:45:21.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:21.390+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7a64162a40 con 0x7f7a641f3a20 2024-09-10T20:45:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.038+0000 7f7a627fc640 1 -- 172.21.15.28:0/1053912200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v99) ==== 104+0+0 (secure 0 0 0) 0x7f7a5c0629b0 con 0x7f7a641f3a20 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7a500787e0 msgr2=0x7f7a5007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7a500787e0 0x7f7a5007aca0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7a54006180 tx=0x7f7a54006ef0 comp rx=0 tx=0).stop 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a641f3a20 msgr2=0x7f7a641f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a641f3a20 0x7f7a641f3e80 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7a640d4090 tx=0x7f7a5c011c60 comp rx=0 tx=0).stop 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 shutdown_connections 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7a500787e0 0x7f7a5007aca0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a641f3a20 0x7f7a641f3e80 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7a640b6f20 0x7f7a640ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 --2- 172.21.15.28:0/1053912200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7a4c004800 0x7f7a640ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 >> 172.21.15.28:0/1053912200 conn(0x7f7a640d40f0 msgr2=0x7f7a640cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 shutdown_connections 2024-09-10T20:45:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.042+0000 7f7a6bf10640 1 -- 172.21.15.28:0/1053912200 wait complete. 2024-09-10T20:45:22.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:45:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.172+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/2361825750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc4003f70 msgr2=0x7f9cc4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.172+0000 7f9ce2e9b640 1 --2- 172.21.15.28:0/2361825750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc4003f70 0x7f9cc4004800 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0001eb0 tx=0x7f9cc4005830 comp rx=0 tx=0).stop 2024-09-10T20:45:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.172+0000 7f9ce1438640 1 -- 172.21.15.28:0/2361825750 reap_dead start 2024-09-10T20:45:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.172+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/2361825750 shutdown_connections 2024-09-10T20:45:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.172+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/2361825750 >> 172.21.15.28:0/2361825750 conn(0x7f9cdc0d0410 msgr2=0x7f9cdc0d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.172+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/2361825750 shutdown_connections 2024-09-10T20:45:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.172+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/2361825750 wait complete. 2024-09-10T20:45:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.173+0000 7f9ce2e9b640 1 Processor -- start 2024-09-10T20:45:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.173+0000 7f9ce2e9b640 1 -- start start 2024-09-10T20:45:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce2e9b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc4003f70 0x7f9cdc14b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce2e9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cdc0c6940 0x7f9cdc1f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce2e9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cdc1f4120 0x7f9cdc14ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce2e9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc0739e0 con 0x7f9cdc1f4120 2024-09-10T20:45:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce2e9b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc0730b0 con 0x7f9cdc0c6940 2024-09-10T20:45:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce2e9b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9cdc06d850 con 0x7f9cc4003f70 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cdc1f4120 0x7f9cdc14ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cdc1f4120 0x7f9cdc14ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51618/0 (socket says 172.21.15.28:51618) 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 -- 172.21.15.28:0/1026761149 learned_addr learned my addr 172.21.15.28:0/1026761149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce0c37640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc4003f70 0x7f9cdc14b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9cdbfff640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cdc0c6940 0x7f9cdc1f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 -- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc4003f70 msgr2=0x7f9cdc14b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc4003f70 0x7f9cdc14b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 -- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cdc0c6940 msgr2=0x7f9cdc1f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cdc0c6940 0x7f9cdc1f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.174+0000 7f9ce1438640 1 -- 172.21.15.28:0/1026761149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc0c6dc0 con 0x7f9cdc1f4120 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.175+0000 7f9ce1438640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cdc1f4120 0x7f9cdc14ae40 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc073f20 tx=0x7f9cd4001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.175+0000 7f9cd97fa640 1 -- 172.21.15.28:0/1026761149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4015020 con 0x7f9cdc1f4120 2024-09-10T20:45:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.175+0000 7f9cd97fa640 1 -- 172.21.15.28:0/1026761149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cd40119a0 con 0x7f9cdc1f4120 2024-09-10T20:45:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.175+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cdc0b3400 con 0x7f9cdc1f4120 2024-09-10T20:45:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.175+0000 7f9cd97fa640 1 -- 172.21.15.28:0/1026761149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4011df0 con 0x7f9cdc1f4120 2024-09-10T20:45:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.175+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cdc0b5b30 con 0x7f9cdc1f4120 2024-09-10T20:45:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.176+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cdc15e8d0 con 0x7f9cdc1f4120 2024-09-10T20:45:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.176+0000 7f9cd97fa640 1 -- 172.21.15.28:0/1026761149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f9cd40103c0 con 0x7f9cdc1f4120 2024-09-10T20:45:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.177+0000 7f9cd97fa640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cc8078810 0x7f9cc807acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.177+0000 7f9cd97fa640 1 -- 172.21.15.28:0/1026761149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5881+0+0 (secure 0 0 0) 0x7f9cd4098eb0 con 0x7f9cdc1f4120 2024-09-10T20:45:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.177+0000 7f9ce0c37640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cc8078810 0x7f9cc807acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.177+0000 7f9ce0c37640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cc8078810 0x7f9cc807acd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009c50 tx=0x7f9ccc005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.181+0000 7f9cd97fa640 1 -- 172.21.15.28:0/1026761149 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9cd401f030 con 0x7f9cdc1f4120 2024-09-10T20:45:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:22.366+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9cdc0b1840 con 0x7f9cdc1f4120 2024-09-10T20:45:23.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.042+0000 7f9cd97fa640 1 -- 172.21.15.28:0/1026761149 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) ==== 181+0+0 (secure 0 0 0) 0x7f9cd4002080 con 0x7f9cdc1f4120 2024-09-10T20:45:23.047 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:45:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.046+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cc8078810 msgr2=0x7f9cc807acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.046+0000 7f9ce2e9b640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cc8078810 0x7f9cc807acd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009c50 tx=0x7f9ccc005f90 comp rx=0 tx=0).stop 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.046+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cdc1f4120 msgr2=0x7f9cdc14ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.046+0000 7f9ce2e9b640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cdc1f4120 0x7f9cdc14ae40 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc073f20 tx=0x7f9cd4001a80 comp rx=0 tx=0).stop 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 shutdown_connections 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9cc8078810 0x7f9cc807acd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cdc1f4120 0x7f9cdc14ae40 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cdc0c6940 0x7f9cdc1f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 --2- 172.21.15.28:0/1026761149 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc4003f70 0x7f9cdc14b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 >> 172.21.15.28:0/1026761149 conn(0x7f9cdc0d0410 msgr2=0x7f9cdc0cce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 shutdown_connections 2024-09-10T20:45:23.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.047+0000 7f9ce2e9b640 1 -- 172.21.15.28:0/1026761149 wait complete. 2024-09-10T20:45:23.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:45:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 -- 172.21.15.28:0/634292665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc162000 msgr2=0x7f9edc0d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 --2- 172.21.15.28:0/634292665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc162000 0x7f9edc0d4a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4005fd0 tx=0x7f9ed402a670 comp rx=0 tx=0).stop 2024-09-10T20:45:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 -- 172.21.15.28:0/634292665 shutdown_connections 2024-09-10T20:45:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 --2- 172.21.15.28:0/634292665 >> v2:172.21.15.159:3300/0 conn(0x7f9edc161850 0x7f9edc0d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 --2- 172.21.15.28:0/634292665 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc162000 0x7f9edc0d4a80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 -- 172.21.15.28:0/634292665 >> 172.21.15.28:0/634292665 conn(0x7f9edc0d3f90 msgr2=0x7f9edc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 -- 172.21.15.28:0/634292665 shutdown_connections 2024-09-10T20:45:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.180+0000 7f9ee436b640 1 -- 172.21.15.28:0/634292665 wait complete. 2024-09-10T20:45:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.181+0000 7f9ee436b640 1 Processor -- start 2024-09-10T20:45:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.181+0000 7f9ee436b640 1 -- start start 2024-09-10T20:45:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee436b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc0b4c20 0x7f9edc1f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee436b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc161850 0x7f9edc144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee436b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc162000 0x7f9edc0cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee436b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc0d2320 con 0x7f9edc162000 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee436b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc0d21a0 con 0x7f9edc0b4c20 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee436b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9edc0d24a0 con 0x7f9edc161850 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc162000 0x7f9edc0cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc162000 0x7f9edc0cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51632/0 (socket says 172.21.15.28:51632) 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 -- 172.21.15.28:0/4229768289 learned_addr learned my addr 172.21.15.28:0/4229768289 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2107640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc0b4c20 0x7f9edc1f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee1906640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc161850 0x7f9edc144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 -- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc161850 msgr2=0x7f9edc144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc161850 0x7f9edc144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 -- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc0b4c20 msgr2=0x7f9edc1f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc0b4c20 0x7f9edc1f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 -- 172.21.15.28:0/4229768289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc1f2760 con 0x7f9edc162000 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.182+0000 7f9ee2908640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc162000 0x7f9edc0cd7d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009cf0 tx=0x7f9ecc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.183+0000 7f9edaffd640 1 -- 172.21.15.28:0/4229768289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc003b70 con 0x7f9edc162000 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.183+0000 7f9edaffd640 1 -- 172.21.15.28:0/4229768289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ecc003e00 con 0x7f9edc162000 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.183+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9edc147e60 con 0x7f9edc162000 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.183+0000 7f9edaffd640 1 -- 172.21.15.28:0/4229768289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc004280 con 0x7f9edc162000 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.183+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9edc15ec40 con 0x7f9edc162000 2024-09-10T20:45:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.184+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9edc15f420 con 0x7f9edc162000 2024-09-10T20:45:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.184+0000 7f9edaffd640 1 -- 172.21.15.28:0/4229768289 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f9ecc004420 con 0x7f9edc162000 2024-09-10T20:45:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.185+0000 7f9edaffd640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ec80787e0 0x7f9ec807aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.185+0000 7f9ee2107640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ec80787e0 0x7f9ec807aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.185+0000 7f9edaffd640 1 -- 172.21.15.28:0/4229768289 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5443+0+0 (secure 0 0 0) 0x7f9ecc099490 con 0x7f9edc162000 2024-09-10T20:45:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.185+0000 7f9ee2107640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ec80787e0 0x7f9ec807aca0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4033000 tx=0x7f9ed4004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.190+0000 7f9edaffd640 1 -- 172.21.15.28:0/4229768289 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9edc15f420 con 0x7f9edc162000 2024-09-10T20:45:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:23.377+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9edc0cf570 con 0x7f9edc162000 2024-09-10T20:45:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.044+0000 7f9edaffd640 1 -- 172.21.15.28:0/4229768289 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) ==== 169+0+0 (secure 0 0 0) 0x7f9ecc0633f0 con 0x7f9edc162000 2024-09-10T20:45:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ec80787e0 msgr2=0x7f9ec807aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ec80787e0 0x7f9ec807aca0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4033000 tx=0x7f9ed4004730 comp rx=0 tx=0).stop 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc162000 msgr2=0x7f9edc0cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc162000 0x7f9edc0cd7d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009cf0 tx=0x7f9ecc00c6a0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 shutdown_connections 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9ec80787e0 0x7f9ec807aca0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9edc162000 0x7f9edc0cd7d0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9edc161850 0x7f9edc144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 --2- 172.21.15.28:0/4229768289 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9edc0b4c20 0x7f9edc1f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 >> 172.21.15.28:0/4229768289 conn(0x7f9edc0d3f90 msgr2=0x7f9edc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 shutdown_connections 2024-09-10T20:45:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.048+0000 7f9ee436b640 1 -- 172.21.15.28:0/4229768289 wait complete. 2024-09-10T20:45:24.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 -- 172.21.15.28:0/4069303724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0d05d0 msgr2=0x7f2abc0b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/4069303724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0d05d0 0x7f2abc0b2e80 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009ca0 tx=0x7f2ab402aa30 comp rx=0 tx=0).stop 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 -- 172.21.15.28:0/4069303724 shutdown_connections 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/4069303724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0d05d0 0x7f2abc0b2e80 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/4069303724 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0cc570 0x7f2abc0d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 -- 172.21.15.28:0/4069303724 >> 172.21.15.28:0/4069303724 conn(0x7f2abc06cf00 msgr2=0x7f2abc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 -- 172.21.15.28:0/4069303724 shutdown_connections 2024-09-10T20:45:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.176+0000 7f2ac17ad640 1 -- 172.21.15.28:0/4069303724 wait complete. 2024-09-10T20:45:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.177+0000 7f2ac17ad640 1 Processor -- start 2024-09-10T20:45:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.177+0000 7f2ac17ad640 1 -- start start 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0cc570 0x7f2abc1f3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0cd3d0 0x7f2abc152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0d05d0 0x7f2abc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc0b33c0 con 0x7f2abc0cc570 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc0b4b30 con 0x7f2abc0cd3d0 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc0b3540 con 0x7f2abc0d05d0 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0cc570 0x7f2abc1f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0cc570 0x7f2abc1f3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51650/0 (socket says 172.21.15.28:51650) 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 -- 172.21.15.28:0/939794659 learned_addr learned my addr 172.21.15.28:0/939794659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abb7fe640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0d05d0 0x7f2abc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ab3fff640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0cd3d0 0x7f2abc152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 -- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0d05d0 msgr2=0x7f2abc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0d05d0 0x7f2abc1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 -- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0cd3d0 msgr2=0x7f2abc152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0cd3d0 0x7f2abc152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 -- 172.21.15.28:0/939794659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc155e40 con 0x7f2abc0cc570 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abb7fe640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0d05d0 0x7f2abc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ab3fff640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0cd3d0 0x7f2abc152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2abaffd640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0cc570 0x7f2abc1f3700 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c440 tx=0x7f2aac00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ab37fe640 1 -- 172.21.15.28:0/939794659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac017070 con 0x7f2abc0cc570 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ab37fe640 1 -- 172.21.15.28:0/939794659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2aac0035f0 con 0x7f2abc0cc570 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ab37fe640 1 -- 172.21.15.28:0/939794659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac003a40 con 0x7f2abc0cc570 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc0afa60 con 0x7f2abc0cc570 2024-09-10T20:45:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.178+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc152ff0 con 0x7f2abc0cc570 2024-09-10T20:45:24.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.179+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc0b3a00 con 0x7f2abc0cc570 2024-09-10T20:45:24.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.180+0000 7f2ab37fe640 1 -- 172.21.15.28:0/939794659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f2aac003be0 con 0x7f2abc0cc570 2024-09-10T20:45:24.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.181+0000 7f2ab37fe640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a9c0787e0 0x7f2a9c07aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.181+0000 7f2ab37fe640 1 -- 172.21.15.28:0/939794659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7f2aac09a390 con 0x7f2abc0cc570 2024-09-10T20:45:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.181+0000 7f2ab3fff640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a9c0787e0 0x7f2a9c07aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.181+0000 7f2ab3fff640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a9c0787e0 0x7f2a9c07aca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2abc155160 tx=0x7f2aa4008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:24.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.185+0000 7f2ab37fe640 1 -- 172.21.15.28:0/939794659 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2aac0621d0 con 0x7f2abc0cc570 2024-09-10T20:45:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.373+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f2abc0b3040 con 0x7f2abc0cc570 2024-09-10T20:45:24.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.376+0000 7f2ab37fe640 1 -- 172.21.15.28:0/939794659 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v45) ==== 71+0+0 (secure 0 0 0) 0x7f2aac0010c0 con 0x7f2abc0cc570 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.378+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a9c0787e0 msgr2=0x7f2a9c07aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a9c0787e0 0x7f2a9c07aca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2abc155160 tx=0x7f2aa4008040 comp rx=0 tx=0).stop 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0cc570 msgr2=0x7f2abc1f3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0cc570 0x7f2abc1f3700 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c440 tx=0x7f2aac00c900 comp rx=0 tx=0).stop 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 shutdown_connections 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2a9c0787e0 0x7f2a9c07aca0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2abc0d05d0 0x7f2abc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2abc0cd3d0 0x7f2abc152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 --2- 172.21.15.28:0/939794659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc0cc570 0x7f2abc1f3700 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 >> 172.21.15.28:0/939794659 conn(0x7f2abc06cf00 msgr2=0x7f2abc15b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 shutdown_connections 2024-09-10T20:45:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.379+0000 7f2ac17ad640 1 -- 172.21.15.28:0/939794659 wait complete. 2024-09-10T20:45:24.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1' 2024-09-10T20:45:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 -- 172.21.15.28:0/1473972326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440d05d0 msgr2=0x7fb1440b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 --2- 172.21.15.28:0/1473972326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440d05d0 0x7fb1440b2e80 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009ca0 tx=0x7fb13c02aac0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 -- 172.21.15.28:0/1473972326 shutdown_connections 2024-09-10T20:45:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 --2- 172.21.15.28:0/1473972326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440d05d0 0x7fb1440b2e80 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 --2- 172.21.15.28:0/1473972326 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1440cc570 0x7fb1440d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 -- 172.21.15.28:0/1473972326 >> 172.21.15.28:0/1473972326 conn(0x7fb14406cf00 msgr2=0x7fb14406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 -- 172.21.15.28:0/1473972326 shutdown_connections 2024-09-10T20:45:24.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.495+0000 7fb1497f3640 1 -- 172.21.15.28:0/1473972326 wait complete. 2024-09-10T20:45:24.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.496+0000 7fb1497f3640 1 Processor -- start 2024-09-10T20:45:24.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.496+0000 7fb1497f3640 1 -- start start 2024-09-10T20:45:24.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1497f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440cc570 0x7fb1440c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1497f3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1440cd3d0 0x7fb14414cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1497f3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1440d05d0 0x7fb144073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1497f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1440b3540 con 0x7fb1440cc570 2024-09-10T20:45:24.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1497f3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb1440b33c0 con 0x7fb1440d05d0 2024-09-10T20:45:24.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1497f3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb1440b36c0 con 0x7fb1440cd3d0 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440cc570 0x7fb1440c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440cc570 0x7fb1440c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51674/0 (socket says 172.21.15.28:51674) 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 -- 172.21.15.28:0/4256600370 learned_addr learned my addr 172.21.15.28:0/4256600370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1427fc640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1440cd3d0 0x7fb14414cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1437fe640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1440d05d0 0x7fb144073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 -- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1440cd3d0 msgr2=0x7fb14414cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1440cd3d0 0x7fb14414cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 -- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1440d05d0 msgr2=0x7fb144073470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1440d05d0 0x7fb144073470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb142ffd640 1 -- 172.21.15.28:0/4256600370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1440ca170 con 0x7fb1440cc570 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.497+0000 7fb1427fc640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1440cd3d0 0x7fb14414cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.498+0000 7fb142ffd640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440cc570 0x7fb1440c9c30 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c450 tx=0x7fb13400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.498+0000 7fb12b7fe640 1 -- 172.21.15.28:0/4256600370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134017070 con 0x7fb1440cc570 2024-09-10T20:45:24.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.498+0000 7fb12b7fe640 1 -- 172.21.15.28:0/4256600370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1340053c0 con 0x7fb1440cc570 2024-09-10T20:45:24.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.498+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1440af570 con 0x7fb1440cc570 2024-09-10T20:45:24.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.498+0000 7fb12b7fe640 1 -- 172.21.15.28:0/4256600370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134003850 con 0x7fb1440cc570 2024-09-10T20:45:24.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.498+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1440aed50 con 0x7fb1440cc570 2024-09-10T20:45:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.499+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1440ae860 con 0x7fb1440cc570 2024-09-10T20:45:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.500+0000 7fb12b7fe640 1 -- 172.21.15.28:0/4256600370 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fb1340039f0 con 0x7fb1440cc570 2024-09-10T20:45:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.500+0000 7fb12b7fe640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb1300787e0 0x7fb13007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.500+0000 7fb12b7fe640 1 -- 172.21.15.28:0/4256600370 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7fb134097c70 con 0x7fb1440cc570 2024-09-10T20:45:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.500+0000 7fb1427fc640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb1300787e0 0x7fb13007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.501+0000 7fb1427fc640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb1300787e0 0x7fb13007aca0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1440c92f0 tx=0x7fb13800a5c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:24.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.504+0000 7fb12b7fe640 1 -- 172.21.15.28:0/4256600370 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb134062220 con 0x7fb1440cc570 2024-09-10T20:45:24.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:24.765+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]} v 0) -- 0x7fb1440b3ae0 con 0x7fb1440cc570 2024-09-10T20:45:25.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.048+0000 7fb12b7fe640 1 -- 172.21.15.28:0/4256600370 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]}]=0 v400) ==== 149+0+0 (secure 0 0 0) 0x7fb134061b20 con 0x7fb1440cc570 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb1300787e0 msgr2=0x7fb13007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb1300787e0 0x7fb13007aca0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1440c92f0 tx=0x7fb13800a5c0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440cc570 msgr2=0x7fb1440c9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440cc570 0x7fb1440c9c30 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c450 tx=0x7fb13400c910 comp rx=0 tx=0).stop 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 shutdown_connections 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb1300787e0 0x7fb13007aca0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1440d05d0 0x7fb144073470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb1440cd3d0 0x7fb14414cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 --2- 172.21.15.28:0/4256600370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1440cc570 0x7fb1440c9c30 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 >> 172.21.15.28:0/4256600370 conn(0x7fb14406cf00 msgr2=0x7fb14409b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 shutdown_connections 2024-09-10T20:45:25.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.051+0000 7fb1497f3640 1 -- 172.21.15.28:0/4256600370 wait complete. 2024-09-10T20:45:25.066 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- rather than going through a full upgrade cycle, ... ok 2024-09-10T20:45:25.066 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-10T20:45:25.066 INFO:tasks.cephfs_test_runner:test_subvolume_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:45:25.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 -- 172.21.15.28:0/3729410678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0d05d0 msgr2=0x7fa33c0b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 --2- 172.21.15.28:0/3729410678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0d05d0 0x7fa33c0b2e80 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa334005fd0 tx=0x7fa33402aae0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 -- 172.21.15.28:0/3729410678 shutdown_connections 2024-09-10T20:45:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 --2- 172.21.15.28:0/3729410678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0d05d0 0x7fa33c0b2e80 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 --2- 172.21.15.28:0/3729410678 >> v2:172.21.15.159:3300/0 conn(0x7fa33c0cc570 0x7fa33c0d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 -- 172.21.15.28:0/3729410678 >> 172.21.15.28:0/3729410678 conn(0x7fa33c06cf00 msgr2=0x7fa33c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 -- 172.21.15.28:0/3729410678 shutdown_connections 2024-09-10T20:45:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.169+0000 7fa3438f6640 1 -- 172.21.15.28:0/3729410678 wait complete. 2024-09-10T20:45:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.170+0000 7fa3438f6640 1 Processor -- start 2024-09-10T20:45:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.170+0000 7fa3438f6640 1 -- start start 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa3438f6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa33c0cc570 0x7fa33c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa3438f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0cd3d0 0x7fa33c0cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa3438f6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa33c0d05d0 0x7fa33c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa3438f6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa33c0b33c0 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa3438f6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa33c0cd7b0 con 0x7fa33c0cc570 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa3438f6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa33c0b3540 con 0x7fa33c0d05d0 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0cd3d0 0x7fa33c0cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0cd3d0 0x7fa33c0cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51696/0 (socket says 172.21.15.28:51696) 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 -- 172.21.15.28:0/937541270 learned_addr learned my addr 172.21.15.28:0/937541270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa341692640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa33c0cc570 0x7fa33c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa341e93640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa33c0d05d0 0x7fa33c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 -- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa33c0d05d0 msgr2=0x7fa33c0cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa33c0d05d0 0x7fa33c0cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 -- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa33c0cc570 msgr2=0x7fa33c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa33c0cc570 0x7fa33c070280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.171+0000 7fa340e91640 1 -- 172.21.15.28:0/937541270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33c14a730 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.172+0000 7fa340e91640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0cd3d0 0x7fa33c0cb200 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e9f0 tx=0x7fa32c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.172+0000 7fa331ffb640 1 -- 172.21.15.28:0/937541270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00cd80 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.172+0000 7fa331ffb640 1 -- 172.21.15.28:0/937541270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa32c010430 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.172+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33c14aa10 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.172+0000 7fa331ffb640 1 -- 172.21.15.28:0/937541270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c010840 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.172+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa33c14afd0 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.173+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa33c0b3a30 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.173+0000 7fa331ffb640 1 -- 172.21.15.28:0/937541270 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fa32c0109e0 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.174+0000 7fa331ffb640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3200787e0 0x7fa32007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.174+0000 7fa331ffb640 1 -- 172.21.15.28:0/937541270 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7fa32c0990b0 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.174+0000 7fa341692640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3200787e0 0x7fa32007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.174+0000 7fa341692640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3200787e0 0x7fa32007aca0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3380017c0 tx=0x7fa338009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.178+0000 7fa331ffb640 1 -- 172.21.15.28:0/937541270 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa32c0630d0 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.371+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa33c0b2f40 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.372+0000 7fa331ffb640 1 -- 172.21.15.28:0/937541270 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+13086 (secure 0 0 0) 0x7fa32c067b00 con 0x7fa33c0cd3d0 2024-09-10T20:45:25.376 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:25.376 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":143,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:24.043582+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.159:6801/103941194":"2024-09-11T20:45:21.035515+0000","172.21.15.159:6800/103941194":"2024-09-11T20:45:21.035515+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3200787e0 msgr2=0x7fa32007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3200787e0 0x7fa32007aca0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3380017c0 tx=0x7fa338009040 comp rx=0 tx=0).stop 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0cd3d0 msgr2=0x7fa33c0cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0cd3d0 0x7fa33c0cb200 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e9f0 tx=0x7fa32c00eeb0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 shutdown_connections 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa3200787e0 0x7fa32007aca0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa33c0d05d0 0x7fa33c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c0cd3d0 0x7fa33c0cb200 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 --2- 172.21.15.28:0/937541270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa33c0cc570 0x7fa33c070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 >> 172.21.15.28:0/937541270 conn(0x7fa33c06cf00 msgr2=0x7fa33c15d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 shutdown_connections 2024-09-10T20:45:25.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.375+0000 7fa3438f6640 1 -- 172.21.15.28:0/937541270 wait complete. 2024-09-10T20:45:25.390 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:45:25.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 -- 172.21.15.28:0/1800356433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c161770 msgr2=0x7f3e7c0d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 --2- 172.21.15.28:0/1800356433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c161770 0x7f3e7c0d20e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c08d410 tx=0x7f3e6c02a4d0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 -- 172.21.15.28:0/1800356433 shutdown_connections 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 --2- 172.21.15.28:0/1800356433 >> v2:172.21.15.159:3300/0 conn(0x7f3e7c0d4b90 0x7f3e7c0b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 --2- 172.21.15.28:0/1800356433 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e7c161b50 0x7f3e7c0d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 --2- 172.21.15.28:0/1800356433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c161770 0x7f3e7c0d20e0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 -- 172.21.15.28:0/1800356433 >> 172.21.15.28:0/1800356433 conn(0x7f3e7c0d4070 msgr2=0x7f3e7c0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 -- 172.21.15.28:0/1800356433 shutdown_connections 2024-09-10T20:45:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.494+0000 7f3e82a60640 1 -- 172.21.15.28:0/1800356433 wait complete. 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.496+0000 7f3e82a60640 1 Processor -- start 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.496+0000 7f3e82a60640 1 -- start start 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c0d4b90 0x7f3e7c15d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e7c161770 0x7f3e7c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e7c161b50 0x7f3e7c0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0b3f30 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0b3db0 con 0x7f3e7c161b50 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c0b40b0 con 0x7f3e7c161770 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c0d4b90 0x7f3e7c15d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c0d4b90 0x7f3e7c15d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51722/0 (socket says 172.21.15.28:51722) 2024-09-10T20:45:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 -- 172.21.15.28:0/95329469 learned_addr learned my addr 172.21.15.28:0/95329469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7b7fe640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e7c161770 0x7f3e7c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e80ffd640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e7c161b50 0x7f3e7c0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 -- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e7c161770 msgr2=0x7f3e7c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e7c161770 0x7f3e7c0cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 -- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e7c161b50 msgr2=0x7f3e7c0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e7c161b50 0x7f3e7c0cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 -- 172.21.15.28:0/95329469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c1f7cb0 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7b7fe640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e7c161770 0x7f3e7c0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e7bfff640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c0d4b90 0x7f3e7c15d9c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009850 tx=0x7f3e6c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e78ff9640 1 -- 172.21.15.28:0/95329469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c03f070 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e78ff9640 1 -- 172.21.15.28:0/95329469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e6c0047a0 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7c1f8250 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e78ff9640 1 -- 172.21.15.28:0/95329469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c035030 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.497+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7c1f8810 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.498+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e7c0b45d0 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.499+0000 7f3e78ff9640 1 -- 172.21.15.28:0/95329469 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f3e6c004bb0 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.500+0000 7f3e78ff9640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3e680787e0 0x7f3e6807aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.500+0000 7f3e7b7fe640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3e680787e0 0x7f3e6807aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.500+0000 7f3e78ff9640 1 -- 172.21.15.28:0/95329469 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7f3e6c0bbe70 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.500+0000 7f3e7b7fe640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3e680787e0 0x7f3e6807aca0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3e70005fd0 tx=0x7f3e7000f360 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.504+0000 7f3e78ff9640 1 -- 172.21.15.28:0/95329469 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3e6c083cb0 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.687+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e7c06d530 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.688+0000 7f3e78ff9640 1 -- 172.21.15.28:0/95329469 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) ==== 93+0+10 (secure 0 0 0) 0x7f3e6c0835b0 con 0x7f3e7c0d4b90 2024-09-10T20:45:25.692 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:45:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3e680787e0 msgr2=0x7f3e6807aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3e680787e0 0x7f3e6807aca0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3e70005fd0 tx=0x7f3e7000f360 comp rx=0 tx=0).stop 2024-09-10T20:45:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c0d4b90 msgr2=0x7f3e7c15d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c0d4b90 0x7f3e7c15d9c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009850 tx=0x7f3e6c003910 comp rx=0 tx=0).stop 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 shutdown_connections 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3e680787e0 0x7f3e6807aca0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3e7c161b50 0x7f3e7c0cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3e7c161770 0x7f3e7c0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 --2- 172.21.15.28:0/95329469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e7c0d4b90 0x7f3e7c15d9c0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 >> 172.21.15.28:0/95329469 conn(0x7f3e7c0d4070 msgr2=0x7f3e7c0ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 shutdown_connections 2024-09-10T20:45:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.690+0000 7f3e82a60640 1 -- 172.21.15.28:0/95329469 wait complete. 2024-09-10T20:45:25.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2' 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 -- 172.21.15.28:0/507530338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c15ee70 msgr2=0x7fb25c0d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 --2- 172.21.15.28:0/507530338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c15ee70 0x7fb25c0d0140 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fb25c0b2960 tx=0x7fb24c02a560 comp rx=0 tx=0).stop 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 -- 172.21.15.28:0/507530338 shutdown_connections 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 --2- 172.21.15.28:0/507530338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c15ee70 0x7fb25c0d0140 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 --2- 172.21.15.28:0/507530338 >> v2:172.21.15.159:3300/0 conn(0x7fb25c1554e0 0x7fb25c15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 --2- 172.21.15.28:0/507530338 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb25c155100 0x7fb25c15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 -- 172.21.15.28:0/507530338 >> 172.21.15.28:0/507530338 conn(0x7fb25c06cf00 msgr2=0x7fb25c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 -- 172.21.15.28:0/507530338 shutdown_connections 2024-09-10T20:45:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.807+0000 7fb261847640 1 -- 172.21.15.28:0/507530338 wait complete. 2024-09-10T20:45:25.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.808+0000 7fb261847640 1 Processor -- start 2024-09-10T20:45:25.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.808+0000 7fb261847640 1 -- start start 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb261847640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c155100 0x7fb25c140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb261847640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb25c1554e0 0x7fb25c1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb261847640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb25c15ee70 0x7fb25c1f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb261847640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c072bd0 con 0x7fb25c155100 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb261847640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c0d0ab0 con 0x7fb25c15ee70 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb261847640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb25c072d50 con 0x7fb25c1554e0 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c155100 0x7fb25c140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c155100 0x7fb25c140390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51750/0 (socket says 172.21.15.28:51750) 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 -- 172.21.15.28:0/122478460 learned_addr learned my addr 172.21.15.28:0/122478460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25b7fe640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb25c15ee70 0x7fb25c1f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25a7fc640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb25c1554e0 0x7fb25c1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 -- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb25c1554e0 msgr2=0x7fb25c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb25c1554e0 0x7fb25c1408d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 -- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb25c15ee70 msgr2=0x7fb25c1f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb25c15ee70 0x7fb25c1f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 -- 172.21.15.28:0/122478460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c141500 con 0x7fb25c155100 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb25affd640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c155100 0x7fb25c140390 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fb25400eb00 tx=0x7fb254009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb253fff640 1 -- 172.21.15.28:0/122478460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254004b20 con 0x7fb25c155100 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb253fff640 1 -- 172.21.15.28:0/122478460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb254004db0 con 0x7fb25c155100 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25c1417c0 con 0x7fb25c155100 2024-09-10T20:45:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.809+0000 7fb253fff640 1 -- 172.21.15.28:0/122478460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254005280 con 0x7fb25c155100 2024-09-10T20:45:25.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.810+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25c141d80 con 0x7fb25c155100 2024-09-10T20:45:25.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.810+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb25c15e930 con 0x7fb25c155100 2024-09-10T20:45:25.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.811+0000 7fb253fff640 1 -- 172.21.15.28:0/122478460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fb254005420 con 0x7fb25c155100 2024-09-10T20:45:25.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.812+0000 7fb253fff640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb23c0787e0 0x7fb23c07aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:25.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.812+0000 7fb253fff640 1 -- 172.21.15.28:0/122478460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7fb254099460 con 0x7fb25c155100 2024-09-10T20:45:25.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.812+0000 7fb25a7fc640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb23c0787e0 0x7fb23c07aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:25.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.812+0000 7fb25a7fc640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb23c0787e0 0x7fb23c07aca0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb24400b3c0 tx=0x7fb244005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:25.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:25.816+0000 7fb253fff640 1 -- 172.21.15.28:0/122478460 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb254063a10 con 0x7fb25c155100 2024-09-10T20:45:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:26.074+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]} v 0) -- 0x7fb25c140e10 con 0x7fb25c155100 2024-09-10T20:45:27.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.054+0000 7fb253fff640 1 -- 172.21.15.28:0/122478460 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]}]=0 v402) ==== 148+0+0 (secure 0 0 0) 0x7fb25406c130 con 0x7fb25c155100 2024-09-10T20:45:27.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb23c0787e0 msgr2=0x7fb23c07aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb23c0787e0 0x7fb23c07aca0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb24400b3c0 tx=0x7fb244005f90 comp rx=0 tx=0).stop 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c155100 msgr2=0x7fb25c140390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c155100 0x7fb25c140390 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fb25400eb00 tx=0x7fb254009f70 comp rx=0 tx=0).stop 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 shutdown_connections 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb23c0787e0 0x7fb23c07aca0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb25c15ee70 0x7fb25c1f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb25c1554e0 0x7fb25c1408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 --2- 172.21.15.28:0/122478460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25c155100 0x7fb25c140390 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.057+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 >> 172.21.15.28:0/122478460 conn(0x7fb25c06cf00 msgr2=0x7fb25c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.058+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 shutdown_connections 2024-09-10T20:45:27.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.058+0000 7fb261847640 1 -- 172.21.15.28:0/122478460 wait complete. 2024-09-10T20:45:27.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.183+0000 7ffac0471640 1 -- 172.21.15.28:0/2421608575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab80cf780 msgr2=0x7ffab80cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.183+0000 7ffac0471640 1 --2- 172.21.15.28:0/2421608575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab80cf780 0x7ffab80cfbf0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7ffab808c020 tx=0x7ffaac02aa30 comp rx=0 tx=0).stop 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.184+0000 7ffac0471640 1 -- 172.21.15.28:0/2421608575 shutdown_connections 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.184+0000 7ffac0471640 1 --2- 172.21.15.28:0/2421608575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab80cf780 0x7ffab80cfbf0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.184+0000 7ffac0471640 1 --2- 172.21.15.28:0/2421608575 >> v2:172.21.15.159:3300/0 conn(0x7ffab815de70 0x7ffab80d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.184+0000 7ffac0471640 1 -- 172.21.15.28:0/2421608575 >> 172.21.15.28:0/2421608575 conn(0x7ffab80d3fb0 msgr2=0x7ffab80d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.184+0000 7ffac0471640 1 -- 172.21.15.28:0/2421608575 shutdown_connections 2024-09-10T20:45:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.184+0000 7ffac0471640 1 -- 172.21.15.28:0/2421608575 wait complete. 2024-09-10T20:45:27.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.185+0000 7ffac0471640 1 Processor -- start 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.185+0000 7ffac0471640 1 -- start start 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.185+0000 7ffac0471640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffab80cf780 0x7ffab81f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.185+0000 7ffac0471640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab815de70 0x7ffab80b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffac0471640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab8161b50 0x7ffab80b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffac0471640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffab80b7690 con 0x7ffab815de70 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffac0471640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffab80b7510 con 0x7ffab8161b50 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffac0471640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffab80b7810 con 0x7ffab80cf780 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab815de70 0x7ffab80b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab815de70 0x7ffab80b0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51764/0 (socket says 172.21.15.28:51764) 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 -- 172.21.15.28:0/2119731858 learned_addr learned my addr 172.21.15.28:0/2119731858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabea0e640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab8161b50 0x7ffab80b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabe20d640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffab80cf780 0x7ffab81f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 -- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffab80cf780 msgr2=0x7ffab81f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffab80cf780 0x7ffab81f53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 -- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab8161b50 msgr2=0x7ffab80b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab8161b50 0x7ffab80b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 -- 172.21.15.28:0/2119731858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab81607c0 con 0x7ffab815de70 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabe20d640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffab80cf780 0x7ffab81f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffabda0c640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab815de70 0x7ffab80b0ae0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009bf0 tx=0x7ffaac003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffab6ffd640 1 -- 172.21.15.28:0/2119731858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac047070 con 0x7ffab815de70 2024-09-10T20:45:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffab6ffd640 1 -- 172.21.15.28:0/2119731858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffaac004250 con 0x7ffab815de70 2024-09-10T20:45:27.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab80b17f0 con 0x7ffab815de70 2024-09-10T20:45:27.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffab6ffd640 1 -- 172.21.15.28:0/2119731858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac033050 con 0x7ffab815de70 2024-09-10T20:45:27.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.186+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab80b1510 con 0x7ffab815de70 2024-09-10T20:45:27.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.187+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab80cfbf0 con 0x7ffab815de70 2024-09-10T20:45:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.188+0000 7ffab6ffd640 1 -- 172.21.15.28:0/2119731858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7ffaac004660 con 0x7ffab815de70 2024-09-10T20:45:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.189+0000 7ffab6ffd640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffaa40787e0 0x7ffaa407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.189+0000 7ffabe20d640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffaa40787e0 0x7ffaa407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.189+0000 7ffab6ffd640 1 -- 172.21.15.28:0/2119731858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7ffaac0bc0a0 con 0x7ffab815de70 2024-09-10T20:45:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.189+0000 7ffabe20d640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffaa40787e0 0x7ffaa407aca0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80058c0 tx=0x7ffaa8005d50 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:27.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.193+0000 7ffab6ffd640 1 -- 172.21.15.28:0/2119731858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffaac0842f0 con 0x7ffab815de70 2024-09-10T20:45:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.379+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7ffab80b9f30 con 0x7ffab815de70 2024-09-10T20:45:27.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.379+0000 7ffab6ffd640 1 -- 172.21.15.28:0/2119731858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v45) ==== 59+0+1998 (secure 0 0 0) 0x7ffaac083bf0 con 0x7ffab815de70 2024-09-10T20:45:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:45:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:45:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:45:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:45:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:45:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:45:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffaa40787e0 msgr2=0x7ffaa407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffaa40787e0 0x7ffaa407aca0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80058c0 tx=0x7ffaa8005d50 comp rx=0 tx=0).stop 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab815de70 msgr2=0x7ffab80b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab815de70 0x7ffab80b0ae0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009bf0 tx=0x7ffaac003f10 comp rx=0 tx=0).stop 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 shutdown_connections 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffaa40787e0 0x7ffaa407aca0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab8161b50 0x7ffab80b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab815de70 0x7ffab80b0ae0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 --2- 172.21.15.28:0/2119731858 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffab80cf780 0x7ffab81f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.381+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 >> 172.21.15.28:0/2119731858 conn(0x7ffab80d3fb0 msgr2=0x7ffab8075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:27.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.382+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 shutdown_connections 2024-09-10T20:45:27.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.382+0000 7ffac0471640 1 -- 172.21.15.28:0/2119731858 wait complete. 2024-09-10T20:45:27.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 -- 172.21.15.28:0/876122020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 msgr2=0x7f42d015e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 --2- 172.21.15.28:0/876122020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d015e8f0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f42d00d1dd0 tx=0x7f42c802a9a0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 -- 172.21.15.28:0/876122020 shutdown_connections 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 --2- 172.21.15.28:0/876122020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d015e8f0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 --2- 172.21.15.28:0/876122020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42d015bd80 0x7f42d0073790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 --2- 172.21.15.28:0/876122020 >> v2:172.21.15.159:3300/0 conn(0x7f42d0151db0 0x7f42d00706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 -- 172.21.15.28:0/876122020 >> 172.21.15.28:0/876122020 conn(0x7f42d00d0d50 msgr2=0x7f42d00d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 -- 172.21.15.28:0/876122020 shutdown_connections 2024-09-10T20:45:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.502+0000 7f42d656b640 1 -- 172.21.15.28:0/876122020 wait complete. 2024-09-10T20:45:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.503+0000 7f42d656b640 1 Processor -- start 2024-09-10T20:45:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.503+0000 7f42d656b640 1 -- start start 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42d656b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d01f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42d656b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42d0151db0 0x7f42d01464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42d656b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42d015bd80 0x7f42d00cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42d656b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42d015f1c0 con 0x7f42d0073cd0 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42d656b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f42d015f040 con 0x7f42d015bd80 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42d656b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f42d015f340 con 0x7f42d0151db0 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d01f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d01f59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51780/0 (socket says 172.21.15.28:51780) 2024-09-10T20:45:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 -- 172.21.15.28:0/1440169996 learned_addr learned my addr 172.21.15.28:0/1440169996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42d4b08640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42d015bd80 0x7f42d00cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cf7fe640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42d0151db0 0x7f42d01464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 -- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42d0151db0 msgr2=0x7f42d01464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42d0151db0 0x7f42d01464b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 -- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42d015bd80 msgr2=0x7f42d00cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42d015bd80 0x7f42d00cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.504+0000 7f42cffff640 1 -- 172.21.15.28:0/1440169996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d0146c00 con 0x7f42d0073cd0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.505+0000 7f42cffff640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d01f59c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f42c0005940 tx=0x7f42c000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.505+0000 7f42ccff9640 1 -- 172.21.15.28:0/1440169996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c0007bb0 con 0x7f42d0073cd0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.505+0000 7f42ccff9640 1 -- 172.21.15.28:0/1440169996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42c0007e40 con 0x7f42d0073cd0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.505+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d0146ee0 con 0x7f42d0073cd0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.505+0000 7f42ccff9640 1 -- 172.21.15.28:0/1440169996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c000f6b0 con 0x7f42d0073cd0 2024-09-10T20:45:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.505+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d00c6930 con 0x7f42d0073cd0 2024-09-10T20:45:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.506+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d00706e0 con 0x7f42d0073cd0 2024-09-10T20:45:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.506+0000 7f42ccff9640 1 -- 172.21.15.28:0/1440169996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f42c0020030 con 0x7f42d0073cd0 2024-09-10T20:45:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.507+0000 7f42ccff9640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42bc0787e0 0x7f42bc07aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.507+0000 7f42ccff9640 1 -- 172.21.15.28:0/1440169996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7f42c009b300 con 0x7f42d0073cd0 2024-09-10T20:45:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.507+0000 7f42cf7fe640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42bc0787e0 0x7f42bc07aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.508+0000 7f42cf7fe640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42bc0787e0 0x7f42bc07aca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f42d015bbc0 tx=0x7f42c4005c20 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.511+0000 7f42ccff9640 1 -- 172.21.15.28:0/1440169996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f42c0014030 con 0x7f42d0073cd0 2024-09-10T20:45:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.695+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f42d00c6b00 con 0x7f42d0073cd0 2024-09-10T20:45:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.699+0000 7f42ccff9640 1 -- 172.21.15.28:0/1440169996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7f42c00630b0 con 0x7f42d0073cd0 2024-09-10T20:45:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42bc0787e0 msgr2=0x7f42bc07aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42bc0787e0 0x7f42bc07aca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f42d015bbc0 tx=0x7f42c4005c20 comp rx=0 tx=0).stop 2024-09-10T20:45:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 msgr2=0x7f42d01f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d01f59c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f42c0005940 tx=0x7f42c000cd60 comp rx=0 tx=0).stop 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 shutdown_connections 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f42bc0787e0 0x7f42bc07aca0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42d015bd80 0x7f42d00cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42d0151db0 0x7f42d01464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 --2- 172.21.15.28:0/1440169996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d0073cd0 0x7f42d01f59c0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 >> 172.21.15.28:0/1440169996 conn(0x7f42d00d0d50 msgr2=0x7f42d00cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 shutdown_connections 2024-09-10T20:45:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:27.701+0000 7f42d656b640 1 -- 172.21.15.28:0/1440169996 wait complete. 2024-09-10T20:45:27.716 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:45:27.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:27.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:27.811 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:45:27.811 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:27.811 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:27.828 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:45:27.828 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:27.828 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:27.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:45:27.846 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:27.847 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:27.884 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:45:27.884 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:27.884 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:27.905 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:45:27.905 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:45:27.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 -- 172.21.15.28:0/4061569120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80162260 msgr2=0x7f0d800cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 --2- 172.21.15.28:0/4061569120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80162260 0x7f0d800cfbf0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f0d8008c020 tx=0x7f0d7002aa30 comp rx=0 tx=0).stop 2024-09-10T20:45:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 -- 172.21.15.28:0/4061569120 shutdown_connections 2024-09-10T20:45:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 --2- 172.21.15.28:0/4061569120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80162260 0x7f0d800cfbf0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 --2- 172.21.15.28:0/4061569120 >> v2:172.21.15.159:3300/0 conn(0x7f0d8015de70 0x7f0d800d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 -- 172.21.15.28:0/4061569120 >> 172.21.15.28:0/4061569120 conn(0x7f0d800d3fb0 msgr2=0x7f0d800d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 -- 172.21.15.28:0/4061569120 shutdown_connections 2024-09-10T20:45:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.025+0000 7f0d85672640 1 -- 172.21.15.28:0/4061569120 wait complete. 2024-09-10T20:45:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.026+0000 7f0d85672640 1 Processor -- start 2024-09-10T20:45:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d85672640 1 -- start start 2024-09-10T20:45:28.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d85672640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d800cf780 0x7f0d801f8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d85672640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d8015de70 0x7f0d800b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d85672640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d80162260 0x7f0d801f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d85672640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d800b6f80 con 0x7f0d800cf780 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d85672640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0d800b6e00 con 0x7f0d80162260 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d85672640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0d800b7100 con 0x7f0d8015de70 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d800cf780 0x7f0d801f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d800cf780 0x7f0d801f8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51796/0 (socket says 172.21.15.28:51796) 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.027+0000 7f0d7effd640 1 -- 172.21.15.28:0/2723399268 learned_addr learned my addr 172.21.15.28:0/2723399268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7f7fe640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d80162260 0x7f0d801f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7e7fc640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d8015de70 0x7f0d800b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7effd640 1 -- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d8015de70 msgr2=0x7f0d800b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7effd640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d8015de70 0x7f0d800b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7effd640 1 -- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d80162260 msgr2=0x7f0d801f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7effd640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d80162260 0x7f0d801f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7effd640 1 -- 172.21.15.28:0/2723399268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d801f8d90 con 0x7f0d800cf780 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7f7fe640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d80162260 0x7f0d801f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d7effd640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d800cf780 0x7f0d801f8850 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800c970 tx=0x7f0d7800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d777fe640 1 -- 172.21.15.28:0/2723399268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78007bb0 con 0x7f0d800cf780 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d777fe640 1 -- 172.21.15.28:0/2723399268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d78005020 con 0x7f0d800cf780 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d800cd090 con 0x7f0d800cf780 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d777fe640 1 -- 172.21.15.28:0/2723399268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78007e60 con 0x7f0d800cf780 2024-09-10T20:45:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.028+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d800cd930 con 0x7f0d800cf780 2024-09-10T20:45:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.029+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d800d2b90 con 0x7f0d800cf780 2024-09-10T20:45:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.030+0000 7f0d777fe640 1 -- 172.21.15.28:0/2723399268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f0d78005470 con 0x7f0d800cf780 2024-09-10T20:45:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.030+0000 7f0d777fe640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0d600787e0 0x7f0d6007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.030+0000 7f0d777fe640 1 -- 172.21.15.28:0/2723399268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7f0d78098aa0 con 0x7f0d800cf780 2024-09-10T20:45:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.030+0000 7f0d7e7fc640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0d600787e0 0x7f0d6007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.031+0000 7f0d7e7fc640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0d600787e0 0x7f0d6007aca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0d8015beb0 tx=0x7f0d68009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:28.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.034+0000 7f0d777fe640 1 -- 172.21.15.28:0/2723399268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0d801f8d90 con 0x7f0d800cf780 2024-09-10T20:45:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.257+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d800b9f30 con 0x7f0d800cf780 2024-09-10T20:45:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.258+0000 7f0d777fe640 1 -- 172.21.15.28:0/2723399268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+3914 (secure 0 0 0) 0x7f0d78014030 con 0x7f0d800cf780 2024-09-10T20:45:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:28.262 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"btime":"2024-09-10T20:45:22:037056+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6980,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/1118874936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":1118874936},{"type":"v1","addr":"172.21.15.159:6801","nonce":1118874936}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":7336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4242811460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":4242811460},{"type":"v1","addr":"172.21.15.28:6834","nonce":4242811460}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/914514353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":914514353},{"type":"v1","addr":"172.21.15.159:6835","nonce":914514353}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8334,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2842589826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2842589826},{"type":"v1","addr":"172.21.15.28:6835","nonce":2842589826}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[]} 2024-09-10T20:45:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 99 2024-09-10T20:45:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0d600787e0 msgr2=0x7f0d6007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0d600787e0 0x7f0d6007aca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0d8015beb0 tx=0x7f0d68009040 comp rx=0 tx=0).stop 2024-09-10T20:45:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d800cf780 msgr2=0x7f0d801f8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d800cf780 0x7f0d801f8850 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800c970 tx=0x7f0d7800ce30 comp rx=0 tx=0).stop 2024-09-10T20:45:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 shutdown_connections 2024-09-10T20:45:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0d600787e0 0x7f0d6007aca0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d80162260 0x7f0d801f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d8015de70 0x7f0d800b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 --2- 172.21.15.28:0/2723399268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d800cf780 0x7f0d801f8850 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 >> 172.21.15.28:0/2723399268 conn(0x7f0d800d3fb0 msgr2=0x7f0d8006eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 shutdown_connections 2024-09-10T20:45:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.260+0000 7f0d85672640 1 -- 172.21.15.28:0/2723399268 wait complete. 2024-09-10T20:45:28.275 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:45:28.275 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:45:28.276 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:45:28.276 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:28.276 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:45:28.276 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:45:28.276 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:45:28.276 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:28.276 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:45:28.276 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:45:28.277 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:45:28.277 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:28.277 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:45:28.277 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:45:28.277 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:45:28.277 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:28.285 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:45:28.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:45:28.286 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:45:28.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:45:28.288 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:45:28.288 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:45:28.289 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:45:28.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:45:28.328 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:45:28.329 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:28.330 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:45:28.330 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:28.331 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:45:28.331 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:28.332 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:45:28.332 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:28.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:45:28.407 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:45:28.407 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:45:28.412 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:45:28.412 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:45:28.413 INFO:tasks.ceph.mds.b.smithi159.stderr:2024-09-10T20:45:28.408+0000 7f6069f2e640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-09-10T20:45:28.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.439+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/221130091 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 msgr2=0x7fa1b40b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.439+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/221130091 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 0x7fa1b40b6e00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40d1d00 tx=0x7fa1ac02a890 comp rx=0 tx=0).stop 2024-09-10T20:45:28.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.439+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/221130091 shutdown_connections 2024-09-10T20:45:28.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.439+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/221130091 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 0x7fa1b40b6e00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.439+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/221130091 >> 172.21.15.28:0/221130091 conn(0x7fa1b40d3fb0 msgr2=0x7fa1b40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:28.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.439+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/221130091 shutdown_connections 2024-09-10T20:45:28.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.439+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/221130091 wait complete. 2024-09-10T20:45:28.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 Processor -- start 2024-09-10T20:45:28.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 -- start start 2024-09-10T20:45:28.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b40d2b90 0x7fa1b415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1b415de70 0x7fa1b40a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 0x7fa1b41f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b41629b0 con 0x7fa1b40d2b90 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b415e460 con 0x7fa1b4162260 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1bbbbb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa1b40b7340 con 0x7fa1b415de70 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.441+0000 7fa1b9957640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b40d2b90 0x7fa1b415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b40d2b90 0x7fa1b415b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51840/0 (socket says 172.21.15.28:51840) 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 -- 172.21.15.28:0/2109476833 learned_addr learned my addr 172.21.15.28:0/2109476833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1ba158640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 0x7fa1b41f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9156640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1b415de70 0x7fa1b40a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 -- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1b415de70 msgr2=0x7fa1b40a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1b415de70 0x7fa1b40a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 -- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 msgr2=0x7fa1b41f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 0x7fa1b41f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:28.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 -- 172.21.15.28:0/2109476833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b4160a00 con 0x7fa1b40d2b90 2024-09-10T20:45:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9156640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1b415de70 0x7fa1b40a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1b9957640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b40d2b90 0x7fa1b415b5c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000c450 tx=0x7fa1b000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1aa7fc640 1 -- 172.21.15.28:0/2109476833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00052d0 con 0x7fa1b40d2b90 2024-09-10T20:45:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1aa7fc640 1 -- 172.21.15.28:0/2109476833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1b0005470 con 0x7fa1b40d2b90 2024-09-10T20:45:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.442+0000 7fa1aa7fc640 1 -- 172.21.15.28:0/2109476833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0012870 con 0x7fa1b40d2b90 2024-09-10T20:45:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.443+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b40a1b60 con 0x7fa1b40d2b90 2024-09-10T20:45:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.443+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b4160f80 con 0x7fa1b40d2b90 2024-09-10T20:45:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.444+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b415bb50 con 0x7fa1b40d2b90 2024-09-10T20:45:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.444+0000 7fa1aa7fc640 1 -- 172.21.15.28:0/2109476833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fa1b00035f0 con 0x7fa1b40d2b90 2024-09-10T20:45:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.445+0000 7fa1aa7fc640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1980787e0 0x7fa19807aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.445+0000 7fa1aa7fc640 1 -- 172.21.15.28:0/2109476833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 4971+0+0 (secure 0 0 0) 0x7fa1b0098150 con 0x7fa1b40d2b90 2024-09-10T20:45:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.445+0000 7fa1b9156640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1980787e0 0x7fa19807aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.445+0000 7fa1b9156640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1980787e0 0x7fa19807aca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4001eb0 tx=0x7fa1a4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:28.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.449+0000 7fa1aa7fc640 1 -- 172.21.15.28:0/2109476833 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa1b0062680 con 0x7fa1b40d2b90 2024-09-10T20:45:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:28.648+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa1b40b9f30 con 0x7fa1b40d2b90 2024-09-10T20:45:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.074+0000 7fa1aa7fc640 1 -- 172.21.15.28:0/2109476833 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v145) ==== 97+0+0 (secure 0 0 0) 0x7fa1b0061f80 con 0x7fa1b40d2b90 2024-09-10T20:45:29.078 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:45:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1980787e0 msgr2=0x7fa19807aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1980787e0 0x7fa19807aca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4001eb0 tx=0x7fa1a4009040 comp rx=0 tx=0).stop 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b40d2b90 msgr2=0x7fa1b415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b40d2b90 0x7fa1b415b5c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000c450 tx=0x7fa1b000c910 comp rx=0 tx=0).stop 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 shutdown_connections 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa1980787e0 0x7fa19807aca0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa1b4162260 0x7fa1b41f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1b415de70 0x7fa1b40a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 --2- 172.21.15.28:0/2109476833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b40d2b90 0x7fa1b415b5c0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 >> 172.21.15.28:0/2109476833 conn(0x7fa1b40d3fb0 msgr2=0x7fa1b40cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 shutdown_connections 2024-09-10T20:45:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.077+0000 7fa1bbbbb640 1 -- 172.21.15.28:0/2109476833 wait complete. 2024-09-10T20:45:29.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:45:29.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2123988895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 msgr2=0x7f4ae40d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2123988895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40d2b90 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40d4d80 tx=0x7f4ad402a560 comp rx=0 tx=0).stop 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2123988895 shutdown_connections 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2123988895 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ae4161b50 0x7f4ae4161f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2123988895 >> v2:172.21.15.159:3300/0 conn(0x7f4ae40cf780 0x7f4ae40cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2123988895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40d2b90 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2123988895 >> 172.21.15.28:0/2123988895 conn(0x7f4ae40d3fb0 msgr2=0x7f4ae40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2123988895 shutdown_connections 2024-09-10T20:45:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.196+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2123988895 wait complete. 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.198+0000 7f4ae8a8c640 1 Processor -- start 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.198+0000 7f4ae8a8c640 1 -- start start 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.198+0000 7f4ae8a8c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ae40cf780 0x7f4ae40b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae8a8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae8a8c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ae4161b50 0x7f4ae4152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae8a8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae40b6e00 con 0x7f4ae415de70 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae8a8c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae415e460 con 0x7f4ae40cf780 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae8a8c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae40b6f80 con 0x7f4ae4161b50 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51854/0 (socket says 172.21.15.28:51854) 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 -- 172.21.15.28:0/2171251500 learned_addr learned my addr 172.21.15.28:0/2171251500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae2d9d640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ae4161b50 0x7f4ae4152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae259c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ae40cf780 0x7f4ae40b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 -- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ae4161b50 msgr2=0x7f4ae4152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ae4161b50 0x7f4ae4152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 -- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ae40cf780 msgr2=0x7f4ae40b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ae40cf780 0x7f4ae40b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 -- 172.21.15.28:0/2171251500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae40b46d0 con 0x7f4ae415de70 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adbfff640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40b2720 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f4acc00b4a0 tx=0x7f4acc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adaffd640 1 -- 172.21.15.28:0/2171251500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc014070 con 0x7f4ae415de70 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adaffd640 1 -- 172.21.15.28:0/2171251500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4acc010700 con 0x7f4ae415de70 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae40b0c80 con 0x7f4ae415de70 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4adaffd640 1 -- 172.21.15.28:0/2171251500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc010b50 con 0x7f4ae415de70 2024-09-10T20:45:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.199+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae40b1240 con 0x7f4ae415de70 2024-09-10T20:45:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.200+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae40cd7a0 con 0x7f4ae415de70 2024-09-10T20:45:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.201+0000 7f4adaffd640 1 -- 172.21.15.28:0/2171251500 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f4acc017360 con 0x7f4ae415de70 2024-09-10T20:45:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.201+0000 7f4adaffd640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ac40787e0 0x7f4ac407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.202+0000 7f4adaffd640 1 -- 172.21.15.28:0/2171251500 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 4885+0+0 (secure 0 0 0) 0x7f4acc098360 con 0x7f4ae415de70 2024-09-10T20:45:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.202+0000 7f4ae259c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ac40787e0 0x7f4ac407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.202+0000 7f4ae259c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ac40787e0 0x7f4ac407aca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4033000 tx=0x7f4ad4004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:29.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.206+0000 7f4adaffd640 1 -- 172.21.15.28:0/2171251500 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4ae40cd7a0 con 0x7f4ae415de70 2024-09-10T20:45:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.469+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4ae40b1410 con 0x7f4ae415de70 2024-09-10T20:45:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.469+0000 7f4adaffd640 1 -- 172.21.15.28:0/2171251500 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) ==== 80+0+5304 (secure 0 0 0) 0x7f4acc062910 con 0x7f4ae415de70 2024-09-10T20:45:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:29.473 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:45:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:45:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:45:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:45:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:45:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:45:29.486 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ac40787e0 msgr2=0x7f4ac407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ac40787e0 0x7f4ac407aca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4033000 tx=0x7f4ad4004730 comp rx=0 tx=0).stop 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 msgr2=0x7f4ae40b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40b2720 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f4acc00b4a0 tx=0x7f4acc00b960 comp rx=0 tx=0).stop 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 shutdown_connections 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4ac40787e0 0x7f4ac407aca0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4ae4161b50 0x7f4ae4152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ae415de70 0x7f4ae40b2720 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 --2- 172.21.15.28:0/2171251500 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4ae40cf780 0x7f4ae40b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 >> 172.21.15.28:0/2171251500 conn(0x7f4ae40d3fb0 msgr2=0x7f4ae40cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 shutdown_connections 2024-09-10T20:45:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.472+0000 7f4ae8a8c640 1 -- 172.21.15.28:0/2171251500 wait complete. 2024-09-10T20:45:29.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 -- 172.21.15.28:0/4257353666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded40b1320 msgr2=0x7fded40d03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 --2- 172.21.15.28:0/4257353666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded40b1320 0x7fded40d03f0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fded4072460 tx=0x7fdec802a440 comp rx=0 tx=0).stop 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 -- 172.21.15.28:0/4257353666 shutdown_connections 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 --2- 172.21.15.28:0/4257353666 >> v2:172.21.15.159:3300/0 conn(0x7fded4071a30 0x7fded40b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 --2- 172.21.15.28:0/4257353666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded40b1320 0x7fded40d03f0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 --2- 172.21.15.28:0/4257353666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fded4070b90 0x7fded4070f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 -- 172.21.15.28:0/4257353666 >> 172.21.15.28:0/4257353666 conn(0x7fded40d1e10 msgr2=0x7fded40d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 -- 172.21.15.28:0/4257353666 shutdown_connections 2024-09-10T20:45:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.590+0000 7fded9153640 1 -- 172.21.15.28:0/4257353666 wait complete. 2024-09-10T20:45:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.591+0000 7fded9153640 1 Processor -- start 2024-09-10T20:45:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.591+0000 7fded9153640 1 -- start start 2024-09-10T20:45:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded9153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded4070b90 0x7fded41f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded9153640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fded4071a30 0x7fded40cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded9153640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fded40b1320 0x7fded414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded9153640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fded40b0bb0 con 0x7fded4070b90 2024-09-10T20:45:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded9153640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fded40b0a30 con 0x7fded4071a30 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded9153640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fded40b0d30 con 0x7fded40b1320 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded4070b90 0x7fded41f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded4070b90 0x7fded41f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51858/0 (socket says 172.21.15.28:51858) 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.592+0000 7fded2d9d640 1 -- 172.21.15.28:0/2639944407 learned_addr learned my addr 172.21.15.28:0/2639944407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded259c640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fded4071a30 0x7fded40cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded2d9d640 1 -- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fded40b1320 msgr2=0x7fded414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded359e640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fded40b1320 0x7fded414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded2d9d640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fded40b1320 0x7fded414e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded2d9d640 1 -- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fded4071a30 msgr2=0x7fded40cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded2d9d640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fded4071a30 0x7fded40cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded2d9d640 1 -- 172.21.15.28:0/2639944407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded414f380 con 0x7fded4070b90 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded259c640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fded4071a30 0x7fded40cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded359e640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fded40b1320 0x7fded414e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded2d9d640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded4070b90 0x7fded41f6b50 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fdec400d750 tx=0x7fdec400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fdebb7fe640 1 -- 172.21.15.28:0/2639944407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4017070 con 0x7fded4070b90 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fdebb7fe640 1 -- 172.21.15.28:0/2639944407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdec4009d20 con 0x7fded4070b90 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded414f660 con 0x7fded4070b90 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fdebb7fe640 1 -- 172.21.15.28:0/2639944407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4004280 con 0x7fded4070b90 2024-09-10T20:45:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.593+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded4152b30 con 0x7fded4070b90 2024-09-10T20:45:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.594+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded41f6780 con 0x7fded4070b90 2024-09-10T20:45:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.595+0000 7fdebb7fe640 1 -- 172.21.15.28:0/2639944407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fdec4005020 con 0x7fded4070b90 2024-09-10T20:45:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.595+0000 7fdebb7fe640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdec00787e0 0x7fdec007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.595+0000 7fdebb7fe640 1 -- 172.21.15.28:0/2639944407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 4885+0+0 (secure 0 0 0) 0x7fdec4098ae0 con 0x7fded4070b90 2024-09-10T20:45:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.595+0000 7fded259c640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdec00787e0 0x7fdec007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.596+0000 7fded259c640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdec00787e0 0x7fdec007aca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdec8033000 tx=0x7fdec8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.600+0000 7fdebb7fe640 1 -- 172.21.15.28:0/2639944407 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdec4020020 con 0x7fded4070b90 2024-09-10T20:45:29.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.788+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fded40b04f0 con 0x7fded4070b90 2024-09-10T20:45:29.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.788+0000 7fdebb7fe640 1 -- 172.21.15.28:0/2639944407 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v147) ==== 84+0+65 (secure 0 0 0) 0x7fdec4063090 con 0x7fded4070b90 2024-09-10T20:45:29.793 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:29.793 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:45:29.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:29.793 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:45:29.793 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:45:29.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:29.793 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdec00787e0 msgr2=0x7fdec007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdec00787e0 0x7fdec007aca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdec8033000 tx=0x7fdec8004730 comp rx=0 tx=0).stop 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded4070b90 msgr2=0x7fded41f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded4070b90 0x7fded41f6b50 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fdec400d750 tx=0x7fdec400dc10 comp rx=0 tx=0).stop 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 shutdown_connections 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdec00787e0 0x7fdec007aca0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fded40b1320 0x7fded414e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fded4071a30 0x7fded40cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 --2- 172.21.15.28:0/2639944407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded4070b90 0x7fded41f6b50 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 >> 172.21.15.28:0/2639944407 conn(0x7fded40d1e10 msgr2=0x7fded40b7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 shutdown_connections 2024-09-10T20:45:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.790+0000 7fded9153640 1 -- 172.21.15.28:0/2639944407 wait complete. 2024-09-10T20:45:29.805 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:45:29.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:45:29.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.910+0000 7fb42d223640 1 -- 172.21.15.28:0/3979021969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 msgr2=0x7fb428166d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.910+0000 7fb42d223640 1 --2- 172.21.15.28:0/3979021969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb428166d80 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fb420005fd0 tx=0x7fb42002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.910+0000 7fb42d223640 1 -- 172.21.15.28:0/3979021969 shutdown_connections 2024-09-10T20:45:29.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.910+0000 7fb42d223640 1 --2- 172.21.15.28:0/3979021969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb428166d80 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.910+0000 7fb42d223640 1 -- 172.21.15.28:0/3979021969 >> 172.21.15.28:0/3979021969 conn(0x7fb4280d0410 msgr2=0x7fb4280d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:29.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.910+0000 7fb42d223640 1 -- 172.21.15.28:0/3979021969 shutdown_connections 2024-09-10T20:45:29.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.910+0000 7fb42d223640 1 -- 172.21.15.28:0/3979021969 wait complete. 2024-09-10T20:45:29.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.911+0000 7fb42d223640 1 Processor -- start 2024-09-10T20:45:29.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.911+0000 7fb42d223640 1 -- start start 2024-09-10T20:45:29.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42d223640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb4280b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42d223640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4280d0f30 0x7fb428153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42d223640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb42815bfc0 0x7fb428153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42d223640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb428167440 con 0x7fb4280b10d0 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42d223640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb4281672c0 con 0x7fb42815bfc0 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42d223640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb4281675c0 con 0x7fb4280d0f30 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb426d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb4280b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb426d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb4280b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51870/0 (socket says 172.21.15.28:51870) 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb426d9d640 1 -- 172.21.15.28:0/1327409602 learned_addr learned my addr 172.21.15.28:0/1327409602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42759e640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb42815bfc0 0x7fb428153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.912+0000 7fb42659c640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4280d0f30 0x7fb428153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb426d9d640 1 -- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4280d0f30 msgr2=0x7fb428153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb426d9d640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4280d0f30 0x7fb428153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb426d9d640 1 -- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb42815bfc0 msgr2=0x7fb428153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb426d9d640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb42815bfc0 0x7fb428153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:29.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb426d9d640 1 -- 172.21.15.28:0/1327409602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4281f64e0 con 0x7fb4280b10d0 2024-09-10T20:45:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb426d9d640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb4280b36e0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fb41800c3a0 tx=0x7fb41800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb40f7fe640 1 -- 172.21.15.28:0/1327409602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb418011790 con 0x7fb4280b10d0 2024-09-10T20:45:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb40f7fe640 1 -- 172.21.15.28:0/1327409602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb418017070 con 0x7fb4280b10d0 2024-09-10T20:45:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4281f67c0 con 0x7fb4280b10d0 2024-09-10T20:45:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb40f7fe640 1 -- 172.21.15.28:0/1327409602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb418011cc0 con 0x7fb4280b10d0 2024-09-10T20:45:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.913+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4281f6d80 con 0x7fb4280b10d0 2024-09-10T20:45:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.914+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb428072680 con 0x7fb4280b10d0 2024-09-10T20:45:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.915+0000 7fb40f7fe640 1 -- 172.21.15.28:0/1327409602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fb41801c460 con 0x7fb4280b10d0 2024-09-10T20:45:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.915+0000 7fb40f7fe640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4140787e0 0x7fb41407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.915+0000 7fb40f7fe640 1 -- 172.21.15.28:0/1327409602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 4885+0+0 (secure 0 0 0) 0x7fb418098370 con 0x7fb4280b10d0 2024-09-10T20:45:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.915+0000 7fb42659c640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4140787e0 0x7fb41407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.916+0000 7fb42659c640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4140787e0 0x7fb41407aca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb4280d00d0 tx=0x7fb41c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:29.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:29.919+0000 7fb40f7fe640 1 -- 172.21.15.28:0/1327409602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb428072680 con 0x7fb4280b10d0 2024-09-10T20:45:30.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:30.108+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb42815aba0 con 0x7fb4280b10d0 2024-09-10T20:45:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.078+0000 7fb40f7fe640 1 -- 172.21.15.28:0/1327409602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v148) ==== 138+0+0 (secure 0 0 0) 0x7fb418062920 con 0x7fb4280b10d0 2024-09-10T20:45:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:45:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.082+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4140787e0 msgr2=0x7fb41407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.082+0000 7fb42d223640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4140787e0 0x7fb41407aca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb4280d00d0 tx=0x7fb41c009040 comp rx=0 tx=0).stop 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.082+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 msgr2=0x7fb4280b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.082+0000 7fb42d223640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb4280b36e0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fb41800c3a0 tx=0x7fb41800c860 comp rx=0 tx=0).stop 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.082+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 shutdown_connections 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.082+0000 7fb42d223640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb4140787e0 0x7fb41407aca0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.083+0000 7fb42d223640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb42815bfc0 0x7fb428153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.083+0000 7fb42d223640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb4280d0f30 0x7fb428153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.083+0000 7fb42d223640 1 --2- 172.21.15.28:0/1327409602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4280b10d0 0x7fb4280b36e0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.083+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 >> 172.21.15.28:0/1327409602 conn(0x7fb4280d0410 msgr2=0x7fb42815af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.083+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 shutdown_connections 2024-09-10T20:45:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.083+0000 7fb42d223640 1 -- 172.21.15.28:0/1327409602 wait complete. 2024-09-10T20:45:31.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.218+0000 7f6a69e9b640 1 -- 172.21.15.28:0/1119115203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 msgr2=0x7f6a640cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.218+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/1119115203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a640cfa80 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f6a640cef50 tx=0x7f6a5c02aa00 comp rx=0 tx=0).stop 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.218+0000 7f6a69e9b640 1 -- 172.21.15.28:0/1119115203 shutdown_connections 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.218+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/1119115203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a640cfa80 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.218+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/1119115203 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a640b0c90 0x7f6a640cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.218+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/1119115203 >> v2:172.21.15.159:3300/0 conn(0x7f6a6415e130 0x7f6a640d4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.219+0000 7f6a69e9b640 1 -- 172.21.15.28:0/1119115203 >> 172.21.15.28:0/1119115203 conn(0x7f6a640d4040 msgr2=0x7f6a640d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.219+0000 7f6a69e9b640 1 -- 172.21.15.28:0/1119115203 shutdown_connections 2024-09-10T20:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.219+0000 7f6a69e9b640 1 -- 172.21.15.28:0/1119115203 wait complete. 2024-09-10T20:45:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.220+0000 7f6a69e9b640 1 Processor -- start 2024-09-10T20:45:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.220+0000 7f6a69e9b640 1 -- start start 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a69e9b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a640b0c90 0x7f6a640cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a69e9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a641f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a69e9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6415e130 0x7f6a640cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a69e9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a640d0f20 con 0x7f6a640b10d0 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a69e9b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6a640d0da0 con 0x7f6a6415e130 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a69e9b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6a640d10a0 con 0x7f6a640b0c90 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a641f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a641f4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40616/0 (socket says 172.21.15.28:40616) 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 -- 172.21.15.28:0/53610593 learned_addr learned my addr 172.21.15.28:0/53610593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a63fff640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6415e130 0x7f6a640cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a637fe640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a640b0c90 0x7f6a640cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 -- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a640b0c90 msgr2=0x7f6a640cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a640b0c90 0x7f6a640cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 -- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6415e130 msgr2=0x7f6a640cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6415e130 0x7f6a640cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 -- 172.21.15.28:0/53610593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a641f52e0 con 0x7f6a640b10d0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.221+0000 7f6a62ffd640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a641f4a80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800d750 tx=0x7f6a5800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.222+0000 7f6a4bfff640 1 -- 172.21.15.28:0/53610593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58017070 con 0x7f6a640b10d0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.222+0000 7f6a4bfff640 1 -- 172.21.15.28:0/53610593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a58009cc0 con 0x7f6a640b10d0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.222+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a641f5620 con 0x7f6a640b10d0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.222+0000 7f6a4bfff640 1 -- 172.21.15.28:0/53610593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800de60 con 0x7f6a640b10d0 2024-09-10T20:45:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.222+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a640cb3f0 con 0x7f6a640b10d0 2024-09-10T20:45:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.223+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a640d1630 con 0x7f6a640b10d0 2024-09-10T20:45:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.224+0000 7f6a4bfff640 1 -- 172.21.15.28:0/53610593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f6a58005280 con 0x7f6a640b10d0 2024-09-10T20:45:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.224+0000 7f6a4bfff640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6a500787e0 0x7f6a5007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.224+0000 7f6a4bfff640 1 -- 172.21.15.28:0/53610593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5255+0+0 (secure 0 0 0) 0x7f6a58098730 con 0x7f6a640b10d0 2024-09-10T20:45:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.224+0000 7f6a637fe640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6a500787e0 0x7f6a5007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.225+0000 7f6a637fe640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6a500787e0 0x7f6a5007aca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400bb50 tx=0x7f6a5400b2e0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.229+0000 7f6a4bfff640 1 -- 172.21.15.28:0/53610593 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6a641f5620 con 0x7f6a640b10d0 2024-09-10T20:45:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:31.415+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6a640cb5c0 con 0x7f6a640b10d0 2024-09-10T20:45:32.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.081+0000 7f6a4bfff640 1 -- 172.21.15.28:0/53610593 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v149) ==== 170+0+0 (secure 0 0 0) 0x7f6a5809c0d0 con 0x7f6a640b10d0 2024-09-10T20:45:32.085 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:45:32.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6a500787e0 msgr2=0x7f6a5007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:32.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6a500787e0 0x7f6a5007aca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400bb50 tx=0x7f6a5400b2e0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 msgr2=0x7f6a641f4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a641f4a80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800d750 tx=0x7f6a5800dc10 comp rx=0 tx=0).stop 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 shutdown_connections 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6a500787e0 0x7f6a5007aca0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6a6415e130 0x7f6a640cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a640b10d0 0x7f6a641f4a80 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 --2- 172.21.15.28:0/53610593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6a640b0c90 0x7f6a640cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.084+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 >> 172.21.15.28:0/53610593 conn(0x7f6a640d4040 msgr2=0x7f6a640b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.085+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 shutdown_connections 2024-09-10T20:45:32.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.085+0000 7f6a69e9b640 1 -- 172.21.15.28:0/53610593 wait complete. 2024-09-10T20:45:32.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:45:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.222+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/1851660923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 msgr2=0x7fbbb80706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.222+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/1851660923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb80706e0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fbba8009930 tx=0x7fbba802a5c0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.222+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/1851660923 shutdown_connections 2024-09-10T20:45:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.222+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/1851660923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb80706e0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.222+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/1851660923 >> 172.21.15.28:0/1851660923 conn(0x7fbbb80d0d50 msgr2=0x7fbbb80d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.222+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/1851660923 shutdown_connections 2024-09-10T20:45:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.222+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/1851660923 wait complete. 2024-09-10T20:45:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.223+0000 7fbbbd2c6640 1 Processor -- start 2024-09-10T20:45:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbbd2c6640 1 -- start start 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbbd2c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbbb8073ea0 0x7fbbb80b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbbd2c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb81f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbbd2c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbbb815bd80 0x7fbbb8153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbbd2c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80ccc40 con 0x7fbbb8151db0 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbbd2c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80ccac0 con 0x7fbbb815bd80 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbbd2c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb80ccdc0 con 0x7fbbb8073ea0 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbb67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb81f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbb67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb81f4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40630/0 (socket says 172.21.15.28:40630) 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.224+0000 7fbbb67fc640 1 -- 172.21.15.28:0/64704448 learned_addr learned my addr 172.21.15.28:0/64704448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb77fe640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbbb815bd80 0x7fbbb8153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb6ffd640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbbb8073ea0 0x7fbbb80b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb67fc640 1 -- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbbb8073ea0 msgr2=0x7fbbb80b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb67fc640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbbb8073ea0 0x7fbbb80b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb67fc640 1 -- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbbb815bd80 msgr2=0x7fbbb8153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb67fc640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbbb815bd80 0x7fbbb8153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb67fc640 1 -- 172.21.15.28:0/64704448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb80b4160 con 0x7fbbb8151db0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb77fe640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbbb815bd80 0x7fbbb8153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb6ffd640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbbb8073ea0 0x7fbbb80b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbb67fc640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb81f4ab0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00b420 tx=0x7fbbac00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbb9f7fe640 1 -- 172.21.15.28:0/64704448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac011830 con 0x7fbbb8151db0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbb9f7fe640 1 -- 172.21.15.28:0/64704448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbac016070 con 0x7fbbb8151db0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb80b4440 con 0x7fbbb8151db0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbb9f7fe640 1 -- 172.21.15.28:0/64704448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac011d60 con 0x7fbbb8151db0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.225+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb80b4a00 con 0x7fbbb8151db0 2024-09-10T20:45:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.226+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb80706e0 con 0x7fbbb8151db0 2024-09-10T20:45:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.227+0000 7fbb9f7fe640 1 -- 172.21.15.28:0/64704448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fbbac01b460 con 0x7fbbb8151db0 2024-09-10T20:45:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.228+0000 7fbb9f7fe640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbba40787e0 0x7fbba407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.228+0000 7fbb9f7fe640 1 -- 172.21.15.28:0/64704448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 5637+0+0 (secure 0 0 0) 0x7fbbac0980c0 con 0x7fbbb8151db0 2024-09-10T20:45:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.228+0000 7fbbb6ffd640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbba40787e0 0x7fbba407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.228+0000 7fbbb6ffd640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbba40787e0 0x7fbba407aca0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbba8033000 tx=0x7fbba8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.232+0000 7fbb9f7fe640 1 -- 172.21.15.28:0/64704448 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbbac020020 con 0x7fbbb8151db0 2024-09-10T20:45:32.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:32.456+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbbb80749a0 con 0x7fbbb8151db0 2024-09-10T20:45:32.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:45:32.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:45:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.103+0000 7fbb9f7fe640 1 -- 172.21.15.28:0/64704448 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v102) ==== 174+0+0 (secure 0 0 0) 0x7fbbac062650 con 0x7fbbb8151db0 2024-09-10T20:45:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 24 and data pool 25 2024-09-10T20:45:33.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.110+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbba40787e0 msgr2=0x7fbba407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.110+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbba40787e0 0x7fbba407aca0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbba8033000 tx=0x7fbba8004730 comp rx=0 tx=0).stop 2024-09-10T20:45:33.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.110+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 msgr2=0x7fbbb81f4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.110+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb81f4ab0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00b420 tx=0x7fbbac00b8e0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.111+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 shutdown_connections 2024-09-10T20:45:33.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.111+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbba40787e0 0x7fbba407aca0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.112+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbbb815bd80 0x7fbbb8153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.112+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8151db0 0x7fbbb81f4ab0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.112+0000 7fbbbd2c6640 1 --2- 172.21.15.28:0/64704448 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbbb8073ea0 0x7fbbb80b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.112+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 >> 172.21.15.28:0/64704448 conn(0x7fbbb80d0d50 msgr2=0x7fbbb80b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.116+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 shutdown_connections 2024-09-10T20:45:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.116+0000 7fbbbd2c6640 1 -- 172.21.15.28:0/64704448 wait complete. 2024-09-10T20:45:33.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/85504140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e015bc40 msgr2=0x7fb2e015e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/85504140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e015bc40 0x7fb2e015e6e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4005fd0 tx=0x7fb2d402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/85504140 shutdown_connections 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/85504140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e015bc40 0x7fb2e015e6e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/85504140 >> v2:172.21.15.159:3300/0 conn(0x7fb2e015ec20 0x7fb2e0070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/85504140 >> 172.21.15.28:0/85504140 conn(0x7fb2e006dbf0 msgr2=0x7fb2e006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/85504140 shutdown_connections 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.241+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/85504140 wait complete. 2024-09-10T20:45:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 Processor -- start 2024-09-10T20:45:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 -- start start 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e00d0c80 0x7fb2e01f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2e015bc40 0x7fb2e014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e015ec20 0x7fb2e00cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e0070940 con 0x7fb2e015ec20 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e00707c0 con 0x7fb2e015bc40 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.243+0000 7fb2e4ab8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e0070ac0 con 0x7fb2e00d0c80 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e015ec20 0x7fb2e00cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e015ec20 0x7fb2e00cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40640/0 (socket says 172.21.15.28:40640) 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 -- 172.21.15.28:0/3120854945 learned_addr learned my addr 172.21.15.28:0/3120854945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2de59c640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e00d0c80 0x7fb2e01f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ddd9b640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2e015bc40 0x7fb2e014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 -- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e00d0c80 msgr2=0x7fb2e01f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e00d0c80 0x7fb2e01f55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 -- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2e015bc40 msgr2=0x7fb2e014f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2e015bc40 0x7fb2e014f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 -- 172.21.15.28:0/3120854945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e014fe00 con 0x7fb2e015ec20 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ded9d640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e015ec20 0x7fb2e00cd070 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800ec20 tx=0x7fb2d800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ceffd640 1 -- 172.21.15.28:0/3120854945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8003b70 con 0x7fb2e015ec20 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ceffd640 1 -- 172.21.15.28:0/3120854945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2d8003e00 con 0x7fb2e015ec20 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e014f0d0 con 0x7fb2e015ec20 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2ceffd640 1 -- 172.21.15.28:0/3120854945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8004280 con 0x7fb2e015ec20 2024-09-10T20:45:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.244+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e00cf230 con 0x7fb2e015ec20 2024-09-10T20:45:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.245+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e015e6e0 con 0x7fb2e015ec20 2024-09-10T20:45:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.246+0000 7fb2ceffd640 1 -- 172.21.15.28:0/3120854945 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fb2d8004420 con 0x7fb2e015ec20 2024-09-10T20:45:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.246+0000 7fb2ceffd640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2c00787e0 0x7fb2c007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.247+0000 7fb2ceffd640 1 -- 172.21.15.28:0/3120854945 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5737+0+0 (secure 0 0 0) 0x7fb2d809bd80 con 0x7fb2e015ec20 2024-09-10T20:45:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.247+0000 7fb2de59c640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2c00787e0 0x7fb2c007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.247+0000 7fb2de59c640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2c00787e0 0x7fb2c007aca0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00017c0 tx=0x7fb2d0009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.251+0000 7fb2ceffd640 1 -- 172.21.15.28:0/3120854945 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb2d8063270 con 0x7fb2e015ec20 2024-09-10T20:45:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.437+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2e00d0a60 con 0x7fb2e015ec20 2024-09-10T20:45:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.438+0000 7fb2ceffd640 1 -- 172.21.15.28:0/3120854945 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) ==== 74+0+16740 (secure 0 0 0) 0x7fb2d8067ca0 con 0x7fb2e015ec20 2024-09-10T20:45:33.442 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:33.442 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":150,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:32.457195+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:45:30.112884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-10T20:45:31.419730+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2c00787e0 msgr2=0x7fb2c007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2c00787e0 0x7fb2c007aca0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00017c0 tx=0x7fb2d0009040 comp rx=0 tx=0).stop 2024-09-10T20:45:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e015ec20 msgr2=0x7fb2e00cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e015ec20 0x7fb2e00cd070 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800ec20 tx=0x7fb2d800c6a0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 shutdown_connections 2024-09-10T20:45:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb2c00787e0 0x7fb2c007aca0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e015ec20 0x7fb2e00cd070 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb2e015bc40 0x7fb2e014f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 --2- 172.21.15.28:0/3120854945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb2e00d0c80 0x7fb2e01f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 >> 172.21.15.28:0/3120854945 conn(0x7fb2e006dbf0 msgr2=0x7fb2e015c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 shutdown_connections 2024-09-10T20:45:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.440+0000 7fb2e4ab8640 1 -- 172.21.15.28:0/3120854945 wait complete. 2024-09-10T20:45:33.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.557+0000 7f41217bb640 1 -- 172.21.15.28:0/3582932809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4108003a50 msgr2=0x7f4108003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.557+0000 7f41217bb640 1 --2- 172.21.15.28:0/3582932809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4108003a50 0x7f4108003e70 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f411400c610 tx=0x7f4108005830 comp rx=0 tx=0).stop 2024-09-10T20:45:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.557+0000 7f41217bb640 1 -- 172.21.15.28:0/3582932809 shutdown_connections 2024-09-10T20:45:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.557+0000 7f41217bb640 1 --2- 172.21.15.28:0/3582932809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4108003a50 0x7f4108003e70 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.557+0000 7f41217bb640 1 -- 172.21.15.28:0/3582932809 >> 172.21.15.28:0/3582932809 conn(0x7f411c0d1e40 msgr2=0x7f411c0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.557+0000 7f41217bb640 1 -- 172.21.15.28:0/3582932809 shutdown_connections 2024-09-10T20:45:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.557+0000 7f41217bb640 1 -- 172.21.15.28:0/3582932809 wait complete. 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.558+0000 7f41217bb640 1 Processor -- start 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.558+0000 7f41217bb640 1 -- start start 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41217bb640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4108003a50 0x7f411c09e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41217bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f411c1f5490 0x7f411c06f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41217bb640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f411c09c740 0x7f411c09cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41217bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0d0cc0 con 0x7f411c1f5490 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41217bb640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0d0b40 con 0x7f4108003a50 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41217bb640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f411c0cfec0 con 0x7f411c09c740 2024-09-10T20:45:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f411c1f5490 0x7f411c06f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f411c1f5490 0x7f411c06f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40670/0 (socket says 172.21.15.28:40670) 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 -- 172.21.15.28:0/458532028 learned_addr learned my addr 172.21.15.28:0/458532028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411b7fe640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f411c09c740 0x7f411c09cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411affd640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4108003a50 0x7f411c09e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 -- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f411c09c740 msgr2=0x7f411c09cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f411c09c740 0x7f411c09cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 -- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4108003a50 msgr2=0x7f411c09e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4108003a50 0x7f411c09e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 -- 172.21.15.28:0/458532028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c09bec0 con 0x7f411c1f5490 2024-09-10T20:45:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f411a7fc640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f411c1f5490 0x7f411c06f700 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f4104003ed0 tx=0x7f4104001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41137fe640 1 -- 172.21.15.28:0/458532028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104015070 con 0x7f411c1f5490 2024-09-10T20:45:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41137fe640 1 -- 172.21.15.28:0/458532028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4104011840 con 0x7f411c1f5490 2024-09-10T20:45:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.559+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411c1f5910 con 0x7f411c1f5490 2024-09-10T20:45:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.560+0000 7f41137fe640 1 -- 172.21.15.28:0/458532028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104011c70 con 0x7f411c1f5490 2024-09-10T20:45:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.560+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411c09f120 con 0x7f411c1f5490 2024-09-10T20:45:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.561+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f411c0d0320 con 0x7f411c1f5490 2024-09-10T20:45:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.561+0000 7f41137fe640 1 -- 172.21.15.28:0/458532028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f41040103c0 con 0x7f411c1f5490 2024-09-10T20:45:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.562+0000 7f41137fe640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40fc078810 0x7f40fc07acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.562+0000 7f41137fe640 1 -- 172.21.15.28:0/458532028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5737+0+0 (secure 0 0 0) 0x7f4104099410 con 0x7f411c1f5490 2024-09-10T20:45:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.562+0000 7f411affd640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40fc078810 0x7f40fc07acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.562+0000 7f411affd640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40fc078810 0x7f40fc07acd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4114033000 tx=0x7f411c0cda10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.566+0000 7f41137fe640 1 -- 172.21.15.28:0/458532028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f41040a2080 con 0x7f411c1f5490 2024-09-10T20:45:33.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.755+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f411c0cf600 con 0x7f411c1f5490 2024-09-10T20:45:33.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.756+0000 7f41137fe640 1 -- 172.21.15.28:0/458532028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) ==== 74+0+16740 (secure 0 0 0) 0x7f41040639c0 con 0x7f411c1f5490 2024-09-10T20:45:33.760 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:33.760 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":150,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:32.457195+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:45:30.112884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-10T20:45:31.419730+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40fc078810 msgr2=0x7f40fc07acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40fc078810 0x7f40fc07acd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4114033000 tx=0x7f411c0cda10 comp rx=0 tx=0).stop 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f411c1f5490 msgr2=0x7f411c06f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f411c1f5490 0x7f411c06f700 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f4104003ed0 tx=0x7f4104001630 comp rx=0 tx=0).stop 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 shutdown_connections 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40fc078810 0x7f40fc07acd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f411c09c740 0x7f411c09cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f411c1f5490 0x7f411c06f700 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 --2- 172.21.15.28:0/458532028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4108003a50 0x7f411c09e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 >> 172.21.15.28:0/458532028 conn(0x7f411c0d1e40 msgr2=0x7f411c15e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:33.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 shutdown_connections 2024-09-10T20:45:33.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.759+0000 7f41217bb640 1 -- 172.21.15.28:0/458532028 wait complete. 2024-09-10T20:45:33.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:45:33.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.879+0000 7fd6455ad640 1 -- 172.21.15.28:0/885982806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd640161770 msgr2=0x7fd6400d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.879+0000 7fd6455ad640 1 --2- 172.21.15.28:0/885982806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd640161770 0x7fd6400d20e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd630009830 tx=0x7fd63002a4b0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.879+0000 7fd6455ad640 1 -- 172.21.15.28:0/885982806 shutdown_connections 2024-09-10T20:45:33.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.879+0000 7fd6455ad640 1 --2- 172.21.15.28:0/885982806 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd640161770 0x7fd6400d20e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.879+0000 7fd6455ad640 1 -- 172.21.15.28:0/885982806 >> 172.21.15.28:0/885982806 conn(0x7fd6400d4070 msgr2=0x7fd6400d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:33.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.879+0000 7fd6455ad640 1 -- 172.21.15.28:0/885982806 shutdown_connections 2024-09-10T20:45:33.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.879+0000 7fd6455ad640 1 -- 172.21.15.28:0/885982806 wait complete. 2024-09-10T20:45:33.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.880+0000 7fd6455ad640 1 Processor -- start 2024-09-10T20:45:33.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd6455ad640 1 -- start start 2024-09-10T20:45:33.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd6455ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6400d4b90 0x7fd6401548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd6455ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd640161770 0x7fd6400d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd6455ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd640161b50 0x7fd6400ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd6455ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6400b2dd0 con 0x7fd640161770 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd6455ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd6400b2a50 con 0x7fd6400d4b90 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd6455ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd6400b2f50 con 0x7fd640161b50 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd63e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd640161770 0x7fd6400d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd63e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd640161770 0x7fd6400d1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40676/0 (socket says 172.21.15.28:40676) 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.881+0000 7fd63e7fc640 1 -- 172.21.15.28:0/162519364 learned_addr learned my addr 172.21.15.28:0/162519364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63effd640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6400d4b90 0x7fd6401548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63f7fe640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd640161b50 0x7fd6400ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63e7fc640 1 -- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd640161b50 msgr2=0x7fd6400ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63e7fc640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd640161b50 0x7fd6400ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63e7fc640 1 -- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6400d4b90 msgr2=0x7fd6401548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:33.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63e7fc640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6400d4b90 0x7fd6401548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63e7fc640 1 -- 172.21.15.28:0/162519364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd640154de0 con 0x7fd640161770 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63effd640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6400d4b90 0x7fd6401548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd63e7fc640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd640161770 0x7fd6400d1060 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c430 tx=0x7fd62800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd6377fe640 1 -- 172.21.15.28:0/162519364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628017070 con 0x7fd640161770 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd6377fe640 1 -- 172.21.15.28:0/162519364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6280053c0 con 0x7fd640161770 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6400cb390 con 0x7fd640161770 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd6377fe640 1 -- 172.21.15.28:0/162519364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628003850 con 0x7fd640161770 2024-09-10T20:45:33.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.882+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6400d0860 con 0x7fd640161770 2024-09-10T20:45:33.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.883+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6400cb120 con 0x7fd640161770 2024-09-10T20:45:33.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.884+0000 7fd6377fe640 1 -- 172.21.15.28:0/162519364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fd6280039f0 con 0x7fd640161770 2024-09-10T20:45:33.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.884+0000 7fd6377fe640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6200787e0 0x7fd62007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:33.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.885+0000 7fd6377fe640 1 -- 172.21.15.28:0/162519364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5737+0+0 (secure 0 0 0) 0x7fd62809a6b0 con 0x7fd640161770 2024-09-10T20:45:33.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.885+0000 7fd63effd640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6200787e0 0x7fd62007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:33.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.885+0000 7fd63effd640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6200787e0 0x7fd62007aca0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd630034000 tx=0x7fd630004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:33.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:33.889+0000 7fd6377fe640 1 -- 172.21.15.28:0/162519364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd628062220 con 0x7fd640161770 2024-09-10T20:45:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:34.113+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd6400d20c0 con 0x7fd640161770 2024-09-10T20:45:35.082 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-10T20:45:35.079+0000 7fe898603640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-10T20:45:35.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.117+0000 7fd6377fe640 1 -- 172.21.15.28:0/162519364 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v105) ==== 118+0+0 (secure 0 0 0) 0x7fd628061b20 con 0x7fd640161770 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.120+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6200787e0 msgr2=0x7fd62007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.120+0000 7fd6455ad640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6200787e0 0x7fd62007aca0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd630034000 tx=0x7fd630004730 comp rx=0 tx=0).stop 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.120+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd640161770 msgr2=0x7fd6400d1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.120+0000 7fd6455ad640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd640161770 0x7fd6400d1060 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c430 tx=0x7fd62800c8f0 comp rx=0 tx=0).stop 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 shutdown_connections 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd6200787e0 0x7fd62007aca0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd640161b50 0x7fd6400ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd640161770 0x7fd6400d1060 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 --2- 172.21.15.28:0/162519364 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd6400d4b90 0x7fd6401548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 >> 172.21.15.28:0/162519364 conn(0x7fd6400d4070 msgr2=0x7fd6400cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 shutdown_connections 2024-09-10T20:45:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.121+0000 7fd6455ad640 1 -- 172.21.15.28:0/162519364 wait complete. 2024-09-10T20:45:35.138 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:45:35.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:45:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.251+0000 7f1354a39640 1 -- 172.21.15.28:0/1016533438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 msgr2=0x7f133c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.251+0000 7f1354a39640 1 --2- 172.21.15.28:0/1016533438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 0x7f133c0051c0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f1348001ea0 tx=0x7f133c005830 comp rx=0 tx=0).stop 2024-09-10T20:45:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.251+0000 7f134ed9d640 1 -- 172.21.15.28:0/1016533438 reap_dead start 2024-09-10T20:45:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.251+0000 7f1354a39640 1 -- 172.21.15.28:0/1016533438 shutdown_connections 2024-09-10T20:45:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.251+0000 7f1354a39640 1 -- 172.21.15.28:0/1016533438 >> 172.21.15.28:0/1016533438 conn(0x7f13500d4070 msgr2=0x7f13500d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.251+0000 7f1354a39640 1 -- 172.21.15.28:0/1016533438 shutdown_connections 2024-09-10T20:45:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.251+0000 7f1354a39640 1 -- 172.21.15.28:0/1016533438 wait complete. 2024-09-10T20:45:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.253+0000 7f1354a39640 1 Processor -- start 2024-09-10T20:45:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.253+0000 7f1354a39640 1 -- start start 2024-09-10T20:45:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f1354a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 0x7f13501578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f1354a39640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13500cca90 0x7f13501f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f1354a39640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135006dc60 0x7f135006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f1354a39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13500b2f50 con 0x7f133c004d40 2024-09-10T20:45:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f1354a39640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f13500b2dd0 con 0x7f135006dc60 2024-09-10T20:45:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f1354a39640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f13500b30d0 con 0x7f13500cca90 2024-09-10T20:45:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 0x7f13501578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 0x7f13501578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40684/0 (socket says 172.21.15.28:40684) 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 -- 172.21.15.28:0/255863337 learned_addr learned my addr 172.21.15.28:0/255863337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f1347fff640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13500cca90 0x7f13501f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134ed9d640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135006dc60 0x7f135006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 -- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13500cca90 msgr2=0x7f13501f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13500cca90 0x7f13501f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 -- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135006dc60 msgr2=0x7f135006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135006dc60 0x7f135006e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.254+0000 7f134e59c640 1 -- 172.21.15.28:0/255863337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135015b6f0 con 0x7f133c004d40 2024-09-10T20:45:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.255+0000 7f134e59c640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 0x7f13501578f0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1340006600 tx=0x7f13400039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.255+0000 7f1346ffd640 1 -- 172.21.15.28:0/255863337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340015070 con 0x7f133c004d40 2024-09-10T20:45:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.255+0000 7f1346ffd640 1 -- 172.21.15.28:0/255863337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1340011840 con 0x7f133c004d40 2024-09-10T20:45:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.255+0000 7f1346ffd640 1 -- 172.21.15.28:0/255863337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340011c90 con 0x7f133c004d40 2024-09-10T20:45:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.255+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13500ccfe0 con 0x7f133c004d40 2024-09-10T20:45:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.255+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13500cd700 con 0x7f133c004d40 2024-09-10T20:45:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.256+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13500d48d0 con 0x7f133c004d40 2024-09-10T20:45:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.256+0000 7f1346ffd640 1 -- 172.21.15.28:0/255863337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f13400103c0 con 0x7f133c004d40 2024-09-10T20:45:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.257+0000 7f1346ffd640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13300787e0 0x7f133007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.257+0000 7f1347fff640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13300787e0 0x7f133007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.257+0000 7f1346ffd640 1 -- 172.21.15.28:0/255863337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f1340099500 con 0x7f133c004d40 2024-09-10T20:45:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.257+0000 7f1347fff640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13300787e0 0x7f133007aca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f13500ccf10 tx=0x7f1338005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.261+0000 7f1346ffd640 1 -- 172.21.15.28:0/255863337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f13500d48d0 con 0x7f133c004d40 2024-09-10T20:45:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:35.487+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f13500d1700 con 0x7f133c004d40 2024-09-10T20:45:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.118+0000 7f1346ffd640 1 -- 172.21.15.28:0/255863337 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v106) ==== 122+0+0 (secure 0 0 0) 0x7f13500d1700 con 0x7f133c004d40 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.121+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13300787e0 msgr2=0x7f133007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.121+0000 7f1354a39640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13300787e0 0x7f133007aca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f13500ccf10 tx=0x7f1338005f90 comp rx=0 tx=0).stop 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.121+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 msgr2=0x7f13501578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.121+0000 7f1354a39640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 0x7f13501578f0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1340006600 tx=0x7f13400039c0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 shutdown_connections 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13300787e0 0x7f133007aca0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f135006dc60 0x7f135006e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13500cca90 0x7f13501f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 --2- 172.21.15.28:0/255863337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f133c004d40 0x7f13501578f0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 >> 172.21.15.28:0/255863337 conn(0x7f13500d4070 msgr2=0x7f13500b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 shutdown_connections 2024-09-10T20:45:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.122+0000 7f1354a39640 1 -- 172.21.15.28:0/255863337 wait complete. 2024-09-10T20:45:36.139 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:45:36.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 -- 172.21.15.28:0/894097052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8155100 msgr2=0x7f46f815bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 --2- 172.21.15.28:0/894097052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8155100 0x7f46f815bd50 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f46f80cbd10 tx=0x7f46e802a560 comp rx=0 tx=0).stop 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 -- 172.21.15.28:0/894097052 shutdown_connections 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 --2- 172.21.15.28:0/894097052 >> v2:172.21.15.159:3300/0 conn(0x7f46f815ee70 0x7f46f80d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 --2- 172.21.15.28:0/894097052 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f81554e0 0x7f46f815e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 --2- 172.21.15.28:0/894097052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8155100 0x7f46f815bd50 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 -- 172.21.15.28:0/894097052 >> 172.21.15.28:0/894097052 conn(0x7f46f806cf00 msgr2=0x7f46f806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 -- 172.21.15.28:0/894097052 shutdown_connections 2024-09-10T20:45:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.247+0000 7f46fd732640 1 -- 172.21.15.28:0/894097052 wait complete. 2024-09-10T20:45:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 Processor -- start 2024-09-10T20:45:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 -- start start 2024-09-10T20:45:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46f8155100 0x7f46f81f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f81554e0 0x7f46f80ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f815ee70 0x7f46f81f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8072bd0 con 0x7f46f815ee70 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f46f80d0ab0 con 0x7f46f8155100 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.249+0000 7f46fd732640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f46f8072d50 con 0x7f46f81554e0 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f815ee70 0x7f46f81f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f815ee70 0x7f46f81f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40690/0 (socket says 172.21.15.28:40690) 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 -- 172.21.15.28:0/632399703 learned_addr learned my addr 172.21.15.28:0/632399703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f6ffd640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46f8155100 0x7f46f81f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46effff640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f81554e0 0x7f46f80ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 -- 172.21.15.28:0/632399703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f81554e0 msgr2=0x7f46f80ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f81554e0 0x7f46f80ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 -- 172.21.15.28:0/632399703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46f8155100 msgr2=0x7f46f81f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46f8155100 0x7f46f81f30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 -- 172.21.15.28:0/632399703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f81f35f0 con 0x7f46f815ee70 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46f77fe640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f815ee70 0x7f46f81f5ee0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f46f000e9d0 tx=0x7f46f000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46ef7fe640 1 -- 172.21.15.28:0/632399703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f000cd80 con 0x7f46f815ee70 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46ef7fe640 1 -- 172.21.15.28:0/632399703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46f0004c30 con 0x7f46f815ee70 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f80ad0e0 con 0x7f46f815ee70 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46ef7fe640 1 -- 172.21.15.28:0/632399703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0005280 con 0x7f46f815ee70 2024-09-10T20:45:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.250+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f80ad6a0 con 0x7f46f815ee70 2024-09-10T20:45:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.251+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f815e930 con 0x7f46f815ee70 2024-09-10T20:45:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.252+0000 7f46ef7fe640 1 -- 172.21.15.28:0/632399703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f46f000f040 con 0x7f46f815ee70 2024-09-10T20:45:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.253+0000 7f46ef7fe640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f46d80787e0 0x7f46d807aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.253+0000 7f46f6ffd640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f46d80787e0 0x7f46d807aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.253+0000 7f46ef7fe640 1 -- 172.21.15.28:0/632399703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f46f009b660 con 0x7f46f815ee70 2024-09-10T20:45:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.253+0000 7f46f6ffd640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f46d80787e0 0x7f46d807aca0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f46e8033000 tx=0x7f46e8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:36.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.257+0000 7f46ef7fe640 1 -- 172.21.15.28:0/632399703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f46f0016030 con 0x7f46f815ee70 2024-09-10T20:45:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.480+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46f80ae6c0 con 0x7f46f815ee70 2024-09-10T20:45:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.480+0000 7f46ef7fe640 1 -- 172.21.15.28:0/632399703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5236 (secure 0 0 0) 0x7f46f0062ac0 con 0x7f46f815ee70 2024-09-10T20:45:36.485 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:36.485 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"btime":"2024-09-10T20:45:36:118423+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3007122398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3007122398},{"type":"v1","addr":"172.21.15.28:6834","nonce":3007122398}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":7031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3280811834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3280811834},{"type":"v1","addr":"172.21.15.159:6835","nonce":3280811834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":7630,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/632114757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":632114757},{"type":"v1","addr":"172.21.15.159:6834","nonce":632114757}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:45:32.457163+0000","modified":"2024-09-10T20:45:35.489078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8730},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8730":{"gid":8730,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1892964400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1892964400},{"type":"v1","addr":"172.21.15.28:6835","nonce":1892964400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8730,"qdb_cluster":[8730]},"id":12}]} 2024-09-10T20:45:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 106 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f46d80787e0 msgr2=0x7f46d807aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f46d80787e0 0x7f46d807aca0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f46e8033000 tx=0x7f46e8004730 comp rx=0 tx=0).stop 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f815ee70 msgr2=0x7f46f81f5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f815ee70 0x7f46f81f5ee0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f46f000e9d0 tx=0x7f46f000ee90 comp rx=0 tx=0).stop 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 shutdown_connections 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f46d80787e0 0x7f46d807aca0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f815ee70 0x7f46f81f5ee0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f46f81554e0 0x7f46f80ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 --2- 172.21.15.28:0/632399703 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f46f8155100 0x7f46f81f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 >> 172.21.15.28:0/632399703 conn(0x7f46f806cf00 msgr2=0x7f46f806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:36.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 shutdown_connections 2024-09-10T20:45:36.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.483+0000 7f46fd732640 1 -- 172.21.15.28:0/632399703 wait complete. 2024-09-10T20:45:36.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.600+0000 7f3f7b433640 1 -- 172.21.15.28:0/1713803072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c004800 msgr2=0x7f3f5c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.600+0000 7f3f7b433640 1 --2- 172.21.15.28:0/1713803072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c004800 0x7f3f5c004c80 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006fc0 tx=0x7f3f5c005830 comp rx=0 tx=0).stop 2024-09-10T20:45:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.600+0000 7f3f799d0640 1 -- 172.21.15.28:0/1713803072 reap_dead start 2024-09-10T20:45:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.600+0000 7f3f7b433640 1 -- 172.21.15.28:0/1713803072 shutdown_connections 2024-09-10T20:45:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.600+0000 7f3f7b433640 1 -- 172.21.15.28:0/1713803072 >> 172.21.15.28:0/1713803072 conn(0x7f3f740d3fb0 msgr2=0x7f3f740d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.600+0000 7f3f7b433640 1 -- 172.21.15.28:0/1713803072 shutdown_connections 2024-09-10T20:45:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.600+0000 7f3f7b433640 1 -- 172.21.15.28:0/1713803072 wait complete. 2024-09-10T20:45:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.601+0000 7f3f7b433640 1 Processor -- start 2024-09-10T20:45:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.601+0000 7f3f7b433640 1 -- start start 2024-09-10T20:45:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f7b433640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f5c004800 0x7f3f7414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f7b433640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7414d0b0 0x7f3f740a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f7b433640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f740a12a0 0x7f3f7414fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f7b433640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f740b6e00 con 0x7f3f740a12a0 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f7b433640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7415e460 con 0x7f3f5c004800 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f7b433640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3f740b6f80 con 0x7f3f7414d0b0 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f740a12a0 0x7f3f7414fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f740a12a0 0x7f3f7414fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40708/0 (socket says 172.21.15.28:40708) 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 -- 172.21.15.28:0/3606027951 learned_addr learned my addr 172.21.15.28:0/3606027951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f791cf640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f5c004800 0x7f3f7414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f789ce640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7414d0b0 0x7f3f740a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 -- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7414d0b0 msgr2=0x7f3f740a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7414d0b0 0x7f3f740a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 -- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f5c004800 msgr2=0x7f3f7414da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f5c004800 0x7f3f7414da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.602+0000 7f3f799d0640 1 -- 172.21.15.28:0/3606027951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7414df40 con 0x7f3f740a12a0 2024-09-10T20:45:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.603+0000 7f3f799d0640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f740a12a0 0x7f3f7414fa50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f3f740ce590 tx=0x7f3f6c011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.603+0000 7f3f71ffb640 1 -- 172.21.15.28:0/3606027951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c01f070 con 0x7f3f740a12a0 2024-09-10T20:45:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.603+0000 7f3f71ffb640 1 -- 172.21.15.28:0/3606027951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f6c00fb40 con 0x7f3f740a12a0 2024-09-10T20:45:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.603+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f740a1720 con 0x7f3f740a12a0 2024-09-10T20:45:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.603+0000 7f3f71ffb640 1 -- 172.21.15.28:0/3606027951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c012450 con 0x7f3f740a12a0 2024-09-10T20:45:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.603+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f740cb2d0 con 0x7f3f740a12a0 2024-09-10T20:45:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.604+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f74161b50 con 0x7f3f740a12a0 2024-09-10T20:45:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.605+0000 7f3f71ffb640 1 -- 172.21.15.28:0/3606027951 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f3f6c0125f0 con 0x7f3f740a12a0 2024-09-10T20:45:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.605+0000 7f3f71ffb640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f600787e0 0x7f3f6007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.605+0000 7f3f71ffb640 1 -- 172.21.15.28:0/3606027951 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f3f6c099110 con 0x7f3f740a12a0 2024-09-10T20:45:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.605+0000 7f3f791cf640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f600787e0 0x7f3f6007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.606+0000 7f3f791cf640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f600787e0 0x7f3f6007aca0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3f64006d90 tx=0x7f3f64006c80 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.609+0000 7f3f71ffb640 1 -- 172.21.15.28:0/3606027951 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3f6c0636c0 con 0x7f3f740a12a0 2024-09-10T20:45:36.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.796+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f740b9f30 con 0x7f3f740a12a0 2024-09-10T20:45:36.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.797+0000 7f3f71ffb640 1 -- 172.21.15.28:0/3606027951 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16727 (secure 0 0 0) 0x7f3f6c062fc0 con 0x7f3f740a12a0 2024-09-10T20:45:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:36.801 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:34.103863+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:45:30.112884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-10T20:45:31.419730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:36.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f600787e0 msgr2=0x7f3f6007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f600787e0 0x7f3f6007aca0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3f64006d90 tx=0x7f3f64006c80 comp rx=0 tx=0).stop 2024-09-10T20:45:36.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f740a12a0 msgr2=0x7f3f7414fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f740a12a0 0x7f3f7414fa50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f3f740ce590 tx=0x7f3f6c011b10 comp rx=0 tx=0).stop 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 shutdown_connections 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3f600787e0 0x7f3f6007aca0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f740a12a0 0x7f3f7414fa50 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f7414d0b0 0x7f3f740a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 --2- 172.21.15.28:0/3606027951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f5c004800 0x7f3f7414da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 >> 172.21.15.28:0/3606027951 conn(0x7f3f740d3fb0 msgr2=0x7f3f740b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 shutdown_connections 2024-09-10T20:45:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.799+0000 7f3f7b433640 1 -- 172.21.15.28:0/3606027951 wait complete. 2024-09-10T20:45:36.815 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:45:36.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:45:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 -- 172.21.15.28:0/600571172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 msgr2=0x7f752c15bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 --2- 172.21.15.28:0/600571172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c15bc40 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f751c005fd0 tx=0x7f751c02a530 comp rx=0 tx=0).stop 2024-09-10T20:45:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 -- 172.21.15.28:0/600571172 shutdown_connections 2024-09-10T20:45:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 --2- 172.21.15.28:0/600571172 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f752c070280 0x7f752c070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 --2- 172.21.15.28:0/600571172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c15bc40 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 -- 172.21.15.28:0/600571172 >> 172.21.15.28:0/600571172 conn(0x7f752c06dbf0 msgr2=0x7f752c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 -- 172.21.15.28:0/600571172 shutdown_connections 2024-09-10T20:45:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.916+0000 7f7531860640 1 -- 172.21.15.28:0/600571172 wait complete. 2024-09-10T20:45:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.917+0000 7f7531860640 1 Processor -- start 2024-09-10T20:45:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f7531860640 1 -- start start 2024-09-10T20:45:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f7531860640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f752c070280 0x7f752c1433b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f7531860640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f7531860640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f752c0d1060 0x7f752c15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f7531860640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f752c15ef70 con 0x7f752c0d0c80 2024-09-10T20:45:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f7531860640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f752c15edf0 con 0x7f752c0d1060 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f7531860640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f752c070bc0 con 0x7f752c070280 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f752a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f752a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c144b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40734/0 (socket says 172.21.15.28:40734) 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f752a7fc640 1 -- 172.21.15.28:0/781803230 learned_addr learned my addr 172.21.15.28:0/781803230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f752affd640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f752c070280 0x7f752c1433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.918+0000 7f752b7fe640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f752c0d1060 0x7f752c15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f752a7fc640 1 -- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f752c070280 msgr2=0x7f752c1433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f752a7fc640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f752c070280 0x7f752c1433b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f752a7fc640 1 -- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f752c0d1060 msgr2=0x7f752c15a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f752a7fc640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f752c0d1060 0x7f752c15a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f752a7fc640 1 -- 172.21.15.28:0/781803230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c143ae0 con 0x7f752c0d0c80 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f752b7fe640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f752c0d1060 0x7f752c15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f752a7fc640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c144b80 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f752000b3f0 tx=0x7f752000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f7513fff640 1 -- 172.21.15.28:0/781803230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520015020 con 0x7f752c0d0c80 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f7513fff640 1 -- 172.21.15.28:0/781803230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7520011920 con 0x7f752c0d0c80 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752c143dc0 con 0x7f752c0d0c80 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f7513fff640 1 -- 172.21.15.28:0/781803230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520011d70 con 0x7f752c0d0c80 2024-09-10T20:45:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.919+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752c144660 con 0x7f752c0d0c80 2024-09-10T20:45:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.920+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f752c1441b0 con 0x7f752c0d0c80 2024-09-10T20:45:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.921+0000 7f7513fff640 1 -- 172.21.15.28:0/781803230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f752001a400 con 0x7f752c0d0c80 2024-09-10T20:45:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.921+0000 7f7513fff640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f75180787e0 0x7f751807aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.922+0000 7f752affd640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f75180787e0 0x7f751807aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.922+0000 7f7513fff640 1 -- 172.21.15.28:0/781803230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f7520098460 con 0x7f752c0d0c80 2024-09-10T20:45:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.922+0000 7f752affd640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f75180787e0 0x7f751807aca0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f751c033000 tx=0x7f751c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:36.926+0000 7f7513fff640 1 -- 172.21.15.28:0/781803230 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f752001f030 con 0x7f752c0d0c80 2024-09-10T20:45:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.109+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f752c070680 con 0x7f75180787e0 2024-09-10T20:45:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.113+0000 7f7513fff640 1 -- 172.21.15.28:0/781803230 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252479 (secure 0 0 0) 0x7f752c070680 con 0x7f75180787e0 2024-09-10T20:45:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:37.122 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:45:37.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f75180787e0 msgr2=0x7f751807aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f75180787e0 0x7f751807aca0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f751c033000 tx=0x7f751c004730 comp rx=0 tx=0).stop 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 msgr2=0x7f752c144b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c144b80 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f752000b3f0 tx=0x7f752000b8b0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 shutdown_connections 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f75180787e0 0x7f751807aca0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f752c0d1060 0x7f752c15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f752c0d0c80 0x7f752c144b80 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 --2- 172.21.15.28:0/781803230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f752c070280 0x7f752c1433b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 >> 172.21.15.28:0/781803230 conn(0x7f752c06dbf0 msgr2=0x7f752c06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 shutdown_connections 2024-09-10T20:45:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.117+0000 7f7531860640 1 -- 172.21.15.28:0/781803230 wait complete. 2024-09-10T20:45:37.132 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-10T20:45:36.333138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282128,"kb_used_data":15968,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692592,"statfs":{"total":773094113280,"available":772805214208,"internally_reserved":0,"allocated":16351232,"data_stored":7861564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.253090"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107482+0000","last_change":"2024-09-10T20:45:33.114526+0000","last_active":"2024-09-10T20:45:34.107482+0000","last_peered":"2024-09-10T20:45:34.107482+0000","last_clean":"2024-09-10T20:45:34.107482+0000","last_became_active":"2024-09-10T20:45:33.114248+0000","last_became_peered":"2024-09-10T20:45:33.114248+0000","last_unstale":"2024-09-10T20:45:34.107482+0000","last_undegraded":"2024-09-10T20:45:34.107482+0000","last_fullsized":"2024-09-10T20:45:34.107482+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:30:00.736809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.114983+0000","last_change":"2024-09-10T20:45:33.108874+0000","last_active":"2024-09-10T20:45:33.114983+0000","last_peered":"2024-09-10T20:45:33.114983+0000","last_clean":"2024-09-10T20:45:33.114983+0000","last_became_active":"2024-09-10T20:45:33.108632+0000","last_became_peered":"2024-09-10T20:45:33.108632+0000","last_unstale":"2024-09-10T20:45:33.114983+0000","last_undegraded":"2024-09-10T20:45:33.114983+0000","last_fullsized":"2024-09-10T20:45:33.114983+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:55:32.635136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109612+0000","last_change":"2024-09-10T20:45:33.109173+0000","last_active":"2024-09-10T20:45:34.109612+0000","last_peered":"2024-09-10T20:45:34.109612+0000","last_clean":"2024-09-10T20:45:34.109612+0000","last_became_active":"2024-09-10T20:45:33.108839+0000","last_became_peered":"2024-09-10T20:45:33.108839+0000","last_unstale":"2024-09-10T20:45:34.109612+0000","last_undegraded":"2024-09-10T20:45:34.109612+0000","last_fullsized":"2024-09-10T20:45:34.109612+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:48.286241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117601+0000","last_change":"2024-09-10T20:45:33.111808+0000","last_active":"2024-09-10T20:45:33.117601+0000","last_peered":"2024-09-10T20:45:33.117601+0000","last_clean":"2024-09-10T20:45:33.117601+0000","last_became_active":"2024-09-10T20:45:33.111528+0000","last_became_peered":"2024-09-10T20:45:33.111528+0000","last_unstale":"2024-09-10T20:45:33.117601+0000","last_undegraded":"2024-09-10T20:45:33.117601+0000","last_fullsized":"2024-09-10T20:45:33.117601+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:42:09.243456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193384+0000","last_change":"2024-09-10T20:45:33.123880+0000","last_active":"2024-09-10T20:45:34.193384+0000","last_peered":"2024-09-10T20:45:34.193384+0000","last_clean":"2024-09-10T20:45:34.193384+0000","last_became_active":"2024-09-10T20:45:33.123511+0000","last_became_peered":"2024-09-10T20:45:33.123511+0000","last_unstale":"2024-09-10T20:45:34.193384+0000","last_undegraded":"2024-09-10T20:45:34.193384+0000","last_fullsized":"2024-09-10T20:45:34.193384+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:51:57.969974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108166+0000","last_change":"2024-09-10T20:45:33.118940+0000","last_active":"2024-09-10T20:45:34.108166+0000","last_peered":"2024-09-10T20:45:34.108166+0000","last_clean":"2024-09-10T20:45:34.108166+0000","last_became_active":"2024-09-10T20:45:33.118666+0000","last_became_peered":"2024-09-10T20:45:33.118666+0000","last_unstale":"2024-09-10T20:45:34.108166+0000","last_undegraded":"2024-09-10T20:45:34.108166+0000","last_fullsized":"2024-09-10T20:45:34.108166+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:20.315176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.116779+0000","last_change":"2024-09-10T20:45:33.114261+0000","last_active":"2024-09-10T20:45:33.116779+0000","last_peered":"2024-09-10T20:45:33.116779+0000","last_clean":"2024-09-10T20:45:33.116779+0000","last_became_active":"2024-09-10T20:45:33.114005+0000","last_became_peered":"2024-09-10T20:45:33.114005+0000","last_unstale":"2024-09-10T20:45:33.116779+0000","last_undegraded":"2024-09-10T20:45:33.116779+0000","last_fullsized":"2024-09-10T20:45:33.116779+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:09.715076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193061+0000","last_change":"2024-09-10T20:45:33.121844+0000","last_active":"2024-09-10T20:45:34.193061+0000","last_peered":"2024-09-10T20:45:34.193061+0000","last_clean":"2024-09-10T20:45:34.193061+0000","last_became_active":"2024-09-10T20:45:33.121553+0000","last_became_peered":"2024-09-10T20:45:33.121553+0000","last_unstale":"2024-09-10T20:45:34.193061+0000","last_undegraded":"2024-09-10T20:45:34.193061+0000","last_fullsized":"2024-09-10T20:45:34.193061+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:42:04.153517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117537+0000","last_change":"2024-09-10T20:45:33.113860+0000","last_active":"2024-09-10T20:45:33.117537+0000","last_peered":"2024-09-10T20:45:33.117537+0000","last_clean":"2024-09-10T20:45:33.117537+0000","last_became_active":"2024-09-10T20:45:33.113577+0000","last_became_peered":"2024-09-10T20:45:33.113577+0000","last_unstale":"2024-09-10T20:45:33.117537+0000","last_undegraded":"2024-09-10T20:45:33.117537+0000","last_fullsized":"2024-09-10T20:45:33.117537+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:18:20.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119541+0000","last_change":"2024-09-10T20:45:33.117159+0000","last_active":"2024-09-10T20:45:34.119541+0000","last_peered":"2024-09-10T20:45:34.119541+0000","last_clean":"2024-09-10T20:45:34.119541+0000","last_became_active":"2024-09-10T20:45:33.114548+0000","last_became_peered":"2024-09-10T20:45:33.114548+0000","last_unstale":"2024-09-10T20:45:34.119541+0000","last_undegraded":"2024-09-10T20:45:34.119541+0000","last_fullsized":"2024-09-10T20:45:34.119541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:36:03.874654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107559+0000","last_change":"2024-09-10T20:45:33.104844+0000","last_active":"2024-09-10T20:45:34.107559+0000","last_peered":"2024-09-10T20:45:34.107559+0000","last_clean":"2024-09-10T20:45:34.107559+0000","last_became_active":"2024-09-10T20:45:33.104580+0000","last_became_peered":"2024-09-10T20:45:33.104580+0000","last_unstale":"2024-09-10T20:45:34.107559+0000","last_undegraded":"2024-09-10T20:45:34.107559+0000","last_fullsized":"2024-09-10T20:45:34.107559+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:24.644915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118181+0000","last_change":"2024-09-10T20:45:33.112679+0000","last_active":"2024-09-10T20:45:33.118181+0000","last_peered":"2024-09-10T20:45:33.118181+0000","last_clean":"2024-09-10T20:45:33.118181+0000","last_became_active":"2024-09-10T20:45:33.112207+0000","last_became_peered":"2024-09-10T20:45:33.112207+0000","last_unstale":"2024-09-10T20:45:33.118181+0000","last_undegraded":"2024-09-10T20:45:33.118181+0000","last_fullsized":"2024-09-10T20:45:33.118181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:02.899888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107225+0000","last_change":"2024-09-10T20:45:33.115303+0000","last_active":"2024-09-10T20:45:34.107225+0000","last_peered":"2024-09-10T20:45:34.107225+0000","last_clean":"2024-09-10T20:45:34.107225+0000","last_became_active":"2024-09-10T20:45:33.115038+0000","last_became_peered":"2024-09-10T20:45:33.115038+0000","last_unstale":"2024-09-10T20:45:34.107225+0000","last_undegraded":"2024-09-10T20:45:34.107225+0000","last_fullsized":"2024-09-10T20:45:34.107225+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:45:07.998515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118337+0000","last_change":"2024-09-10T20:45:33.115315+0000","last_active":"2024-09-10T20:45:33.118337+0000","last_peered":"2024-09-10T20:45:33.118337+0000","last_clean":"2024-09-10T20:45:33.118337+0000","last_became_active":"2024-09-10T20:45:33.115028+0000","last_became_peered":"2024-09-10T20:45:33.115028+0000","last_unstale":"2024-09-10T20:45:33.118337+0000","last_undegraded":"2024-09-10T20:45:33.118337+0000","last_fullsized":"2024-09-10T20:45:33.118337+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:52:31.236437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107261+0000","last_change":"2024-09-10T20:45:33.107179+0000","last_active":"2024-09-10T20:45:34.107261+0000","last_peered":"2024-09-10T20:45:34.107261+0000","last_clean":"2024-09-10T20:45:34.107261+0000","last_became_active":"2024-09-10T20:45:33.106852+0000","last_became_peered":"2024-09-10T20:45:33.106852+0000","last_unstale":"2024-09-10T20:45:34.107261+0000","last_undegraded":"2024-09-10T20:45:34.107261+0000","last_fullsized":"2024-09-10T20:45:34.107261+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:23:07.143302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108672+0000","last_change":"2024-09-10T20:45:33.110475+0000","last_active":"2024-09-10T20:45:34.108672+0000","last_peered":"2024-09-10T20:45:34.108672+0000","last_clean":"2024-09-10T20:45:34.108672+0000","last_became_active":"2024-09-10T20:45:33.110209+0000","last_became_peered":"2024-09-10T20:45:33.110209+0000","last_unstale":"2024-09-10T20:45:34.108672+0000","last_undegraded":"2024-09-10T20:45:34.108672+0000","last_fullsized":"2024-09-10T20:45:34.108672+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:38.326547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108543+0000","last_change":"2024-09-10T20:45:33.117253+0000","last_active":"2024-09-10T20:45:34.108543+0000","last_peered":"2024-09-10T20:45:34.108543+0000","last_clean":"2024-09-10T20:45:34.108543+0000","last_became_active":"2024-09-10T20:45:33.116957+0000","last_became_peered":"2024-09-10T20:45:33.116957+0000","last_unstale":"2024-09-10T20:45:34.108543+0000","last_undegraded":"2024-09-10T20:45:34.108543+0000","last_fullsized":"2024-09-10T20:45:34.108543+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:08:11.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.116655+0000","last_change":"2024-09-10T20:45:33.112949+0000","last_active":"2024-09-10T20:45:33.116655+0000","last_peered":"2024-09-10T20:45:33.116655+0000","last_clean":"2024-09-10T20:45:33.116655+0000","last_became_active":"2024-09-10T20:45:33.112547+0000","last_became_peered":"2024-09-10T20:45:33.112547+0000","last_unstale":"2024-09-10T20:45:33.116655+0000","last_undegraded":"2024-09-10T20:45:33.116655+0000","last_fullsized":"2024-09-10T20:45:33.116655+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:40.389722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109640+0000","last_change":"2024-09-10T20:45:33.116286+0000","last_active":"2024-09-10T20:45:34.109640+0000","last_peered":"2024-09-10T20:45:34.109640+0000","last_clean":"2024-09-10T20:45:34.109640+0000","last_became_active":"2024-09-10T20:45:33.115337+0000","last_became_peered":"2024-09-10T20:45:33.115337+0000","last_unstale":"2024-09-10T20:45:34.109640+0000","last_undegraded":"2024-09-10T20:45:34.109640+0000","last_fullsized":"2024-09-10T20:45:34.109640+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:12.987501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193654+0000","last_change":"2024-09-10T20:45:33.124344+0000","last_active":"2024-09-10T20:45:34.193654+0000","last_peered":"2024-09-10T20:45:34.193654+0000","last_clean":"2024-09-10T20:45:34.193654+0000","last_became_active":"2024-09-10T20:45:33.124076+0000","last_became_peered":"2024-09-10T20:45:33.124076+0000","last_unstale":"2024-09-10T20:45:34.193654+0000","last_undegraded":"2024-09-10T20:45:34.193654+0000","last_fullsized":"2024-09-10T20:45:34.193654+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:25:41.018499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108914+0000","last_change":"2024-09-10T20:45:33.109115+0000","last_active":"2024-09-10T20:45:34.108914+0000","last_peered":"2024-09-10T20:45:34.108914+0000","last_clean":"2024-09-10T20:45:34.108914+0000","last_became_active":"2024-09-10T20:45:33.108872+0000","last_became_peered":"2024-09-10T20:45:33.108872+0000","last_unstale":"2024-09-10T20:45:34.108914+0000","last_undegraded":"2024-09-10T20:45:34.108914+0000","last_fullsized":"2024-09-10T20:45:34.108914+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:19.170809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193660+0000","last_change":"2024-09-10T20:45:33.125182+0000","last_active":"2024-09-10T20:45:34.193660+0000","last_peered":"2024-09-10T20:45:34.193660+0000","last_clean":"2024-09-10T20:45:34.193660+0000","last_became_active":"2024-09-10T20:45:33.124901+0000","last_became_peered":"2024-09-10T20:45:33.124901+0000","last_unstale":"2024-09-10T20:45:34.193660+0000","last_undegraded":"2024-09-10T20:45:34.193660+0000","last_fullsized":"2024-09-10T20:45:34.193660+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:49:02.439836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119182+0000","last_change":"2024-09-10T20:45:33.118532+0000","last_active":"2024-09-10T20:45:34.119182+0000","last_peered":"2024-09-10T20:45:34.119182+0000","last_clean":"2024-09-10T20:45:34.119182+0000","last_became_active":"2024-09-10T20:45:33.118241+0000","last_became_peered":"2024-09-10T20:45:33.118241+0000","last_unstale":"2024-09-10T20:45:34.119182+0000","last_undegraded":"2024-09-10T20:45:34.119182+0000","last_fullsized":"2024-09-10T20:45:34.119182+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:14.258490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193056+0000","last_change":"2024-09-10T20:45:33.117013+0000","last_active":"2024-09-10T20:45:34.193056+0000","last_peered":"2024-09-10T20:45:34.193056+0000","last_clean":"2024-09-10T20:45:34.193056+0000","last_became_active":"2024-09-10T20:45:33.115968+0000","last_became_peered":"2024-09-10T20:45:33.115968+0000","last_unstale":"2024-09-10T20:45:34.193056+0000","last_undegraded":"2024-09-10T20:45:34.193056+0000","last_fullsized":"2024-09-10T20:45:34.193056+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:51.716077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.114838+0000","last_change":"2024-09-10T20:45:33.108132+0000","last_active":"2024-09-10T20:45:33.114838+0000","last_peered":"2024-09-10T20:45:33.114838+0000","last_clean":"2024-09-10T20:45:33.114838+0000","last_became_active":"2024-09-10T20:45:33.107852+0000","last_became_peered":"2024-09-10T20:45:33.107852+0000","last_unstale":"2024-09-10T20:45:33.114838+0000","last_undegraded":"2024-09-10T20:45:33.114838+0000","last_fullsized":"2024-09-10T20:45:33.114838+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:30.269625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.119524+0000","last_change":"2024-09-10T20:45:33.117567+0000","last_active":"2024-09-10T20:45:33.119524+0000","last_peered":"2024-09-10T20:45:33.119524+0000","last_clean":"2024-09-10T20:45:33.119524+0000","last_became_active":"2024-09-10T20:45:33.117242+0000","last_became_peered":"2024-09-10T20:45:33.117242+0000","last_unstale":"2024-09-10T20:45:33.119524+0000","last_undegraded":"2024-09-10T20:45:33.119524+0000","last_fullsized":"2024-09-10T20:45:33.119524+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:58.644697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109805+0000","last_change":"2024-09-10T20:45:33.116564+0000","last_active":"2024-09-10T20:45:34.109805+0000","last_peered":"2024-09-10T20:45:34.109805+0000","last_clean":"2024-09-10T20:45:34.109805+0000","last_became_active":"2024-09-10T20:45:33.114935+0000","last_became_peered":"2024-09-10T20:45:33.114935+0000","last_unstale":"2024-09-10T20:45:34.109805+0000","last_undegraded":"2024-09-10T20:45:34.109805+0000","last_fullsized":"2024-09-10T20:45:34.109805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:28.467385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119454+0000","last_change":"2024-09-10T20:45:33.111540+0000","last_active":"2024-09-10T20:45:34.119454+0000","last_peered":"2024-09-10T20:45:34.119454+0000","last_clean":"2024-09-10T20:45:34.119454+0000","last_became_active":"2024-09-10T20:45:33.111191+0000","last_became_peered":"2024-09-10T20:45:33.111191+0000","last_unstale":"2024-09-10T20:45:34.119454+0000","last_undegraded":"2024-09-10T20:45:34.119454+0000","last_fullsized":"2024-09-10T20:45:34.119454+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:44.514005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.119565+0000","last_change":"2024-09-10T20:45:33.118504+0000","last_active":"2024-09-10T20:45:33.119565+0000","last_peered":"2024-09-10T20:45:33.119565+0000","last_clean":"2024-09-10T20:45:33.119565+0000","last_became_active":"2024-09-10T20:45:33.118245+0000","last_became_peered":"2024-09-10T20:45:33.118245+0000","last_unstale":"2024-09-10T20:45:33.119565+0000","last_undegraded":"2024-09-10T20:45:33.119565+0000","last_fullsized":"2024-09-10T20:45:33.119565+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:04.497879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.119801+0000","last_change":"2024-09-10T20:45:33.118716+0000","last_active":"2024-09-10T20:45:33.119801+0000","last_peered":"2024-09-10T20:45:33.119801+0000","last_clean":"2024-09-10T20:45:33.119801+0000","last_became_active":"2024-09-10T20:45:33.118444+0000","last_became_peered":"2024-09-10T20:45:33.118444+0000","last_unstale":"2024-09-10T20:45:33.119801+0000","last_undegraded":"2024-09-10T20:45:33.119801+0000","last_fullsized":"2024-09-10T20:45:33.119801+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:37:04.613463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193218+0000","last_change":"2024-09-10T20:45:33.126939+0000","last_active":"2024-09-10T20:45:34.193218+0000","last_peered":"2024-09-10T20:45:34.193218+0000","last_clean":"2024-09-10T20:45:34.193218+0000","last_became_active":"2024-09-10T20:45:33.126662+0000","last_became_peered":"2024-09-10T20:45:33.126662+0000","last_unstale":"2024-09-10T20:45:34.193218+0000","last_undegraded":"2024-09-10T20:45:34.193218+0000","last_fullsized":"2024-09-10T20:45:34.193218+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:34:00.496803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193343+0000","last_change":"2024-09-10T20:45:33.122334+0000","last_active":"2024-09-10T20:45:34.193343+0000","last_peered":"2024-09-10T20:45:34.193343+0000","last_clean":"2024-09-10T20:45:34.193343+0000","last_became_active":"2024-09-10T20:45:33.122082+0000","last_became_peered":"2024-09-10T20:45:33.122082+0000","last_unstale":"2024-09-10T20:45:34.193343+0000","last_undegraded":"2024-09-10T20:45:34.193343+0000","last_fullsized":"2024-09-10T20:45:34.193343+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:50.182824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193477+0000","last_change":"2024-09-10T20:45:33.124372+0000","last_active":"2024-09-10T20:45:34.193477+0000","last_peered":"2024-09-10T20:45:34.193477+0000","last_clean":"2024-09-10T20:45:34.193477+0000","last_became_active":"2024-09-10T20:45:33.124071+0000","last_became_peered":"2024-09-10T20:45:33.124071+0000","last_unstale":"2024-09-10T20:45:34.193477+0000","last_undegraded":"2024-09-10T20:45:34.193477+0000","last_fullsized":"2024-09-10T20:45:34.193477+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:27:38.846539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1f","version":"150'4","reported_seq":19,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109288+0000","last_change":"2024-09-10T20:45:32.105640+0000","last_active":"2024-09-10T20:45:34.109288+0000","last_peered":"2024-09-10T20:45:34.109288+0000","last_clean":"2024-09-10T20:45:34.109288+0000","last_became_active":"2024-09-10T20:45:32.105358+0000","last_became_peered":"2024-09-10T20:45:32.105358+0000","last_unstale":"2024-09-10T20:45:34.109288+0000","last_undegraded":"2024-09-10T20:45:34.109288+0000","last_fullsized":"2024-09-10T20:45:34.109288+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:21:17.129046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119691+0000","last_change":"2024-09-10T20:45:33.112965+0000","last_active":"2024-09-10T20:45:34.119691+0000","last_peered":"2024-09-10T20:45:34.119691+0000","last_clean":"2024-09-10T20:45:34.119691+0000","last_became_active":"2024-09-10T20:45:33.112481+0000","last_became_peered":"2024-09-10T20:45:33.112481+0000","last_unstale":"2024-09-10T20:45:34.119691+0000","last_undegraded":"2024-09-10T20:45:34.119691+0000","last_fullsized":"2024-09-10T20:45:34.119691+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:00.691119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1e","version":"150'1","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.126959+0000","last_change":"2024-09-10T20:45:32.104788+0000","last_active":"2024-09-10T20:45:33.126959+0000","last_peered":"2024-09-10T20:45:33.126959+0000","last_clean":"2024-09-10T20:45:33.126959+0000","last_became_active":"2024-09-10T20:45:32.104482+0000","last_became_peered":"2024-09-10T20:45:32.104482+0000","last_unstale":"2024-09-10T20:45:33.126959+0000","last_undegraded":"2024-09-10T20:45:33.126959+0000","last_fullsized":"2024-09-10T20:45:33.126959+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:33.739830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.115081+0000","last_change":"2024-09-10T20:45:33.109794+0000","last_active":"2024-09-10T20:45:33.115081+0000","last_peered":"2024-09-10T20:45:33.115081+0000","last_clean":"2024-09-10T20:45:33.115081+0000","last_became_active":"2024-09-10T20:45:33.109531+0000","last_became_peered":"2024-09-10T20:45:33.109531+0000","last_unstale":"2024-09-10T20:45:33.115081+0000","last_undegraded":"2024-09-10T20:45:33.115081+0000","last_fullsized":"2024-09-10T20:45:33.115081+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:25.422218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1d","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193695+0000","last_change":"2024-09-10T20:45:32.100914+0000","last_active":"2024-09-10T20:45:34.193695+0000","last_peered":"2024-09-10T20:45:34.193695+0000","last_clean":"2024-09-10T20:45:34.193695+0000","last_became_active":"2024-09-10T20:45:32.100261+0000","last_became_peered":"2024-09-10T20:45:32.100261+0000","last_unstale":"2024-09-10T20:45:34.193695+0000","last_undegraded":"2024-09-10T20:45:34.193695+0000","last_fullsized":"2024-09-10T20:45:34.193695+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:20.473374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118243+0000","last_change":"2024-09-10T20:45:33.114563+0000","last_active":"2024-09-10T20:45:33.118243+0000","last_peered":"2024-09-10T20:45:33.118243+0000","last_clean":"2024-09-10T20:45:33.118243+0000","last_became_active":"2024-09-10T20:45:33.114290+0000","last_became_peered":"2024-09-10T20:45:33.114290+0000","last_unstale":"2024-09-10T20:45:33.118243+0000","last_undegraded":"2024-09-10T20:45:33.118243+0000","last_fullsized":"2024-09-10T20:45:33.118243+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:10:25.521055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1c","version":"150'2","reported_seq":15,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.126585+0000","last_change":"2024-09-10T20:45:32.105077+0000","last_active":"2024-09-10T20:45:33.126585+0000","last_peered":"2024-09-10T20:45:33.126585+0000","last_clean":"2024-09-10T20:45:33.126585+0000","last_became_active":"2024-09-10T20:45:32.104752+0000","last_became_peered":"2024-09-10T20:45:32.104752+0000","last_unstale":"2024-09-10T20:45:33.126585+0000","last_undegraded":"2024-09-10T20:45:33.126585+0000","last_fullsized":"2024-09-10T20:45:33.126585+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:51:59.116943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108191+0000","last_change":"2024-09-10T20:45:33.119936+0000","last_active":"2024-09-10T20:45:34.108191+0000","last_peered":"2024-09-10T20:45:34.108191+0000","last_clean":"2024-09-10T20:45:34.108191+0000","last_became_active":"2024-09-10T20:45:33.119633+0000","last_became_peered":"2024-09-10T20:45:33.119633+0000","last_unstale":"2024-09-10T20:45:34.108191+0000","last_undegraded":"2024-09-10T20:45:34.108191+0000","last_fullsized":"2024-09-10T20:45:34.108191+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:09.028915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193571+0000","last_change":"2024-09-10T20:45:32.101597+0000","last_active":"2024-09-10T20:45:34.193571+0000","last_peered":"2024-09-10T20:45:34.193571+0000","last_clean":"2024-09-10T20:45:34.193571+0000","last_became_active":"2024-09-10T20:45:32.101285+0000","last_became_peered":"2024-09-10T20:45:32.101285+0000","last_unstale":"2024-09-10T20:45:34.193571+0000","last_undegraded":"2024-09-10T20:45:34.193571+0000","last_fullsized":"2024-09-10T20:45:34.193571+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:27.417706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193355+0000","last_change":"2024-09-10T20:45:33.126688+0000","last_active":"2024-09-10T20:45:34.193355+0000","last_peered":"2024-09-10T20:45:34.193355+0000","last_clean":"2024-09-10T20:45:34.193355+0000","last_became_active":"2024-09-10T20:45:33.126158+0000","last_became_peered":"2024-09-10T20:45:33.126158+0000","last_unstale":"2024-09-10T20:45:34.193355+0000","last_undegraded":"2024-09-10T20:45:34.193355+0000","last_fullsized":"2024-09-10T20:45:34.193355+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:09.824283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193600+0000","last_change":"2024-09-10T20:45:32.100239+0000","last_active":"2024-09-10T20:45:34.193600+0000","last_peered":"2024-09-10T20:45:34.193600+0000","last_clean":"2024-09-10T20:45:34.193600+0000","last_became_active":"2024-09-10T20:45:32.099917+0000","last_became_peered":"2024-09-10T20:45:32.099917+0000","last_unstale":"2024-09-10T20:45:34.193600+0000","last_undegraded":"2024-09-10T20:45:34.193600+0000","last_fullsized":"2024-09-10T20:45:34.193600+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:05.758751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107391+0000","last_change":"2024-09-10T20:45:33.120260+0000","last_active":"2024-09-10T20:45:34.107391+0000","last_peered":"2024-09-10T20:45:34.107391+0000","last_clean":"2024-09-10T20:45:34.107391+0000","last_became_active":"2024-09-10T20:45:33.119988+0000","last_became_peered":"2024-09-10T20:45:33.119988+0000","last_unstale":"2024-09-10T20:45:34.107391+0000","last_undegraded":"2024-09-10T20:45:34.107391+0000","last_fullsized":"2024-09-10T20:45:34.107391+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:58:02.211223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.104317+0000","last_change":"2024-09-10T20:45:32.093986+0000","last_active":"2024-09-10T20:45:33.104317+0000","last_peered":"2024-09-10T20:45:33.104317+0000","last_clean":"2024-09-10T20:45:33.104317+0000","last_became_active":"2024-09-10T20:45:32.093670+0000","last_became_peered":"2024-09-10T20:45:32.093670+0000","last_unstale":"2024-09-10T20:45:33.104317+0000","last_undegraded":"2024-09-10T20:45:33.104317+0000","last_fullsized":"2024-09-10T20:45:33.104317+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:05.838651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108803+0000","last_change":"2024-09-10T20:45:33.107865+0000","last_active":"2024-09-10T20:45:34.108803+0000","last_peered":"2024-09-10T20:45:34.108803+0000","last_clean":"2024-09-10T20:45:34.108803+0000","last_became_active":"2024-09-10T20:45:33.107260+0000","last_became_peered":"2024-09-10T20:45:33.107260+0000","last_unstale":"2024-09-10T20:45:34.108803+0000","last_undegraded":"2024-09-10T20:45:34.108803+0000","last_fullsized":"2024-09-10T20:45:34.108803+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:43.513040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107460+0000","last_change":"2024-09-10T20:45:32.102614+0000","last_active":"2024-09-10T20:45:34.107460+0000","last_peered":"2024-09-10T20:45:34.107460+0000","last_clean":"2024-09-10T20:45:34.107460+0000","last_became_active":"2024-09-10T20:45:32.102337+0000","last_became_peered":"2024-09-10T20:45:32.102337+0000","last_unstale":"2024-09-10T20:45:34.107460+0000","last_undegraded":"2024-09-10T20:45:34.107460+0000","last_fullsized":"2024-09-10T20:45:34.107460+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:47.515120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119824+0000","last_change":"2024-09-10T20:45:33.118749+0000","last_active":"2024-09-10T20:45:34.119824+0000","last_peered":"2024-09-10T20:45:34.119824+0000","last_clean":"2024-09-10T20:45:34.119824+0000","last_became_active":"2024-09-10T20:45:33.118223+0000","last_became_peered":"2024-09-10T20:45:33.118223+0000","last_unstale":"2024-09-10T20:45:34.119824+0000","last_undegraded":"2024-09-10T20:45:34.119824+0000","last_fullsized":"2024-09-10T20:45:34.119824+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:05:21.215639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119667+0000","last_change":"2024-09-10T20:45:32.105851+0000","last_active":"2024-09-10T20:45:34.119667+0000","last_peered":"2024-09-10T20:45:34.119667+0000","last_clean":"2024-09-10T20:45:34.119667+0000","last_became_active":"2024-09-10T20:45:32.105471+0000","last_became_peered":"2024-09-10T20:45:32.105471+0000","last_unstale":"2024-09-10T20:45:34.119667+0000","last_undegraded":"2024-09-10T20:45:34.119667+0000","last_fullsized":"2024-09-10T20:45:34.119667+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:58.457158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118887+0000","last_change":"2024-09-10T20:45:33.117535+0000","last_active":"2024-09-10T20:45:33.118887+0000","last_peered":"2024-09-10T20:45:33.118887+0000","last_clean":"2024-09-10T20:45:33.118887+0000","last_became_active":"2024-09-10T20:45:33.117274+0000","last_became_peered":"2024-09-10T20:45:33.117274+0000","last_unstale":"2024-09-10T20:45:33.118887+0000","last_undegraded":"2024-09-10T20:45:33.118887+0000","last_fullsized":"2024-09-10T20:45:33.118887+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:04.130253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.15","version":"150'3","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193745+0000","last_change":"2024-09-10T20:45:32.101232+0000","last_active":"2024-09-10T20:45:34.193745+0000","last_peered":"2024-09-10T20:45:34.193745+0000","last_clean":"2024-09-10T20:45:34.193745+0000","last_became_active":"2024-09-10T20:45:32.100599+0000","last_became_peered":"2024-09-10T20:45:32.100599+0000","last_unstale":"2024-09-10T20:45:34.193745+0000","last_undegraded":"2024-09-10T20:45:34.193745+0000","last_fullsized":"2024-09-10T20:45:34.193745+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:41:33.804543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119571+0000","last_change":"2024-09-10T20:45:33.119799+0000","last_active":"2024-09-10T20:45:34.119571+0000","last_peered":"2024-09-10T20:45:34.119571+0000","last_clean":"2024-09-10T20:45:34.119571+0000","last_became_active":"2024-09-10T20:45:33.119525+0000","last_became_peered":"2024-09-10T20:45:33.119525+0000","last_unstale":"2024-09-10T20:45:34.119571+0000","last_undegraded":"2024-09-10T20:45:34.119571+0000","last_fullsized":"2024-09-10T20:45:34.119571+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:29:55.073845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.14","version":"150'6","reported_seq":21,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193644+0000","last_change":"2024-09-10T20:45:32.101310+0000","last_active":"2024-09-10T20:45:34.193644+0000","last_peered":"2024-09-10T20:45:34.193644+0000","last_clean":"2024-09-10T20:45:34.193644+0000","last_became_active":"2024-09-10T20:45:32.100870+0000","last_became_peered":"2024-09-10T20:45:32.100870+0000","last_unstale":"2024-09-10T20:45:34.193644+0000","last_undegraded":"2024-09-10T20:45:34.193644+0000","last_fullsized":"2024-09-10T20:45:34.193644+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:32.955412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119347+0000","last_change":"2024-09-10T20:45:33.113848+0000","last_active":"2024-09-10T20:45:34.119347+0000","last_peered":"2024-09-10T20:45:34.119347+0000","last_clean":"2024-09-10T20:45:34.119347+0000","last_became_active":"2024-09-10T20:45:33.113510+0000","last_became_peered":"2024-09-10T20:45:33.113510+0000","last_unstale":"2024-09-10T20:45:34.119347+0000","last_undegraded":"2024-09-10T20:45:34.119347+0000","last_fullsized":"2024-09-10T20:45:34.119347+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:00.364074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.13","version":"150'2","reported_seq":15,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.128234+0000","last_change":"2024-09-10T20:45:32.105323+0000","last_active":"2024-09-10T20:45:33.128234+0000","last_peered":"2024-09-10T20:45:33.128234+0000","last_clean":"2024-09-10T20:45:33.128234+0000","last_became_active":"2024-09-10T20:45:32.105020+0000","last_became_peered":"2024-09-10T20:45:32.105020+0000","last_unstale":"2024-09-10T20:45:33.128234+0000","last_undegraded":"2024-09-10T20:45:33.128234+0000","last_fullsized":"2024-09-10T20:45:33.128234+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:21:32.402201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117028+0000","last_change":"2024-09-10T20:45:33.110010+0000","last_active":"2024-09-10T20:45:33.117028+0000","last_peered":"2024-09-10T20:45:33.117028+0000","last_clean":"2024-09-10T20:45:33.117028+0000","last_became_active":"2024-09-10T20:45:33.108937+0000","last_became_peered":"2024-09-10T20:45:33.108937+0000","last_unstale":"2024-09-10T20:45:33.117028+0000","last_undegraded":"2024-09-10T20:45:33.117028+0000","last_fullsized":"2024-09-10T20:45:33.117028+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:21:36.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119324+0000","last_change":"2024-09-10T20:45:32.101592+0000","last_active":"2024-09-10T20:45:34.119324+0000","last_peered":"2024-09-10T20:45:34.119324+0000","last_clean":"2024-09-10T20:45:34.119324+0000","last_became_active":"2024-09-10T20:45:32.101288+0000","last_became_peered":"2024-09-10T20:45:32.101288+0000","last_unstale":"2024-09-10T20:45:34.119324+0000","last_undegraded":"2024-09-10T20:45:34.119324+0000","last_fullsized":"2024-09-10T20:45:34.119324+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:33.739830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.115302+0000","last_change":"2024-09-10T20:45:33.110681+0000","last_active":"2024-09-10T20:45:33.115302+0000","last_peered":"2024-09-10T20:45:33.115302+0000","last_clean":"2024-09-10T20:45:33.115302+0000","last_became_active":"2024-09-10T20:45:33.110294+0000","last_became_peered":"2024-09-10T20:45:33.110294+0000","last_unstale":"2024-09-10T20:45:33.115302+0000","last_undegraded":"2024-09-10T20:45:33.115302+0000","last_fullsized":"2024-09-10T20:45:33.115302+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:54:26.905862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193736+0000","last_change":"2024-09-10T20:45:32.107778+0000","last_active":"2024-09-10T20:45:34.193736+0000","last_peered":"2024-09-10T20:45:34.193736+0000","last_clean":"2024-09-10T20:45:34.193736+0000","last_became_active":"2024-09-10T20:45:32.107430+0000","last_became_peered":"2024-09-10T20:45:32.107430+0000","last_unstale":"2024-09-10T20:45:34.193736+0000","last_undegraded":"2024-09-10T20:45:34.193736+0000","last_fullsized":"2024-09-10T20:45:34.193736+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:19:52.391022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193618+0000","last_change":"2024-09-10T20:45:33.123262+0000","last_active":"2024-09-10T20:45:34.193618+0000","last_peered":"2024-09-10T20:45:34.193618+0000","last_clean":"2024-09-10T20:45:34.193618+0000","last_became_active":"2024-09-10T20:45:33.121534+0000","last_became_peered":"2024-09-10T20:45:33.121534+0000","last_unstale":"2024-09-10T20:45:34.193618+0000","last_undegraded":"2024-09-10T20:45:34.193618+0000","last_fullsized":"2024-09-10T20:45:34.193618+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:23.769449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"150'8","reported_seq":21,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.129672+0000","last_change":"2024-09-10T20:45:32.104079+0000","last_active":"2024-09-10T20:45:33.129672+0000","last_peered":"2024-09-10T20:45:33.129672+0000","last_clean":"2024-09-10T20:45:33.129672+0000","last_became_active":"2024-09-10T20:45:32.103764+0000","last_became_peered":"2024-09-10T20:45:32.103764+0000","last_unstale":"2024-09-10T20:45:33.129672+0000","last_undegraded":"2024-09-10T20:45:33.129672+0000","last_fullsized":"2024-09-10T20:45:33.129672+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:39.635963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119593+0000","last_change":"2024-09-10T20:45:33.118866+0000","last_active":"2024-09-10T20:45:34.119593+0000","last_peered":"2024-09-10T20:45:34.119593+0000","last_clean":"2024-09-10T20:45:34.119593+0000","last_became_active":"2024-09-10T20:45:33.117849+0000","last_became_peered":"2024-09-10T20:45:33.117849+0000","last_unstale":"2024-09-10T20:45:34.119593+0000","last_undegraded":"2024-09-10T20:45:34.119593+0000","last_fullsized":"2024-09-10T20:45:34.119593+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:03.850771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.f","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193720+0000","last_change":"2024-09-10T20:45:32.100578+0000","last_active":"2024-09-10T20:45:34.193720+0000","last_peered":"2024-09-10T20:45:34.193720+0000","last_clean":"2024-09-10T20:45:34.193720+0000","last_became_active":"2024-09-10T20:45:32.100285+0000","last_became_peered":"2024-09-10T20:45:32.100285+0000","last_unstale":"2024-09-10T20:45:34.193720+0000","last_undegraded":"2024-09-10T20:45:34.193720+0000","last_fullsized":"2024-09-10T20:45:34.193720+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:35:49.837811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118850+0000","last_change":"2024-09-10T20:45:33.116598+0000","last_active":"2024-09-10T20:45:33.118850+0000","last_peered":"2024-09-10T20:45:33.118850+0000","last_clean":"2024-09-10T20:45:33.118850+0000","last_became_active":"2024-09-10T20:45:33.116312+0000","last_became_peered":"2024-09-10T20:45:33.116312+0000","last_unstale":"2024-09-10T20:45:33.118850+0000","last_undegraded":"2024-09-10T20:45:33.118850+0000","last_fullsized":"2024-09-10T20:45:33.118850+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:45.994824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.e","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108277+0000","last_change":"2024-09-10T20:45:32.093961+0000","last_active":"2024-09-10T20:45:34.108277+0000","last_peered":"2024-09-10T20:45:34.108277+0000","last_clean":"2024-09-10T20:45:34.108277+0000","last_became_active":"2024-09-10T20:45:32.093685+0000","last_became_peered":"2024-09-10T20:45:32.093685+0000","last_unstale":"2024-09-10T20:45:34.108277+0000","last_undegraded":"2024-09-10T20:45:34.108277+0000","last_fullsized":"2024-09-10T20:45:34.108277+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:34.299156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.115101+0000","last_change":"2024-09-10T20:45:33.111138+0000","last_active":"2024-09-10T20:45:33.115101+0000","last_peered":"2024-09-10T20:45:33.115101+0000","last_clean":"2024-09-10T20:45:33.115101+0000","last_became_active":"2024-09-10T20:45:33.110869+0000","last_became_peered":"2024-09-10T20:45:33.110869+0000","last_unstale":"2024-09-10T20:45:33.115101+0000","last_undegraded":"2024-09-10T20:45:33.115101+0000","last_fullsized":"2024-09-10T20:45:33.115101+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:09.847175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.d","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107384+0000","last_change":"2024-09-10T20:45:32.097103+0000","last_active":"2024-09-10T20:45:34.107384+0000","last_peered":"2024-09-10T20:45:34.107384+0000","last_clean":"2024-09-10T20:45:34.107384+0000","last_became_active":"2024-09-10T20:45:32.096822+0000","last_became_peered":"2024-09-10T20:45:32.096822+0000","last_unstale":"2024-09-10T20:45:34.107384+0000","last_undegraded":"2024-09-10T20:45:34.107384+0000","last_fullsized":"2024-09-10T20:45:34.107384+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:56:57.528882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118215+0000","last_change":"2024-09-10T20:45:33.113822+0000","last_active":"2024-09-10T20:45:33.118215+0000","last_peered":"2024-09-10T20:45:33.118215+0000","last_clean":"2024-09-10T20:45:33.118215+0000","last_became_active":"2024-09-10T20:45:33.113543+0000","last_became_peered":"2024-09-10T20:45:33.113543+0000","last_unstale":"2024-09-10T20:45:33.118215+0000","last_undegraded":"2024-09-10T20:45:33.118215+0000","last_fullsized":"2024-09-10T20:45:33.118215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:00.405043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107370+0000","last_change":"2024-09-10T20:45:32.099053+0000","last_active":"2024-09-10T20:45:34.107370+0000","last_peered":"2024-09-10T20:45:34.107370+0000","last_clean":"2024-09-10T20:45:34.107370+0000","last_became_active":"2024-09-10T20:45:32.098034+0000","last_became_peered":"2024-09-10T20:45:32.098034+0000","last_unstale":"2024-09-10T20:45:34.107370+0000","last_undegraded":"2024-09-10T20:45:34.107370+0000","last_fullsized":"2024-09-10T20:45:34.107370+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:51:38.133565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108496+0000","last_change":"2024-09-10T20:45:33.108637+0000","last_active":"2024-09-10T20:45:34.108496+0000","last_peered":"2024-09-10T20:45:34.108496+0000","last_clean":"2024-09-10T20:45:34.108496+0000","last_became_active":"2024-09-10T20:45:33.107586+0000","last_became_peered":"2024-09-10T20:45:33.107586+0000","last_unstale":"2024-09-10T20:45:34.108496+0000","last_undegraded":"2024-09-10T20:45:34.108496+0000","last_fullsized":"2024-09-10T20:45:34.108496+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:44.419441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.b","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193617+0000","last_change":"2024-09-10T20:45:32.103774+0000","last_active":"2024-09-10T20:45:34.193617+0000","last_peered":"2024-09-10T20:45:34.193617+0000","last_clean":"2024-09-10T20:45:34.193617+0000","last_became_active":"2024-09-10T20:45:32.103494+0000","last_became_peered":"2024-09-10T20:45:32.103494+0000","last_unstale":"2024-09-10T20:45:34.193617+0000","last_undegraded":"2024-09-10T20:45:34.193617+0000","last_fullsized":"2024-09-10T20:45:34.193617+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:53:41.319739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117042+0000","last_change":"2024-09-10T20:45:33.114706+0000","last_active":"2024-09-10T20:45:33.117042+0000","last_peered":"2024-09-10T20:45:33.117042+0000","last_clean":"2024-09-10T20:45:33.117042+0000","last_became_active":"2024-09-10T20:45:33.114434+0000","last_became_peered":"2024-09-10T20:45:33.114434+0000","last_unstale":"2024-09-10T20:45:33.117042+0000","last_undegraded":"2024-09-10T20:45:33.117042+0000","last_fullsized":"2024-09-10T20:45:33.117042+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:39.730807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108305+0000","last_change":"2024-09-10T20:45:32.089519+0000","last_active":"2024-09-10T20:45:34.108305+0000","last_peered":"2024-09-10T20:45:34.108305+0000","last_clean":"2024-09-10T20:45:34.108305+0000","last_became_active":"2024-09-10T20:45:32.089261+0000","last_became_peered":"2024-09-10T20:45:32.089261+0000","last_unstale":"2024-09-10T20:45:34.108305+0000","last_undegraded":"2024-09-10T20:45:34.108305+0000","last_fullsized":"2024-09-10T20:45:34.108305+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:54:12.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107195+0000","last_change":"2024-09-10T20:45:33.111558+0000","last_active":"2024-09-10T20:45:34.107195+0000","last_peered":"2024-09-10T20:45:34.107195+0000","last_clean":"2024-09-10T20:45:34.107195+0000","last_became_active":"2024-09-10T20:45:33.111289+0000","last_became_peered":"2024-09-10T20:45:33.111289+0000","last_unstale":"2024-09-10T20:45:34.107195+0000","last_undegraded":"2024-09-10T20:45:34.107195+0000","last_fullsized":"2024-09-10T20:45:34.107195+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:10:52.100150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107962+0000","last_change":"2024-09-10T20:45:32.095470+0000","last_active":"2024-09-10T20:45:34.107962+0000","last_peered":"2024-09-10T20:45:34.107962+0000","last_clean":"2024-09-10T20:45:34.107962+0000","last_became_active":"2024-09-10T20:45:32.095205+0000","last_became_peered":"2024-09-10T20:45:32.095205+0000","last_unstale":"2024-09-10T20:45:34.107962+0000","last_undegraded":"2024-09-10T20:45:34.107962+0000","last_fullsized":"2024-09-10T20:45:34.107962+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:15:38.235973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119248+0000","last_change":"2024-09-10T20:45:33.113477+0000","last_active":"2024-09-10T20:45:34.119248+0000","last_peered":"2024-09-10T20:45:34.119248+0000","last_clean":"2024-09-10T20:45:34.119248+0000","last_became_active":"2024-09-10T20:45:33.112598+0000","last_became_peered":"2024-09-10T20:45:33.112598+0000","last_unstale":"2024-09-10T20:45:34.119248+0000","last_undegraded":"2024-09-10T20:45:34.119248+0000","last_fullsized":"2024-09-10T20:45:34.119248+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:07.408588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107078+0000","last_change":"2024-09-10T20:45:32.094828+0000","last_active":"2024-09-10T20:45:34.107078+0000","last_peered":"2024-09-10T20:45:34.107078+0000","last_clean":"2024-09-10T20:45:34.107078+0000","last_became_active":"2024-09-10T20:45:32.094378+0000","last_became_peered":"2024-09-10T20:45:32.094378+0000","last_unstale":"2024-09-10T20:45:34.107078+0000","last_undegraded":"2024-09-10T20:45:34.107078+0000","last_fullsized":"2024-09-10T20:45:34.107078+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:08.348638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109039+0000","last_change":"2024-09-10T20:45:33.105239+0000","last_active":"2024-09-10T20:45:34.109039+0000","last_peered":"2024-09-10T20:45:34.109039+0000","last_clean":"2024-09-10T20:45:34.109039+0000","last_became_active":"2024-09-10T20:45:33.104892+0000","last_became_peered":"2024-09-10T20:45:33.104892+0000","last_unstale":"2024-09-10T20:45:34.109039+0000","last_undegraded":"2024-09-10T20:45:34.109039+0000","last_fullsized":"2024-09-10T20:45:34.109039+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:14.498352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107559+0000","last_change":"2024-09-10T20:45:32.098195+0000","last_active":"2024-09-10T20:45:34.107559+0000","last_peered":"2024-09-10T20:45:34.107559+0000","last_clean":"2024-09-10T20:45:34.107559+0000","last_became_active":"2024-09-10T20:45:32.097860+0000","last_became_peered":"2024-09-10T20:45:32.097860+0000","last_unstale":"2024-09-10T20:45:34.107559+0000","last_undegraded":"2024-09-10T20:45:34.107559+0000","last_fullsized":"2024-09-10T20:45:34.107559+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:54:18.509515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109175+0000","last_change":"2024-09-10T20:45:33.112713+0000","last_active":"2024-09-10T20:45:34.109175+0000","last_peered":"2024-09-10T20:45:34.109175+0000","last_clean":"2024-09-10T20:45:34.109175+0000","last_became_active":"2024-09-10T20:45:33.112455+0000","last_became_peered":"2024-09-10T20:45:33.112455+0000","last_unstale":"2024-09-10T20:45:34.109175+0000","last_undegraded":"2024-09-10T20:45:34.109175+0000","last_fullsized":"2024-09-10T20:45:34.109175+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:56.477740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"151'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.118590+0000","last_change":"2024-09-10T20:45:32.095563+0000","last_active":"2024-09-10T20:45:34.118590+0000","last_peered":"2024-09-10T20:45:34.118590+0000","last_clean":"2024-09-10T20:45:34.118590+0000","last_became_active":"2024-09-10T20:45:32.095292+0000","last_became_peered":"2024-09-10T20:45:32.095292+0000","last_unstale":"2024-09-10T20:45:34.118590+0000","last_undegraded":"2024-09-10T20:45:34.118590+0000","last_fullsized":"2024-09-10T20:45:34.118590+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:08.710035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108776+0000","last_change":"2024-09-10T20:45:33.113117+0000","last_active":"2024-09-10T20:45:34.108776+0000","last_peered":"2024-09-10T20:45:34.108776+0000","last_clean":"2024-09-10T20:45:34.108776+0000","last_became_active":"2024-09-10T20:45:33.112773+0000","last_became_peered":"2024-09-10T20:45:33.112773+0000","last_unstale":"2024-09-10T20:45:34.108776+0000","last_undegraded":"2024-09-10T20:45:34.108776+0000","last_fullsized":"2024-09-10T20:45:34.108776+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:24.667998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109760+0000","last_change":"2024-09-10T20:45:32.097347+0000","last_active":"2024-09-10T20:45:34.109760+0000","last_peered":"2024-09-10T20:45:34.109760+0000","last_clean":"2024-09-10T20:45:34.109760+0000","last_became_active":"2024-09-10T20:45:32.095888+0000","last_became_peered":"2024-09-10T20:45:32.095888+0000","last_unstale":"2024-09-10T20:45:34.109760+0000","last_undegraded":"2024-09-10T20:45:34.109760+0000","last_fullsized":"2024-09-10T20:45:34.109760+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:27:31.905240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193438+0000","last_change":"2024-09-10T20:45:33.121934+0000","last_active":"2024-09-10T20:45:34.193438+0000","last_peered":"2024-09-10T20:45:34.193438+0000","last_clean":"2024-09-10T20:45:34.193438+0000","last_became_active":"2024-09-10T20:45:33.121588+0000","last_became_peered":"2024-09-10T20:45:33.121588+0000","last_unstale":"2024-09-10T20:45:34.193438+0000","last_undegraded":"2024-09-10T20:45:34.193438+0000","last_fullsized":"2024-09-10T20:45:34.193438+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:19.484832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193757+0000","last_change":"2024-09-10T20:45:32.102574+0000","last_active":"2024-09-10T20:45:34.193757+0000","last_peered":"2024-09-10T20:45:34.193757+0000","last_clean":"2024-09-10T20:45:34.193757+0000","last_became_active":"2024-09-10T20:45:32.102293+0000","last_became_peered":"2024-09-10T20:45:32.102293+0000","last_unstale":"2024-09-10T20:45:34.193757+0000","last_undegraded":"2024-09-10T20:45:34.193757+0000","last_fullsized":"2024-09-10T20:45:34.193757+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:47:51.409191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.125494+0000","last_change":"2024-09-10T20:45:33.120957+0000","last_active":"2024-09-10T20:45:33.125494+0000","last_peered":"2024-09-10T20:45:33.125494+0000","last_clean":"2024-09-10T20:45:33.125494+0000","last_became_active":"2024-09-10T20:45:33.120572+0000","last_became_peered":"2024-09-10T20:45:33.120572+0000","last_unstale":"2024-09-10T20:45:33.125494+0000","last_undegraded":"2024-09-10T20:45:33.125494+0000","last_fullsized":"2024-09-10T20:45:33.125494+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:54.583764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.103241+0000","last_change":"2024-09-10T20:45:32.102494+0000","last_active":"2024-09-10T20:45:33.103241+0000","last_peered":"2024-09-10T20:45:33.103241+0000","last_clean":"2024-09-10T20:45:33.103241+0000","last_became_active":"2024-09-10T20:45:32.102235+0000","last_became_peered":"2024-09-10T20:45:32.102235+0000","last_unstale":"2024-09-10T20:45:33.103241+0000","last_undegraded":"2024-09-10T20:45:33.103241+0000","last_fullsized":"2024-09-10T20:45:33.103241+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:52.741100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107242+0000","last_change":"2024-09-10T20:45:33.108626+0000","last_active":"2024-09-10T20:45:34.107242+0000","last_peered":"2024-09-10T20:45:34.107242+0000","last_clean":"2024-09-10T20:45:34.107242+0000","last_became_active":"2024-09-10T20:45:33.108328+0000","last_became_peered":"2024-09-10T20:45:33.108328+0000","last_unstale":"2024-09-10T20:45:34.107242+0000","last_undegraded":"2024-09-10T20:45:34.107242+0000","last_fullsized":"2024-09-10T20:45:34.107242+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:32:18.145006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.102974+0000","last_change":"2024-09-10T20:45:32.096057+0000","last_active":"2024-09-10T20:45:33.102974+0000","last_peered":"2024-09-10T20:45:33.102974+0000","last_clean":"2024-09-10T20:45:33.102974+0000","last_became_active":"2024-09-10T20:45:32.095774+0000","last_became_peered":"2024-09-10T20:45:32.095774+0000","last_unstale":"2024-09-10T20:45:33.102974+0000","last_undegraded":"2024-09-10T20:45:33.102974+0000","last_fullsized":"2024-09-10T20:45:33.102974+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:28.439601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118456+0000","last_change":"2024-09-10T20:45:33.114644+0000","last_active":"2024-09-10T20:45:33.118456+0000","last_peered":"2024-09-10T20:45:33.118456+0000","last_clean":"2024-09-10T20:45:33.118456+0000","last_became_active":"2024-09-10T20:45:33.114266+0000","last_became_peered":"2024-09-10T20:45:33.114266+0000","last_unstale":"2024-09-10T20:45:33.118456+0000","last_undegraded":"2024-09-10T20:45:33.118456+0000","last_fullsized":"2024-09-10T20:45:33.118456+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:53:24.555776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.102963+0000","last_change":"2024-09-10T20:45:32.095328+0000","last_active":"2024-09-10T20:45:33.102963+0000","last_peered":"2024-09-10T20:45:33.102963+0000","last_clean":"2024-09-10T20:45:33.102963+0000","last_became_active":"2024-09-10T20:45:32.095058+0000","last_became_peered":"2024-09-10T20:45:32.095058+0000","last_unstale":"2024-09-10T20:45:33.102963+0000","last_undegraded":"2024-09-10T20:45:33.102963+0000","last_fullsized":"2024-09-10T20:45:33.102963+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:13.020977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193589+0000","last_change":"2024-09-10T20:45:33.124485+0000","last_active":"2024-09-10T20:45:34.193589+0000","last_peered":"2024-09-10T20:45:34.193589+0000","last_clean":"2024-09-10T20:45:34.193589+0000","last_became_active":"2024-09-10T20:45:33.124210+0000","last_became_peered":"2024-09-10T20:45:33.124210+0000","last_unstale":"2024-09-10T20:45:34.193589+0000","last_undegraded":"2024-09-10T20:45:34.193589+0000","last_fullsized":"2024-09-10T20:45:34.193589+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:40.051038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107916+0000","last_change":"2024-09-10T20:45:32.090837+0000","last_active":"2024-09-10T20:45:34.107916+0000","last_peered":"2024-09-10T20:45:34.107916+0000","last_clean":"2024-09-10T20:45:34.107916+0000","last_became_active":"2024-09-10T20:45:32.090366+0000","last_became_peered":"2024-09-10T20:45:32.090366+0000","last_unstale":"2024-09-10T20:45:34.107916+0000","last_undegraded":"2024-09-10T20:45:34.107916+0000","last_fullsized":"2024-09-10T20:45:34.107916+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:39:12.557544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108054+0000","last_change":"2024-09-10T20:45:33.110200+0000","last_active":"2024-09-10T20:45:34.108054+0000","last_peered":"2024-09-10T20:45:34.108054+0000","last_clean":"2024-09-10T20:45:34.108054+0000","last_became_active":"2024-09-10T20:45:33.109900+0000","last_became_peered":"2024-09-10T20:45:33.109900+0000","last_unstale":"2024-09-10T20:45:34.108054+0000","last_undegraded":"2024-09-10T20:45:34.108054+0000","last_fullsized":"2024-09-10T20:45:34.108054+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:54.427154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.110953+0000","last_change":"2024-09-10T20:45:32.106666+0000","last_active":"2024-09-10T20:45:33.110953+0000","last_peered":"2024-09-10T20:45:33.110953+0000","last_clean":"2024-09-10T20:45:33.110953+0000","last_became_active":"2024-09-10T20:45:32.106188+0000","last_became_peered":"2024-09-10T20:45:32.106188+0000","last_unstale":"2024-09-10T20:45:33.110953+0000","last_undegraded":"2024-09-10T20:45:33.110953+0000","last_fullsized":"2024-09-10T20:45:33.110953+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:58.457158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":377,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.102784+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:45:33.102784+0000","last_peered":"2024-09-10T20:45:33.102784+0000","last_clean":"2024-09-10T20:45:33.102784+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:45:33.102784+0000","last_undegraded":"2024-09-10T20:45:33.102784+0000","last_fullsized":"2024-09-10T20:45:33.102784+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542238,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1916928,"data_stored":840971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1888256,"data_stored":833009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542237,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2498560,"data_stored":1430219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1888256,"data_stored":832953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509535,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1900544,"data_stored":834601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509534,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1888256,"data_stored":832989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509534,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37768,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334072,"statfs":{"total":96636764160,"available":96598089728,"internally_reserved":0,"allocated":2478080,"data_stored":1423287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509534,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":833535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:45:37.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 -- 172.21.15.28:0/2342973005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4162090 msgr2=0x7ff3c40b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 --2- 172.21.15.28:0/2342973005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4162090 0x7ff3c40b6e00 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40d4d80 tx=0x7ff3bc02aac0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 -- 172.21.15.28:0/2342973005 shutdown_connections 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 --2- 172.21.15.28:0/2342973005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4162090 0x7ff3c40b6e00 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 --2- 172.21.15.28:0/2342973005 >> v2:172.21.15.159:3300/0 conn(0x7ff3c40d2b90 0x7ff3c4161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 -- 172.21.15.28:0/2342973005 >> 172.21.15.28:0/2342973005 conn(0x7ff3c40d3fb0 msgr2=0x7ff3c40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 -- 172.21.15.28:0/2342973005 shutdown_connections 2024-09-10T20:45:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.274+0000 7ff3c9724640 1 -- 172.21.15.28:0/2342973005 wait complete. 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.275+0000 7ff3c9724640 1 Processor -- start 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.275+0000 7ff3c9724640 1 -- start start 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c9724640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c40d2b90 0x7ff3c409dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c9724640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3c415de70 0x7ff3c409f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c9724640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3c4162090 0x7ff3c40a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c9724640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c415e3d0 con 0x7ff3c40d2b90 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c9724640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c415e250 con 0x7ff3c4162090 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c9724640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c40b7340 con 0x7ff3c415de70 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c40d2b90 0x7ff3c409dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c40d2b90 0x7ff3c409dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40762/0 (socket says 172.21.15.28:40762) 2024-09-10T20:45:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c2ffd640 1 -- 172.21.15.28:0/1677693642 learned_addr learned my addr 172.21.15.28:0/1677693642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c37fe640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3c4162090 0x7ff3c40a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.276+0000 7ff3c27fc640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3c415de70 0x7ff3c409f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c2ffd640 1 -- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3c415de70 msgr2=0x7ff3c409f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c2ffd640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3c415de70 0x7ff3c409f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c2ffd640 1 -- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3c4162090 msgr2=0x7ff3c40a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c2ffd640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3c4162090 0x7ff3c40a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c2ffd640 1 -- 172.21.15.28:0/1677693642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c41f5820 con 0x7ff3c40d2b90 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c2ffd640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c40d2b90 0x7ff3c409dbf0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c450 tx=0x7ff3b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3ab7fe640 1 -- 172.21.15.28:0/1677693642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b40052d0 con 0x7ff3c40d2b90 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3ab7fe640 1 -- 172.21.15.28:0/1677693642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3b4005470 con 0x7ff3c40d2b90 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c41f4c10 con 0x7ff3c40d2b90 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3ab7fe640 1 -- 172.21.15.28:0/1677693642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4012870 con 0x7ff3c40d2b90 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.277+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c41f51d0 con 0x7ff3c40d2b90 2024-09-10T20:45:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.278+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c40b7860 con 0x7ff3c40d2b90 2024-09-10T20:45:37.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.279+0000 7ff3ab7fe640 1 -- 172.21.15.28:0/1677693642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7ff3b40035f0 con 0x7ff3c40d2b90 2024-09-10T20:45:37.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.279+0000 7ff3ab7fe640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3b0078810 0x7ff3b007acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.279+0000 7ff3ab7fe640 1 -- 172.21.15.28:0/1677693642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7ff3b4097f80 con 0x7ff3c40d2b90 2024-09-10T20:45:37.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.279+0000 7ff3c27fc640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3b0078810 0x7ff3b007acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:37.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.280+0000 7ff3c27fc640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3b0078810 0x7ff3b007acd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40b9040 tx=0x7ff3b800b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:37.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.283+0000 7ff3ab7fe640 1 -- 172.21.15.28:0/1677693642 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff3b4061ee0 con 0x7ff3c40d2b90 2024-09-10T20:45:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.465+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff3c40d1360 con 0x7ff3b0078810 2024-09-10T20:45:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.470+0000 7ff3ab7fe640 1 -- 172.21.15.28:0/1677693642 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252479 (secure 0 0 0) 0x7ff3c40d1360 con 0x7ff3b0078810 2024-09-10T20:45:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:45:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3b0078810 msgr2=0x7ff3b007acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3b0078810 0x7ff3b007acd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40b9040 tx=0x7ff3b800b390 comp rx=0 tx=0).stop 2024-09-10T20:45:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c40d2b90 msgr2=0x7ff3c409dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c40d2b90 0x7ff3c409dbf0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c450 tx=0x7ff3b400c910 comp rx=0 tx=0).stop 2024-09-10T20:45:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 shutdown_connections 2024-09-10T20:45:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff3b0078810 0x7ff3b007acd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff3c4162090 0x7ff3c40a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff3c415de70 0x7ff3c409f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 --2- 172.21.15.28:0/1677693642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c40d2b90 0x7ff3c409dbf0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 >> 172.21.15.28:0/1677693642 conn(0x7ff3c40d3fb0 msgr2=0x7ff3c40b93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 shutdown_connections 2024-09-10T20:45:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.473+0000 7ff3c9724640 1 -- 172.21.15.28:0/1677693642 wait complete. 2024-09-10T20:45:37.487 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-10T20:45:36.333138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282128,"kb_used_data":15968,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754692592,"statfs":{"total":773094113280,"available":772805214208,"internally_reserved":0,"allocated":16351232,"data_stored":7861564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.253090"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107482+0000","last_change":"2024-09-10T20:45:33.114526+0000","last_active":"2024-09-10T20:45:34.107482+0000","last_peered":"2024-09-10T20:45:34.107482+0000","last_clean":"2024-09-10T20:45:34.107482+0000","last_became_active":"2024-09-10T20:45:33.114248+0000","last_became_peered":"2024-09-10T20:45:33.114248+0000","last_unstale":"2024-09-10T20:45:34.107482+0000","last_undegraded":"2024-09-10T20:45:34.107482+0000","last_fullsized":"2024-09-10T20:45:34.107482+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:30:00.736809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.114983+0000","last_change":"2024-09-10T20:45:33.108874+0000","last_active":"2024-09-10T20:45:33.114983+0000","last_peered":"2024-09-10T20:45:33.114983+0000","last_clean":"2024-09-10T20:45:33.114983+0000","last_became_active":"2024-09-10T20:45:33.108632+0000","last_became_peered":"2024-09-10T20:45:33.108632+0000","last_unstale":"2024-09-10T20:45:33.114983+0000","last_undegraded":"2024-09-10T20:45:33.114983+0000","last_fullsized":"2024-09-10T20:45:33.114983+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:55:32.635136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109612+0000","last_change":"2024-09-10T20:45:33.109173+0000","last_active":"2024-09-10T20:45:34.109612+0000","last_peered":"2024-09-10T20:45:34.109612+0000","last_clean":"2024-09-10T20:45:34.109612+0000","last_became_active":"2024-09-10T20:45:33.108839+0000","last_became_peered":"2024-09-10T20:45:33.108839+0000","last_unstale":"2024-09-10T20:45:34.109612+0000","last_undegraded":"2024-09-10T20:45:34.109612+0000","last_fullsized":"2024-09-10T20:45:34.109612+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:48.286241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117601+0000","last_change":"2024-09-10T20:45:33.111808+0000","last_active":"2024-09-10T20:45:33.117601+0000","last_peered":"2024-09-10T20:45:33.117601+0000","last_clean":"2024-09-10T20:45:33.117601+0000","last_became_active":"2024-09-10T20:45:33.111528+0000","last_became_peered":"2024-09-10T20:45:33.111528+0000","last_unstale":"2024-09-10T20:45:33.117601+0000","last_undegraded":"2024-09-10T20:45:33.117601+0000","last_fullsized":"2024-09-10T20:45:33.117601+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:42:09.243456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193384+0000","last_change":"2024-09-10T20:45:33.123880+0000","last_active":"2024-09-10T20:45:34.193384+0000","last_peered":"2024-09-10T20:45:34.193384+0000","last_clean":"2024-09-10T20:45:34.193384+0000","last_became_active":"2024-09-10T20:45:33.123511+0000","last_became_peered":"2024-09-10T20:45:33.123511+0000","last_unstale":"2024-09-10T20:45:34.193384+0000","last_undegraded":"2024-09-10T20:45:34.193384+0000","last_fullsized":"2024-09-10T20:45:34.193384+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:51:57.969974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108166+0000","last_change":"2024-09-10T20:45:33.118940+0000","last_active":"2024-09-10T20:45:34.108166+0000","last_peered":"2024-09-10T20:45:34.108166+0000","last_clean":"2024-09-10T20:45:34.108166+0000","last_became_active":"2024-09-10T20:45:33.118666+0000","last_became_peered":"2024-09-10T20:45:33.118666+0000","last_unstale":"2024-09-10T20:45:34.108166+0000","last_undegraded":"2024-09-10T20:45:34.108166+0000","last_fullsized":"2024-09-10T20:45:34.108166+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:20.315176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.116779+0000","last_change":"2024-09-10T20:45:33.114261+0000","last_active":"2024-09-10T20:45:33.116779+0000","last_peered":"2024-09-10T20:45:33.116779+0000","last_clean":"2024-09-10T20:45:33.116779+0000","last_became_active":"2024-09-10T20:45:33.114005+0000","last_became_peered":"2024-09-10T20:45:33.114005+0000","last_unstale":"2024-09-10T20:45:33.116779+0000","last_undegraded":"2024-09-10T20:45:33.116779+0000","last_fullsized":"2024-09-10T20:45:33.116779+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:09.715076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193061+0000","last_change":"2024-09-10T20:45:33.121844+0000","last_active":"2024-09-10T20:45:34.193061+0000","last_peered":"2024-09-10T20:45:34.193061+0000","last_clean":"2024-09-10T20:45:34.193061+0000","last_became_active":"2024-09-10T20:45:33.121553+0000","last_became_peered":"2024-09-10T20:45:33.121553+0000","last_unstale":"2024-09-10T20:45:34.193061+0000","last_undegraded":"2024-09-10T20:45:34.193061+0000","last_fullsized":"2024-09-10T20:45:34.193061+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:42:04.153517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117537+0000","last_change":"2024-09-10T20:45:33.113860+0000","last_active":"2024-09-10T20:45:33.117537+0000","last_peered":"2024-09-10T20:45:33.117537+0000","last_clean":"2024-09-10T20:45:33.117537+0000","last_became_active":"2024-09-10T20:45:33.113577+0000","last_became_peered":"2024-09-10T20:45:33.113577+0000","last_unstale":"2024-09-10T20:45:33.117537+0000","last_undegraded":"2024-09-10T20:45:33.117537+0000","last_fullsized":"2024-09-10T20:45:33.117537+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:18:20.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119541+0000","last_change":"2024-09-10T20:45:33.117159+0000","last_active":"2024-09-10T20:45:34.119541+0000","last_peered":"2024-09-10T20:45:34.119541+0000","last_clean":"2024-09-10T20:45:34.119541+0000","last_became_active":"2024-09-10T20:45:33.114548+0000","last_became_peered":"2024-09-10T20:45:33.114548+0000","last_unstale":"2024-09-10T20:45:34.119541+0000","last_undegraded":"2024-09-10T20:45:34.119541+0000","last_fullsized":"2024-09-10T20:45:34.119541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:36:03.874654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107559+0000","last_change":"2024-09-10T20:45:33.104844+0000","last_active":"2024-09-10T20:45:34.107559+0000","last_peered":"2024-09-10T20:45:34.107559+0000","last_clean":"2024-09-10T20:45:34.107559+0000","last_became_active":"2024-09-10T20:45:33.104580+0000","last_became_peered":"2024-09-10T20:45:33.104580+0000","last_unstale":"2024-09-10T20:45:34.107559+0000","last_undegraded":"2024-09-10T20:45:34.107559+0000","last_fullsized":"2024-09-10T20:45:34.107559+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:24.644915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118181+0000","last_change":"2024-09-10T20:45:33.112679+0000","last_active":"2024-09-10T20:45:33.118181+0000","last_peered":"2024-09-10T20:45:33.118181+0000","last_clean":"2024-09-10T20:45:33.118181+0000","last_became_active":"2024-09-10T20:45:33.112207+0000","last_became_peered":"2024-09-10T20:45:33.112207+0000","last_unstale":"2024-09-10T20:45:33.118181+0000","last_undegraded":"2024-09-10T20:45:33.118181+0000","last_fullsized":"2024-09-10T20:45:33.118181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:02.899888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107225+0000","last_change":"2024-09-10T20:45:33.115303+0000","last_active":"2024-09-10T20:45:34.107225+0000","last_peered":"2024-09-10T20:45:34.107225+0000","last_clean":"2024-09-10T20:45:34.107225+0000","last_became_active":"2024-09-10T20:45:33.115038+0000","last_became_peered":"2024-09-10T20:45:33.115038+0000","last_unstale":"2024-09-10T20:45:34.107225+0000","last_undegraded":"2024-09-10T20:45:34.107225+0000","last_fullsized":"2024-09-10T20:45:34.107225+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:45:07.998515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118337+0000","last_change":"2024-09-10T20:45:33.115315+0000","last_active":"2024-09-10T20:45:33.118337+0000","last_peered":"2024-09-10T20:45:33.118337+0000","last_clean":"2024-09-10T20:45:33.118337+0000","last_became_active":"2024-09-10T20:45:33.115028+0000","last_became_peered":"2024-09-10T20:45:33.115028+0000","last_unstale":"2024-09-10T20:45:33.118337+0000","last_undegraded":"2024-09-10T20:45:33.118337+0000","last_fullsized":"2024-09-10T20:45:33.118337+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:52:31.236437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107261+0000","last_change":"2024-09-10T20:45:33.107179+0000","last_active":"2024-09-10T20:45:34.107261+0000","last_peered":"2024-09-10T20:45:34.107261+0000","last_clean":"2024-09-10T20:45:34.107261+0000","last_became_active":"2024-09-10T20:45:33.106852+0000","last_became_peered":"2024-09-10T20:45:33.106852+0000","last_unstale":"2024-09-10T20:45:34.107261+0000","last_undegraded":"2024-09-10T20:45:34.107261+0000","last_fullsized":"2024-09-10T20:45:34.107261+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:23:07.143302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108672+0000","last_change":"2024-09-10T20:45:33.110475+0000","last_active":"2024-09-10T20:45:34.108672+0000","last_peered":"2024-09-10T20:45:34.108672+0000","last_clean":"2024-09-10T20:45:34.108672+0000","last_became_active":"2024-09-10T20:45:33.110209+0000","last_became_peered":"2024-09-10T20:45:33.110209+0000","last_unstale":"2024-09-10T20:45:34.108672+0000","last_undegraded":"2024-09-10T20:45:34.108672+0000","last_fullsized":"2024-09-10T20:45:34.108672+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:38.326547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108543+0000","last_change":"2024-09-10T20:45:33.117253+0000","last_active":"2024-09-10T20:45:34.108543+0000","last_peered":"2024-09-10T20:45:34.108543+0000","last_clean":"2024-09-10T20:45:34.108543+0000","last_became_active":"2024-09-10T20:45:33.116957+0000","last_became_peered":"2024-09-10T20:45:33.116957+0000","last_unstale":"2024-09-10T20:45:34.108543+0000","last_undegraded":"2024-09-10T20:45:34.108543+0000","last_fullsized":"2024-09-10T20:45:34.108543+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:08:11.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.116655+0000","last_change":"2024-09-10T20:45:33.112949+0000","last_active":"2024-09-10T20:45:33.116655+0000","last_peered":"2024-09-10T20:45:33.116655+0000","last_clean":"2024-09-10T20:45:33.116655+0000","last_became_active":"2024-09-10T20:45:33.112547+0000","last_became_peered":"2024-09-10T20:45:33.112547+0000","last_unstale":"2024-09-10T20:45:33.116655+0000","last_undegraded":"2024-09-10T20:45:33.116655+0000","last_fullsized":"2024-09-10T20:45:33.116655+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:40.389722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109640+0000","last_change":"2024-09-10T20:45:33.116286+0000","last_active":"2024-09-10T20:45:34.109640+0000","last_peered":"2024-09-10T20:45:34.109640+0000","last_clean":"2024-09-10T20:45:34.109640+0000","last_became_active":"2024-09-10T20:45:33.115337+0000","last_became_peered":"2024-09-10T20:45:33.115337+0000","last_unstale":"2024-09-10T20:45:34.109640+0000","last_undegraded":"2024-09-10T20:45:34.109640+0000","last_fullsized":"2024-09-10T20:45:34.109640+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:12.987501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193654+0000","last_change":"2024-09-10T20:45:33.124344+0000","last_active":"2024-09-10T20:45:34.193654+0000","last_peered":"2024-09-10T20:45:34.193654+0000","last_clean":"2024-09-10T20:45:34.193654+0000","last_became_active":"2024-09-10T20:45:33.124076+0000","last_became_peered":"2024-09-10T20:45:33.124076+0000","last_unstale":"2024-09-10T20:45:34.193654+0000","last_undegraded":"2024-09-10T20:45:34.193654+0000","last_fullsized":"2024-09-10T20:45:34.193654+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:25:41.018499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108914+0000","last_change":"2024-09-10T20:45:33.109115+0000","last_active":"2024-09-10T20:45:34.108914+0000","last_peered":"2024-09-10T20:45:34.108914+0000","last_clean":"2024-09-10T20:45:34.108914+0000","last_became_active":"2024-09-10T20:45:33.108872+0000","last_became_peered":"2024-09-10T20:45:33.108872+0000","last_unstale":"2024-09-10T20:45:34.108914+0000","last_undegraded":"2024-09-10T20:45:34.108914+0000","last_fullsized":"2024-09-10T20:45:34.108914+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:19.170809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193660+0000","last_change":"2024-09-10T20:45:33.125182+0000","last_active":"2024-09-10T20:45:34.193660+0000","last_peered":"2024-09-10T20:45:34.193660+0000","last_clean":"2024-09-10T20:45:34.193660+0000","last_became_active":"2024-09-10T20:45:33.124901+0000","last_became_peered":"2024-09-10T20:45:33.124901+0000","last_unstale":"2024-09-10T20:45:34.193660+0000","last_undegraded":"2024-09-10T20:45:34.193660+0000","last_fullsized":"2024-09-10T20:45:34.193660+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:49:02.439836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119182+0000","last_change":"2024-09-10T20:45:33.118532+0000","last_active":"2024-09-10T20:45:34.119182+0000","last_peered":"2024-09-10T20:45:34.119182+0000","last_clean":"2024-09-10T20:45:34.119182+0000","last_became_active":"2024-09-10T20:45:33.118241+0000","last_became_peered":"2024-09-10T20:45:33.118241+0000","last_unstale":"2024-09-10T20:45:34.119182+0000","last_undegraded":"2024-09-10T20:45:34.119182+0000","last_fullsized":"2024-09-10T20:45:34.119182+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:14.258490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193056+0000","last_change":"2024-09-10T20:45:33.117013+0000","last_active":"2024-09-10T20:45:34.193056+0000","last_peered":"2024-09-10T20:45:34.193056+0000","last_clean":"2024-09-10T20:45:34.193056+0000","last_became_active":"2024-09-10T20:45:33.115968+0000","last_became_peered":"2024-09-10T20:45:33.115968+0000","last_unstale":"2024-09-10T20:45:34.193056+0000","last_undegraded":"2024-09-10T20:45:34.193056+0000","last_fullsized":"2024-09-10T20:45:34.193056+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:17:51.716077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.114838+0000","last_change":"2024-09-10T20:45:33.108132+0000","last_active":"2024-09-10T20:45:33.114838+0000","last_peered":"2024-09-10T20:45:33.114838+0000","last_clean":"2024-09-10T20:45:33.114838+0000","last_became_active":"2024-09-10T20:45:33.107852+0000","last_became_peered":"2024-09-10T20:45:33.107852+0000","last_unstale":"2024-09-10T20:45:33.114838+0000","last_undegraded":"2024-09-10T20:45:33.114838+0000","last_fullsized":"2024-09-10T20:45:33.114838+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:22:30.269625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.119524+0000","last_change":"2024-09-10T20:45:33.117567+0000","last_active":"2024-09-10T20:45:33.119524+0000","last_peered":"2024-09-10T20:45:33.119524+0000","last_clean":"2024-09-10T20:45:33.119524+0000","last_became_active":"2024-09-10T20:45:33.117242+0000","last_became_peered":"2024-09-10T20:45:33.117242+0000","last_unstale":"2024-09-10T20:45:33.119524+0000","last_undegraded":"2024-09-10T20:45:33.119524+0000","last_fullsized":"2024-09-10T20:45:33.119524+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:58.644697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109805+0000","last_change":"2024-09-10T20:45:33.116564+0000","last_active":"2024-09-10T20:45:34.109805+0000","last_peered":"2024-09-10T20:45:34.109805+0000","last_clean":"2024-09-10T20:45:34.109805+0000","last_became_active":"2024-09-10T20:45:33.114935+0000","last_became_peered":"2024-09-10T20:45:33.114935+0000","last_unstale":"2024-09-10T20:45:34.109805+0000","last_undegraded":"2024-09-10T20:45:34.109805+0000","last_fullsized":"2024-09-10T20:45:34.109805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:28.467385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119454+0000","last_change":"2024-09-10T20:45:33.111540+0000","last_active":"2024-09-10T20:45:34.119454+0000","last_peered":"2024-09-10T20:45:34.119454+0000","last_clean":"2024-09-10T20:45:34.119454+0000","last_became_active":"2024-09-10T20:45:33.111191+0000","last_became_peered":"2024-09-10T20:45:33.111191+0000","last_unstale":"2024-09-10T20:45:34.119454+0000","last_undegraded":"2024-09-10T20:45:34.119454+0000","last_fullsized":"2024-09-10T20:45:34.119454+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:44.514005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.119565+0000","last_change":"2024-09-10T20:45:33.118504+0000","last_active":"2024-09-10T20:45:33.119565+0000","last_peered":"2024-09-10T20:45:33.119565+0000","last_clean":"2024-09-10T20:45:33.119565+0000","last_became_active":"2024-09-10T20:45:33.118245+0000","last_became_peered":"2024-09-10T20:45:33.118245+0000","last_unstale":"2024-09-10T20:45:33.119565+0000","last_undegraded":"2024-09-10T20:45:33.119565+0000","last_fullsized":"2024-09-10T20:45:33.119565+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:04.497879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.119801+0000","last_change":"2024-09-10T20:45:33.118716+0000","last_active":"2024-09-10T20:45:33.119801+0000","last_peered":"2024-09-10T20:45:33.119801+0000","last_clean":"2024-09-10T20:45:33.119801+0000","last_became_active":"2024-09-10T20:45:33.118444+0000","last_became_peered":"2024-09-10T20:45:33.118444+0000","last_unstale":"2024-09-10T20:45:33.119801+0000","last_undegraded":"2024-09-10T20:45:33.119801+0000","last_fullsized":"2024-09-10T20:45:33.119801+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:37:04.613463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193218+0000","last_change":"2024-09-10T20:45:33.126939+0000","last_active":"2024-09-10T20:45:34.193218+0000","last_peered":"2024-09-10T20:45:34.193218+0000","last_clean":"2024-09-10T20:45:34.193218+0000","last_became_active":"2024-09-10T20:45:33.126662+0000","last_became_peered":"2024-09-10T20:45:33.126662+0000","last_unstale":"2024-09-10T20:45:34.193218+0000","last_undegraded":"2024-09-10T20:45:34.193218+0000","last_fullsized":"2024-09-10T20:45:34.193218+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:34:00.496803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193343+0000","last_change":"2024-09-10T20:45:33.122334+0000","last_active":"2024-09-10T20:45:34.193343+0000","last_peered":"2024-09-10T20:45:34.193343+0000","last_clean":"2024-09-10T20:45:34.193343+0000","last_became_active":"2024-09-10T20:45:33.122082+0000","last_became_peered":"2024-09-10T20:45:33.122082+0000","last_unstale":"2024-09-10T20:45:34.193343+0000","last_undegraded":"2024-09-10T20:45:34.193343+0000","last_fullsized":"2024-09-10T20:45:34.193343+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:50.182824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193477+0000","last_change":"2024-09-10T20:45:33.124372+0000","last_active":"2024-09-10T20:45:34.193477+0000","last_peered":"2024-09-10T20:45:34.193477+0000","last_clean":"2024-09-10T20:45:34.193477+0000","last_became_active":"2024-09-10T20:45:33.124071+0000","last_became_peered":"2024-09-10T20:45:33.124071+0000","last_unstale":"2024-09-10T20:45:34.193477+0000","last_undegraded":"2024-09-10T20:45:34.193477+0000","last_fullsized":"2024-09-10T20:45:34.193477+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:27:38.846539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1f","version":"150'4","reported_seq":19,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109288+0000","last_change":"2024-09-10T20:45:32.105640+0000","last_active":"2024-09-10T20:45:34.109288+0000","last_peered":"2024-09-10T20:45:34.109288+0000","last_clean":"2024-09-10T20:45:34.109288+0000","last_became_active":"2024-09-10T20:45:32.105358+0000","last_became_peered":"2024-09-10T20:45:32.105358+0000","last_unstale":"2024-09-10T20:45:34.109288+0000","last_undegraded":"2024-09-10T20:45:34.109288+0000","last_fullsized":"2024-09-10T20:45:34.109288+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:21:17.129046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119691+0000","last_change":"2024-09-10T20:45:33.112965+0000","last_active":"2024-09-10T20:45:34.119691+0000","last_peered":"2024-09-10T20:45:34.119691+0000","last_clean":"2024-09-10T20:45:34.119691+0000","last_became_active":"2024-09-10T20:45:33.112481+0000","last_became_peered":"2024-09-10T20:45:33.112481+0000","last_unstale":"2024-09-10T20:45:34.119691+0000","last_undegraded":"2024-09-10T20:45:34.119691+0000","last_fullsized":"2024-09-10T20:45:34.119691+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:00.691119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1e","version":"150'1","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.126959+0000","last_change":"2024-09-10T20:45:32.104788+0000","last_active":"2024-09-10T20:45:33.126959+0000","last_peered":"2024-09-10T20:45:33.126959+0000","last_clean":"2024-09-10T20:45:33.126959+0000","last_became_active":"2024-09-10T20:45:32.104482+0000","last_became_peered":"2024-09-10T20:45:32.104482+0000","last_unstale":"2024-09-10T20:45:33.126959+0000","last_undegraded":"2024-09-10T20:45:33.126959+0000","last_fullsized":"2024-09-10T20:45:33.126959+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:33.739830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.115081+0000","last_change":"2024-09-10T20:45:33.109794+0000","last_active":"2024-09-10T20:45:33.115081+0000","last_peered":"2024-09-10T20:45:33.115081+0000","last_clean":"2024-09-10T20:45:33.115081+0000","last_became_active":"2024-09-10T20:45:33.109531+0000","last_became_peered":"2024-09-10T20:45:33.109531+0000","last_unstale":"2024-09-10T20:45:33.115081+0000","last_undegraded":"2024-09-10T20:45:33.115081+0000","last_fullsized":"2024-09-10T20:45:33.115081+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:25.422218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1d","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193695+0000","last_change":"2024-09-10T20:45:32.100914+0000","last_active":"2024-09-10T20:45:34.193695+0000","last_peered":"2024-09-10T20:45:34.193695+0000","last_clean":"2024-09-10T20:45:34.193695+0000","last_became_active":"2024-09-10T20:45:32.100261+0000","last_became_peered":"2024-09-10T20:45:32.100261+0000","last_unstale":"2024-09-10T20:45:34.193695+0000","last_undegraded":"2024-09-10T20:45:34.193695+0000","last_fullsized":"2024-09-10T20:45:34.193695+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:20.473374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118243+0000","last_change":"2024-09-10T20:45:33.114563+0000","last_active":"2024-09-10T20:45:33.118243+0000","last_peered":"2024-09-10T20:45:33.118243+0000","last_clean":"2024-09-10T20:45:33.118243+0000","last_became_active":"2024-09-10T20:45:33.114290+0000","last_became_peered":"2024-09-10T20:45:33.114290+0000","last_unstale":"2024-09-10T20:45:33.118243+0000","last_undegraded":"2024-09-10T20:45:33.118243+0000","last_fullsized":"2024-09-10T20:45:33.118243+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:10:25.521055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1c","version":"150'2","reported_seq":15,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.126585+0000","last_change":"2024-09-10T20:45:32.105077+0000","last_active":"2024-09-10T20:45:33.126585+0000","last_peered":"2024-09-10T20:45:33.126585+0000","last_clean":"2024-09-10T20:45:33.126585+0000","last_became_active":"2024-09-10T20:45:32.104752+0000","last_became_peered":"2024-09-10T20:45:32.104752+0000","last_unstale":"2024-09-10T20:45:33.126585+0000","last_undegraded":"2024-09-10T20:45:33.126585+0000","last_fullsized":"2024-09-10T20:45:33.126585+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:51:59.116943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108191+0000","last_change":"2024-09-10T20:45:33.119936+0000","last_active":"2024-09-10T20:45:34.108191+0000","last_peered":"2024-09-10T20:45:34.108191+0000","last_clean":"2024-09-10T20:45:34.108191+0000","last_became_active":"2024-09-10T20:45:33.119633+0000","last_became_peered":"2024-09-10T20:45:33.119633+0000","last_unstale":"2024-09-10T20:45:34.108191+0000","last_undegraded":"2024-09-10T20:45:34.108191+0000","last_fullsized":"2024-09-10T20:45:34.108191+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:09.028915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193571+0000","last_change":"2024-09-10T20:45:32.101597+0000","last_active":"2024-09-10T20:45:34.193571+0000","last_peered":"2024-09-10T20:45:34.193571+0000","last_clean":"2024-09-10T20:45:34.193571+0000","last_became_active":"2024-09-10T20:45:32.101285+0000","last_became_peered":"2024-09-10T20:45:32.101285+0000","last_unstale":"2024-09-10T20:45:34.193571+0000","last_undegraded":"2024-09-10T20:45:34.193571+0000","last_fullsized":"2024-09-10T20:45:34.193571+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:27.417706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193355+0000","last_change":"2024-09-10T20:45:33.126688+0000","last_active":"2024-09-10T20:45:34.193355+0000","last_peered":"2024-09-10T20:45:34.193355+0000","last_clean":"2024-09-10T20:45:34.193355+0000","last_became_active":"2024-09-10T20:45:33.126158+0000","last_became_peered":"2024-09-10T20:45:33.126158+0000","last_unstale":"2024-09-10T20:45:34.193355+0000","last_undegraded":"2024-09-10T20:45:34.193355+0000","last_fullsized":"2024-09-10T20:45:34.193355+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:09.824283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193600+0000","last_change":"2024-09-10T20:45:32.100239+0000","last_active":"2024-09-10T20:45:34.193600+0000","last_peered":"2024-09-10T20:45:34.193600+0000","last_clean":"2024-09-10T20:45:34.193600+0000","last_became_active":"2024-09-10T20:45:32.099917+0000","last_became_peered":"2024-09-10T20:45:32.099917+0000","last_unstale":"2024-09-10T20:45:34.193600+0000","last_undegraded":"2024-09-10T20:45:34.193600+0000","last_fullsized":"2024-09-10T20:45:34.193600+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:05.758751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107391+0000","last_change":"2024-09-10T20:45:33.120260+0000","last_active":"2024-09-10T20:45:34.107391+0000","last_peered":"2024-09-10T20:45:34.107391+0000","last_clean":"2024-09-10T20:45:34.107391+0000","last_became_active":"2024-09-10T20:45:33.119988+0000","last_became_peered":"2024-09-10T20:45:33.119988+0000","last_unstale":"2024-09-10T20:45:34.107391+0000","last_undegraded":"2024-09-10T20:45:34.107391+0000","last_fullsized":"2024-09-10T20:45:34.107391+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:58:02.211223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.104317+0000","last_change":"2024-09-10T20:45:32.093986+0000","last_active":"2024-09-10T20:45:33.104317+0000","last_peered":"2024-09-10T20:45:33.104317+0000","last_clean":"2024-09-10T20:45:33.104317+0000","last_became_active":"2024-09-10T20:45:32.093670+0000","last_became_peered":"2024-09-10T20:45:32.093670+0000","last_unstale":"2024-09-10T20:45:33.104317+0000","last_undegraded":"2024-09-10T20:45:33.104317+0000","last_fullsized":"2024-09-10T20:45:33.104317+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:05.838651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108803+0000","last_change":"2024-09-10T20:45:33.107865+0000","last_active":"2024-09-10T20:45:34.108803+0000","last_peered":"2024-09-10T20:45:34.108803+0000","last_clean":"2024-09-10T20:45:34.108803+0000","last_became_active":"2024-09-10T20:45:33.107260+0000","last_became_peered":"2024-09-10T20:45:33.107260+0000","last_unstale":"2024-09-10T20:45:34.108803+0000","last_undegraded":"2024-09-10T20:45:34.108803+0000","last_fullsized":"2024-09-10T20:45:34.108803+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:25:43.513040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107460+0000","last_change":"2024-09-10T20:45:32.102614+0000","last_active":"2024-09-10T20:45:34.107460+0000","last_peered":"2024-09-10T20:45:34.107460+0000","last_clean":"2024-09-10T20:45:34.107460+0000","last_became_active":"2024-09-10T20:45:32.102337+0000","last_became_peered":"2024-09-10T20:45:32.102337+0000","last_unstale":"2024-09-10T20:45:34.107460+0000","last_undegraded":"2024-09-10T20:45:34.107460+0000","last_fullsized":"2024-09-10T20:45:34.107460+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:47.515120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119824+0000","last_change":"2024-09-10T20:45:33.118749+0000","last_active":"2024-09-10T20:45:34.119824+0000","last_peered":"2024-09-10T20:45:34.119824+0000","last_clean":"2024-09-10T20:45:34.119824+0000","last_became_active":"2024-09-10T20:45:33.118223+0000","last_became_peered":"2024-09-10T20:45:33.118223+0000","last_unstale":"2024-09-10T20:45:34.119824+0000","last_undegraded":"2024-09-10T20:45:34.119824+0000","last_fullsized":"2024-09-10T20:45:34.119824+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:05:21.215639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119667+0000","last_change":"2024-09-10T20:45:32.105851+0000","last_active":"2024-09-10T20:45:34.119667+0000","last_peered":"2024-09-10T20:45:34.119667+0000","last_clean":"2024-09-10T20:45:34.119667+0000","last_became_active":"2024-09-10T20:45:32.105471+0000","last_became_peered":"2024-09-10T20:45:32.105471+0000","last_unstale":"2024-09-10T20:45:34.119667+0000","last_undegraded":"2024-09-10T20:45:34.119667+0000","last_fullsized":"2024-09-10T20:45:34.119667+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:58.457158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118887+0000","last_change":"2024-09-10T20:45:33.117535+0000","last_active":"2024-09-10T20:45:33.118887+0000","last_peered":"2024-09-10T20:45:33.118887+0000","last_clean":"2024-09-10T20:45:33.118887+0000","last_became_active":"2024-09-10T20:45:33.117274+0000","last_became_peered":"2024-09-10T20:45:33.117274+0000","last_unstale":"2024-09-10T20:45:33.118887+0000","last_undegraded":"2024-09-10T20:45:33.118887+0000","last_fullsized":"2024-09-10T20:45:33.118887+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:04.130253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.15","version":"150'3","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193745+0000","last_change":"2024-09-10T20:45:32.101232+0000","last_active":"2024-09-10T20:45:34.193745+0000","last_peered":"2024-09-10T20:45:34.193745+0000","last_clean":"2024-09-10T20:45:34.193745+0000","last_became_active":"2024-09-10T20:45:32.100599+0000","last_became_peered":"2024-09-10T20:45:32.100599+0000","last_unstale":"2024-09-10T20:45:34.193745+0000","last_undegraded":"2024-09-10T20:45:34.193745+0000","last_fullsized":"2024-09-10T20:45:34.193745+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:41:33.804543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119571+0000","last_change":"2024-09-10T20:45:33.119799+0000","last_active":"2024-09-10T20:45:34.119571+0000","last_peered":"2024-09-10T20:45:34.119571+0000","last_clean":"2024-09-10T20:45:34.119571+0000","last_became_active":"2024-09-10T20:45:33.119525+0000","last_became_peered":"2024-09-10T20:45:33.119525+0000","last_unstale":"2024-09-10T20:45:34.119571+0000","last_undegraded":"2024-09-10T20:45:34.119571+0000","last_fullsized":"2024-09-10T20:45:34.119571+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:29:55.073845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.14","version":"150'6","reported_seq":21,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193644+0000","last_change":"2024-09-10T20:45:32.101310+0000","last_active":"2024-09-10T20:45:34.193644+0000","last_peered":"2024-09-10T20:45:34.193644+0000","last_clean":"2024-09-10T20:45:34.193644+0000","last_became_active":"2024-09-10T20:45:32.100870+0000","last_became_peered":"2024-09-10T20:45:32.100870+0000","last_unstale":"2024-09-10T20:45:34.193644+0000","last_undegraded":"2024-09-10T20:45:34.193644+0000","last_fullsized":"2024-09-10T20:45:34.193644+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:32.955412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119347+0000","last_change":"2024-09-10T20:45:33.113848+0000","last_active":"2024-09-10T20:45:34.119347+0000","last_peered":"2024-09-10T20:45:34.119347+0000","last_clean":"2024-09-10T20:45:34.119347+0000","last_became_active":"2024-09-10T20:45:33.113510+0000","last_became_peered":"2024-09-10T20:45:33.113510+0000","last_unstale":"2024-09-10T20:45:34.119347+0000","last_undegraded":"2024-09-10T20:45:34.119347+0000","last_fullsized":"2024-09-10T20:45:34.119347+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:00.364074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.13","version":"150'2","reported_seq":15,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.128234+0000","last_change":"2024-09-10T20:45:32.105323+0000","last_active":"2024-09-10T20:45:33.128234+0000","last_peered":"2024-09-10T20:45:33.128234+0000","last_clean":"2024-09-10T20:45:33.128234+0000","last_became_active":"2024-09-10T20:45:32.105020+0000","last_became_peered":"2024-09-10T20:45:32.105020+0000","last_unstale":"2024-09-10T20:45:33.128234+0000","last_undegraded":"2024-09-10T20:45:33.128234+0000","last_fullsized":"2024-09-10T20:45:33.128234+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:21:32.402201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117028+0000","last_change":"2024-09-10T20:45:33.110010+0000","last_active":"2024-09-10T20:45:33.117028+0000","last_peered":"2024-09-10T20:45:33.117028+0000","last_clean":"2024-09-10T20:45:33.117028+0000","last_became_active":"2024-09-10T20:45:33.108937+0000","last_became_peered":"2024-09-10T20:45:33.108937+0000","last_unstale":"2024-09-10T20:45:33.117028+0000","last_undegraded":"2024-09-10T20:45:33.117028+0000","last_fullsized":"2024-09-10T20:45:33.117028+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:21:36.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119324+0000","last_change":"2024-09-10T20:45:32.101592+0000","last_active":"2024-09-10T20:45:34.119324+0000","last_peered":"2024-09-10T20:45:34.119324+0000","last_clean":"2024-09-10T20:45:34.119324+0000","last_became_active":"2024-09-10T20:45:32.101288+0000","last_became_peered":"2024-09-10T20:45:32.101288+0000","last_unstale":"2024-09-10T20:45:34.119324+0000","last_undegraded":"2024-09-10T20:45:34.119324+0000","last_fullsized":"2024-09-10T20:45:34.119324+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:33.739830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.115302+0000","last_change":"2024-09-10T20:45:33.110681+0000","last_active":"2024-09-10T20:45:33.115302+0000","last_peered":"2024-09-10T20:45:33.115302+0000","last_clean":"2024-09-10T20:45:33.115302+0000","last_became_active":"2024-09-10T20:45:33.110294+0000","last_became_peered":"2024-09-10T20:45:33.110294+0000","last_unstale":"2024-09-10T20:45:33.115302+0000","last_undegraded":"2024-09-10T20:45:33.115302+0000","last_fullsized":"2024-09-10T20:45:33.115302+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:54:26.905862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193736+0000","last_change":"2024-09-10T20:45:32.107778+0000","last_active":"2024-09-10T20:45:34.193736+0000","last_peered":"2024-09-10T20:45:34.193736+0000","last_clean":"2024-09-10T20:45:34.193736+0000","last_became_active":"2024-09-10T20:45:32.107430+0000","last_became_peered":"2024-09-10T20:45:32.107430+0000","last_unstale":"2024-09-10T20:45:34.193736+0000","last_undegraded":"2024-09-10T20:45:34.193736+0000","last_fullsized":"2024-09-10T20:45:34.193736+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:19:52.391022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193618+0000","last_change":"2024-09-10T20:45:33.123262+0000","last_active":"2024-09-10T20:45:34.193618+0000","last_peered":"2024-09-10T20:45:34.193618+0000","last_clean":"2024-09-10T20:45:34.193618+0000","last_became_active":"2024-09-10T20:45:33.121534+0000","last_became_peered":"2024-09-10T20:45:33.121534+0000","last_unstale":"2024-09-10T20:45:34.193618+0000","last_undegraded":"2024-09-10T20:45:34.193618+0000","last_fullsized":"2024-09-10T20:45:34.193618+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:35:23.769449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"150'8","reported_seq":21,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.129672+0000","last_change":"2024-09-10T20:45:32.104079+0000","last_active":"2024-09-10T20:45:33.129672+0000","last_peered":"2024-09-10T20:45:33.129672+0000","last_clean":"2024-09-10T20:45:33.129672+0000","last_became_active":"2024-09-10T20:45:32.103764+0000","last_became_peered":"2024-09-10T20:45:32.103764+0000","last_unstale":"2024-09-10T20:45:33.129672+0000","last_undegraded":"2024-09-10T20:45:33.129672+0000","last_fullsized":"2024-09-10T20:45:33.129672+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:39.635963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119593+0000","last_change":"2024-09-10T20:45:33.118866+0000","last_active":"2024-09-10T20:45:34.119593+0000","last_peered":"2024-09-10T20:45:34.119593+0000","last_clean":"2024-09-10T20:45:34.119593+0000","last_became_active":"2024-09-10T20:45:33.117849+0000","last_became_peered":"2024-09-10T20:45:33.117849+0000","last_unstale":"2024-09-10T20:45:34.119593+0000","last_undegraded":"2024-09-10T20:45:34.119593+0000","last_fullsized":"2024-09-10T20:45:34.119593+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:03.850771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.f","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193720+0000","last_change":"2024-09-10T20:45:32.100578+0000","last_active":"2024-09-10T20:45:34.193720+0000","last_peered":"2024-09-10T20:45:34.193720+0000","last_clean":"2024-09-10T20:45:34.193720+0000","last_became_active":"2024-09-10T20:45:32.100285+0000","last_became_peered":"2024-09-10T20:45:32.100285+0000","last_unstale":"2024-09-10T20:45:34.193720+0000","last_undegraded":"2024-09-10T20:45:34.193720+0000","last_fullsized":"2024-09-10T20:45:34.193720+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:35:49.837811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118850+0000","last_change":"2024-09-10T20:45:33.116598+0000","last_active":"2024-09-10T20:45:33.118850+0000","last_peered":"2024-09-10T20:45:33.118850+0000","last_clean":"2024-09-10T20:45:33.118850+0000","last_became_active":"2024-09-10T20:45:33.116312+0000","last_became_peered":"2024-09-10T20:45:33.116312+0000","last_unstale":"2024-09-10T20:45:33.118850+0000","last_undegraded":"2024-09-10T20:45:33.118850+0000","last_fullsized":"2024-09-10T20:45:33.118850+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:45.994824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.e","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108277+0000","last_change":"2024-09-10T20:45:32.093961+0000","last_active":"2024-09-10T20:45:34.108277+0000","last_peered":"2024-09-10T20:45:34.108277+0000","last_clean":"2024-09-10T20:45:34.108277+0000","last_became_active":"2024-09-10T20:45:32.093685+0000","last_became_peered":"2024-09-10T20:45:32.093685+0000","last_unstale":"2024-09-10T20:45:34.108277+0000","last_undegraded":"2024-09-10T20:45:34.108277+0000","last_fullsized":"2024-09-10T20:45:34.108277+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:34.299156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.115101+0000","last_change":"2024-09-10T20:45:33.111138+0000","last_active":"2024-09-10T20:45:33.115101+0000","last_peered":"2024-09-10T20:45:33.115101+0000","last_clean":"2024-09-10T20:45:33.115101+0000","last_became_active":"2024-09-10T20:45:33.110869+0000","last_became_peered":"2024-09-10T20:45:33.110869+0000","last_unstale":"2024-09-10T20:45:33.115101+0000","last_undegraded":"2024-09-10T20:45:33.115101+0000","last_fullsized":"2024-09-10T20:45:33.115101+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:43:09.847175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.d","version":"150'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107384+0000","last_change":"2024-09-10T20:45:32.097103+0000","last_active":"2024-09-10T20:45:34.107384+0000","last_peered":"2024-09-10T20:45:34.107384+0000","last_clean":"2024-09-10T20:45:34.107384+0000","last_became_active":"2024-09-10T20:45:32.096822+0000","last_became_peered":"2024-09-10T20:45:32.096822+0000","last_unstale":"2024-09-10T20:45:34.107384+0000","last_undegraded":"2024-09-10T20:45:34.107384+0000","last_fullsized":"2024-09-10T20:45:34.107384+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:56:57.528882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118215+0000","last_change":"2024-09-10T20:45:33.113822+0000","last_active":"2024-09-10T20:45:33.118215+0000","last_peered":"2024-09-10T20:45:33.118215+0000","last_clean":"2024-09-10T20:45:33.118215+0000","last_became_active":"2024-09-10T20:45:33.113543+0000","last_became_peered":"2024-09-10T20:45:33.113543+0000","last_unstale":"2024-09-10T20:45:33.118215+0000","last_undegraded":"2024-09-10T20:45:33.118215+0000","last_fullsized":"2024-09-10T20:45:33.118215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:00.405043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107370+0000","last_change":"2024-09-10T20:45:32.099053+0000","last_active":"2024-09-10T20:45:34.107370+0000","last_peered":"2024-09-10T20:45:34.107370+0000","last_clean":"2024-09-10T20:45:34.107370+0000","last_became_active":"2024-09-10T20:45:32.098034+0000","last_became_peered":"2024-09-10T20:45:32.098034+0000","last_unstale":"2024-09-10T20:45:34.107370+0000","last_undegraded":"2024-09-10T20:45:34.107370+0000","last_fullsized":"2024-09-10T20:45:34.107370+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:51:38.133565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108496+0000","last_change":"2024-09-10T20:45:33.108637+0000","last_active":"2024-09-10T20:45:34.108496+0000","last_peered":"2024-09-10T20:45:34.108496+0000","last_clean":"2024-09-10T20:45:34.108496+0000","last_became_active":"2024-09-10T20:45:33.107586+0000","last_became_peered":"2024-09-10T20:45:33.107586+0000","last_unstale":"2024-09-10T20:45:34.108496+0000","last_undegraded":"2024-09-10T20:45:34.108496+0000","last_fullsized":"2024-09-10T20:45:34.108496+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:44.419441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.b","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193617+0000","last_change":"2024-09-10T20:45:32.103774+0000","last_active":"2024-09-10T20:45:34.193617+0000","last_peered":"2024-09-10T20:45:34.193617+0000","last_clean":"2024-09-10T20:45:34.193617+0000","last_became_active":"2024-09-10T20:45:32.103494+0000","last_became_peered":"2024-09-10T20:45:32.103494+0000","last_unstale":"2024-09-10T20:45:34.193617+0000","last_undegraded":"2024-09-10T20:45:34.193617+0000","last_fullsized":"2024-09-10T20:45:34.193617+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:53:41.319739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.117042+0000","last_change":"2024-09-10T20:45:33.114706+0000","last_active":"2024-09-10T20:45:33.117042+0000","last_peered":"2024-09-10T20:45:33.117042+0000","last_clean":"2024-09-10T20:45:33.117042+0000","last_became_active":"2024-09-10T20:45:33.114434+0000","last_became_peered":"2024-09-10T20:45:33.114434+0000","last_unstale":"2024-09-10T20:45:33.117042+0000","last_undegraded":"2024-09-10T20:45:33.117042+0000","last_fullsized":"2024-09-10T20:45:33.117042+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:39.730807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108305+0000","last_change":"2024-09-10T20:45:32.089519+0000","last_active":"2024-09-10T20:45:34.108305+0000","last_peered":"2024-09-10T20:45:34.108305+0000","last_clean":"2024-09-10T20:45:34.108305+0000","last_became_active":"2024-09-10T20:45:32.089261+0000","last_became_peered":"2024-09-10T20:45:32.089261+0000","last_unstale":"2024-09-10T20:45:34.108305+0000","last_undegraded":"2024-09-10T20:45:34.108305+0000","last_fullsized":"2024-09-10T20:45:34.108305+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:54:12.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107195+0000","last_change":"2024-09-10T20:45:33.111558+0000","last_active":"2024-09-10T20:45:34.107195+0000","last_peered":"2024-09-10T20:45:34.107195+0000","last_clean":"2024-09-10T20:45:34.107195+0000","last_became_active":"2024-09-10T20:45:33.111289+0000","last_became_peered":"2024-09-10T20:45:33.111289+0000","last_unstale":"2024-09-10T20:45:34.107195+0000","last_undegraded":"2024-09-10T20:45:34.107195+0000","last_fullsized":"2024-09-10T20:45:34.107195+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:10:52.100150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107962+0000","last_change":"2024-09-10T20:45:32.095470+0000","last_active":"2024-09-10T20:45:34.107962+0000","last_peered":"2024-09-10T20:45:34.107962+0000","last_clean":"2024-09-10T20:45:34.107962+0000","last_became_active":"2024-09-10T20:45:32.095205+0000","last_became_peered":"2024-09-10T20:45:32.095205+0000","last_unstale":"2024-09-10T20:45:34.107962+0000","last_undegraded":"2024-09-10T20:45:34.107962+0000","last_fullsized":"2024-09-10T20:45:34.107962+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:15:38.235973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.119248+0000","last_change":"2024-09-10T20:45:33.113477+0000","last_active":"2024-09-10T20:45:34.119248+0000","last_peered":"2024-09-10T20:45:34.119248+0000","last_clean":"2024-09-10T20:45:34.119248+0000","last_became_active":"2024-09-10T20:45:33.112598+0000","last_became_peered":"2024-09-10T20:45:33.112598+0000","last_unstale":"2024-09-10T20:45:34.119248+0000","last_undegraded":"2024-09-10T20:45:34.119248+0000","last_fullsized":"2024-09-10T20:45:34.119248+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:07.408588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107078+0000","last_change":"2024-09-10T20:45:32.094828+0000","last_active":"2024-09-10T20:45:34.107078+0000","last_peered":"2024-09-10T20:45:34.107078+0000","last_clean":"2024-09-10T20:45:34.107078+0000","last_became_active":"2024-09-10T20:45:32.094378+0000","last_became_peered":"2024-09-10T20:45:32.094378+0000","last_unstale":"2024-09-10T20:45:34.107078+0000","last_undegraded":"2024-09-10T20:45:34.107078+0000","last_fullsized":"2024-09-10T20:45:34.107078+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:08.348638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109039+0000","last_change":"2024-09-10T20:45:33.105239+0000","last_active":"2024-09-10T20:45:34.109039+0000","last_peered":"2024-09-10T20:45:34.109039+0000","last_clean":"2024-09-10T20:45:34.109039+0000","last_became_active":"2024-09-10T20:45:33.104892+0000","last_became_peered":"2024-09-10T20:45:33.104892+0000","last_unstale":"2024-09-10T20:45:34.109039+0000","last_undegraded":"2024-09-10T20:45:34.109039+0000","last_fullsized":"2024-09-10T20:45:34.109039+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:14.498352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"150'2","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107559+0000","last_change":"2024-09-10T20:45:32.098195+0000","last_active":"2024-09-10T20:45:34.107559+0000","last_peered":"2024-09-10T20:45:34.107559+0000","last_clean":"2024-09-10T20:45:34.107559+0000","last_became_active":"2024-09-10T20:45:32.097860+0000","last_became_peered":"2024-09-10T20:45:32.097860+0000","last_unstale":"2024-09-10T20:45:34.107559+0000","last_undegraded":"2024-09-10T20:45:34.107559+0000","last_fullsized":"2024-09-10T20:45:34.107559+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:54:18.509515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109175+0000","last_change":"2024-09-10T20:45:33.112713+0000","last_active":"2024-09-10T20:45:34.109175+0000","last_peered":"2024-09-10T20:45:34.109175+0000","last_clean":"2024-09-10T20:45:34.109175+0000","last_became_active":"2024-09-10T20:45:33.112455+0000","last_became_peered":"2024-09-10T20:45:33.112455+0000","last_unstale":"2024-09-10T20:45:34.109175+0000","last_undegraded":"2024-09-10T20:45:34.109175+0000","last_fullsized":"2024-09-10T20:45:34.109175+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:01:56.477740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"151'1","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.118590+0000","last_change":"2024-09-10T20:45:32.095563+0000","last_active":"2024-09-10T20:45:34.118590+0000","last_peered":"2024-09-10T20:45:34.118590+0000","last_clean":"2024-09-10T20:45:34.118590+0000","last_became_active":"2024-09-10T20:45:32.095292+0000","last_became_peered":"2024-09-10T20:45:32.095292+0000","last_unstale":"2024-09-10T20:45:34.118590+0000","last_undegraded":"2024-09-10T20:45:34.118590+0000","last_fullsized":"2024-09-10T20:45:34.118590+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:08.710035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108776+0000","last_change":"2024-09-10T20:45:33.113117+0000","last_active":"2024-09-10T20:45:34.108776+0000","last_peered":"2024-09-10T20:45:34.108776+0000","last_clean":"2024-09-10T20:45:34.108776+0000","last_became_active":"2024-09-10T20:45:33.112773+0000","last_became_peered":"2024-09-10T20:45:33.112773+0000","last_unstale":"2024-09-10T20:45:34.108776+0000","last_undegraded":"2024-09-10T20:45:34.108776+0000","last_fullsized":"2024-09-10T20:45:34.108776+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:24.667998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.109760+0000","last_change":"2024-09-10T20:45:32.097347+0000","last_active":"2024-09-10T20:45:34.109760+0000","last_peered":"2024-09-10T20:45:34.109760+0000","last_clean":"2024-09-10T20:45:34.109760+0000","last_became_active":"2024-09-10T20:45:32.095888+0000","last_became_peered":"2024-09-10T20:45:32.095888+0000","last_unstale":"2024-09-10T20:45:34.109760+0000","last_undegraded":"2024-09-10T20:45:34.109760+0000","last_fullsized":"2024-09-10T20:45:34.109760+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:27:31.905240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193438+0000","last_change":"2024-09-10T20:45:33.121934+0000","last_active":"2024-09-10T20:45:34.193438+0000","last_peered":"2024-09-10T20:45:34.193438+0000","last_clean":"2024-09-10T20:45:34.193438+0000","last_became_active":"2024-09-10T20:45:33.121588+0000","last_became_peered":"2024-09-10T20:45:33.121588+0000","last_unstale":"2024-09-10T20:45:34.193438+0000","last_undegraded":"2024-09-10T20:45:34.193438+0000","last_fullsized":"2024-09-10T20:45:34.193438+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:57:19.484832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193757+0000","last_change":"2024-09-10T20:45:32.102574+0000","last_active":"2024-09-10T20:45:34.193757+0000","last_peered":"2024-09-10T20:45:34.193757+0000","last_clean":"2024-09-10T20:45:34.193757+0000","last_became_active":"2024-09-10T20:45:32.102293+0000","last_became_peered":"2024-09-10T20:45:32.102293+0000","last_unstale":"2024-09-10T20:45:34.193757+0000","last_undegraded":"2024-09-10T20:45:34.193757+0000","last_fullsized":"2024-09-10T20:45:34.193757+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:47:51.409191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.125494+0000","last_change":"2024-09-10T20:45:33.120957+0000","last_active":"2024-09-10T20:45:33.125494+0000","last_peered":"2024-09-10T20:45:33.125494+0000","last_clean":"2024-09-10T20:45:33.125494+0000","last_became_active":"2024-09-10T20:45:33.120572+0000","last_became_peered":"2024-09-10T20:45:33.120572+0000","last_unstale":"2024-09-10T20:45:33.125494+0000","last_undegraded":"2024-09-10T20:45:33.125494+0000","last_fullsized":"2024-09-10T20:45:33.125494+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:54.583764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.103241+0000","last_change":"2024-09-10T20:45:32.102494+0000","last_active":"2024-09-10T20:45:33.103241+0000","last_peered":"2024-09-10T20:45:33.103241+0000","last_clean":"2024-09-10T20:45:33.103241+0000","last_became_active":"2024-09-10T20:45:32.102235+0000","last_became_peered":"2024-09-10T20:45:32.102235+0000","last_unstale":"2024-09-10T20:45:33.103241+0000","last_undegraded":"2024-09-10T20:45:33.103241+0000","last_fullsized":"2024-09-10T20:45:33.103241+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:52.741100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107242+0000","last_change":"2024-09-10T20:45:33.108626+0000","last_active":"2024-09-10T20:45:34.107242+0000","last_peered":"2024-09-10T20:45:34.107242+0000","last_clean":"2024-09-10T20:45:34.107242+0000","last_became_active":"2024-09-10T20:45:33.108328+0000","last_became_peered":"2024-09-10T20:45:33.108328+0000","last_unstale":"2024-09-10T20:45:34.107242+0000","last_undegraded":"2024-09-10T20:45:34.107242+0000","last_fullsized":"2024-09-10T20:45:34.107242+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:32:18.145006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.102974+0000","last_change":"2024-09-10T20:45:32.096057+0000","last_active":"2024-09-10T20:45:33.102974+0000","last_peered":"2024-09-10T20:45:33.102974+0000","last_clean":"2024-09-10T20:45:33.102974+0000","last_became_active":"2024-09-10T20:45:32.095774+0000","last_became_peered":"2024-09-10T20:45:32.095774+0000","last_unstale":"2024-09-10T20:45:33.102974+0000","last_undegraded":"2024-09-10T20:45:33.102974+0000","last_fullsized":"2024-09-10T20:45:33.102974+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:28.439601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.118456+0000","last_change":"2024-09-10T20:45:33.114644+0000","last_active":"2024-09-10T20:45:33.118456+0000","last_peered":"2024-09-10T20:45:33.118456+0000","last_clean":"2024-09-10T20:45:33.118456+0000","last_became_active":"2024-09-10T20:45:33.114266+0000","last_became_peered":"2024-09-10T20:45:33.114266+0000","last_unstale":"2024-09-10T20:45:33.118456+0000","last_undegraded":"2024-09-10T20:45:33.118456+0000","last_fullsized":"2024-09-10T20:45:33.118456+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:53:24.555776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.102963+0000","last_change":"2024-09-10T20:45:32.095328+0000","last_active":"2024-09-10T20:45:33.102963+0000","last_peered":"2024-09-10T20:45:33.102963+0000","last_clean":"2024-09-10T20:45:33.102963+0000","last_became_active":"2024-09-10T20:45:32.095058+0000","last_became_peered":"2024-09-10T20:45:32.095058+0000","last_unstale":"2024-09-10T20:45:33.102963+0000","last_undegraded":"2024-09-10T20:45:33.102963+0000","last_fullsized":"2024-09-10T20:45:33.102963+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:15:13.020977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.193589+0000","last_change":"2024-09-10T20:45:33.124485+0000","last_active":"2024-09-10T20:45:34.193589+0000","last_peered":"2024-09-10T20:45:34.193589+0000","last_clean":"2024-09-10T20:45:34.193589+0000","last_became_active":"2024-09-10T20:45:33.124210+0000","last_became_peered":"2024-09-10T20:45:33.124210+0000","last_unstale":"2024-09-10T20:45:34.193589+0000","last_undegraded":"2024-09-10T20:45:34.193589+0000","last_fullsized":"2024-09-10T20:45:34.193589+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:40.051038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.107916+0000","last_change":"2024-09-10T20:45:32.090837+0000","last_active":"2024-09-10T20:45:34.107916+0000","last_peered":"2024-09-10T20:45:34.107916+0000","last_clean":"2024-09-10T20:45:34.107916+0000","last_became_active":"2024-09-10T20:45:32.090366+0000","last_became_peered":"2024-09-10T20:45:32.090366+0000","last_unstale":"2024-09-10T20:45:34.107916+0000","last_undegraded":"2024-09-10T20:45:34.107916+0000","last_fullsized":"2024-09-10T20:45:34.107916+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:39:12.557544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-10T20:45:34.108054+0000","last_change":"2024-09-10T20:45:33.110200+0000","last_active":"2024-09-10T20:45:34.108054+0000","last_peered":"2024-09-10T20:45:34.108054+0000","last_clean":"2024-09-10T20:45:34.108054+0000","last_became_active":"2024-09-10T20:45:33.109900+0000","last_became_peered":"2024-09-10T20:45:33.109900+0000","last_unstale":"2024-09-10T20:45:34.108054+0000","last_undegraded":"2024-09-10T20:45:34.108054+0000","last_fullsized":"2024-09-10T20:45:34.108054+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:32.079462+0000","last_clean_scrub_stamp":"2024-09-10T20:45:32.079462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:54.427154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.110953+0000","last_change":"2024-09-10T20:45:32.106666+0000","last_active":"2024-09-10T20:45:33.110953+0000","last_peered":"2024-09-10T20:45:33.110953+0000","last_clean":"2024-09-10T20:45:33.110953+0000","last_became_active":"2024-09-10T20:45:32.106188+0000","last_became_peered":"2024-09-10T20:45:32.106188+0000","last_unstale":"2024-09-10T20:45:33.110953+0000","last_undegraded":"2024-09-10T20:45:33.110953+0000","last_fullsized":"2024-09-10T20:45:33.110953+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:45:31.074421+0000","last_clean_scrub_stamp":"2024-09-10T20:45:31.074421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:58.457158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":377,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-10T20:45:33.102784+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:45:33.102784+0000","last_peered":"2024-09-10T20:45:33.102784+0000","last_clean":"2024-09-10T20:45:33.102784+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:45:33.102784+0000","last_undegraded":"2024-09-10T20:45:33.102784+0000","last_fullsized":"2024-09-10T20:45:33.102784+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542238,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1916928,"data_stored":840971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1888256,"data_stored":833009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542237,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2498560,"data_stored":1430219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1888256,"data_stored":832953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509535,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1900544,"data_stored":834601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509534,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1888256,"data_stored":832989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509534,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37768,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334072,"statfs":{"total":96636764160,"available":96598089728,"internally_reserved":0,"allocated":2478080,"data_stored":1423287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509534,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":1892352,"data_stored":833535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:45:37.491 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:45:37.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:45:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 -- 172.21.15.28:0/366202919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc815fd20 msgr2=0x7f6dc8072610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/366202919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc815fd20 0x7f6dc8072610 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80b29b0 tx=0x7f6dc002a980 comp rx=0 tx=0).stop 2024-09-10T20:45:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 -- 172.21.15.28:0/366202919 shutdown_connections 2024-09-10T20:45:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/366202919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc815fd20 0x7f6dc8072610 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/366202919 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc80d05a0 0x7f6dc815f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/366202919 >> v2:172.21.15.159:3300/0 conn(0x7f6dc80706d0 0x7f6dc80d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 -- 172.21.15.28:0/366202919 >> 172.21.15.28:0/366202919 conn(0x7f6dc806cf00 msgr2=0x7f6dc806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 -- 172.21.15.28:0/366202919 shutdown_connections 2024-09-10T20:45:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.636+0000 7f6dcfb82640 1 -- 172.21.15.28:0/366202919 wait complete. 2024-09-10T20:45:37.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.637+0000 7f6dcfb82640 1 Processor -- start 2024-09-10T20:45:37.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.637+0000 7f6dcfb82640 1 -- start start 2024-09-10T20:45:37.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcfb82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80706d0 0x7f6dc809bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcfb82640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc80d05a0 0x7f6dc809c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcfb82640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc815fd20 0x7f6dc81f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcfb82640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8072b50 con 0x7f6dc80706d0 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcfb82640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc815d1a0 con 0x7f6dc80d05a0 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcfb82640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc8072cd0 con 0x7f6dc815fd20 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcd91e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80706d0 0x7f6dc809bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcd91e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80706d0 0x7f6dc809bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40780/0 (socket says 172.21.15.28:40780) 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.638+0000 7f6dcd91e640 1 -- 172.21.15.28:0/142001761 learned_addr learned my addr 172.21.15.28:0/142001761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcd91e640 1 -- 172.21.15.28:0/142001761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc815fd20 msgr2=0x7f6dc81f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcd11d640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc80d05a0 0x7f6dc809c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcd91e640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc815fd20 0x7f6dc81f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcd91e640 1 -- 172.21.15.28:0/142001761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc80d05a0 msgr2=0x7f6dc809c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcd91e640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc80d05a0 0x7f6dc809c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcd91e640 1 -- 172.21.15.28:0/142001761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc8152990 con 0x7f6dc80706d0 2024-09-10T20:45:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcd91e640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80706d0 0x7f6dc809bf10 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4005940 tx=0x7f6dc400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dbe7fc640 1 -- 172.21.15.28:0/142001761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4007bb0 con 0x7f6dc80706d0 2024-09-10T20:45:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dbe7fc640 1 -- 172.21.15.28:0/142001761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dc4007e40 con 0x7f6dc80706d0 2024-09-10T20:45:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc809ca80 con 0x7f6dc80706d0 2024-09-10T20:45:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dbe7fc640 1 -- 172.21.15.28:0/142001761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc400f6b0 con 0x7f6dc80706d0 2024-09-10T20:45:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.639+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc809d040 con 0x7f6dc80706d0 2024-09-10T20:45:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.640+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc8073290 con 0x7f6dc80706d0 2024-09-10T20:45:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.641+0000 7f6dbe7fc640 1 -- 172.21.15.28:0/142001761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f6dc4020030 con 0x7f6dc80706d0 2024-09-10T20:45:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.642+0000 7f6dbe7fc640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6dac0787e0 0x7f6dac07aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.642+0000 7f6dcd11d640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6dac0787e0 0x7f6dac07aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.642+0000 7f6dbe7fc640 1 -- 172.21.15.28:0/142001761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f6dc4098aa0 con 0x7f6dc80706d0 2024-09-10T20:45:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.642+0000 7f6dcd11d640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6dac0787e0 0x7f6dac07aca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6db8005fd0 tx=0x7f6db8005ce0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.646+0000 7f6dbe7fc640 1 -- 172.21.15.28:0/142001761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6dc4014030 con 0x7f6dc80706d0 2024-09-10T20:45:37.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.903+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f6dc8070180 con 0x7f6dc80706d0 2024-09-10T20:45:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.905+0000 7f6dbe7fc640 1 -- 172.21.15.28:0/142001761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v36) ==== 190+0+147 (secure 0 0 0) 0x7f6dc4063050 con 0x7f6dc80706d0 2024-09-10T20:45:37.909 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:45:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:45:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:45:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:45:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:45:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:45:37.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6dac0787e0 msgr2=0x7f6dac07aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6dac0787e0 0x7f6dac07aca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6db8005fd0 tx=0x7f6db8005ce0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80706d0 msgr2=0x7f6dc809bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80706d0 0x7f6dc809bf10 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4005940 tx=0x7f6dc400cd60 comp rx=0 tx=0).stop 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 shutdown_connections 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6dac0787e0 0x7f6dac07aca0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6dc815fd20 0x7f6dc81f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6dc80d05a0 0x7f6dc809c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 --2- 172.21.15.28:0/142001761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80706d0 0x7f6dc809bf10 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 >> 172.21.15.28:0/142001761 conn(0x7f6dc806cf00 msgr2=0x7f6dc815ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 shutdown_connections 2024-09-10T20:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:37.907+0000 7f6dcfb82640 1 -- 172.21.15.28:0/142001761 wait complete. 2024-09-10T20:45:37.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 -- 172.21.15.28:0/1180565361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 msgr2=0x7f4dd80b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/1180565361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80b6e00 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80d4d80 tx=0x7f4dd002aac0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 -- 172.21.15.28:0/1180565361 shutdown_connections 2024-09-10T20:45:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/1180565361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80b6e00 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/1180565361 >> v2:172.21.15.159:3300/0 conn(0x7f4dd80d2b90 0x7f4dd8161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 -- 172.21.15.28:0/1180565361 >> 172.21.15.28:0/1180565361 conn(0x7f4dd80d3fb0 msgr2=0x7f4dd80d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 -- 172.21.15.28:0/1180565361 shutdown_connections 2024-09-10T20:45:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.023+0000 7f4ddde1f640 1 -- 172.21.15.28:0/1180565361 wait complete. 2024-09-10T20:45:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.025+0000 7f4ddde1f640 1 Processor -- start 2024-09-10T20:45:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.025+0000 7f4ddde1f640 1 -- start start 2024-09-10T20:45:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.025+0000 7f4ddde1f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dd80d2b90 0x7f4dd809dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4ddde1f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dd815de70 0x7f4dd809f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4ddde1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4ddde1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd815e3d0 con 0x7f4dd8162090 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4ddde1f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd815e250 con 0x7f4dd80d2b90 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4ddde1f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd80b7340 con 0x7f4dd815de70 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40790/0 (socket says 172.21.15.28:40790) 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 -- 172.21.15.28:0/3134231065 learned_addr learned my addr 172.21.15.28:0/3134231065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd77fe640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dd80d2b90 0x7f4dd809dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd6ffd640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dd815de70 0x7f4dd809f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 -- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dd815de70 msgr2=0x7f4dd809f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dd815de70 0x7f4dd809f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 -- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dd80d2b90 msgr2=0x7f4dd809dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dd80d2b90 0x7f4dd809dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 -- 172.21.15.28:0/3134231065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd81f5820 con 0x7f4dd8162090 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dd7fff640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80a1330 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0009d60 tx=0x7f4dd0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dbffff640 1 -- 172.21.15.28:0/3134231065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0044070 con 0x7f4dd8162090 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dbffff640 1 -- 172.21.15.28:0/3134231065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4dd0004250 con 0x7f4dd8162090 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd81f4c10 con 0x7f4dd8162090 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.026+0000 7f4dbffff640 1 -- 172.21.15.28:0/3134231065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0004680 con 0x7f4dd8162090 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.027+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd81f51d0 con 0x7f4dd8162090 2024-09-10T20:45:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.027+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd80b7860 con 0x7f4dd8162090 2024-09-10T20:45:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.028+0000 7f4dbffff640 1 -- 172.21.15.28:0/3134231065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f4dd002c840 con 0x7f4dd8162090 2024-09-10T20:45:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.029+0000 7f4dbffff640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4dc4078810 0x7f4dc407acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.029+0000 7f4dbffff640 1 -- 172.21.15.28:0/3134231065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f4dd00bb530 con 0x7f4dd8162090 2024-09-10T20:45:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.029+0000 7f4dd77fe640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4dc4078810 0x7f4dc407acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.029+0000 7f4dd77fe640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4dc4078810 0x7f4dc407acd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80b8c40 tx=0x7f4dc8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.033+0000 7f4dbffff640 1 -- 172.21.15.28:0/3134231065 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4dd0032030 con 0x7f4dd8162090 2024-09-10T20:45:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.254+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dd80b9f30 con 0x7f4dd8162090 2024-09-10T20:45:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.254+0000 7f4dbffff640 1 -- 172.21.15.28:0/3134231065 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5236 (secure 0 0 0) 0x7f4dd00b9e20 con 0x7f4dd8162090 2024-09-10T20:45:38.259 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:38.259 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"btime":"2024-09-10T20:45:36:118423+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3007122398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3007122398},{"type":"v1","addr":"172.21.15.28:6834","nonce":3007122398}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":7031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3280811834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3280811834},{"type":"v1","addr":"172.21.15.159:6835","nonce":3280811834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":7630,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/632114757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":632114757},{"type":"v1","addr":"172.21.15.159:6834","nonce":632114757}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:45:32.457163+0000","modified":"2024-09-10T20:45:35.489078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8730},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8730":{"gid":8730,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1892964400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1892964400},{"type":"v1","addr":"172.21.15.28:6835","nonce":1892964400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8730,"qdb_cluster":[8730]},"id":12}]} 2024-09-10T20:45:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 106 2024-09-10T20:45:38.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.256+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4dc4078810 msgr2=0x7f4dc407acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.256+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4dc4078810 0x7f4dc407acd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80b8c40 tx=0x7f4dc8009040 comp rx=0 tx=0).stop 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.256+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 msgr2=0x7f4dd80a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.256+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80a1330 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0009d60 tx=0x7f4dd0003f10 comp rx=0 tx=0).stop 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.256+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 shutdown_connections 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.257+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4dc4078810 0x7f4dc407acd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.257+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd8162090 0x7f4dd80a1330 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.257+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4dd815de70 0x7f4dd809f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.257+0000 7f4ddde1f640 1 --2- 172.21.15.28:0/3134231065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4dd80d2b90 0x7f4dd809dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.257+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 >> 172.21.15.28:0/3134231065 conn(0x7f4dd80d3fb0 msgr2=0x7f4dd80b93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.257+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 shutdown_connections 2024-09-10T20:45:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.257+0000 7f4ddde1f640 1 -- 172.21.15.28:0/3134231065 wait complete. 2024-09-10T20:45:38.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:45:32.457163+0000', 'modified': '2024-09-10T20:45:35.489078+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8730}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8730': {'gid': 8730, 'name': 'a', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1892964400', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1892964400}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1892964400}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8730, 'qdb_cluster': [8730]} 2024-09-10T20:45:38.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:45:38.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-10T20:45:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 -- 172.21.15.28:0/3609300839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 msgr2=0x7fc5800b5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 --2- 172.21.15.28:0/3609300839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 0x7fc5800b5ac0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc578005fd0 tx=0x7fc57802a8b0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 -- 172.21.15.28:0/3609300839 shutdown_connections 2024-09-10T20:45:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 --2- 172.21.15.28:0/3609300839 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5800d05d0 0x7fc5800d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 --2- 172.21.15.28:0/3609300839 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 0x7fc5800b5ac0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 -- 172.21.15.28:0/3609300839 >> 172.21.15.28:0/3609300839 conn(0x7fc5800d1e40 msgr2=0x7fc5800d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 -- 172.21.15.28:0/3609300839 shutdown_connections 2024-09-10T20:45:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.785+0000 7fc587ea2640 1 -- 172.21.15.28:0/3609300839 wait complete. 2024-09-10T20:45:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.786+0000 7fc587ea2640 1 Processor -- start 2024-09-10T20:45:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.786+0000 7fc587ea2640 1 -- start start 2024-09-10T20:45:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc587ea2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 0x7fc58006f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc587ea2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5800d05d0 0x7fc5801f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc587ea2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58015bf30 0x7fc58009c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc587ea2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5800b6000 con 0x7fc5800d05d0 2024-09-10T20:45:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc587ea2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc5800d0d50 con 0x7fc5800b5610 2024-09-10T20:45:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc587ea2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc5800b6180 con 0x7fc58015bf30 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc58543d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5800d05d0 0x7fc5801f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc58543d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5800d05d0 0x7fc5801f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40796/0 (socket says 172.21.15.28:40796) 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc58543d640 1 -- 172.21.15.28:0/610175018 learned_addr learned my addr 172.21.15.28:0/610175018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc58643f640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58015bf30 0x7fc58009c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.787+0000 7fc585c3e640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 0x7fc58006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc58543d640 1 -- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58015bf30 msgr2=0x7fc58009c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc58543d640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58015bf30 0x7fc58009c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc58543d640 1 -- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 msgr2=0x7fc58006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc58543d640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 0x7fc58006f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc58543d640 1 -- 172.21.15.28:0/610175018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58009cf80 con 0x7fc5800d05d0 2024-09-10T20:45:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc58543d640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5800d05d0 0x7fc5801f55a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc57000ea60 tx=0x7fc57000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc5767fc640 1 -- 172.21.15.28:0/610175018 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57000cde0 con 0x7fc5800d05d0 2024-09-10T20:45:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc5767fc640 1 -- 172.21.15.28:0/610175018 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc570004020 con 0x7fc5800d05d0 2024-09-10T20:45:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc5767fc640 1 -- 172.21.15.28:0/610175018 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570004450 con 0x7fc5800d05d0 2024-09-10T20:45:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58009d260 con 0x7fc5800d05d0 2024-09-10T20:45:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.788+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58009e880 con 0x7fc5800d05d0 2024-09-10T20:45:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.790+0000 7fc5767fc640 1 -- 172.21.15.28:0/610175018 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fc5700035f0 con 0x7fc5800d05d0 2024-09-10T20:45:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.790+0000 7fc5767fc640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5640787e0 0x7fc56407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.790+0000 7fc5767fc640 1 -- 172.21.15.28:0/610175018 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7fc57009b940 con 0x7fc5800d05d0 2024-09-10T20:45:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.790+0000 7fc585c3e640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5640787e0 0x7fc56407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.791+0000 7fc585c3e640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5640787e0 0x7fc56407aca0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c001370 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.791+0000 7fc58ac70740 1 Processor -- start 2024-09-10T20:45:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.791+0000 7fc58ac70740 1 -- start start 2024-09-10T20:45:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563cce87b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563cced01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cce855f0 0x5563cce85a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58ac70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5563ccdfc820 con 0x5563ccecfdf0 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x5563ccdebc50 con 0x5563cceb01b0 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x5563ccdefa60 con 0x5563cce855f0 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563cced01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563cced01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40808/0 (socket says 172.21.15.28:40808) 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 -- 172.21.15.28:0/1206607485 learned_addr learned my addr 172.21.15.28:0/1206607485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc585c3e640 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563cce87b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58643f640 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cce855f0 0x5563cce85a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 -- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cce855f0 msgr2=0x5563cce85a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cce855f0 0x5563cce85a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 -- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 msgr2=0x5563cce87b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563cce87b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.792+0000 7fc58543d640 1 -- 172.21.15.28:0/1206607485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563ccb4d850 con 0x5563ccecfdf0 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.793+0000 7fc58543d640 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563cced01f0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc570062d40 tx=0x7fc57005f530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ec7f9defac1e75 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.793+0000 7fc5637fe640 1 -- 172.21.15.28:0/1206607485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570088020 con 0x5563ccecfdf0 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.793+0000 7fc5637fe640 1 -- 172.21.15.28:0/1206607485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc570062db0 con 0x5563ccecfdf0 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.793+0000 7fc5637fe640 1 -- 172.21.15.28:0/1206607485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570062230 con 0x5563ccecfdf0 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.793+0000 7fc58ac70740 1 -- 172.21.15.28:0/1206607485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 msgr2=0x5563cced01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.793+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563cced01f0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc570062d40 tx=0x7fc57005f530 comp rx=0 tx=0).stop 2024-09-10T20:45:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 -- 172.21.15.28:0/1206607485 shutdown_connections 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cce855f0 0x5563cce85a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563cced01f0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1206607485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563cce87b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 -- 172.21.15.28:0/1206607485 >> 172.21.15.28:0/1206607485 conn(0x5563cce866a0 msgr2=0x5563cce8a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 -- 172.21.15.28:0/1206607485 shutdown_connections 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 -- 172.21.15.28:0/1206607485 wait complete. 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 -- start start 2024-09-10T20:45:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.794+0000 7fc58ac70740 1 Processor -- start 2024-09-10T20:45:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563ccecfdf0 0x5563ccecac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb3d10 0x5563ccecb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58ac70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5563ccb5c620 con 0x5563cceb01b0 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x5563ccd915f0 con 0x5563cceb3d10 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x5563cce84b30 con 0x5563ccecfdf0 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58ac70740 10 client.0 _ll_register_callbacks cb 0x5563ccd950a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc585c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc585c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40812/0 (socket says 172.21.15.28:40812) 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc585c3e640 1 -- 172.21.15.28:0/1492339543 learned_addr learned my addr 172.21.15.28:0/1492339543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58543d640 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563ccecfdf0 0x5563ccecac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.795+0000 7fc58643f640 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb3d10 0x5563ccecb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc585c3e640 1 -- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563ccecfdf0 msgr2=0x5563ccecac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc585c3e640 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563ccecfdf0 0x5563ccecac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc585c3e640 1 -- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb3d10 msgr2=0x5563ccecb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc585c3e640 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb3d10 0x5563ccecb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc585c3e640 1 -- 172.21.15.28:0/1492339543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563ccbc46b0 con 0x5563cceb01b0 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc585c3e640 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7e70 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fc57c014730 tx=0x7fc57c004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654d10eb37f0d515 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc560ff9640 1 -- 172.21.15.28:0/1492339543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c025070 con 0x5563cceb01b0 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc560ff9640 1 -- 172.21.15.28:0/1492339543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7fc57c015700 con 0x5563cceb01b0 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc570062db0 con 0x5563cceb01b0 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc560ff9640 1 -- 172.21.15.28:0/1492339543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fc57c01b040 con 0x5563cceb01b0 2024-09-10T20:45:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 10 client.8844 fetch_fsmap learned FSMap version 106 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) -- 0x5563ccee2f00 con 0x5563cceb01b0 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc560ff9640 1 -- 172.21.15.28:0/1492339543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc570062db0 con 0x5563cceb01b0 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc560ff9640 1 -- 172.21.15.28:0/1492339543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 106) ==== 3502+0+0 (secure 0 0 0) 0x7fc57c033080 con 0x5563cceb01b0 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 10 client.8844 fetch_fsmap finished waiting for FSMap version 106 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 10 client.8844 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 20 client.8844 populate_metadata read hostname 'smithi028' 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccedf1c0 0x5563ccec9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.796+0000 7fc58ac70740 4 client.8844 mds_command: new command op to 8730 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.797+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 --> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563ccdfe770 con 0x5563ccedf1c0 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.797+0000 7fc58543d640 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccedf1c0 0x5563ccec9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.797+0000 7fc560ff9640 10 client.8844 ms_handle_connect on v2:172.21.15.28:6833/1892964400 2024-09-10T20:45:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.797+0000 7fc58543d640 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccedf1c0 0x5563ccec9430 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6672664627691390 server_cookie=5696863c0e34e0c2 in_seq=0 out_seq=0 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc560ff9640 1 -- 172.21.15.28:0/1492339543 <== mds.0 v2:172.21.15.28:6833/1892964400 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5563ccdfe770 con 0x5563ccedf1c0 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc560ff9640 10 client.8844 handle_command_reply: tid=0 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 client.8844 shutdown 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 msgr2=0x5563ccea7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7e70 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fc57c014730 tx=0x7fc57c004620 comp rx=0 tx=0).stop 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 shutdown_connections 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb3d10 0x5563ccecb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccedf1c0 0x5563ccec9430 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563ccecfdf0 0x5563ccecac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1492339543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7e70 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc560ff9640 0 client.8844 ms_handle_reset on v2:172.21.15.28:6833/1892964400 2024-09-10T20:45:38.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 >> 172.21.15.28:0/1492339543 conn(0x5563cce855f0 msgr2=0x5563cce8a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 shutdown_connections 2024-09-10T20:45:38.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 1 -- 172.21.15.28:0/1492339543 wait complete. 2024-09-10T20:45:38.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.798+0000 7fc58ac70740 20 client.8844 trim_cache size 0 max 16384 2024-09-10T20:45:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.816+0000 7fc58ac70740 1 Processor -- start 2024-09-10T20:45:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.816+0000 7fc58ac70740 1 -- start start 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 0x5563ccea7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563ccecfdf0 0x5563ccef28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc58ac70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801f8de0 con 0x5563cceb01b0 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801f8140 con 0x5563ccecfdf0 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc5801f8790 con 0x5563cceb3d10 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40826/0 (socket says 172.21.15.28:40826) 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 -- 172.21.15.28:0/4131042125 learned_addr learned my addr 172.21.15.28:0/4131042125 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 -- 172.21.15.28:0/4131042125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 msgr2=0x5563ccea7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 --2- 172.21.15.28:0/4131042125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 0x5563ccea7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 -- 172.21.15.28:0/4131042125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563ccecfdf0 msgr2=0x5563ccef28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 --2- 172.21.15.28:0/4131042125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563ccecfdf0 0x5563ccef28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.817+0000 7fc585c3e640 1 -- 172.21.15.28:0/4131042125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563cceb13a0 con 0x5563cceb01b0 2024-09-10T20:45:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc585c3e640 1 --2- 172.21.15.28:0/4131042125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7970 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fc57c015cd0 tx=0x7fc57c015d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5ec0fedc280c7a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc562ffd640 1 -- 172.21.15.28:0/4131042125 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c025070 con 0x5563cceb01b0 2024-09-10T20:45:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc562ffd640 1 -- 172.21.15.28:0/4131042125 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc57c01c040 con 0x5563cceb01b0 2024-09-10T20:45:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc562ffd640 1 -- 172.21.15.28:0/4131042125 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c006af0 con 0x5563cceb01b0 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 -- 172.21.15.28:0/4131042125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 msgr2=0x5563ccea7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 --2- 172.21.15.28:0/4131042125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7970 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fc57c015cd0 tx=0x7fc57c015d00 comp rx=0 tx=0).stop 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 -- 172.21.15.28:0/4131042125 shutdown_connections 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 --2- 172.21.15.28:0/4131042125 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563ccecfdf0 0x5563ccef28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 --2- 172.21.15.28:0/4131042125 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 0x5563ccea7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 --2- 172.21.15.28:0/4131042125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563cceb01b0 0x5563ccea7970 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 -- 172.21.15.28:0/4131042125 >> 172.21.15.28:0/4131042125 conn(0x5563cce855f0 msgr2=0x5563cce8a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 -- 172.21.15.28:0/4131042125 shutdown_connections 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.818+0000 7fc58ac70740 1 -- 172.21.15.28:0/4131042125 wait complete. 2024-09-10T20:45:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.819+0000 7fc58ac70740 1 -- start start 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.819+0000 7fc58ac70740 1 Processor -- start 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563ccf4f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 0x5563ccea7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58ac70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563ccea7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58ac70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5563ccedf040 con 0x5563ccecfdf0 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x5563cceb1890 con 0x5563cceb01b0 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58ac70740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x5563ccedf5d0 con 0x5563cceb3d10 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58ac70740 10 client.0 _ll_register_callbacks cb 0x7fc57c00fdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:45:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563ccea7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563ccea7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40828/0 (socket says 172.21.15.28:40828) 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 -- 172.21.15.28:0/1133441645 learned_addr learned my addr 172.21.15.28:0/1133441645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc585c3e640 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563ccf4f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58543d640 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 0x5563ccea7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 -- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 msgr2=0x5563ccea7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 0x5563ccea7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 -- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 msgr2=0x5563ccf4f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563ccf4f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.820+0000 7fc58643f640 1 -- 172.21.15.28:0/1133441645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563cceaf8d0 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58643f640 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563ccea7eb0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fc578009cd0 tx=0x7fc578005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=decc401f6bde1ca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc560ff9640 1 -- 172.21.15.28:0/1133441645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57804c060 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc560ff9640 1 -- 172.21.15.28:0/1133441645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7fc578004450 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc57c01c040 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc560ff9640 1 -- 172.21.15.28:0/1133441645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fc578052050 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 10 client.8850 fetch_fsmap learned FSMap version 106 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) -- 0x5563ccee8e30 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc560ff9640 1 -- 172.21.15.28:0/1133441645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc57c01c040 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc560ff9640 1 -- 172.21.15.28:0/1133441645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 106) ==== 3502+0+0 (secure 0 0 0) 0x7fc578055020 con 0x5563ccecfdf0 2024-09-10T20:45:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 10 client.8850 fetch_fsmap finished waiting for FSMap version 106 2024-09-10T20:45:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 10 client.8850 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:45:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 20 client.8850 populate_metadata read hostname 'smithi028' 2024-09-10T20:45:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccee48d0 0x5563ccf53290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 4 client.8850 mds_command: new command op to 8730 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:45:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 --> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] -- command(tid 0: {"prefix": "status"}) -- 0x5563ccecc9c0 con 0x5563ccee48d0 2024-09-10T20:45:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.821+0000 7fc585c3e640 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccee48d0 0x5563ccf53290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc560ff9640 10 client.8850 ms_handle_connect on v2:172.21.15.28:6833/1892964400 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc585c3e640 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccee48d0 0x5563ccf53290 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b936adb804b62a5c server_cookie=3f88975c42f0dab5 in_seq=0 out_seq=0 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc560ff9640 1 -- 172.21.15.28:0/1133441645 <== mds.0 v2:172.21.15.28:6833/1892964400 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x5563ccecc9c0 con 0x5563ccee48d0 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc560ff9640 10 client.8850 handle_command_reply: tid=0 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 client.8850 shutdown 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 msgr2=0x5563ccea7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563ccea7eb0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fc578009cd0 tx=0x7fc578005e50 comp rx=0 tx=0).stop 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 shutdown_connections 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563ccecfdf0 0x5563ccea7eb0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.28:6833/1892964400,v1:172.21.15.28:6835/1892964400] conn(0x5563ccee48d0 0x5563ccf53290 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5563cceb3d10 0x5563ccea7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 --2- 172.21.15.28:0/1133441645 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x5563cceb01b0 0x5563ccf4f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 >> 172.21.15.28:0/1133441645 conn(0x5563cce855f0 msgr2=0x5563ccecaf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc560ff9640 0 client.8850 ms_handle_reset on v2:172.21.15.28:6833/1892964400 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 shutdown_connections 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 1 -- 172.21.15.28:0/1133441645 wait complete. 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.822+0000 7fc58ac70740 20 client.8850 trim_cache size 0 max 16384 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:45:38.834 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8730, 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7170404960000001, 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 106, 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 151, 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 151, 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.416582845000001 2024-09-10T20:45:38.835 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:45:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5640787e0 msgr2=0x7fc56407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5640787e0 0x7fc56407aca0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c001370 comp rx=0 tx=0).stop 2024-09-10T20:45:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5800d05d0 msgr2=0x7fc5801f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5800d05d0 0x7fc5801f55a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc57000ea60 tx=0x7fc57000ef20 comp rx=0 tx=0).stop 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 shutdown_connections 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc5640787e0 0x7fc56407aca0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc58015bf30 0x7fc58009c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5800d05d0 0x7fc5801f55a0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 --2- 172.21.15.28:0/610175018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5800b5610 0x7fc58006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 >> 172.21.15.28:0/610175018 conn(0x7fc5800d1e40 msgr2=0x7fc580070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 shutdown_connections 2024-09-10T20:45:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.823+0000 7fc587ea2640 1 -- 172.21.15.28:0/610175018 wait complete. 2024-09-10T20:45:38.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:45:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.940+0000 7fe4c38a3640 1 -- 172.21.15.28:0/614127538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8004800 msgr2=0x7fe4a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.940+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/614127538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8004800 0x7fe4a8004c80 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac001690 tx=0x7fe4a8005830 comp rx=0 tx=0).stop 2024-09-10T20:45:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.940+0000 7fe4c38a3640 1 -- 172.21.15.28:0/614127538 shutdown_connections 2024-09-10T20:45:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.940+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/614127538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8004800 0x7fe4a8004c80 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.940+0000 7fe4c38a3640 1 -- 172.21.15.28:0/614127538 >> 172.21.15.28:0/614127538 conn(0x7fe4bc06cf00 msgr2=0x7fe4bc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.940+0000 7fe4c38a3640 1 -- 172.21.15.28:0/614127538 shutdown_connections 2024-09-10T20:45:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.940+0000 7fe4c38a3640 1 -- 172.21.15.28:0/614127538 wait complete. 2024-09-10T20:45:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.941+0000 7fe4c38a3640 1 Processor -- start 2024-09-10T20:45:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.941+0000 7fe4c38a3640 1 -- start start 2024-09-10T20:45:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c38a3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4a8004800 0x7fe4bc152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c38a3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4bc1f8710 0x7fe4bc14f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c38a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4bc14f9f0 0x7fe4bc09cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c38a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc0d08d0 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c38a3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc0d0750 con 0x7fe4a8004800 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c38a3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe4bc15e930 con 0x7fe4bc1f8710 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4bc14f9f0 0x7fe4bc09cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c163f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4a8004800 0x7fe4bc152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4bc14f9f0 0x7fe4bc09cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40852/0 (socket says 172.21.15.28:40852) 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 -- 172.21.15.28:0/1520680140 learned_addr learned my addr 172.21.15.28:0/1520680140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c0e3e640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4bc1f8710 0x7fe4bc14f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 -- 172.21.15.28:0/1520680140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4bc1f8710 msgr2=0x7fe4bc14f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4bc1f8710 0x7fe4bc14f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 -- 172.21.15.28:0/1520680140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4a8004800 msgr2=0x7fe4bc152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4a8004800 0x7fe4bc152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 -- 172.21.15.28:0/1520680140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4bc09dbd0 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.942+0000 7fe4c1e40640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4bc14f9f0 0x7fe4bc09cf30 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0b4150 tx=0x7fe4b4010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.943+0000 7fe4b1ffb640 1 -- 172.21.15.28:0/1520680140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4003970 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.943+0000 7fe4b1ffb640 1 -- 172.21.15.28:0/1520680140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4b4015070 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.943+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4bc1f7c50 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.943+0000 7fe4b1ffb640 1 -- 172.21.15.28:0/1520680140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4004030 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.943+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4bc1f8210 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.944+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4bc15bd50 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.945+0000 7fe4b1ffb640 1 -- 172.21.15.28:0/1520680140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fe4b40041d0 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.945+0000 7fe4b1ffb640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4a0078810 0x7fe4a007acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.945+0000 7fe4c163f640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4a0078810 0x7fe4a007acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.945+0000 7fe4b1ffb640 1 -- 172.21.15.28:0/1520680140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7fe4b4099080 con 0x7fe4bc14f9f0 2024-09-10T20:45:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.946+0000 7fe4c163f640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4a0078810 0x7fe4a007acd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8001220 tx=0x7fe4b80067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:38.950+0000 7fe4b1ffb640 1 -- 172.21.15.28:0/1520680140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe4b4062fe0 con 0x7fe4bc14f9f0 2024-09-10T20:45:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.177+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe4bc06d530 con 0x7fe4bc14f9f0 2024-09-10T20:45:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.177+0000 7fe4b1ffb640 1 -- 172.21.15.28:0/1520680140 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v106) ==== 53+0+73 (secure 0 0 0) 0x7fe4b4067a10 con 0x7fe4bc14f9f0 2024-09-10T20:45:39.181 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:45:39.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4a0078810 msgr2=0x7fe4a007acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:39.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4a0078810 0x7fe4a007acd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8001220 tx=0x7fe4b80067a0 comp rx=0 tx=0).stop 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4bc14f9f0 msgr2=0x7fe4bc09cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4bc14f9f0 0x7fe4bc09cf30 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0b4150 tx=0x7fe4b4010f40 comp rx=0 tx=0).stop 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 shutdown_connections 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe4a0078810 0x7fe4a007acd0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4bc14f9f0 0x7fe4bc09cf30 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4bc1f8710 0x7fe4bc14f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 --2- 172.21.15.28:0/1520680140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4a8004800 0x7fe4bc152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 >> 172.21.15.28:0/1520680140 conn(0x7fe4bc06cf00 msgr2=0x7fe4bc0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.179+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 shutdown_connections 2024-09-10T20:45:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39.180+0000 7fe4c38a3640 1 -- 172.21.15.28:0/1520680140 wait complete. 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:45:39.194 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:45:39.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:39.195 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42291sec preferred_lft 42291sec 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:45:39.213 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:45:39.213 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:39.213 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:45:39.213 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:39.213 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:45:39.213 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:39.213 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:39.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:39.214 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:45:39.276 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:45:39.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:39.277 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:45:39.335 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:45:39.335 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:45:39.335 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:45:39.335 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:45:39.335 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:45:39.336 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:45:39.336 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:45:39.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:45:39.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:45:39.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:45:39.336 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:45:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:45:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:39.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:39.465 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:45:39.481 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:45:39.481 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:45:39.481 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:45:39.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:39.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:45:39.547 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:39.548 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:39.603 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:45:39.604 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:39.604 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:39.659 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:45:39.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:39.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:39.744 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:39.744 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:45:39.744 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:45:39.753 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:39.753 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:45:39.753 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:45:39.753 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:39.753 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:45:39.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:45:39.760 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:45:39.760 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:45:39.760 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:45:39.760 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:45:39.760 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:45:39.761 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:45:39.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:39.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:39.788 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:45:39.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:45:39.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:client7103 2024-09-10T20:45:39.891 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:39.891 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:39.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 -- 172.21.15.28:0/4244068322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b4b40 msgr2=0x7fb7680ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 --2- 172.21.15.28:0/4244068322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b4b40 0x7fb7680ce030 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb760005fd0 tx=0x7fb76002a8b0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 -- 172.21.15.28:0/4244068322 shutdown_connections 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 --2- 172.21.15.28:0/4244068322 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b4b40 0x7fb7680ce030 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 --2- 172.21.15.28:0/4244068322 >> v2:172.21.15.159:3300/0 conn(0x7fb7680b2ba0 0x7fb7680b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 -- 172.21.15.28:0/4244068322 >> 172.21.15.28:0/4244068322 conn(0x7fb76806cf00 msgr2=0x7fb76806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 -- 172.21.15.28:0/4244068322 shutdown_connections 2024-09-10T20:45:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.048+0000 7fb77027c640 1 -- 172.21.15.28:0/4244068322 wait complete. 2024-09-10T20:45:40.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.049+0000 7fb77027c640 1 Processor -- start 2024-09-10T20:45:40.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.049+0000 7fb77027c640 1 -- start start 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb77027c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b2ba0 0x7fb768157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb77027c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7680b4b40 0x7fb768155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb77027c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76815df70 0x7fb768166b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb77027c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7680ce900 con 0x7fb76815df70 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb77027c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb7680ce780 con 0x7fb7680b4b40 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb77027c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb7680cea80 con 0x7fb7680b2ba0 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76815df70 0x7fb768166b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e018640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b2ba0 0x7fb768157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76815df70 0x7fb768166b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40868/0 (socket says 172.21.15.28:40868) 2024-09-10T20:45:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 -- 172.21.15.28:0/3378650572 learned_addr learned my addr 172.21.15.28:0/3378650572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76d817640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7680b4b40 0x7fb768155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 -- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b2ba0 msgr2=0x7fb768157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b2ba0 0x7fb768157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 -- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7680b4b40 msgr2=0x7fb768155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7680b4b40 0x7fb768155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e819640 1 -- 172.21.15.28:0/3378650572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7681577f0 con 0x7fb76815df70 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.050+0000 7fb76e018640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b2ba0 0x7fb768157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.051+0000 7fb76e819640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76815df70 0x7fb768166b10 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fb760009cd0 tx=0x7fb760031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.051+0000 7fb75effd640 1 -- 172.21.15.28:0/3378650572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760040070 con 0x7fb76815df70 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.051+0000 7fb75effd640 1 -- 172.21.15.28:0/3378650572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb760004040 con 0x7fb76815df70 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.051+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb768157ad0 con 0x7fb76815df70 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.051+0000 7fb75effd640 1 -- 172.21.15.28:0/3378650572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760004470 con 0x7fb76815df70 2024-09-10T20:45:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.051+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb768158090 con 0x7fb76815df70 2024-09-10T20:45:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.052+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7680b4600 con 0x7fb76815df70 2024-09-10T20:45:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.053+0000 7fb75effd640 1 -- 172.21.15.28:0/3378650572 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7fb760004610 con 0x7fb76815df70 2024-09-10T20:45:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.053+0000 7fb75effd640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb74c0787e0 0x7fb74c07aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.053+0000 7fb75effd640 1 -- 172.21.15.28:0/3378650572 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7fb7600b9d10 con 0x7fb76815df70 2024-09-10T20:45:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.053+0000 7fb76e018640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb74c0787e0 0x7fb74c07aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.054+0000 7fb76e018640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb74c0787e0 0x7fb74c07aca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb7680cdd20 tx=0x7fb764009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:40.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.057+0000 7fb75effd640 1 -- 172.21.15.28:0/3378650572 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb7600842c0 con 0x7fb76815df70 2024-09-10T20:45:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.283+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb7680cf0b0 con 0x7fb76815df70 2024-09-10T20:45:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.284+0000 7fb75effd640 1 -- 172.21.15.28:0/3378650572 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v106) ==== 53+0+73 (secure 0 0 0) 0x7fb760083bc0 con 0x7fb76815df70 2024-09-10T20:45:40.287 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb74c0787e0 msgr2=0x7fb74c07aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb74c0787e0 0x7fb74c07aca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb7680cdd20 tx=0x7fb764009040 comp rx=0 tx=0).stop 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76815df70 msgr2=0x7fb768166b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76815df70 0x7fb768166b10 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fb760009cd0 tx=0x7fb760031760 comp rx=0 tx=0).stop 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 shutdown_connections 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb74c0787e0 0x7fb74c07aca0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76815df70 0x7fb768166b10 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7680b4b40 0x7fb768155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 --2- 172.21.15.28:0/3378650572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb7680b2ba0 0x7fb768157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 >> 172.21.15.28:0/3378650572 conn(0x7fb76806cf00 msgr2=0x7fb76806e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 shutdown_connections 2024-09-10T20:45:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:40.286+0000 7fb77027c640 1 -- 172.21.15.28:0/3378650572 wait complete. 2024-09-10T20:45:40.300 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:45:40.301 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:45:40.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:40.301 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:45:40.322 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:45:40.322 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:45:40.322 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:45:40.322 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:45:40.323 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42309sec preferred_lft 42309sec 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:45:40.324 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:45:40.325 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:45:40.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:40.326 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:45:40.393 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:45:40.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:40.394 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:45:40.454 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:45:40.454 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:45:40.454 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:45:40.454 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:45:40.454 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:45:40.454 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:45:40.454 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:45:40.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:45:40.455 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:45:40.455 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:45:40.455 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:45:40.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:45:40.581 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:40.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:40.588 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:45:40.603 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:45:40.603 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:45:40.603 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:45:40.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:40.604 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:45:40.669 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:40.670 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:40.725 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:45:40.726 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:45:40.726 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:40.781 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:45:40.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:40.781 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:40.864 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:40.864 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:45:40.865 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:45:40.873 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:40.873 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:45:40.873 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:45:40.873 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:40.873 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:45:40.879 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:45:40.880 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:45:40.880 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:45:40.880 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:45:40.880 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:45:40.881 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:45:40.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:40.881 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:40.947 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:45:41.036 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:45:41.038 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:45:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:41.044 INFO:teuthology.orchestra.run.smithi159.stdout:client7112 2024-09-10T20:45:41.044 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:41.044 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:41.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:45:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.181+0000 7f40347c6640 1 -- 172.21.15.28:0/867671632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c0b5030 msgr2=0x7f402c15f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.181+0000 7f40347c6640 1 --2- 172.21.15.28:0/867671632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c0b5030 0x7f402c15f850 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4028009830 tx=0x7f402802a510 comp rx=0 tx=0).stop 2024-09-10T20:45:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.182+0000 7f40347c6640 1 -- 172.21.15.28:0/867671632 shutdown_connections 2024-09-10T20:45:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.182+0000 7f40347c6640 1 --2- 172.21.15.28:0/867671632 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402c06e390 0x7f402c15cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.182+0000 7f40347c6640 1 --2- 172.21.15.28:0/867671632 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c0b5030 0x7f402c15f850 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.182+0000 7f40347c6640 1 -- 172.21.15.28:0/867671632 >> 172.21.15.28:0/867671632 conn(0x7f402c0cfc80 msgr2=0x7f402c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.182+0000 7f40347c6640 1 -- 172.21.15.28:0/867671632 shutdown_connections 2024-09-10T20:45:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.182+0000 7f40347c6640 1 -- 172.21.15.28:0/867671632 wait complete. 2024-09-10T20:45:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.183+0000 7f40347c6640 1 Processor -- start 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.183+0000 7f40347c6640 1 -- start start 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f40347c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402c06e390 0x7f402c153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f40347c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c073d40 0x7f402c0c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f40347c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402c0b5030 0x7f402c154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f40347c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f402c0ca9e0 con 0x7f402c06e390 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f40347c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f402c0cab60 con 0x7f402c073d40 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f40347c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f402c0cace0 con 0x7f402c0b5030 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402c06e390 0x7f402c153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402c06e390 0x7f402c153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56228/0 (socket says 172.21.15.28:56228) 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 -- 172.21.15.28:0/3222937200 learned_addr learned my addr 172.21.15.28:0/3222937200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4031d61640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c073d40 0x7f402c0c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032d63640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402c0b5030 0x7f402c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 -- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402c0b5030 msgr2=0x7f402c154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402c0b5030 0x7f402c154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 -- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c073d40 msgr2=0x7f402c0c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c073d40 0x7f402c0c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 -- 172.21.15.28:0/3222937200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c1557d0 con 0x7f402c06e390 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.184+0000 7f4032562640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402c06e390 0x7f402c153020 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f4028000c00 tx=0x7f4028033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.185+0000 7f4022ffd640 1 -- 172.21.15.28:0/3222937200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028031d60 con 0x7f402c06e390 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.185+0000 7f4022ffd640 1 -- 172.21.15.28:0/3222937200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4028033e10 con 0x7f402c06e390 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.185+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402c155ab0 con 0x7f402c06e390 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.185+0000 7f4022ffd640 1 -- 172.21.15.28:0/3222937200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028033310 con 0x7f402c06e390 2024-09-10T20:45:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.185+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402c153df0 con 0x7f402c06e390 2024-09-10T20:45:41.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.186+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f402c074460 con 0x7f402c06e390 2024-09-10T20:45:41.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.186+0000 7f4022ffd640 1 -- 172.21.15.28:0/3222937200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f402804b030 con 0x7f402c06e390 2024-09-10T20:45:41.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.187+0000 7f4022ffd640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40100787e0 0x7f401007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.187+0000 7f4022ffd640 1 -- 172.21.15.28:0/3222937200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f40280b9450 con 0x7f402c06e390 2024-09-10T20:45:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.187+0000 7f4031d61640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40100787e0 0x7f401007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.188+0000 7f4031d61640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40100787e0 0x7f401007aca0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f402c1f6a40 tx=0x7f401c005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.191+0000 7f4022ffd640 1 -- 172.21.15.28:0/3222937200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f402803f030 con 0x7f402c06e390 2024-09-10T20:45:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:41.414+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f402c0cb830 con 0x7f402c06e390 2024-09-10T20:45:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.139+0000 7f4022ffd640 1 -- 172.21.15.28:0/3222937200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v107) ==== 136+0+0 (secure 0 0 0) 0x7f40280833b0 con 0x7f402c06e390 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40100787e0 msgr2=0x7f401007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40100787e0 0x7f401007aca0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f402c1f6a40 tx=0x7f401c005f90 comp rx=0 tx=0).stop 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402c06e390 msgr2=0x7f402c153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402c06e390 0x7f402c153020 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f4028000c00 tx=0x7f4028033c20 comp rx=0 tx=0).stop 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 shutdown_connections 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f40100787e0 0x7f401007aca0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f402c0b5030 0x7f402c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.142+0000 7f40347c6640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f402c073d40 0x7f402c0c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.143+0000 7f40347c6640 1 --2- 172.21.15.28:0/3222937200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f402c06e390 0x7f402c153020 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.143+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 >> 172.21.15.28:0/3222937200 conn(0x7f402c0cfc80 msgr2=0x7f402c0b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:42.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.143+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 shutdown_connections 2024-09-10T20:45:42.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.143+0000 7f40347c6640 1 -- 172.21.15.28:0/3222937200 wait complete. 2024-09-10T20:45:42.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:45:42.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 -- 172.21.15.28:0/425913811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc815bc40 msgr2=0x7f3cc815e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/425913811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc815bc40 0x7f3cc815e6e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f3cc80d1d10 tx=0x7f3cb002a560 comp rx=0 tx=0).stop 2024-09-10T20:45:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 -- 172.21.15.28:0/425913811 shutdown_connections 2024-09-10T20:45:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/425913811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc815bc40 0x7f3cc815e6e0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/425913811 >> v2:172.21.15.159:3300/0 conn(0x7f3cc815ec20 0x7f3cc8070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 -- 172.21.15.28:0/425913811 >> 172.21.15.28:0/425913811 conn(0x7f3cc806dbf0 msgr2=0x7f3cc806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 -- 172.21.15.28:0/425913811 shutdown_connections 2024-09-10T20:45:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.269+0000 7f3ccd32b640 1 -- 172.21.15.28:0/425913811 wait complete. 2024-09-10T20:45:42.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 Processor -- start 2024-09-10T20:45:42.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 -- start start 2024-09-10T20:45:42.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc80d0c80 0x7f3cc8143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cc815bc40 0x7f3cc8142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cc815ec20 0x7f3cc809d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc8070940 con 0x7f3cc80d0c80 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc80707c0 con 0x7f3cc815ec20 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3ccd32b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc8070ac0 con 0x7f3cc815bc40 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.271+0000 7f3cc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc80d0c80 0x7f3cc8143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc80d0c80 0x7f3cc8143220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56260/0 (socket says 172.21.15.28:56260) 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 -- 172.21.15.28:0/2506895466 learned_addr learned my addr 172.21.15.28:0/2506895466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc77fe640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cc815ec20 0x7f3cc809d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc67fc640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cc815bc40 0x7f3cc8142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 -- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cc815bc40 msgr2=0x7f3cc8142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cc815bc40 0x7f3cc8142b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 -- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cc815ec20 msgr2=0x7f3cc809d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cc815ec20 0x7f3cc809d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 -- 172.21.15.28:0/2506895466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc8145590 con 0x7f3cc80d0c80 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cc6ffd640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc80d0c80 0x7f3cc8143220 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c870 tx=0x7f3cc000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cbf7fe640 1 -- 172.21.15.28:0/2506895466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0003b70 con 0x7f3cc80d0c80 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cbf7fe640 1 -- 172.21.15.28:0/2506895466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cc0003d10 con 0x7f3cc80d0c80 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc8145870 con 0x7f3cc80d0c80 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3cbf7fe640 1 -- 172.21.15.28:0/2506895466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000aab0 con 0x7f3cc80d0c80 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.272+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc8145e30 con 0x7f3cc80d0c80 2024-09-10T20:45:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.273+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc815e6e0 con 0x7f3cc80d0c80 2024-09-10T20:45:42.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.274+0000 7f3cbf7fe640 1 -- 172.21.15.28:0/2506895466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f3cc000ac50 con 0x7f3cc80d0c80 2024-09-10T20:45:42.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.275+0000 7f3cbf7fe640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ca80787e0 0x7f3ca807aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.275+0000 7f3cbf7fe640 1 -- 172.21.15.28:0/2506895466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f3cc00982e0 con 0x7f3cc80d0c80 2024-09-10T20:45:42.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.275+0000 7f3cc67fc640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ca80787e0 0x7f3ca807aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.275+0000 7f3cc67fc640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ca80787e0 0x7f3ca807aca0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0033000 tx=0x7f3cb0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:42.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.279+0000 7f3cbf7fe640 1 -- 172.21.15.28:0/2506895466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3cc0062240 con 0x7f3cc80d0c80 2024-09-10T20:45:42.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.491+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f3cc815d980 con 0x7f3ca80787e0 2024-09-10T20:45:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.492+0000 7f3cbf7fe640 1 -- 172.21.15.28:0/2506895466 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f3cc815d980 con 0x7f3ca80787e0 2024-09-10T20:45:42.498 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:45:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:45:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:45:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:45:42.498 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:45:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ca80787e0 msgr2=0x7f3ca807aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ca80787e0 0x7f3ca807aca0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0033000 tx=0x7f3cb0004730 comp rx=0 tx=0).stop 2024-09-10T20:45:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc80d0c80 msgr2=0x7f3cc8143220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc80d0c80 0x7f3cc8143220 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c870 tx=0x7f3cc000cd30 comp rx=0 tx=0).stop 2024-09-10T20:45:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 shutdown_connections 2024-09-10T20:45:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3ca80787e0 0x7f3ca807aca0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3cc815ec20 0x7f3cc809d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3cc815bc40 0x7f3cc8142b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 --2- 172.21.15.28:0/2506895466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cc80d0c80 0x7f3cc8143220 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 >> 172.21.15.28:0/2506895466 conn(0x7f3cc806dbf0 msgr2=0x7f3cc815c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 shutdown_connections 2024-09-10T20:45:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.495+0000 7f3ccd32b640 1 -- 172.21.15.28:0/2506895466 wait complete. 2024-09-10T20:45:42.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:45:42.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.615+0000 7f55137b4640 1 -- 172.21.15.28:0/1520466232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0cee60 msgr2=0x7f550c0ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.615+0000 7f55137b4640 1 --2- 172.21.15.28:0/1520466232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0cee60 0x7f550c0ccb20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f54fc009860 tx=0x7f54fc02a5a0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.615+0000 7f55137b4640 1 -- 172.21.15.28:0/1520466232 shutdown_connections 2024-09-10T20:45:42.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.615+0000 7f55137b4640 1 --2- 172.21.15.28:0/1520466232 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0cee60 0x7f550c0ccb20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.615+0000 7f55137b4640 1 -- 172.21.15.28:0/1520466232 >> 172.21.15.28:0/1520466232 conn(0x7f550c0d1f30 msgr2=0x7f550c0d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.615+0000 7f55137b4640 1 -- 172.21.15.28:0/1520466232 shutdown_connections 2024-09-10T20:45:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.615+0000 7f55137b4640 1 -- 172.21.15.28:0/1520466232 wait complete. 2024-09-10T20:45:42.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.616+0000 7f55137b4640 1 Processor -- start 2024-09-10T20:45:42.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.616+0000 7f55137b4640 1 -- start start 2024-09-10T20:45:42.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f55137b4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0b3110 0x7f550c1531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f55137b4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f550c0cee60 0x7f550c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f55137b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f550c15f580 0x7f550c1f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f55137b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f550c0d1030 con 0x7f550c15f580 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f55137b4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f550c0d0eb0 con 0x7f550c0cee60 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f55137b4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f550c0cd770 con 0x7f550c0b3110 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f550c15f580 0x7f550c1f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f550c15f580 0x7f550c1f6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56274/0 (socket says 172.21.15.28:56274) 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 -- 172.21.15.28:0/552840875 learned_addr learned my addr 172.21.15.28:0/552840875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511550640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0b3110 0x7f550c1531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5510d4f640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f550c0cee60 0x7f550c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 -- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0b3110 msgr2=0x7f550c1531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0b3110 0x7f550c1531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 -- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f550c0cee60 msgr2=0x7f550c1f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f550c0cee60 0x7f550c1f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.617+0000 7f5511d51640 1 -- 172.21.15.28:0/552840875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550c155610 con 0x7f550c15f580 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.618+0000 7f5511d51640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f550c15f580 0x7f550c1f6770 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f550400cbc0 tx=0x7f550400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.618+0000 7f5501ffb640 1 -- 172.21.15.28:0/552840875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504016020 con 0x7f550c15f580 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.618+0000 7f5501ffb640 1 -- 172.21.15.28:0/552840875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f550400add0 con 0x7f550c15f580 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.618+0000 7f5501ffb640 1 -- 172.21.15.28:0/552840875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504012680 con 0x7f550c15f580 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.618+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550c1558f0 con 0x7f550c15f580 2024-09-10T20:45:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.618+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550c155eb0 con 0x7f550c15f580 2024-09-10T20:45:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.619+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f550c0cdc90 con 0x7f550c15f580 2024-09-10T20:45:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.619+0000 7f5501ffb640 1 -- 172.21.15.28:0/552840875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f5504012820 con 0x7f550c15f580 2024-09-10T20:45:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.620+0000 7f5501ffb640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f54f00787e0 0x7f54f007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.620+0000 7f5501ffb640 1 -- 172.21.15.28:0/552840875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f550409b6b0 con 0x7f550c15f580 2024-09-10T20:45:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.620+0000 7f5511550640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f54f00787e0 0x7f54f007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.620+0000 7f5511550640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f54f00787e0 0x7f54f007aca0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5508000f60 tx=0x7f5508009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.624+0000 7f5501ffb640 1 -- 172.21.15.28:0/552840875 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5504020030 con 0x7f550c15f580 2024-09-10T20:45:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.811+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f550c0cf240 con 0x7f550c15f580 2024-09-10T20:45:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.811+0000 7f5501ffb640 1 -- 172.21.15.28:0/552840875 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7f5504063230 con 0x7f550c15f580 2024-09-10T20:45:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f54f00787e0 msgr2=0x7f54f007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f54f00787e0 0x7f54f007aca0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5508000f60 tx=0x7f5508009040 comp rx=0 tx=0).stop 2024-09-10T20:45:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f550c15f580 msgr2=0x7f550c1f6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f550c15f580 0x7f550c1f6770 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f550400cbc0 tx=0x7f550400a430 comp rx=0 tx=0).stop 2024-09-10T20:45:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 shutdown_connections 2024-09-10T20:45:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f54f00787e0 0x7f54f007aca0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f550c15f580 0x7f550c1f6770 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f550c0cee60 0x7f550c1f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 --2- 172.21.15.28:0/552840875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f550c0b3110 0x7f550c1531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 >> 172.21.15.28:0/552840875 conn(0x7f550c0d1f30 msgr2=0x7f550c0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 shutdown_connections 2024-09-10T20:45:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:42.814+0000 7f55137b4640 1 -- 172.21.15.28:0/552840875 wait complete. 2024-09-10T20:45:42.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:42.830 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol89510/cb11e0a5-d16d-425e-9197-1d4c3cdd4bc4) 2024-09-10T20:45:42.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:42.862 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:45:42.919 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:45:42.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:42.919 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol89510/cb11e0a5-d16d-425e-9197-1d4c3cdd4bc4) 2024-09-10T20:45:42.989 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:42.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol89510/.meta 2024-09-10T20:45:43.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:43.059 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp24550/subvol9801/62ceb398-ea64-47e5-b68d-7ba0c0e9b67c) 2024-09-10T20:45:43.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:43.127 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:45:43.184 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:45:43.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:43.185 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/subvol_grp24550/subvol9801/62ceb398-ea64-47e5-b68d-7ba0c0e9b67c) 2024-09-10T20:45:43.255 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:43.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/subvol_grp24550/subvol9801/.meta 2024-09-10T20:45:43.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol89510 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 -- 172.21.15.28:0/89088990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0ceae0 msgr2=0x7f2e9c0b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/89088990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0ceae0 0x7f2e9c0b2280 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0b1770 tx=0x7f2e9402a980 comp rx=0 tx=0).stop 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 -- 172.21.15.28:0/89088990 shutdown_connections 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/89088990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0ceae0 0x7f2e9c0b2280 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/89088990 >> v2:172.21.15.159:3300/0 conn(0x7f2e9c0d0440 0x7f2e9c0ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 -- 172.21.15.28:0/89088990 >> 172.21.15.28:0/89088990 conn(0x7f2e9c0d1e60 msgr2=0x7f2e9c0d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 -- 172.21.15.28:0/89088990 shutdown_connections 2024-09-10T20:45:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.466+0000 7f2ea2dec640 1 -- 172.21.15.28:0/89088990 wait complete. 2024-09-10T20:45:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.468+0000 7f2ea2dec640 1 Processor -- start 2024-09-10T20:45:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.468+0000 7f2ea2dec640 1 -- start start 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e9c0ceae0 0x7f2e9c1531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0d0440 0x7f2e9c0c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e9c15bfe0 0x7f2e9c0c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c0b2b50 con 0x7f2e9c0d0440 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c0b29d0 con 0x7f2e9c15bfe0 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9c0b2cd0 con 0x7f2e9c0ceae0 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0d0440 0x7f2e9c0c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0d0440 0x7f2e9c0c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56294/0 (socket says 172.21.15.28:56294) 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 -- 172.21.15.28:0/1034592015 learned_addr learned my addr 172.21.15.28:0/1034592015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea1389640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e9c15bfe0 0x7f2e9c0c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea0b88640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e9c0ceae0 0x7f2e9c1531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 -- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e9c0ceae0 msgr2=0x7f2e9c1531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e9c0ceae0 0x7f2e9c1531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 -- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e9c15bfe0 msgr2=0x7f2e9c0c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e9c15bfe0 0x7f2e9c0c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 -- 172.21.15.28:0/1034592015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9c0c5a30 con 0x7f2e9c0d0440 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea1389640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e9c15bfe0 0x7f2e9c0c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e9bfff640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0d0440 0x7f2e9c0c75c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800d770 tx=0x7f2e8800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e997fa640 1 -- 172.21.15.28:0/1034592015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88017070 con 0x7f2e9c0d0440 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e997fa640 1 -- 172.21.15.28:0/1034592015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e88009d20 con 0x7f2e9c0d0440 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9c0c5d10 con 0x7f2e9c0d0440 2024-09-10T20:45:43.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2e997fa640 1 -- 172.21.15.28:0/1034592015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88005280 con 0x7f2e9c0d0440 2024-09-10T20:45:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.469+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e9c0c62d0 con 0x7f2e9c0d0440 2024-09-10T20:45:43.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.470+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e9c0b3260 con 0x7f2e9c0d0440 2024-09-10T20:45:43.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.471+0000 7f2e997fa640 1 -- 172.21.15.28:0/1034592015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f2e88005420 con 0x7f2e9c0d0440 2024-09-10T20:45:43.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.472+0000 7f2e997fa640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2e800787e0 0x7f2e8007aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.472+0000 7f2e997fa640 1 -- 172.21.15.28:0/1034592015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f2e88098780 con 0x7f2e9c0d0440 2024-09-10T20:45:43.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.472+0000 7f2ea0b88640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2e800787e0 0x7f2e8007aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.472+0000 7f2ea0b88640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2e800787e0 0x7f2e8007aca0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2e90006fd0 tx=0x7f2e90008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:43.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.476+0000 7f2e997fa640 1 -- 172.21.15.28:0/1034592015 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2e88062d30 con 0x7f2e9c0d0440 2024-09-10T20:45:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.687+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol89510", "target": ["mon-mgr", ""]}) -- 0x7f2e9c0713f0 con 0x7f2e800787e0 2024-09-10T20:45:43.701 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:43.692+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:43.702 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:43.692+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:43.702 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:43.692+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:43.702 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:43.692+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:43.702 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:43.692+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.744+0000 7f2e997fa640 1 -- 172.21.15.28:0/1034592015 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f2e9c0713f0 con 0x7f2e800787e0 2024-09-10T20:45:43.749 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/_nogroup/subvol89510/cb11e0a5-d16d-425e-9197-1d4c3cdd4bc4 2024-09-10T20:45:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2e800787e0 msgr2=0x7f2e8007aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2e800787e0 0x7f2e8007aca0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2e90006fd0 tx=0x7f2e90008040 comp rx=0 tx=0).stop 2024-09-10T20:45:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0d0440 msgr2=0x7f2e9c0c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0d0440 0x7f2e9c0c75c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800d770 tx=0x7f2e8800dc30 comp rx=0 tx=0).stop 2024-09-10T20:45:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 shutdown_connections 2024-09-10T20:45:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f2e800787e0 0x7f2e8007aca0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e9c15bfe0 0x7f2e9c0c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0d0440 0x7f2e9c0c75c0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 --2- 172.21.15.28:0/1034592015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e9c0ceae0 0x7f2e9c1531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 >> 172.21.15.28:0/1034592015 conn(0x7f2e9c0d1e60 msgr2=0x7f2e9c071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:43.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 shutdown_connections 2024-09-10T20:45:43.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.746+0000 7f2ea2dec640 1 -- 172.21.15.28:0/1034592015 wait complete. 2024-09-10T20:45:43.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol9801 subvol_grp24550 2024-09-10T20:45:43.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 -- 172.21.15.28:0/4260737767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e80b4c20 msgr2=0x7f07e80d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 --2- 172.21.15.28:0/4260737767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e80b4c20 0x7f07e80d1c60 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f07e0005fd0 tx=0x7f07e002a670 comp rx=0 tx=0).stop 2024-09-10T20:45:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 -- 172.21.15.28:0/4260737767 shutdown_connections 2024-09-10T20:45:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 --2- 172.21.15.28:0/4260737767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e80b4c20 0x7f07e80d1c60 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 --2- 172.21.15.28:0/4260737767 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07e8161850 0x7f07e80d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 -- 172.21.15.28:0/4260737767 >> 172.21.15.28:0/4260737767 conn(0x7f07e80d3f90 msgr2=0x7f07e80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 -- 172.21.15.28:0/4260737767 shutdown_connections 2024-09-10T20:45:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.866+0000 7f07eceb8640 1 -- 172.21.15.28:0/4260737767 wait complete. 2024-09-10T20:45:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.867+0000 7f07eceb8640 1 Processor -- start 2024-09-10T20:45:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.867+0000 7f07eceb8640 1 -- start start 2024-09-10T20:45:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07eceb8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07e80b4c20 0x7f07e815dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07eceb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e8161850 0x7f07e814b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07eceb8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07e8162000 0x7f07e80ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07eceb8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07e80d2320 con 0x7f07e8161850 2024-09-10T20:45:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07eceb8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f07e80d21a0 con 0x7f07e8162000 2024-09-10T20:45:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07eceb8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f07e80d24a0 con 0x7f07e80b4c20 2024-09-10T20:45:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07e5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e8161850 0x7f07e814b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07e5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e8161850 0x7f07e814b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56334/0 (socket says 172.21.15.28:56334) 2024-09-10T20:45:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07e5d9b640 1 -- 172.21.15.28:0/1956191962 learned_addr learned my addr 172.21.15.28:0/1956191962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07e659c640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07e80b4c20 0x7f07e815dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.868+0000 7f07e6d9d640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07e8162000 0x7f07e80ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e5d9b640 1 -- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07e80b4c20 msgr2=0x7f07e815dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e5d9b640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07e80b4c20 0x7f07e815dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e5d9b640 1 -- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07e8162000 msgr2=0x7f07e80ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e5d9b640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07e8162000 0x7f07e80ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e5d9b640 1 -- 172.21.15.28:0/1956191962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e80b3ed0 con 0x7f07e8161850 2024-09-10T20:45:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e659c640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07e80b4c20 0x7f07e815dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e6d9d640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07e8162000 0x7f07e80ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07e5d9b640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e8161850 0x7f07e814b270 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f07d000b4d0 tx=0x7f07d000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07deffd640 1 -- 172.21.15.28:0/1956191962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0014070 con 0x7f07e8161850 2024-09-10T20:45:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07deffd640 1 -- 172.21.15.28:0/1956191962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07d0010960 con 0x7f07e8161850 2024-09-10T20:45:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e80b0ac0 con 0x7f07e8161850 2024-09-10T20:45:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07deffd640 1 -- 172.21.15.28:0/1956191962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0010db0 con 0x7f07e8161850 2024-09-10T20:45:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.869+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e80b1080 con 0x7f07e8161850 2024-09-10T20:45:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.870+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e80ccb70 con 0x7f07e8161850 2024-09-10T20:45:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.871+0000 7f07deffd640 1 -- 172.21.15.28:0/1956191962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f07d000f3c0 con 0x7f07e8161850 2024-09-10T20:45:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.871+0000 7f07deffd640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f07c80787e0 0x7f07c807aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.871+0000 7f07deffd640 1 -- 172.21.15.28:0/1956191962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f07d00986a0 con 0x7f07e8161850 2024-09-10T20:45:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.871+0000 7f07e659c640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f07c80787e0 0x7f07c807aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.872+0000 7f07e659c640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f07c80787e0 0x7f07c807aca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f07e80b43d0 tx=0x7f07d8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:43.876+0000 7f07deffd640 1 -- 172.21.15.28:0/1956191962 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f07d0062c50 con 0x7f07e8161850 2024-09-10T20:45:44.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.089+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol9801", "group_name": "subvol_grp24550", "target": ["mon-mgr", ""]}) -- 0x7f07e815e0f0 con 0x7f07c80787e0 2024-09-10T20:45:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.125+0000 7f07deffd640 1 -- 172.21.15.28:0/1956191962 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+73 (secure 0 0 0) 0x7f07e815e0f0 con 0x7f07c80787e0 2024-09-10T20:45:44.130 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp24550/subvol9801/62ceb398-ea64-47e5-b68d-7ba0c0e9b67c 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f07c80787e0 msgr2=0x7f07c807aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f07c80787e0 0x7f07c807aca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f07e80b43d0 tx=0x7f07d8009040 comp rx=0 tx=0).stop 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e8161850 msgr2=0x7f07e814b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e8161850 0x7f07e814b270 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f07d000b4d0 tx=0x7f07d000b990 comp rx=0 tx=0).stop 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 shutdown_connections 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f07c80787e0 0x7f07c807aca0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f07e8162000 0x7f07e80ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e8161850 0x7f07e814b270 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 --2- 172.21.15.28:0/1956191962 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f07e80b4c20 0x7f07e815dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 >> 172.21.15.28:0/1956191962 conn(0x7f07e80d3f90 msgr2=0x7f07e80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 shutdown_connections 2024-09-10T20:45:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.128+0000 7f07eceb8640 1 -- 172.21.15.28:0/1956191962 wait complete. 2024-09-10T20:45:44.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:44.144 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol89510/.meta) 2024-09-10T20:45:44.170 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:45:44.170 INFO:teuthology.orchestra.run.smithi028.stdout:version = 2 2024-09-10T20:45:44.170 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:45:44.170 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/_nogroup/subvol89510/cb11e0a5-d16d-425e-9197-1d4c3cdd4bc4 2024-09-10T20:45:44.170 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:45:44.170 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:44.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:44.172 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/subvol_grp24550/subvol9801/.meta) 2024-09-10T20:45:44.238 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:45:44.238 INFO:teuthology.orchestra.run.smithi028.stdout:version = 2 2024-09-10T20:45:44.238 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:45:44.238 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/subvol_grp24550/subvol9801/62ceb398-ea64-47e5-b68d-7ba0c0e9b67c 2024-09-10T20:45:44.238 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:45:44.238 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:44.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol89510 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 -- 172.21.15.28:0/545526909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 msgr2=0x7f27780b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 --2- 172.21.15.28:0/545526909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780b1f80 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f277815d260 tx=0x7f276802a560 comp rx=0 tx=0).stop 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 -- 172.21.15.28:0/545526909 shutdown_connections 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 --2- 172.21.15.28:0/545526909 >> v2:172.21.15.159:3300/0 conn(0x7f27780ced20 0x7f27780cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 --2- 172.21.15.28:0/545526909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780b1f80 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 -- 172.21.15.28:0/545526909 >> 172.21.15.28:0/545526909 conn(0x7f277806d110 msgr2=0x7f27780b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 -- 172.21.15.28:0/545526909 shutdown_connections 2024-09-10T20:45:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.384+0000 7f277d156640 1 -- 172.21.15.28:0/545526909 wait complete. 2024-09-10T20:45:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.385+0000 7f277d156640 1 Processor -- start 2024-09-10T20:45:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.385+0000 7f277d156640 1 -- start start 2024-09-10T20:45:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277d156640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27780ced20 0x7f27780b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277d156640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277d156640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27780b0f50 0x7f27780af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277d156640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f277815f9a0 con 0x7f2778152e00 2024-09-10T20:45:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277d156640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f277815f820 con 0x7f27780ced20 2024-09-10T20:45:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277d156640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f277815fb20 con 0x7f27780b0f50 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780aeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56350/0 (socket says 172.21.15.28:56350) 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 -- 172.21.15.28:0/3829368169 learned_addr learned my addr 172.21.15.28:0/3829368169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f2776d9d640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27780ced20 0x7f27780b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277759e640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27780b0f50 0x7f27780af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 -- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27780b0f50 msgr2=0x7f27780af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27780b0f50 0x7f27780af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 -- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27780ced20 msgr2=0x7f27780b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27780ced20 0x7f27780b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277659c640 1 -- 172.21.15.28:0/3829368169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2778141170 con 0x7f2778152e00 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.386+0000 7f277759e640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27780b0f50 0x7f27780af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.387+0000 7f277659c640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780aeae0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c450 tx=0x7f276c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.387+0000 7f275f7fe640 1 -- 172.21.15.28:0/3829368169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c0052d0 con 0x7f2778152e00 2024-09-10T20:45:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.387+0000 7f275f7fe640 1 -- 172.21.15.28:0/3829368169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f276c005470 con 0x7f2778152e00 2024-09-10T20:45:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.387+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2778141450 con 0x7f2778152e00 2024-09-10T20:45:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.387+0000 7f275f7fe640 1 -- 172.21.15.28:0/3829368169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c012870 con 0x7f2778152e00 2024-09-10T20:45:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.387+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2778141a10 con 0x7f2778152e00 2024-09-10T20:45:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.388+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2778160040 con 0x7f2778152e00 2024-09-10T20:45:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.389+0000 7f275f7fe640 1 -- 172.21.15.28:0/3829368169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7f276c0035f0 con 0x7f2778152e00 2024-09-10T20:45:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.389+0000 7f275f7fe640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27640787e0 0x7f276407aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.389+0000 7f275f7fe640 1 -- 172.21.15.28:0/3829368169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f276c09aca0 con 0x7f2778152e00 2024-09-10T20:45:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.389+0000 7f2776d9d640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27640787e0 0x7f276407aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.390+0000 7f2776d9d640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27640787e0 0x7f276407aca0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2768033000 tx=0x7f2768004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.393+0000 7f275f7fe640 1 -- 172.21.15.28:0/3829368169 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f276c062760 con 0x7f2778152e00 2024-09-10T20:45:44.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.605+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol89510", "target": ["mon-mgr", ""]}) -- 0x7f27780b2180 con 0x7f27640787e0 2024-09-10T20:45:44.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.616+0000 7f275f7fe640 1 -- 172.21.15.28:0/3829368169 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f27780b2180 con 0x7f27640787e0 2024-09-10T20:45:44.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27640787e0 msgr2=0x7f276407aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27640787e0 0x7f276407aca0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2768033000 tx=0x7f2768004730 comp rx=0 tx=0).stop 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 msgr2=0x7f27780aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780aeae0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c450 tx=0x7f276c00c910 comp rx=0 tx=0).stop 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 shutdown_connections 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f27640787e0 0x7f276407aca0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f27780b0f50 0x7f27780af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778152e00 0x7f27780aeae0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 --2- 172.21.15.28:0/3829368169 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f27780ced20 0x7f27780b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 >> 172.21.15.28:0/3829368169 conn(0x7f277806d110 msgr2=0x7f277806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.618+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 shutdown_connections 2024-09-10T20:45:44.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.619+0000 7f277d156640 1 -- 172.21.15.28:0/3829368169 wait complete. 2024-09-10T20:45:44.626 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.621+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.626 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.621+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.626 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.621+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.626 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.621+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.626 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.621+0000 7f93d1403640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol9801 subvol_grp24550 2024-09-10T20:45:44.639 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.634+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.639 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.634+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.639 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.634+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.639 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.634+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.639 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:45:44.634+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:45:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.743+0000 7ff7d6885640 1 -- 172.21.15.28:0/1802524398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 msgr2=0x7ff7d00d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.743+0000 7ff7d6885640 1 --2- 172.21.15.28:0/1802524398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d00d4a80 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005fd0 tx=0x7ff7c002a670 comp rx=0 tx=0).stop 2024-09-10T20:45:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.743+0000 7ff7d6885640 1 -- 172.21.15.28:0/1802524398 shutdown_connections 2024-09-10T20:45:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.743+0000 7ff7d6885640 1 --2- 172.21.15.28:0/1802524398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d00d4a80 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.743+0000 7ff7d6885640 1 -- 172.21.15.28:0/1802524398 >> 172.21.15.28:0/1802524398 conn(0x7ff7d00d3f90 msgr2=0x7ff7d00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.743+0000 7ff7d6885640 1 -- 172.21.15.28:0/1802524398 shutdown_connections 2024-09-10T20:45:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.743+0000 7ff7d6885640 1 -- 172.21.15.28:0/1802524398 wait complete. 2024-09-10T20:45:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.745+0000 7ff7d6885640 1 Processor -- start 2024-09-10T20:45:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.745+0000 7ff7d6885640 1 -- start start 2024-09-10T20:45:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.745+0000 7ff7d6885640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d00b4c20 0x7ff7d0155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.745+0000 7ff7d6885640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d0161850 0x7ff7d01f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d6885640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d015f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d6885640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d00d2370 con 0x7ff7d0162000 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d6885640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d00d24f0 con 0x7ff7d00b4c20 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d6885640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d00d2670 con 0x7ff7d0161850 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d015f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d015f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56408/0 (socket says 172.21.15.28:56408) 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 -- 172.21.15.28:0/388948967 learned_addr learned my addr 172.21.15.28:0/388948967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7cffff640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d00b4c20 0x7ff7d0155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7cf7fe640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d0161850 0x7ff7d01f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 -- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d0161850 msgr2=0x7ff7d01f7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d0161850 0x7ff7d01f7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 -- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d00b4c20 msgr2=0x7ff7d0155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d00b4c20 0x7ff7d0155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 -- 172.21.15.28:0/388948967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d01f8560 con 0x7ff7d0162000 2024-09-10T20:45:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d4e22640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d015f410 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c780 tx=0x7ff7c800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8017070 con 0x7ff7d0162000 2024-09-10T20:45:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7c8005070 con 0x7ff7d0162000 2024-09-10T20:45:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d01f8840 con 0x7ff7d0162000 2024-09-10T20:45:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8005500 con 0x7ff7d0162000 2024-09-10T20:45:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.746+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d01f8e00 con 0x7ff7d0162000 2024-09-10T20:45:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.747+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d0160b60 con 0x7ff7d0162000 2024-09-10T20:45:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.748+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 101366+0+0 (secure 0 0 0) 0x7ff7c800a5e0 con 0x7ff7d0162000 2024-09-10T20:45:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.749+0000 7ff7ccff9640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff7bc078810 0x7ff7bc07acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.749+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7ff7c8097f40 con 0x7ff7d0162000 2024-09-10T20:45:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.749+0000 7ff7cffff640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff7bc078810 0x7ff7bc07acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.749+0000 7ff7cffff640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff7bc078810 0x7ff7bc07acd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0033000 tx=0x7ff7c0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.753+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff7c80624f0 con 0x7ff7d0162000 2024-09-10T20:45:44.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:44.963+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol9801", "group_name": "subvol_grp24550", "target": ["mon-mgr", ""]}) -- 0x7ff7d00b9f10 con 0x7ff7bc078810 2024-09-10T20:45:45.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:45.157+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 26) ==== 101417+0+0 (secure 0 0 0) 0x7ff7c805e060 con 0x7ff7d0162000 2024-09-10T20:45:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:47.167+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7ff7c8088010 con 0x7ff7d0162000 2024-09-10T20:45:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.419+0000 7ff7ccff9640 1 -- 172.21.15.28:0/388948967 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff7d00b9f10 con 0x7ff7bc078810 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.423+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff7bc078810 msgr2=0x7ff7bc07acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.423+0000 7ff7d6885640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff7bc078810 0x7ff7bc07acd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0033000 tx=0x7ff7c0004730 comp rx=0 tx=0).stop 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.423+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 msgr2=0x7ff7d015f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.423+0000 7ff7d6885640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d015f410 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c780 tx=0x7ff7c800cc40 comp rx=0 tx=0).stop 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 shutdown_connections 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff7bc078810 0x7ff7bc07acd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7d0162000 0x7ff7d015f410 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7d0161850 0x7ff7d01f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 --2- 172.21.15.28:0/388948967 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7d00b4c20 0x7ff7d0155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 >> 172.21.15.28:0/388948967 conn(0x7ff7d00d3f90 msgr2=0x7ff7d0160ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 shutdown_connections 2024-09-10T20:45:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.424+0000 7ff7d6885640 1 -- 172.21.15.28:0/388948967 wait complete. 2024-09-10T20:45:48.445 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-10T20:45:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:45:48.548 INFO:teuthology.orchestra.run.smithi028.stdout:2 2024-09-10T20:45:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:45:48.556 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:45:48.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:48.610 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:48.656 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:45:48.656 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:48.656 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:48.656 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:48.673 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:48.673 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:48.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:45:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.873+0000 7fb6bd390640 1 -- 172.21.15.28:0/1124993789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 msgr2=0x7fb6b80b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.873+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1124993789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b80b2e80 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0009be0 tx=0x7fb6b002a8d0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.873+0000 7fb6bd390640 1 -- 172.21.15.28:0/1124993789 shutdown_connections 2024-09-10T20:45:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.873+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1124993789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b80b2e80 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.873+0000 7fb6bd390640 1 -- 172.21.15.28:0/1124993789 >> 172.21.15.28:0/1124993789 conn(0x7fb6b806cf00 msgr2=0x7fb6b806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:48.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.873+0000 7fb6bd390640 1 -- 172.21.15.28:0/1124993789 shutdown_connections 2024-09-10T20:45:48.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.873+0000 7fb6bd390640 1 -- 172.21.15.28:0/1124993789 wait complete. 2024-09-10T20:45:48.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.874+0000 7fb6bd390640 1 Processor -- start 2024-09-10T20:45:48.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.874+0000 7fb6bd390640 1 -- start start 2024-09-10T20:45:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6bd390640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b80cc570 0x7fb6b80b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6bd390640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80cd3d0 0x7fb6b80b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6bd390640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b81efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6bd390640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b80b3540 con 0x7fb6b80d07a0 2024-09-10T20:45:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6bd390640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b80b33c0 con 0x7fb6b80cd3d0 2024-09-10T20:45:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6bd390640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb6b80b36c0 con 0x7fb6b80cc570 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b81efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b81efd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56434/0 (socket says 172.21.15.28:56434) 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6b77fe640 1 -- 172.21.15.28:0/1774426480 learned_addr learned my addr 172.21.15.28:0/1774426480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6b67fc640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80cd3d0 0x7fb6b80b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.875+0000 7fb6b6ffd640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b80cc570 0x7fb6b80b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b77fe640 1 -- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b80cc570 msgr2=0x7fb6b80b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b77fe640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b80cc570 0x7fb6b80b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b77fe640 1 -- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80cd3d0 msgr2=0x7fb6b80b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b77fe640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80cd3d0 0x7fb6b80b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b77fe640 1 -- 172.21.15.28:0/1774426480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b80afd80 con 0x7fb6b80d07a0 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b6ffd640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b80cc570 0x7fb6b80b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b67fc640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80cd3d0 0x7fb6b80b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6b77fe640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b81efd00 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0009d00 tx=0x7fb6b0033d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb69f7fe640 1 -- 172.21.15.28:0/1774426480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0031d80 con 0x7fb6b80d07a0 2024-09-10T20:45:48.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb69f7fe640 1 -- 172.21.15.28:0/1774426480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6b0003a60 con 0x7fb6b80d07a0 2024-09-10T20:45:48.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb69f7fe640 1 -- 172.21.15.28:0/1774426480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b00330d0 con 0x7fb6b80d07a0 2024-09-10T20:45:48.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b80b0060 con 0x7fb6b80d07a0 2024-09-10T20:45:48.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.876+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b80b0620 con 0x7fb6b80d07a0 2024-09-10T20:45:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.877+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6b80b3ae0 con 0x7fb6b80d07a0 2024-09-10T20:45:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.878+0000 7fb69f7fe640 1 -- 172.21.15.28:0/1774426480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fb6b004b030 con 0x7fb6b80d07a0 2024-09-10T20:45:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.878+0000 7fb69f7fe640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb6a4078990 0x7fb6a407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.878+0000 7fb6b6ffd640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb6a4078990 0x7fb6a407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.878+0000 7fb69f7fe640 1 -- 172.21.15.28:0/1774426480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7fb6b00b96b0 con 0x7fb6b80d07a0 2024-09-10T20:45:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.879+0000 7fb6b6ffd640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb6a4078990 0x7fb6a407ae50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8003eb0 tx=0x7fb6a8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:48.883+0000 7fb69f7fe640 1 -- 172.21.15.28:0/1774426480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb6b003f030 con 0x7fb6b80d07a0 2024-09-10T20:45:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.079+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb6b80b4cc0 con 0x7fb6b80d07a0 2024-09-10T20:45:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.079+0000 7fb69f7fe640 1 -- 172.21.15.28:0/1774426480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v151) ==== 80+0+0 (secure 0 0 0) 0x7fb6b0083a40 con 0x7fb6b80d07a0 2024-09-10T20:45:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:45:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb6a4078990 msgr2=0x7fb6a407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb6a4078990 0x7fb6a407ae50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8003eb0 tx=0x7fb6a8009040 comp rx=0 tx=0).stop 2024-09-10T20:45:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 msgr2=0x7fb6b81efd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b81efd00 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0009d00 tx=0x7fb6b0033d80 comp rx=0 tx=0).stop 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 shutdown_connections 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb6a4078990 0x7fb6a407ae50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b80d07a0 0x7fb6b81efd00 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80cd3d0 0x7fb6b80b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 --2- 172.21.15.28:0/1774426480 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b80cc570 0x7fb6b80b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 >> 172.21.15.28:0/1774426480 conn(0x7fb6b806cf00 msgr2=0x7fb6b80b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 shutdown_connections 2024-09-10T20:45:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.081+0000 7fb6bd390640 1 -- 172.21.15.28:0/1774426480 wait complete. 2024-09-10T20:45:49.097 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:45:49.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:49.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:45:49.151 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:45:49.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:49.151 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:45:49.178 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:49.179 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:49.179 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:49.195 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:49.195 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:49.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:45:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 -- 172.21.15.28:0/2156193874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc0cf780 msgr2=0x7f6ffc0cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 --2- 172.21.15.28:0/2156193874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc0cf780 0x7f6ffc0cfbf0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc08c020 tx=0x7f6ff002a560 comp rx=0 tx=0).stop 2024-09-10T20:45:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 -- 172.21.15.28:0/2156193874 shutdown_connections 2024-09-10T20:45:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 --2- 172.21.15.28:0/2156193874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc0cf780 0x7f6ffc0cfbf0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 --2- 172.21.15.28:0/2156193874 >> v2:172.21.15.159:3300/0 conn(0x7f6ffc15de70 0x7f6ffc0d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 -- 172.21.15.28:0/2156193874 >> 172.21.15.28:0/2156193874 conn(0x7f6ffc0d3fb0 msgr2=0x7f6ffc0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:49.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 -- 172.21.15.28:0/2156193874 shutdown_connections 2024-09-10T20:45:49.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.355+0000 7f7003184640 1 -- 172.21.15.28:0/2156193874 wait complete. 2024-09-10T20:45:49.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.356+0000 7f7003184640 1 Processor -- start 2024-09-10T20:45:49.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.356+0000 7f7003184640 1 -- start start 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7003184640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ffc0cf780 0x7f6ffc14fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7003184640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc15de70 0x7f6ffc0b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7003184640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ffc161b50 0x7f6ffc0b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7003184640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc0b7690 con 0x7f6ffc15de70 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7003184640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc0b7510 con 0x7f6ffc0cf780 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7003184640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6ffc0b7810 con 0x7f6ffc161b50 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f6ffbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc15de70 0x7f6ffc0b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f6ffbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc15de70 0x7f6ffc0b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56454/0 (socket says 172.21.15.28:56454) 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f6ffbfff640 1 -- 172.21.15.28:0/3365253631 learned_addr learned my addr 172.21.15.28:0/3365253631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7000f20640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ffc0cf780 0x7f6ffc14fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.357+0000 7f7001721640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ffc161b50 0x7f6ffc0b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ffbfff640 1 -- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ffc161b50 msgr2=0x7f6ffc0b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ffbfff640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ffc161b50 0x7f6ffc0b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ffbfff640 1 -- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ffc0cf780 msgr2=0x7f6ffc14fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ffbfff640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ffc0cf780 0x7f6ffc14fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ffbfff640 1 -- 172.21.15.28:0/3365253631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc0cdda0 con 0x7f6ffc15de70 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ffbfff640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc15de70 0x7f6ffc0b36b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00099c0 tx=0x7f6ff00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ff97fa640 1 -- 172.21.15.28:0/3365253631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff003f070 con 0x7f6ffc15de70 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ff97fa640 1 -- 172.21.15.28:0/3365253631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ff00048b0 con 0x7f6ffc15de70 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ffc0ccd00 con 0x7f6ffc15de70 2024-09-10T20:45:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f6ff97fa640 1 -- 172.21.15.28:0/3365253631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0035030 con 0x7f6ffc15de70 2024-09-10T20:45:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.358+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ffc0cd2c0 con 0x7f6ffc15de70 2024-09-10T20:45:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.359+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ffc0cfbf0 con 0x7f6ffc15de70 2024-09-10T20:45:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.360+0000 7f6ff97fa640 1 -- 172.21.15.28:0/3365253631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f6ff0004d00 con 0x7f6ffc15de70 2024-09-10T20:45:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.360+0000 7f6ff97fa640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fe80789c0 0x7f6fe807ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.360+0000 7f6ff97fa640 1 -- 172.21.15.28:0/3365253631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f6ff00b98a0 con 0x7f6ffc15de70 2024-09-10T20:45:49.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.360+0000 7f7000f20640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fe80789c0 0x7f6fe807ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.361+0000 7f7000f20640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fe80789c0 0x7f6fe807ae80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00bbe0 tx=0x7f6fec00b390 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:49.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.365+0000 7f6ff97fa640 1 -- 172.21.15.28:0/3365253631 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6ff00835e0 con 0x7f6ffc15de70 2024-09-10T20:45:49.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.561+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6ffc0b9f30 con 0x7f6ffc15de70 2024-09-10T20:45:49.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.561+0000 7f6ff97fa640 1 -- 172.21.15.28:0/3365253631 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v151) ==== 80+0+0 (secure 0 0 0) 0x7f6ff0088010 con 0x7f6ffc15de70 2024-09-10T20:45:49.566 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.563+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fe80789c0 msgr2=0x7f6fe807ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.563+0000 7f7003184640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fe80789c0 0x7f6fe807ae80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00bbe0 tx=0x7f6fec00b390 comp rx=0 tx=0).stop 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.563+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc15de70 msgr2=0x7f6ffc0b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.563+0000 7f7003184640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc15de70 0x7f6ffc0b36b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00099c0 tx=0x7f6ff00039e0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 shutdown_connections 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fe80789c0 0x7f6fe807ae80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6ffc161b50 0x7f6ffc0b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ffc15de70 0x7f6ffc0b36b0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 --2- 172.21.15.28:0/3365253631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ffc0cf780 0x7f6ffc14fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 >> 172.21.15.28:0/3365253631 conn(0x7f6ffc0d3fb0 msgr2=0x7f6ffc075890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 shutdown_connections 2024-09-10T20:45:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.564+0000 7f7003184640 1 -- 172.21.15.28:0/3365253631 wait complete. 2024-09-10T20:45:49.579 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:45:49.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:49.580 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:45:49.612 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:45:49.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:45:49.612 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:45:49.670 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:49.670 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:49.670 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:49.687 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:45:49.687 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:45:49.688 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:49.688 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:49.728 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:45:49.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 -- 172.21.15.28:0/1136281661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 msgr2=0x7f803015bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 --2- 172.21.15.28:0/1136281661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f803015bd50 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f803009e0b0 tx=0x7f802c02a560 comp rx=0 tx=0).stop 2024-09-10T20:45:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 -- 172.21.15.28:0/1136281661 shutdown_connections 2024-09-10T20:45:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 --2- 172.21.15.28:0/1136281661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f803015ee70 0x7f80300d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 --2- 172.21.15.28:0/1136281661 >> v2:172.21.15.159:3300/0 conn(0x7f80301554e0 0x7f803015e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 --2- 172.21.15.28:0/1136281661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f803015bd50 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 -- 172.21.15.28:0/1136281661 >> 172.21.15.28:0/1136281661 conn(0x7f803006cf00 msgr2=0x7f803006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 -- 172.21.15.28:0/1136281661 shutdown_connections 2024-09-10T20:45:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.828+0000 7f803836c640 1 -- 172.21.15.28:0/1136281661 wait complete. 2024-09-10T20:45:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.829+0000 7f803836c640 1 Processor -- start 2024-09-10T20:45:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.829+0000 7f803836c640 1 -- start start 2024-09-10T20:45:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f803836c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f8030145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f803836c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80301554e0 0x7f8030144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f803836c640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f803015ee70 0x7f8030153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f803836c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8030072bd0 con 0x7f8030155100 2024-09-10T20:45:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f803836c640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f80300d0ab0 con 0x7f803015ee70 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f803836c640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f8030072d50 con 0x7f80301554e0 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f8030145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f8030145270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56464/0 (socket says 172.21.15.28:56464) 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 -- 172.21.15.28:0/2641689847 learned_addr learned my addr 172.21.15.28:0/2641689847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036909640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f803015ee70 0x7f8030153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8035907640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80301554e0 0x7f8030144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 -- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80301554e0 msgr2=0x7f8030144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80301554e0 0x7f8030144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 -- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f803015ee70 msgr2=0x7f8030153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f803015ee70 0x7f8030153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036108640 1 -- 172.21.15.28:0/2641689847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8030145b50 con 0x7f8030155100 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8035907640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80301554e0 0x7f8030144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.830+0000 7f8036909640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f803015ee70 0x7f8030153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.831+0000 7f8036108640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f8030145270 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f802c034040 tx=0x7f802c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.831+0000 7f8026ffd640 1 -- 172.21.15.28:0/2641689847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c03e030 con 0x7f8030155100 2024-09-10T20:45:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.831+0000 7f8026ffd640 1 -- 172.21.15.28:0/2641689847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f802c003e00 con 0x7f8030155100 2024-09-10T20:45:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.831+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8030145e30 con 0x7f8030155100 2024-09-10T20:45:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.831+0000 7f8026ffd640 1 -- 172.21.15.28:0/2641689847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c035040 con 0x7f8030155100 2024-09-10T20:45:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.831+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80301463f0 con 0x7f8030155100 2024-09-10T20:45:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.832+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f803006d2e0 con 0x7f8030155100 2024-09-10T20:45:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.833+0000 7f8026ffd640 1 -- 172.21.15.28:0/2641689847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f802c0049c0 con 0x7f8030155100 2024-09-10T20:45:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.833+0000 7f8026ffd640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8014078990 0x7f801407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.833+0000 7f8035907640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8014078990 0x7f801407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.833+0000 7f8026ffd640 1 -- 172.21.15.28:0/2641689847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f802c0b9260 con 0x7f8030155100 2024-09-10T20:45:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.834+0000 7f8035907640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8014078990 0x7f801407ae50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f802000baa0 tx=0x7f8020005d20 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:49.837+0000 7f8026ffd640 1 -- 172.21.15.28:0/2641689847 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f803006d2e0 con 0x7f8030155100 2024-09-10T20:45:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.058+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80300d0190 con 0x7f8030155100 2024-09-10T20:45:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.058+0000 7f8026ffd640 1 -- 172.21.15.28:0/2641689847 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5236 (secure 0 0 0) 0x7f802c0c2080 con 0x7f8030155100 2024-09-10T20:45:50.063 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:50.063 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":107,"btime":"2024-09-10T20:45:42:135253+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3007122398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3007122398},{"type":"v1","addr":"172.21.15.28:6834","nonce":3007122398}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":7031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3280811834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3280811834},{"type":"v1","addr":"172.21.15.159:6835","nonce":3280811834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":7630,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/632114757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":632114757},{"type":"v1","addr":"172.21.15.159:6834","nonce":632114757}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:45:32.457163+0000","modified":"2024-09-10T20:45:35.489078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8730},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8730":{"gid":8730,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1892964400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1892964400},{"type":"v1","addr":"172.21.15.28:6835","nonce":1892964400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8730,"qdb_cluster":[8730]},"id":12}]} 2024-09-10T20:45:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 107 2024-09-10T20:45:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8014078990 msgr2=0x7f801407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8014078990 0x7f801407ae50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f802000baa0 tx=0x7f8020005d20 comp rx=0 tx=0).stop 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 msgr2=0x7f8030145270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f8030145270 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f802c034040 tx=0x7f802c003910 comp rx=0 tx=0).stop 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 shutdown_connections 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8014078990 0x7f801407ae50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f803015ee70 0x7f8030153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80301554e0 0x7f8030144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 --2- 172.21.15.28:0/2641689847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8030155100 0x7f8030145270 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 >> 172.21.15.28:0/2641689847 conn(0x7f803006cf00 msgr2=0x7f80300703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 shutdown_connections 2024-09-10T20:45:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.060+0000 7f803836c640 1 -- 172.21.15.28:0/2641689847 wait complete. 2024-09-10T20:45:50.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.178+0000 7f9964dd3640 1 -- 172.21.15.28:0/4136350162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 msgr2=0x7f994c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.178+0000 7f9964dd3640 1 --2- 172.21.15.28:0/4136350162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 0x7f994c004800 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f9948009b20 tx=0x7f994c005830 comp rx=0 tx=0).stop 2024-09-10T20:45:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.178+0000 7f995ed9d640 1 -- 172.21.15.28:0/4136350162 reap_dead start 2024-09-10T20:45:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.178+0000 7f9964dd3640 1 -- 172.21.15.28:0/4136350162 shutdown_connections 2024-09-10T20:45:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.178+0000 7f9964dd3640 1 -- 172.21.15.28:0/4136350162 >> 172.21.15.28:0/4136350162 conn(0x7f99600d0420 msgr2=0x7f99600d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.178+0000 7f9964dd3640 1 -- 172.21.15.28:0/4136350162 shutdown_connections 2024-09-10T20:45:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.178+0000 7f9964dd3640 1 -- 172.21.15.28:0/4136350162 wait complete. 2024-09-10T20:45:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.179+0000 7f9964dd3640 1 Processor -- start 2024-09-10T20:45:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.179+0000 7f9964dd3640 1 -- start start 2024-09-10T20:45:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f9964dd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 0x7f996014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f9964dd3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f996014e830 0x7f99600b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f9964dd3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99601f4bf0 0x7f99601e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f9964dd3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f996015f5b0 con 0x7f994c004070 2024-09-10T20:45:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f9964dd3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f996015f430 con 0x7f99601f4bf0 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f9964dd3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f996015e7e0 con 0x7f996014e830 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f995e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 0x7f996014e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f995e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 0x7f996014e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56490/0 (socket says 172.21.15.28:56490) 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f995e59c640 1 -- 172.21.15.28:0/1124552543 learned_addr learned my addr 172.21.15.28:0/1124552543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f995ed9d640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99601f4bf0 0x7f99601e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.180+0000 7f9957fff640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f996014e830 0x7f99600b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f995e59c640 1 -- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f996014e830 msgr2=0x7f99600b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f995e59c640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f996014e830 0x7f99600b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f995e59c640 1 -- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99601f4bf0 msgr2=0x7f99601e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f995e59c640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99601f4bf0 0x7f99601e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f995e59c640 1 -- 172.21.15.28:0/1124552543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99600b0290 con 0x7f994c004070 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f995e59c640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 0x7f996014e1c0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9950004d70 tx=0x7f9950001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f9956ffd640 1 -- 172.21.15.28:0/1124552543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950015020 con 0x7f994c004070 2024-09-10T20:45:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f9956ffd640 1 -- 172.21.15.28:0/1124552543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9950011960 con 0x7f994c004070 2024-09-10T20:45:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f9956ffd640 1 -- 172.21.15.28:0/1124552543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950001d40 con 0x7f994c004070 2024-09-10T20:45:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99600b0570 con 0x7f994c004070 2024-09-10T20:45:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.181+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99600b0b30 con 0x7f994c004070 2024-09-10T20:45:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.182+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99600738b0 con 0x7f994c004070 2024-09-10T20:45:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.183+0000 7f9956ffd640 1 -- 172.21.15.28:0/1124552543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f995001a400 con 0x7f994c004070 2024-09-10T20:45:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.183+0000 7f9956ffd640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99400789c0 0x7f994007ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.183+0000 7f9956ffd640 1 -- 172.21.15.28:0/1124552543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f9950099410 con 0x7f994c004070 2024-09-10T20:45:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.183+0000 7f9957fff640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99400789c0 0x7f994007ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.184+0000 7f9957fff640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99400789c0 0x7f994007ae80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9948035000 tx=0x7f99480046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.187+0000 7f9956ffd640 1 -- 172.21.15.28:0/1124552543 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f995001f020 con 0x7f994c004070 2024-09-10T20:45:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.408+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9960068d10 con 0x7f994c004070 2024-09-10T20:45:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.408+0000 7f9956ffd640 1 -- 172.21.15.28:0/1124552543 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5236 (secure 0 0 0) 0x7f9950063150 con 0x7f994c004070 2024-09-10T20:45:50.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:50.413 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":107,"btime":"2024-09-10T20:45:42:135253+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3007122398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3007122398},{"type":"v1","addr":"172.21.15.28:6834","nonce":3007122398}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":7031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3280811834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3280811834},{"type":"v1","addr":"172.21.15.159:6835","nonce":3280811834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":7630,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/632114757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":632114757},{"type":"v1","addr":"172.21.15.159:6834","nonce":632114757}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:45:32.457163+0000","modified":"2024-09-10T20:45:35.489078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8730},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8730":{"gid":8730,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1892964400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1892964400},{"type":"v1","addr":"172.21.15.28:6835","nonce":1892964400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8730,"qdb_cluster":[8730]},"id":12}]} 2024-09-10T20:45:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 107 2024-09-10T20:45:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99400789c0 msgr2=0x7f994007ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99400789c0 0x7f994007ae80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9948035000 tx=0x7f99480046c0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 msgr2=0x7f996014e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 0x7f996014e1c0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9950004d70 tx=0x7f9950001630 comp rx=0 tx=0).stop 2024-09-10T20:45:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 shutdown_connections 2024-09-10T20:45:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f99400789c0 0x7f994007ae80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99601f4bf0 0x7f99601e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f996014e830 0x7f99600b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 --2- 172.21.15.28:0/1124552543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f994c004070 0x7f996014e1c0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 >> 172.21.15.28:0/1124552543 conn(0x7f99600d0420 msgr2=0x7f99600cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 shutdown_connections 2024-09-10T20:45:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.411+0000 7f9964dd3640 1 -- 172.21.15.28:0/1124552543 wait complete. 2024-09-10T20:45:50.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 -- 172.21.15.28:0/52819957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c15c050 msgr2=0x7fef9c0cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 --2- 172.21.15.28:0/52819957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c15c050 0x7fef9c0cfa10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0cc420 tx=0x7fef8c02a530 comp rx=0 tx=0).stop 2024-09-10T20:45:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 -- 172.21.15.28:0/52819957 shutdown_connections 2024-09-10T20:45:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 --2- 172.21.15.28:0/52819957 >> v2:172.21.15.159:3300/0 conn(0x7fef9c0cef60 0x7fef9c0b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 --2- 172.21.15.28:0/52819957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c15c050 0x7fef9c0cfa10 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 -- 172.21.15.28:0/52819957 >> 172.21.15.28:0/52819957 conn(0x7fef9c0d1e40 msgr2=0x7fef9c0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 -- 172.21.15.28:0/52819957 shutdown_connections 2024-09-10T20:45:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.526+0000 7fefa24ef640 1 -- 172.21.15.28:0/52819957 wait complete. 2024-09-10T20:45:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.527+0000 7fefa24ef640 1 Processor -- start 2024-09-10T20:45:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.527+0000 7fefa24ef640 1 -- start start 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c0cef60 0x7fef9c0c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0d2960 0x7fef9c0c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c15c050 0x7fef9c1f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c0b1920 con 0x7fef9c0d2960 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c0cdce0 con 0x7fef9c0cef60 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fef9c0b1aa0 con 0x7fef9c15c050 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0d2960 0x7fef9c0c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0d2960 0x7fef9c0c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44834/0 (socket says 172.21.15.28:44834) 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 -- 172.21.15.28:0/1775678864 learned_addr learned my addr 172.21.15.28:0/1775678864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa0a8c640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c15c050 0x7fef9c1f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9bfff640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c0cef60 0x7fef9c0c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 -- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c15c050 msgr2=0x7fef9c1f7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c15c050 0x7fef9c1f7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 -- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c0cef60 msgr2=0x7fef9c0c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c0cef60 0x7fef9c0c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 -- 172.21.15.28:0/1775678864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9c0cc200 con 0x7fef9c0d2960 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9bfff640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c0cef60 0x7fef9c0c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef9b7fe640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0d2960 0x7fef9c0c5580 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fef9000c420 tx=0x7fef9000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef98ff9640 1 -- 172.21.15.28:0/1775678864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90006890 con 0x7fef9c0d2960 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef98ff9640 1 -- 172.21.15.28:0/1775678864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef90006b20 con 0x7fef9c0d2960 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9c0caa10 con 0x7fef9c0d2960 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fef98ff9640 1 -- 172.21.15.28:0/1775678864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90007020 con 0x7fef9c0d2960 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.529+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9c0cafd0 con 0x7fef9c0d2960 2024-09-10T20:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.530+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9c0b1f00 con 0x7fef9c0d2960 2024-09-10T20:45:50.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.531+0000 7fef98ff9640 1 -- 172.21.15.28:0/1775678864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fef900071c0 con 0x7fef9c0d2960 2024-09-10T20:45:50.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.531+0000 7fef98ff9640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef88078990 0x7fef8807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.531+0000 7fef9bfff640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef88078990 0x7fef8807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.531+0000 7fef98ff9640 1 -- 172.21.15.28:0/1775678864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7fef900981f0 con 0x7fef9c0d2960 2024-09-10T20:45:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.532+0000 7fef9bfff640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef88078990 0x7fef8807ae50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fef8c034000 tx=0x7fef8c004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.536+0000 7fef98ff9640 1 -- 172.21.15.28:0/1775678864 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fef90062580 con 0x7fef9c0d2960 2024-09-10T20:45:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.720+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef9c0c5130 con 0x7fef9c0d2960 2024-09-10T20:45:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.721+0000 7fef98ff9640 1 -- 172.21.15.28:0/1775678864 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16727 (secure 0 0 0) 0x7fef90061e80 con 0x7fef9c0d2960 2024-09-10T20:45:50.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:50.727 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:34.103863+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:45:30.112884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-10T20:45:31.419730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef88078990 msgr2=0x7fef8807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef88078990 0x7fef8807ae50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fef8c034000 tx=0x7fef8c004730 comp rx=0 tx=0).stop 2024-09-10T20:45:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0d2960 msgr2=0x7fef9c0c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0d2960 0x7fef9c0c5580 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fef9000c420 tx=0x7fef9000c8e0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 shutdown_connections 2024-09-10T20:45:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fef88078990 0x7fef8807ae50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c15c050 0x7fef9c1f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0d2960 0x7fef9c0c5580 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 --2- 172.21.15.28:0/1775678864 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c0cef60 0x7fef9c0c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 >> 172.21.15.28:0/1775678864 conn(0x7fef9c0d1e40 msgr2=0x7fef9c15b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 shutdown_connections 2024-09-10T20:45:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.725+0000 7fefa24ef640 1 -- 172.21.15.28:0/1775678864 wait complete. 2024-09-10T20:45:50.741 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:45:50.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.841+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1914828862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 msgr2=0x7ffa34161fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.841+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1914828862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa34161fc0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7ffa3408c1a0 tx=0x7ffa2c02a560 comp rx=0 tx=0).stop 2024-09-10T20:45:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.841+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1914828862 shutdown_connections 2024-09-10T20:45:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.841+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1914828862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa34161fc0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.841+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1914828862 >> v2:172.21.15.159:3300/0 conn(0x7ffa340cf780 0x7ffa340cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.841+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1914828862 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3415de70 0x7ffa340d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.841+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1914828862 >> 172.21.15.28:0/1914828862 conn(0x7ffa340d3fb0 msgr2=0x7ffa340d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.842+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1914828862 shutdown_connections 2024-09-10T20:45:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.842+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1914828862 wait complete. 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 Processor -- start 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 -- start start 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa340cf780 0x7ffa340a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3415de70 0x7ffa340b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa340b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa340b6e00 con 0x7ffa34161b50 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3415e460 con 0x7ffa340cf780 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.843+0000 7ffa3c0fe640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffa340b6f80 con 0x7ffa3415de70 2024-09-10T20:45:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa340b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa340b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44850/0 (socket says 172.21.15.28:44850) 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 -- 172.21.15.28:0/1473661015 learned_addr learned my addr 172.21.15.28:0/1473661015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa39e9a640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa340cf780 0x7ffa340a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa39699640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3415de70 0x7ffa340b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 -- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3415de70 msgr2=0x7ffa340b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3415de70 0x7ffa340b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 -- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa340cf780 msgr2=0x7ffa340a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa340cf780 0x7ffa340a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 -- 172.21.15.28:0/1473661015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa340a1a50 con 0x7ffa34161b50 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3a69b640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa340b22d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c034040 tx=0x7ffa2c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa327fc640 1 -- 172.21.15.28:0/1473661015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c03e030 con 0x7ffa34161b50 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa327fc640 1 -- 172.21.15.28:0/1473661015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa2c003c20 con 0x7ffa34161b50 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa340b0b80 con 0x7ffa34161b50 2024-09-10T20:45:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa327fc640 1 -- 172.21.15.28:0/1473661015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c035040 con 0x7ffa34161b50 2024-09-10T20:45:50.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.844+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa340b1100 con 0x7ffa34161b50 2024-09-10T20:45:50.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.845+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3414ec80 con 0x7ffa34161b50 2024-09-10T20:45:50.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.847+0000 7ffa327fc640 1 -- 172.21.15.28:0/1473661015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7ffa2c031400 con 0x7ffa34161b50 2024-09-10T20:45:50.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.847+0000 7ffa327fc640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa20078990 0x7ffa2007ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:50.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.847+0000 7ffa327fc640 1 -- 172.21.15.28:0/1473661015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7ffa2c0b9060 con 0x7ffa34161b50 2024-09-10T20:45:50.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.847+0000 7ffa39e9a640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa20078990 0x7ffa2007ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:50.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.847+0000 7ffa39e9a640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa20078990 0x7ffa2007ae50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffa24005fd0 tx=0x7ffa24009500 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:50.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:50.851+0000 7ffa327fc640 1 -- 172.21.15.28:0/1473661015 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffa3414ec80 con 0x7ffa34161b50 2024-09-10T20:45:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.074+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa340d2b90 con 0x7ffa34161b50 2024-09-10T20:45:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.074+0000 7ffa327fc640 1 -- 172.21.15.28:0/1473661015 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5236 (secure 0 0 0) 0x7ffa2c082d40 con 0x7ffa34161b50 2024-09-10T20:45:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:51.079 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":107,"btime":"2024-09-10T20:45:42:135253+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3007122398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3007122398},{"type":"v1","addr":"172.21.15.28:6834","nonce":3007122398}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":7031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3280811834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3280811834},{"type":"v1","addr":"172.21.15.159:6835","nonce":3280811834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":7630,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/632114757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":632114757},{"type":"v1","addr":"172.21.15.159:6834","nonce":632114757}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:45:32.457163+0000","modified":"2024-09-10T20:45:35.489078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8730},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8730":{"gid":8730,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1892964400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1892964400},{"type":"v1","addr":"172.21.15.28:6835","nonce":1892964400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8730,"qdb_cluster":[8730]},"id":12}]} 2024-09-10T20:45:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 107 2024-09-10T20:45:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa20078990 msgr2=0x7ffa2007ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa20078990 0x7ffa2007ae50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffa24005fd0 tx=0x7ffa24009500 comp rx=0 tx=0).stop 2024-09-10T20:45:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 msgr2=0x7ffa340b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa340b22d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c034040 tx=0x7ffa2c003820 comp rx=0 tx=0).stop 2024-09-10T20:45:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 shutdown_connections 2024-09-10T20:45:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa20078990 0x7ffa2007ae50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa34161b50 0x7ffa340b22d0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3415de70 0x7ffa340b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 --2- 172.21.15.28:0/1473661015 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa340cf780 0x7ffa340a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 >> 172.21.15.28:0/1473661015 conn(0x7ffa340d3fb0 msgr2=0x7ffa340cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.076+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 shutdown_connections 2024-09-10T20:45:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.077+0000 7ffa3c0fe640 1 -- 172.21.15.28:0/1473661015 wait complete. 2024-09-10T20:45:51.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 -- 172.21.15.28:0/293646595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00d2960 msgr2=0x7f0cd00ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/293646595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00d2960 0x7f0cd00ce850 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0073cb0 tx=0x7f0cc402a560 comp rx=0 tx=0).stop 2024-09-10T20:45:51.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 -- 172.21.15.28:0/293646595 shutdown_connections 2024-09-10T20:45:51.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/293646595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00d2960 0x7f0cd00ce850 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/293646595 >> v2:172.21.15.159:3300/0 conn(0x7f0cd015c050 0x7f0cd00cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 -- 172.21.15.28:0/293646595 >> 172.21.15.28:0/293646595 conn(0x7f0cd00d1e40 msgr2=0x7f0cd00d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:51.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 -- 172.21.15.28:0/293646595 shutdown_connections 2024-09-10T20:45:51.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.193+0000 7f0cd5b65640 1 -- 172.21.15.28:0/293646595 wait complete. 2024-09-10T20:45:51.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.194+0000 7f0cd5b65640 1 Processor -- start 2024-09-10T20:45:51.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.194+0000 7f0cd5b65640 1 -- start start 2024-09-10T20:45:51.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0cd5b65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00cd9b0 0x7f0cd01594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0cd5b65640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cd00d2960 0x7f0cd00ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0cd5b65640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cd015c050 0x7f0cd009df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0cd5b65640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd00cf120 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0cd5b65640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd00cefa0 con 0x7f0cd015c050 2024-09-10T20:45:51.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0cd5b65640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0cd00b1af0 con 0x7f0cd00d2960 2024-09-10T20:45:51.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00cd9b0 0x7f0cd01594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00cd9b0 0x7f0cd01594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44872/0 (socket says 172.21.15.28:44872) 2024-09-10T20:45:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 -- 172.21.15.28:0/1058317535 learned_addr learned my addr 172.21.15.28:0/1058317535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccffff640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cd015c050 0x7f0cd009df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0cceffd640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cd00d2960 0x7f0cd00ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 -- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cd00d2960 msgr2=0x7f0cd00ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cd00d2960 0x7f0cd00ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 -- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cd015c050 msgr2=0x7f0cd009df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cd015c050 0x7f0cd009df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.195+0000 7f0ccf7fe640 1 -- 172.21.15.28:0/1058317535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd0159a00 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.196+0000 7f0ccf7fe640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00cd9b0 0x7f0cd01594c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000c9a0 tx=0x7f0cc000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:51.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.196+0000 7f0cb7fff640 1 -- 172.21.15.28:0/1058317535 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0007bb0 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.196+0000 7f0cb7fff640 1 -- 172.21.15.28:0/1058317535 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cc0007e40 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.196+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd00cb270 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.196+0000 7f0cb7fff640 1 -- 172.21.15.28:0/1058317535 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000f690 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.196+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd00cb830 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.197+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cd00b2790 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.198+0000 7f0cb7fff640 1 -- 172.21.15.28:0/1058317535 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f0cc0020030 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.198+0000 7f0cb7fff640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cbc078990 0x7f0cbc07ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.198+0000 7f0cb7fff640 1 -- 172.21.15.28:0/1058317535 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f0cc0098cd0 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.198+0000 7f0cceffd640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cbc078990 0x7f0cbc07ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.199+0000 7f0cceffd640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cbc078990 0x7f0cbc07ae50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4033000 tx=0x7f0cc4004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:51.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.203+0000 7f0cb7fff640 1 -- 172.21.15.28:0/1058317535 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0cc0014030 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.422+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cd0072740 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.423+0000 7f0cb7fff640 1 -- 172.21.15.28:0/1058317535 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5236 (secure 0 0 0) 0x7f0cc0062980 con 0x7f0cd00cd9b0 2024-09-10T20:45:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:51.427 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":107,"btime":"2024-09-10T20:45:42:135253+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3007122398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3007122398},{"type":"v1","addr":"172.21.15.28:6834","nonce":3007122398}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":7031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3280811834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3280811834},{"type":"v1","addr":"172.21.15.159:6835","nonce":3280811834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":7630,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/632114757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":632114757},{"type":"v1","addr":"172.21.15.159:6834","nonce":632114757}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:45:32.457163+0000","modified":"2024-09-10T20:45:35.489078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8730},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8730":{"gid":8730,"name":"a","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1892964400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1892964400},{"type":"v1","addr":"172.21.15.28:6835","nonce":1892964400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8730,"qdb_cluster":[8730]},"id":12}]} 2024-09-10T20:45:51.427 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 107 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cbc078990 msgr2=0x7f0cbc07ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cbc078990 0x7f0cbc07ae50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4033000 tx=0x7f0cc4004730 comp rx=0 tx=0).stop 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00cd9b0 msgr2=0x7f0cd01594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00cd9b0 0x7f0cd01594c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000c9a0 tx=0x7f0cc000ce60 comp rx=0 tx=0).stop 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 shutdown_connections 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0cbc078990 0x7f0cbc07ae50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0cd015c050 0x7f0cd009df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0cd00d2960 0x7f0cd00ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 --2- 172.21.15.28:0/1058317535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cd00cd9b0 0x7f0cd01594c0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 >> 172.21.15.28:0/1058317535 conn(0x7f0cd00d1e40 msgr2=0x7f0cd00b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:51.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 shutdown_connections 2024-09-10T20:45:51.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.425+0000 7f0cd5b65640 1 -- 172.21.15.28:0/1058317535 wait complete. 2024-09-10T20:45:51.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.541+0000 7f76a88f6640 1 -- 172.21.15.28:0/407327776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 msgr2=0x7f76900062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.541+0000 7f76a88f6640 1 --2- 172.21.15.28:0/407327776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 0x7f76900062d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f769c009ed0 tx=0x7f7690006940 comp rx=0 tx=0).stop 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.541+0000 7f76a2d9d640 1 -- 172.21.15.28:0/407327776 reap_dead start 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.541+0000 7f76a88f6640 1 -- 172.21.15.28:0/407327776 shutdown_connections 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.541+0000 7f76a88f6640 1 -- 172.21.15.28:0/407327776 >> 172.21.15.28:0/407327776 conn(0x7f76a406cf00 msgr2=0x7f76a406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.541+0000 7f76a88f6640 1 -- 172.21.15.28:0/407327776 shutdown_connections 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.541+0000 7f76a88f6640 1 -- 172.21.15.28:0/407327776 wait complete. 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.543+0000 7f76a88f6640 1 Processor -- start 2024-09-10T20:45:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.543+0000 7f76a88f6640 1 -- start start 2024-09-10T20:45:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.543+0000 7f76a88f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 0x7f76a41593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.543+0000 7f76a88f6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76a41f21f0 0x7f76a40c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a88f6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76a41f2670 0x7f76a40c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a88f6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76a40d06f0 con 0x7f7690005e50 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a88f6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f76a40d0570 con 0x7f76a41f2670 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a88f6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f76a40ce030 con 0x7f76a41f21f0 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 0x7f76a41593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 0x7f76a41593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44878/0 (socket says 172.21.15.28:44878) 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 -- 172.21.15.28:0/1337921880 learned_addr learned my addr 172.21.15.28:0/1337921880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f769bfff640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76a41f21f0 0x7f76a40c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76a41f2670 0x7f76a40c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 -- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76a41f21f0 msgr2=0x7f76a40c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76a41f21f0 0x7f76a40c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 -- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76a41f2670 msgr2=0x7f76a40c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76a41f2670 0x7f76a40c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 -- 172.21.15.28:0/1337921880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a41f2c40 con 0x7f7690005e50 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f769bfff640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76a41f21f0 0x7f76a40c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a259c640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 0x7f76a41593b0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f7694011ae0 tx=0x7f7694009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f769affd640 1 -- 172.21.15.28:0/1337921880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769400fcb0 con 0x7f7690005e50 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f769affd640 1 -- 172.21.15.28:0/1337921880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f769400fe50 con 0x7f7690005e50 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f769affd640 1 -- 172.21.15.28:0/1337921880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694004280 con 0x7f7690005e50 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a41f2f20 con 0x7f7690005e50 2024-09-10T20:45:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.544+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a40c4b10 con 0x7f7690005e50 2024-09-10T20:45:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.545+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a41f3140 con 0x7f7690005e50 2024-09-10T20:45:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.546+0000 7f769affd640 1 -- 172.21.15.28:0/1337921880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f76940035f0 con 0x7f7690005e50 2024-09-10T20:45:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.546+0000 7f769affd640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7684078990 0x7f768407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.547+0000 7f769affd640 1 -- 172.21.15.28:0/1337921880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f76940993f0 con 0x7f7690005e50 2024-09-10T20:45:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.547+0000 7f769bfff640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7684078990 0x7f768407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.547+0000 7f769bfff640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7684078990 0x7f768407ae50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f768c001c00 tx=0x7f768c001b90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.551+0000 7f769affd640 1 -- 172.21.15.28:0/1337921880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7694063780 con 0x7f7690005e50 2024-09-10T20:45:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.737+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76a40b2ba0 con 0x7f7690005e50 2024-09-10T20:45:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.738+0000 7f769affd640 1 -- 172.21.15.28:0/1337921880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16727 (secure 0 0 0) 0x7f7694063080 con 0x7f7690005e50 2024-09-10T20:45:51.742 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:51.742 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:34.103863+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:45:30.112884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-10T20:45:31.419730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7684078990 msgr2=0x7f768407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7684078990 0x7f768407ae50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f768c001c00 tx=0x7f768c001b90 comp rx=0 tx=0).stop 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 msgr2=0x7f76a41593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 0x7f76a41593b0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f7694011ae0 tx=0x7f7694009f10 comp rx=0 tx=0).stop 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 shutdown_connections 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7684078990 0x7f768407ae50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f76a41f2670 0x7f76a40c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f76a41f21f0 0x7f76a40c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 --2- 172.21.15.28:0/1337921880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690005e50 0x7f76a41593b0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 >> 172.21.15.28:0/1337921880 conn(0x7f76a406cf00 msgr2=0x7f76a415ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:51.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 shutdown_connections 2024-09-10T20:45:51.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.740+0000 7f76a88f6640 1 -- 172.21.15.28:0/1337921880 wait complete. 2024-09-10T20:45:51.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:45:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.858+0000 7f79473a1640 1 -- 172.21.15.28:0/1334247644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 msgr2=0x7f792c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.858+0000 7f79473a1640 1 --2- 172.21.15.28:0/1334247644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f792c003e50 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f793c0099d0 tx=0x7f792c005830 comp rx=0 tx=0).stop 2024-09-10T20:45:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.858+0000 7f79473a1640 1 -- 172.21.15.28:0/1334247644 shutdown_connections 2024-09-10T20:45:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.858+0000 7f79473a1640 1 --2- 172.21.15.28:0/1334247644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f792c003e50 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.858+0000 7f79473a1640 1 -- 172.21.15.28:0/1334247644 >> 172.21.15.28:0/1334247644 conn(0x7f79400d3ed0 msgr2=0x7f79400d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.858+0000 7f79473a1640 1 -- 172.21.15.28:0/1334247644 shutdown_connections 2024-09-10T20:45:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.858+0000 7f79473a1640 1 -- 172.21.15.28:0/1334247644 wait complete. 2024-09-10T20:45:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.859+0000 7f79473a1640 1 Processor -- start 2024-09-10T20:45:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.859+0000 7f79473a1640 1 -- start start 2024-09-10T20:45:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f79473a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f794014c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f79473a1640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79401f78a0 0x7f79400b9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f79473a1640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79401f7d20 0x7f79400b97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f79473a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79400d4d80 con 0x7f792c003a50 2024-09-10T20:45:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f79473a1640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f79400d4c00 con 0x7f79401f78a0 2024-09-10T20:45:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f79473a1640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f79400d07c0 con 0x7f79401f7d20 2024-09-10T20:45:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f794639f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f794014c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f794639f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f794014c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44898/0 (socket says 172.21.15.28:44898) 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f794639f640 1 -- 172.21.15.28:0/3052376022 learned_addr learned my addr 172.21.15.28:0/3052376022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f7946ba0640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79401f7d20 0x7f79400b97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.860+0000 7f7945b9e640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79401f78a0 0x7f79400b9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f794639f640 1 -- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79401f7d20 msgr2=0x7f79400b97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f794639f640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79401f7d20 0x7f79400b97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f794639f640 1 -- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79401f78a0 msgr2=0x7f79400b9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f794639f640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79401f78a0 0x7f79400b9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f794639f640 1 -- 172.21.15.28:0/3052376022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79400b6e80 con 0x7f792c003a50 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f794639f640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f794014c8f0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f793c000c00 tx=0x7f793c032e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f7936ffd640 1 -- 172.21.15.28:0/3052376022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c037050 con 0x7f792c003a50 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f7936ffd640 1 -- 172.21.15.28:0/3052376022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f793c043070 con 0x7f792c003a50 2024-09-10T20:45:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79400b7160 con 0x7f792c003a50 2024-09-10T20:45:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f7936ffd640 1 -- 172.21.15.28:0/3052376022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c009510 con 0x7f792c003a50 2024-09-10T20:45:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.861+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79400b7720 con 0x7f792c003a50 2024-09-10T20:45:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.862+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79400d1aa0 con 0x7f792c003a50 2024-09-10T20:45:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.863+0000 7f7936ffd640 1 -- 172.21.15.28:0/3052376022 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f793c050070 con 0x7f792c003a50 2024-09-10T20:45:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.864+0000 7f7936ffd640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7924078990 0x7f792407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.864+0000 7f7936ffd640 1 -- 172.21.15.28:0/3052376022 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5737+0+0 (secure 0 0 0) 0x7f793c0b9d30 con 0x7f792c003a50 2024-09-10T20:45:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.864+0000 7f7945b9e640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7924078990 0x7f792407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.864+0000 7f7945b9e640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7924078990 0x7f792407ae50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f79400b1a00 tx=0x7f79300067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:51.868+0000 7f7936ffd640 1 -- 172.21.15.28:0/3052376022 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f793c0843e0 con 0x7f792c003a50 2024-09-10T20:45:52.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.091+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f794014ce30 con 0x7f792c003a50 2024-09-10T20:45:52.095 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:45:52.092+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:45:52.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.093+0000 7f7936ffd640 1 -- 172.21.15.28:0/3052376022 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v108) ==== 161+0+0 (secure 0 0 0) 0x7f793c083ce0 con 0x7f792c003a50 2024-09-10T20:45:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7924078990 msgr2=0x7f792407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7924078990 0x7f792407ae50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f79400b1a00 tx=0x7f79300067a0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 msgr2=0x7f794014c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f794014c8f0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f793c000c00 tx=0x7f793c032e10 comp rx=0 tx=0).stop 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 shutdown_connections 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7924078990 0x7f792407ae50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f79401f7d20 0x7f79400b97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f79401f78a0 0x7f79400b9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 --2- 172.21.15.28:0/3052376022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f792c003a50 0x7f794014c8f0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 >> 172.21.15.28:0/3052376022 conn(0x7f79400d3ed0 msgr2=0x7f79400d1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 shutdown_connections 2024-09-10T20:45:52.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.096+0000 7f79473a1640 1 -- 172.21.15.28:0/3052376022 wait complete. 2024-09-10T20:45:52.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:45:52.137 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.217+0000 7f834c52b640 1 -- 172.21.15.28:0/1635206155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2a60 msgr2=0x7f83441626e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.217+0000 7f834c52b640 1 --2- 172.21.15.28:0/1635206155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2a60 0x7f83441626e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8340005fd0 tx=0x7f834002aa90 comp rx=0 tx=0).stop 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.218+0000 7f834c52b640 1 -- 172.21.15.28:0/1635206155 shutdown_connections 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.218+0000 7f834c52b640 1 --2- 172.21.15.28:0/1635206155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440d2a60 0x7f83441626e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.218+0000 7f834c52b640 1 --2- 172.21.15.28:0/1635206155 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d2140 0x7f83440d2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.218+0000 7f834c52b640 1 -- 172.21.15.28:0/1635206155 >> 172.21.15.28:0/1635206155 conn(0x7f83440d40f0 msgr2=0x7f83440d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.218+0000 7f834c52b640 1 -- 172.21.15.28:0/1635206155 shutdown_connections 2024-09-10T20:45:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.218+0000 7f834c52b640 1 -- 172.21.15.28:0/1635206155 wait complete. 2024-09-10T20:45:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.219+0000 7f834c52b640 1 Processor -- start 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.219+0000 7f834c52b640 1 -- start start 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834c52b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440b56e0 0x7f83440b8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834c52b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83440d2140 0x7f83440c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834c52b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d2a60 0x7f83440c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834c52b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83440b2590 con 0x7f83440d2140 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834c52b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f83440b2410 con 0x7f83440b56e0 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834c52b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f83440b2710 con 0x7f83440d2a60 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83440d2140 0x7f83440c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83440d2140 0x7f83440c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44942/0 (socket says 172.21.15.28:44942) 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 -- 172.21.15.28:0/2346911435 learned_addr learned my addr 172.21.15.28:0/2346911435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834a2c7640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440b56e0 0x7f83440b8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f834aac8640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d2a60 0x7f83440c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 -- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d2a60 msgr2=0x7f83440c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d2a60 0x7f83440c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 -- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440b56e0 msgr2=0x7f83440b8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440b56e0 0x7f83440b8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 -- 172.21.15.28:0/2346911435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83441f5760 con 0x7f83440d2140 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.220+0000 7f8349ac6640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83440d2140 0x7f83440c7c20 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f833400c420 tx=0x7f833400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.221+0000 7f833affd640 1 -- 172.21.15.28:0/2346911435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334017070 con 0x7f83440d2140 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.221+0000 7f833affd640 1 -- 172.21.15.28:0/2346911435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83340053c0 con 0x7f83440d2140 2024-09-10T20:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.221+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83440b70c0 con 0x7f83440d2140 2024-09-10T20:45:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.221+0000 7f833affd640 1 -- 172.21.15.28:0/2346911435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334003850 con 0x7f83440d2140 2024-09-10T20:45:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.221+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83440b7680 con 0x7f83440d2140 2024-09-10T20:45:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.222+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83440b7880 con 0x7f83440d2140 2024-09-10T20:45:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.222+0000 7f833affd640 1 -- 172.21.15.28:0/2346911435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f83340039f0 con 0x7f83440d2140 2024-09-10T20:45:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.223+0000 7f833affd640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8328078990 0x7f832807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.223+0000 7f833affd640 1 -- 172.21.15.28:0/2346911435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5823+0+0 (secure 0 0 0) 0x7f8334097ed0 con 0x7f83440d2140 2024-09-10T20:45:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.223+0000 7f834a2c7640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8328078990 0x7f832807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.223+0000 7f834a2c7640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8328078990 0x7f832807ae50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f833c009bd0 tx=0x7f833c005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.227+0000 7f833affd640 1 -- 172.21.15.28:0/2346911435 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f83440b7880 con 0x7f83440d2140 2024-09-10T20:45:52.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:52.452+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f83440d1ab0 con 0x7f83440d2140 2024-09-10T20:45:53.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.097+0000 7f833affd640 1 -- 172.21.15.28:0/2346911435 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v109) ==== 104+0+0 (secure 0 0 0) 0x7f8334061b90 con 0x7f83440d2140 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8328078990 msgr2=0x7f832807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8328078990 0x7f832807ae50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f833c009bd0 tx=0x7f833c005f90 comp rx=0 tx=0).stop 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83440d2140 msgr2=0x7f83440c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83440d2140 0x7f83440c7c20 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f833400c420 tx=0x7f833400c8e0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 shutdown_connections 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8328078990 0x7f832807ae50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f83440d2a60 0x7f83440c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83440d2140 0x7f83440c7c20 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 --2- 172.21.15.28:0/2346911435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440b56e0 0x7f83440b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.100+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 >> 172.21.15.28:0/2346911435 conn(0x7f83440d40f0 msgr2=0x7f83440d03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.101+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 shutdown_connections 2024-09-10T20:45:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.101+0000 7f834c52b640 1 -- 172.21.15.28:0/2346911435 wait complete. 2024-09-10T20:45:53.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:45:53.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.273+0000 7fd999184640 1 -- 172.21.15.28:0/1301611276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 msgr2=0x7fd98000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:53.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.273+0000 7fd999184640 1 --2- 172.21.15.28:0/1301611276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd98000b9c0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009b50 tx=0x7fd980005830 comp rx=0 tx=0).stop 2024-09-10T20:45:53.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.273+0000 7fd999184640 1 -- 172.21.15.28:0/1301611276 shutdown_connections 2024-09-10T20:45:53.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.273+0000 7fd999184640 1 --2- 172.21.15.28:0/1301611276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd98000b9c0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.273+0000 7fd999184640 1 -- 172.21.15.28:0/1301611276 >> 172.21.15.28:0/1301611276 conn(0x7fd9940cfc80 msgr2=0x7fd9940d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:53.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.273+0000 7fd999184640 1 -- 172.21.15.28:0/1301611276 shutdown_connections 2024-09-10T20:45:53.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.273+0000 7fd999184640 1 -- 172.21.15.28:0/1301611276 wait complete. 2024-09-10T20:45:53.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.274+0000 7fd999184640 1 Processor -- start 2024-09-10T20:45:53.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd999184640 1 -- start start 2024-09-10T20:45:53.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd999184640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd99414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:53.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd999184640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9941e4670 0x7fd9940b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd999184640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9941e4b10 0x7fd9941e6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd999184640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd994072db0 con 0x7fd9800051c0 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd999184640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd994070150 con 0x7fd9941e4b10 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd999184640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd994072f30 con 0x7fd9941e4670 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd992d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd99414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd992d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd99414fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44964/0 (socket says 172.21.15.28:44964) 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.275+0000 7fd992d9d640 1 -- 172.21.15.28:0/516055769 learned_addr learned my addr 172.21.15.28:0/516055769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd99259c640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9941e4670 0x7fd9940b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd99359e640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9941e4b10 0x7fd9941e6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd992d9d640 1 -- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9941e4670 msgr2=0x7fd9940b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd992d9d640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9941e4670 0x7fd9940b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd992d9d640 1 -- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9941e4b10 msgr2=0x7fd9941e6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:53.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd992d9d640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9941e4b10 0x7fd9941e6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd992d9d640 1 -- 172.21.15.28:0/516055769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9941502a0 con 0x7fd9800051c0 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd99359e640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9941e4b10 0x7fd9941e6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd992d9d640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd99414fd60 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd98400e9c0 tx=0x7fd98400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd98b7fe640 1 -- 172.21.15.28:0/516055769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984015020 con 0x7fd9800051c0 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd98b7fe640 1 -- 172.21.15.28:0/516055769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd98400ce00 con 0x7fd9800051c0 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9940b50b0 con 0x7fd9800051c0 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd98b7fe640 1 -- 172.21.15.28:0/516055769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840116c0 con 0x7fd9800051c0 2024-09-10T20:45:53.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.276+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9940b5670 con 0x7fd9800051c0 2024-09-10T20:45:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.277+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9941f7b10 con 0x7fd9800051c0 2024-09-10T20:45:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.278+0000 7fd98b7fe640 1 -- 172.21.15.28:0/516055769 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fd984011860 con 0x7fd9800051c0 2024-09-10T20:45:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.278+0000 7fd98b7fe640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd974078990 0x7fd97407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.279+0000 7fd98b7fe640 1 -- 172.21.15.28:0/516055769 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5823+0+0 (secure 0 0 0) 0x7fd984099450 con 0x7fd9800051c0 2024-09-10T20:45:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.279+0000 7fd99259c640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd974078990 0x7fd97407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.279+0000 7fd99259c640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd974078990 0x7fd97407ae50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd9941f7a70 tx=0x7fd97c006b10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.283+0000 7fd98b7fe640 1 -- 172.21.15.28:0/516055769 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd98401f020 con 0x7fd9800051c0 2024-09-10T20:45:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:53.467+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd99415e7c0 con 0x7fd9800051c0 2024-09-10T20:45:54.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.192+0000 7fd98b7fe640 1 -- 172.21.15.28:0/516055769 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v153) ==== 181+0+0 (secure 0 0 0) 0x7fd984063780 con 0x7fd9800051c0 2024-09-10T20:45:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:45:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd974078990 msgr2=0x7fd97407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd974078990 0x7fd97407ae50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd9941f7a70 tx=0x7fd97c006b10 comp rx=0 tx=0).stop 2024-09-10T20:45:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 msgr2=0x7fd99414fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd99414fd60 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd98400e9c0 tx=0x7fd98400ee80 comp rx=0 tx=0).stop 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 shutdown_connections 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd974078990 0x7fd97407ae50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9941e4b10 0x7fd9941e6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9941e4670 0x7fd9940b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 --2- 172.21.15.28:0/516055769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9800051c0 0x7fd99414fd60 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 >> 172.21.15.28:0/516055769 conn(0x7fd9940cfc80 msgr2=0x7fd99406ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.197+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 shutdown_connections 2024-09-10T20:45:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.198+0000 7fd999184640 1 -- 172.21.15.28:0/516055769 wait complete. 2024-09-10T20:45:54.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.330+0000 7f21523f8640 1 -- 172.21.15.28:0/1955782257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2134005910 msgr2=0x7f2134005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.330+0000 7f21523f8640 1 --2- 172.21.15.28:0/1955782257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2134005910 0x7f2134005d90 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f2140001830 tx=0x7f2134006940 comp rx=0 tx=0).stop 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.330+0000 7f2150995640 1 -- 172.21.15.28:0/1955782257 reap_dead start 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.330+0000 7f21523f8640 1 -- 172.21.15.28:0/1955782257 shutdown_connections 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.330+0000 7f21523f8640 1 -- 172.21.15.28:0/1955782257 >> 172.21.15.28:0/1955782257 conn(0x7f214c0cfc80 msgr2=0x7f214c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.330+0000 7f21523f8640 1 -- 172.21.15.28:0/1955782257 shutdown_connections 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.331+0000 7f21523f8640 1 -- 172.21.15.28:0/1955782257 wait complete. 2024-09-10T20:45:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.332+0000 7f21523f8640 1 Processor -- start 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.332+0000 7f21523f8640 1 -- start start 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f21523f8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2134005910 0x7f214c1e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f21523f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f214c150f70 0x7f214c1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f21523f8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214c0cc870 0x7f214c0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f21523f8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f214c0cdf00 con 0x7f214c150f70 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f21523f8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f214c072fb0 con 0x7f214c0cc870 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f21523f8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f214c0ce080 con 0x7f2134005910 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f214c150f70 0x7f214c1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f214c150f70 0x7f214c1f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44982/0 (socket says 172.21.15.28:44982) 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 -- 172.21.15.28:0/2479435288 learned_addr learned my addr 172.21.15.28:0/2479435288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214bfff640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2134005910 0x7f214c1e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f2150995640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214c0cc870 0x7f214c0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 -- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2134005910 msgr2=0x7f214c1e6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2134005910 0x7f214c1e6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 -- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214c0cc870 msgr2=0x7f214c0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214c0cc870 0x7f214c0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 -- 172.21.15.28:0/2479435288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214c1f3fb0 con 0x7f214c150f70 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f214b7fe640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f214c150f70 0x7f214c1f3a70 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f2140001830 tx=0x7f21400060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.333+0000 7f2148ff9640 1 -- 172.21.15.28:0/2479435288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140043070 con 0x7f214c150f70 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.334+0000 7f2148ff9640 1 -- 172.21.15.28:0/2479435288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2140003af0 con 0x7f214c150f70 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.334+0000 7f2148ff9640 1 -- 172.21.15.28:0/2479435288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21400064c0 con 0x7f214c150f70 2024-09-10T20:45:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.334+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f214c1e74a0 con 0x7f214c150f70 2024-09-10T20:45:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.334+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f214c1e7a60 con 0x7f214c150f70 2024-09-10T20:45:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.335+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f214c072610 con 0x7f214c150f70 2024-09-10T20:45:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.335+0000 7f2148ff9640 1 -- 172.21.15.28:0/2479435288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f2140033480 con 0x7f214c150f70 2024-09-10T20:45:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.336+0000 7f2148ff9640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f21380789c0 0x7f213807ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.336+0000 7f2148ff9640 1 -- 172.21.15.28:0/2479435288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5385+0+0 (secure 0 0 0) 0x7f21400b8730 con 0x7f214c150f70 2024-09-10T20:45:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.336+0000 7f214bfff640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f21380789c0 0x7f213807ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.336+0000 7f214bfff640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f21380789c0 0x7f213807ae80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f214c06db30 tx=0x7f213c005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.340+0000 7f2148ff9640 1 -- 172.21.15.28:0/2479435288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f214c1f3fb0 con 0x7f214c150f70 2024-09-10T20:45:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:54.533+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f214c0ce7d0 con 0x7f214c150f70 2024-09-10T20:45:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.198+0000 7f2148ff9640 1 -- 172.21.15.28:0/2479435288 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v154) ==== 169+0+0 (secure 0 0 0) 0x7f214008b1e0 con 0x7f214c150f70 2024-09-10T20:45:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:45:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f21380789c0 msgr2=0x7f213807ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f21380789c0 0x7f213807ae80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f214c06db30 tx=0x7f213c005f90 comp rx=0 tx=0).stop 2024-09-10T20:45:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f214c150f70 msgr2=0x7f214c1f3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f214c150f70 0x7f214c1f3a70 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f2140001830 tx=0x7f21400060e0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 shutdown_connections 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f21380789c0 0x7f213807ae80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f214c0cc870 0x7f214c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f214c150f70 0x7f214c1f3a70 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 --2- 172.21.15.28:0/2479435288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2134005910 0x7f214c1e6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.202+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 >> 172.21.15.28:0/2479435288 conn(0x7f214c0cfc80 msgr2=0x7f214c1e44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.203+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 shutdown_connections 2024-09-10T20:45:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.203+0000 7f21523f8640 1 -- 172.21.15.28:0/2479435288 wait complete. 2024-09-10T20:45:55.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:45:55.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.330+0000 7f02350b8640 1 -- 172.21.15.28:0/1786733259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 msgr2=0x7f0218004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.330+0000 7f02350b8640 1 --2- 172.21.15.28:0/1786733259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 0x7f0218004800 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f02240016c0 tx=0x7f0218005830 comp rx=0 tx=0).stop 2024-09-10T20:45:55.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.330+0000 7f022f59e640 1 -- 172.21.15.28:0/1786733259 reap_dead start 2024-09-10T20:45:55.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.330+0000 7f02350b8640 1 -- 172.21.15.28:0/1786733259 shutdown_connections 2024-09-10T20:45:55.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.330+0000 7f02350b8640 1 -- 172.21.15.28:0/1786733259 >> 172.21.15.28:0/1786733259 conn(0x7f023006dbf0 msgr2=0x7f023006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:55.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.330+0000 7f02350b8640 1 -- 172.21.15.28:0/1786733259 shutdown_connections 2024-09-10T20:45:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.330+0000 7f02350b8640 1 -- 172.21.15.28:0/1786733259 wait complete. 2024-09-10T20:45:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.331+0000 7f02350b8640 1 Processor -- start 2024-09-10T20:45:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f02350b8640 1 -- start start 2024-09-10T20:45:55.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f02350b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 0x7f023014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f02350b8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02301f7f30 0x7f023014de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f02350b8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02300aee50 0x7f02300af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f02350b8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02300706b0 con 0x7f0218004000 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f02350b8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f023015f000 con 0x7f02301f7f30 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f02350b8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0230070830 con 0x7f02300aee50 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f022ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 0x7f023014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f022ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 0x7f023014eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45004/0 (socket says 172.21.15.28:45004) 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.332+0000 7f022ed9d640 1 -- 172.21.15.28:0/415040763 learned_addr learned my addr 172.21.15.28:0/415040763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022e59c640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02301f7f30 0x7f023014de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022f59e640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02300aee50 0x7f02300af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022ed9d640 1 -- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02300aee50 msgr2=0x7f02300af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022ed9d640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02300aee50 0x7f02300af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022ed9d640 1 -- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02301f7f30 msgr2=0x7f023014de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022ed9d640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02301f7f30 0x7f023014de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022ed9d640 1 -- 172.21.15.28:0/415040763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023014e360 con 0x7f0218004000 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f022ed9d640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 0x7f023014eaa0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f02200066b0 tx=0x7f02200039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f02177fe640 1 -- 172.21.15.28:0/415040763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220015070 con 0x7f0218004000 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f02177fe640 1 -- 172.21.15.28:0/415040763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0220011940 con 0x7f0218004000 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02301f8430 con 0x7f0218004000 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f02177fe640 1 -- 172.21.15.28:0/415040763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220011d90 con 0x7f0218004000 2024-09-10T20:45:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.333+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02301f89f0 con 0x7f0218004000 2024-09-10T20:45:55.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.334+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0230070d90 con 0x7f0218004000 2024-09-10T20:45:55.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.335+0000 7f02177fe640 1 -- 172.21.15.28:0/415040763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f02200103c0 con 0x7f0218004000 2024-09-10T20:45:55.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.335+0000 7f02177fe640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f021c078990 0x7f021c07ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.336+0000 7f02177fe640 1 -- 172.21.15.28:0/415040763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7f0220099840 con 0x7f0218004000 2024-09-10T20:45:55.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.336+0000 7f022e59c640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f021c078990 0x7f021c07ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.336+0000 7f022e59c640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f021c078990 0x7f021c07ae50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02240016c0 tx=0x7f0224037040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:55.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.340+0000 7f02177fe640 1 -- 172.21.15.28:0/415040763 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f02200634a0 con 0x7f0218004000 2024-09-10T20:45:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.534+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f02300d0a60 con 0x7f0218004000 2024-09-10T20:45:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.537+0000 7f02177fe640 1 -- 172.21.15.28:0/415040763 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v47) ==== 71+0+0 (secure 0 0 0) 0x7f0220067ed0 con 0x7f0218004000 2024-09-10T20:45:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.539+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f021c078990 msgr2=0x7f021c07ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.539+0000 7f02350b8640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f021c078990 0x7f021c07ae50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02240016c0 tx=0x7f0224037040 comp rx=0 tx=0).stop 2024-09-10T20:45:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.539+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 msgr2=0x7f023014eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.539+0000 7f02350b8640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 0x7f023014eaa0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f02200066b0 tx=0x7f02200039c0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 shutdown_connections 2024-09-10T20:45:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f021c078990 0x7f021c07ae50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02300aee50 0x7f02300af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02301f7f30 0x7f023014de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 --2- 172.21.15.28:0/415040763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0218004000 0x7f023014eaa0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 >> 172.21.15.28:0/415040763 conn(0x7f023006dbf0 msgr2=0x7f02300d11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 shutdown_connections 2024-09-10T20:45:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.540+0000 7f02350b8640 1 -- 172.21.15.28:0/415040763 wait complete. 2024-09-10T20:45:55.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2' 2024-09-10T20:45:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.656+0000 7f16b4d65640 1 -- 172.21.15.28:0/2137299136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b00cfec0 msgr2=0x7f16b00d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.656+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2137299136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b00cfec0 0x7f16b00d0330 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f16b0073cb0 tx=0x7f169802a560 comp rx=0 tx=0).stop 2024-09-10T20:45:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.657+0000 7f16b4d65640 1 -- 172.21.15.28:0/2137299136 shutdown_connections 2024-09-10T20:45:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.657+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2137299136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16b00ce5f0 0x7f16b00cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.657+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2137299136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b00cfec0 0x7f16b00d0330 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.657+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2137299136 >> v2:172.21.15.159:3300/0 conn(0x7f16b015bf30 0x7f16b00d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.657+0000 7f16b4d65640 1 -- 172.21.15.28:0/2137299136 >> 172.21.15.28:0/2137299136 conn(0x7f16b00d1e40 msgr2=0x7f16b00d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:55.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.657+0000 7f16b4d65640 1 -- 172.21.15.28:0/2137299136 shutdown_connections 2024-09-10T20:45:55.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.657+0000 7f16b4d65640 1 -- 172.21.15.28:0/2137299136 wait complete. 2024-09-10T20:45:55.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.658+0000 7f16b4d65640 1 Processor -- start 2024-09-10T20:45:55.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.658+0000 7f16b4d65640 1 -- start start 2024-09-10T20:45:55.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16b4d65640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b00ce5f0 0x7f16b014bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16b4d65640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16b00cfec0 0x7f16b009f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16b4d65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b015bf30 0x7f16b00b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16b4d65640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16b00d0cc0 con 0x7f16b015bf30 2024-09-10T20:45:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16b4d65640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f16b00d0b40 con 0x7f16b00ce5f0 2024-09-10T20:45:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16b4d65640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f16b00cefa0 con 0x7f16b00cfec0 2024-09-10T20:45:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b015bf30 0x7f16b00b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b015bf30 0x7f16b00b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45016/0 (socket says 172.21.15.28:45016) 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 -- 172.21.15.28:0/2089045465 learned_addr learned my addr 172.21.15.28:0/2089045465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16a7fff640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16b00cfec0 0x7f16b009f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16ae59c640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b00ce5f0 0x7f16b014bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 -- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16b00cfec0 msgr2=0x7f16b009f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16b00cfec0 0x7f16b009f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 -- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b00ce5f0 msgr2=0x7f16b014bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b00ce5f0 0x7f16b014bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 -- 172.21.15.28:0/2089045465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b014c320 con 0x7f16b015bf30 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.659+0000 7f16aed9d640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b015bf30 0x7f16b00b1ea0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f16a800b970 tx=0x7f16a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.660+0000 7f16a6ffd640 1 -- 172.21.15.28:0/2089045465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8019070 con 0x7f16b015bf30 2024-09-10T20:45:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.660+0000 7f16a6ffd640 1 -- 172.21.15.28:0/2089045465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16a800cc50 con 0x7f16b015bf30 2024-09-10T20:45:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.660+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b014ec50 con 0x7f16b015bf30 2024-09-10T20:45:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.660+0000 7f16a6ffd640 1 -- 172.21.15.28:0/2089045465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a80146a0 con 0x7f16b015bf30 2024-09-10T20:45:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.660+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b00caa30 con 0x7f16b015bf30 2024-09-10T20:45:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.661+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b00d0330 con 0x7f16b015bf30 2024-09-10T20:45:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.661+0000 7f16a6ffd640 1 -- 172.21.15.28:0/2089045465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f16a801d3b0 con 0x7f16b015bf30 2024-09-10T20:45:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.662+0000 7f16a6ffd640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1690078990 0x7f169007ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.662+0000 7f16a6ffd640 1 -- 172.21.15.28:0/2089045465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7f16a80987b0 con 0x7f16b015bf30 2024-09-10T20:45:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.662+0000 7f16ae59c640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1690078990 0x7f169007ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.662+0000 7f16ae59c640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1690078990 0x7f169007ae50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f16a00098e0 tx=0x7f16a0009060 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:55.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.666+0000 7f16a6ffd640 1 -- 172.21.15.28:0/2089045465 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f16a80624f0 con 0x7f16b015bf30 2024-09-10T20:45:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:55.926+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]} v 0) -- 0x7f16b00ce260 con 0x7f16b015bf30 2024-09-10T20:45:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.199+0000 7f16a6ffd640 1 -- 172.21.15.28:0/2089045465 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]}]=0 v431) ==== 145+0+0 (secure 0 0 0) 0x7f16a8066f20 con 0x7f16b015bf30 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1690078990 msgr2=0x7f169007ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1690078990 0x7f169007ae50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f16a00098e0 tx=0x7f16a0009060 comp rx=0 tx=0).stop 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b015bf30 msgr2=0x7f16b00b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b015bf30 0x7f16b00b1ea0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f16a800b970 tx=0x7f16a800be30 comp rx=0 tx=0).stop 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 shutdown_connections 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1690078990 0x7f169007ae50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b015bf30 0x7f16b00b1ea0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f16b00cfec0 0x7f16b009f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 --2- 172.21.15.28:0/2089045465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f16b00ce5f0 0x7f16b014bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 >> 172.21.15.28:0/2089045465 conn(0x7f16b00d1e40 msgr2=0x7f16b015e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 shutdown_connections 2024-09-10T20:45:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.202+0000 7f16b4d65640 1 -- 172.21.15.28:0/2089045465 wait complete. 2024-09-10T20:45:56.217 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-10T20:45:56.217 INFO:tasks.cephfs_test_runner:Starting test: test_unparseable_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate unparseable text .meta file on legacy subvol root doesn't break the system 2024-09-10T20:45:56.217 INFO:tasks.cephfs_test_runner:test_unparseable_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-10T20:45:56.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:45:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.320+0000 7fbf32ac6640 1 -- 172.21.15.28:0/2426499563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 msgr2=0x7fbf140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.320+0000 7fbf32ac6640 1 --2- 172.21.15.28:0/2426499563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 0x7fbf140051c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009ed0 tx=0x7fbf14005830 comp rx=0 tx=0).stop 2024-09-10T20:45:56.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.320+0000 7fbf31063640 1 -- 172.21.15.28:0/2426499563 reap_dead start 2024-09-10T20:45:56.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.320+0000 7fbf32ac6640 1 -- 172.21.15.28:0/2426499563 shutdown_connections 2024-09-10T20:45:56.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.320+0000 7fbf32ac6640 1 -- 172.21.15.28:0/2426499563 >> 172.21.15.28:0/2426499563 conn(0x7fbf2c06cf00 msgr2=0x7fbf2c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:56.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.320+0000 7fbf32ac6640 1 -- 172.21.15.28:0/2426499563 shutdown_connections 2024-09-10T20:45:56.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.320+0000 7fbf32ac6640 1 -- 172.21.15.28:0/2426499563 wait complete. 2024-09-10T20:45:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 Processor -- start 2024-09-10T20:45:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 -- start start 2024-09-10T20:45:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 0x7fbf2c152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c156720 0x7fbf2c151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c073550 0x7fbf2c0739d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c0b2f30 con 0x7fbf14004d40 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c0b2db0 con 0x7fbf2c156720 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.322+0000 7fbf32ac6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fbf2c0b30b0 con 0x7fbf2c073550 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 0x7fbf2c152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 0x7fbf2c152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45034/0 (socket says 172.21.15.28:45034) 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 -- 172.21.15.28:0/1182552904 learned_addr learned my addr 172.21.15.28:0/1182552904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf31063640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c073550 0x7fbf2c0739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf23fff640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c156720 0x7fbf2c151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 -- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c073550 msgr2=0x7fbf2c0739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c073550 0x7fbf2c0739d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 -- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c156720 msgr2=0x7fbf2c151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c156720 0x7fbf2c151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 -- 172.21.15.28:0/1182552904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2c1f2370 con 0x7fbf14004d40 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf23fff640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c156720 0x7fbf2c151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf31063640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c073550 0x7fbf2c0739d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf30862640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 0x7fbf2c152490 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fbf28004f00 tx=0x7fbf28001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf217fa640 1 -- 172.21.15.28:0/1182552904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28015020 con 0x7fbf14004d40 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf217fa640 1 -- 172.21.15.28:0/1182552904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf28011960 con 0x7fbf14004d40 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2c1f2650 con 0x7fbf14004d40 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf217fa640 1 -- 172.21.15.28:0/1182552904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28011db0 con 0x7fbf14004d40 2024-09-10T20:45:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.323+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2c1f2c10 con 0x7fbf14004d40 2024-09-10T20:45:56.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.324+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf2c0d03b0 con 0x7fbf14004d40 2024-09-10T20:45:56.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.325+0000 7fbf217fa640 1 -- 172.21.15.28:0/1182552904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fbf2801a400 con 0x7fbf14004d40 2024-09-10T20:45:56.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.325+0000 7fbf217fa640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf180789c0 0x7fbf1807ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.326+0000 7fbf217fa640 1 -- 172.21.15.28:0/1182552904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7fbf28099620 con 0x7fbf14004d40 2024-09-10T20:45:56.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.326+0000 7fbf23fff640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf180789c0 0x7fbf1807ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.326+0000 7fbf23fff640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf180789c0 0x7fbf1807ae80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0cbd50 tx=0x7fbf1c001760 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:56.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.330+0000 7fbf217fa640 1 -- 172.21.15.28:0/1182552904 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbf2801f020 con 0x7fbf14004d40 2024-09-10T20:45:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.516+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf2c0ced40 con 0x7fbf14004d40 2024-09-10T20:45:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.517+0000 7fbf217fa640 1 -- 172.21.15.28:0/1182552904 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) ==== 74+0+13086 (secure 0 0 0) 0x7fbf2806c190 con 0x7fbf14004d40 2024-09-10T20:45:56.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:56.521 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":154,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:45:55.193826+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1892964400":"2024-09-11T20:45:52.092462+0000","172.21.15.28:6833/1892964400":"2024-09-11T20:45:52.092462+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:45:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf180789c0 msgr2=0x7fbf1807ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf180789c0 0x7fbf1807ae80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0cbd50 tx=0x7fbf1c001760 comp rx=0 tx=0).stop 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 msgr2=0x7fbf2c152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 0x7fbf2c152490 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fbf28004f00 tx=0x7fbf28001630 comp rx=0 tx=0).stop 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 shutdown_connections 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fbf180789c0 0x7fbf1807ae80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf2c073550 0x7fbf2c0739d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf2c156720 0x7fbf2c151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 --2- 172.21.15.28:0/1182552904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf14004d40 0x7fbf2c152490 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 >> 172.21.15.28:0/1182552904 conn(0x7fbf2c06cf00 msgr2=0x7fbf2c15b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 shutdown_connections 2024-09-10T20:45:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.519+0000 7fbf32ac6640 1 -- 172.21.15.28:0/1182552904 wait complete. 2024-09-10T20:45:56.535 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:37:59.990929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:45:56.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:45:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1391547115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe78161770 msgr2=0x7efe780d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1391547115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe78161770 0x7efe780d20e0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7efe68005fd0 tx=0x7efe6802a820 comp rx=0 tx=0).stop 2024-09-10T20:45:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1391547115 shutdown_connections 2024-09-10T20:45:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1391547115 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe78161b50 0x7efe780d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1391547115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe78161770 0x7efe780d20e0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1391547115 >> 172.21.15.28:0/1391547115 conn(0x7efe780d4070 msgr2=0x7efe780d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1391547115 shutdown_connections 2024-09-10T20:45:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.639+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1391547115 wait complete. 2024-09-10T20:45:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.641+0000 7efe7cb2b640 1 Processor -- start 2024-09-10T20:45:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.641+0000 7efe7cb2b640 1 -- start start 2024-09-10T20:45:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.641+0000 7efe7cb2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe780d4b90 0x7efe7815f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7cb2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe78161770 0x7efe780ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7cb2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe78161b50 0x7efe780b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7cb2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe780b2f50 con 0x7efe780d4b90 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7cb2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efe780b2dd0 con 0x7efe78161770 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7cb2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efe780b30d0 con 0x7efe78161b50 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe780d4b90 0x7efe7815f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe780d4b90 0x7efe7815f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45046/0 (socket says 172.21.15.28:45046) 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 -- 172.21.15.28:0/1338077166 learned_addr learned my addr 172.21.15.28:0/1338077166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 -- 172.21.15.28:0/1338077166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe78161b50 msgr2=0x7efe780b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe78161b50 0x7efe780b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 -- 172.21.15.28:0/1338077166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe78161770 msgr2=0x7efe780ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe75d9b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe78161770 0x7efe780ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe78161770 0x7efe780ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 -- 172.21.15.28:0/1338077166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7809e160 con 0x7efe780d4b90 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7659c640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe780d4b90 0x7efe7815f610 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7efe68035040 tx=0x7efe68032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe66ffd640 1 -- 172.21.15.28:0/1338077166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68037040 con 0x7efe780d4b90 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe66ffd640 1 -- 172.21.15.28:0/1338077166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe68042070 con 0x7efe780d4b90 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe781f54f0 con 0x7efe780d4b90 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe66ffd640 1 -- 172.21.15.28:0/1338077166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe680042a0 con 0x7efe780d4b90 2024-09-10T20:45:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.642+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe781f5ab0 con 0x7efe780d4b90 2024-09-10T20:45:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.643+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe780b3870 con 0x7efe780d4b90 2024-09-10T20:45:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.644+0000 7efe66ffd640 1 -- 172.21.15.28:0/1338077166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7efe6803a030 con 0x7efe780d4b90 2024-09-10T20:45:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.645+0000 7efe66ffd640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe58078990 0x7efe5807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.645+0000 7efe75d9b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe58078990 0x7efe5807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.645+0000 7efe66ffd640 1 -- 172.21.15.28:0/1338077166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7efe680b9390 con 0x7efe780d4b90 2024-09-10T20:45:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.645+0000 7efe75d9b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe58078990 0x7efe5807ae50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efe6c005fd0 tx=0x7efe6c005bc0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.649+0000 7efe66ffd640 1 -- 172.21.15.28:0/1338077166 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efe68083720 con 0x7efe780d4b90 2024-09-10T20:45:56.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.835+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efe780d3120 con 0x7efe780d4b90 2024-09-10T20:45:56.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.836+0000 7efe66ffd640 1 -- 172.21.15.28:0/1338077166 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v154) ==== 93+0+10 (secure 0 0 0) 0x7efe68083020 con 0x7efe780d4b90 2024-09-10T20:45:56.840 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T20:45:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe58078990 msgr2=0x7efe5807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe58078990 0x7efe5807ae50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efe6c005fd0 tx=0x7efe6c005bc0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe780d4b90 msgr2=0x7efe7815f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe780d4b90 0x7efe7815f610 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7efe68035040 tx=0x7efe68032c90 comp rx=0 tx=0).stop 2024-09-10T20:45:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 shutdown_connections 2024-09-10T20:45:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efe58078990 0x7efe5807ae50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe78161b50 0x7efe780b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe78161770 0x7efe780ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 --2- 172.21.15.28:0/1338077166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe780d4b90 0x7efe7815f610 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 >> 172.21.15.28:0/1338077166 conn(0x7efe780d4070 msgr2=0x7efe780d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 shutdown_connections 2024-09-10T20:45:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.838+0000 7efe7cb2b640 1 -- 172.21.15.28:0/1338077166 wait complete. 2024-09-10T20:45:56.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade' 2024-09-10T20:45:56.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 -- 172.21.15.28:0/1162738609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc62015de70 msgr2=0x7fc6200d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 --2- 172.21.15.28:0/1162738609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc62015de70 0x7fc6200d2b90 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fc6200d01d0 tx=0x7fc61002a560 comp rx=0 tx=0).stop 2024-09-10T20:45:56.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 -- 172.21.15.28:0/1162738609 shutdown_connections 2024-09-10T20:45:56.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 --2- 172.21.15.28:0/1162738609 >> v2:172.21.15.159:3300/0 conn(0x7fc620162260 0x7fc6200cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 --2- 172.21.15.28:0/1162738609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc62015de70 0x7fc6200d2b90 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 -- 172.21.15.28:0/1162738609 >> 172.21.15.28:0/1162738609 conn(0x7fc6200d3fb0 msgr2=0x7fc6200d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 -- 172.21.15.28:0/1162738609 shutdown_connections 2024-09-10T20:45:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.954+0000 7fc627085640 1 -- 172.21.15.28:0/1162738609 wait complete. 2024-09-10T20:45:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.955+0000 7fc627085640 1 Processor -- start 2024-09-10T20:45:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.955+0000 7fc627085640 1 -- start start 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc627085640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6200cf780 0x7fc62015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc627085640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc62015de70 0x7fc62009e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc627085640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc620162260 0x7fc620160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc627085640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6200b7190 con 0x7fc6200cf780 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc627085640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc6200b7010 con 0x7fc62015de70 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc627085640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc6200b7310 con 0x7fc620162260 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc624e21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6200cf780 0x7fc62015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc624e21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6200cf780 0x7fc62015a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45072/0 (socket says 172.21.15.28:45072) 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.956+0000 7fc624e21640 1 -- 172.21.15.28:0/1655364404 learned_addr learned my addr 172.21.15.28:0/1655364404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc625622640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc620162260 0x7fc620160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc61ffff640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc62015de70 0x7fc62009e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc624e21640 1 -- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc620162260 msgr2=0x7fc620160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc624e21640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc620162260 0x7fc620160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc624e21640 1 -- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc62015de70 msgr2=0x7fc62009e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc624e21640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc62015de70 0x7fc62009e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc624e21640 1 -- 172.21.15.28:0/1655364404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6201f4b20 con 0x7fc6200cf780 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc625622640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc620162260 0x7fc620160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc61ffff640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc62015de70 0x7fc62009e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc624e21640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6200cf780 0x7fc62015a590 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fc610009850 tx=0x7fc6100039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc61d7fa640 1 -- 172.21.15.28:0/1655364404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61003f070 con 0x7fc6200cf780 2024-09-10T20:45:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc61d7fa640 1 -- 172.21.15.28:0/1655364404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc610004890 con 0x7fc6200cf780 2024-09-10T20:45:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc61d7fa640 1 -- 172.21.15.28:0/1655364404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610035030 con 0x7fc6200cf780 2024-09-10T20:45:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62015aad0 con 0x7fc6200cf780 2024-09-10T20:45:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.957+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6201f50a0 con 0x7fc6200cf780 2024-09-10T20:45:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.958+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6200b78b0 con 0x7fc6200cf780 2024-09-10T20:45:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.959+0000 7fc61d7fa640 1 -- 172.21.15.28:0/1655364404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fc610004ca0 con 0x7fc6200cf780 2024-09-10T20:45:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.960+0000 7fc61d7fa640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc60c078990 0x7fc60c07ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.960+0000 7fc61ffff640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc60c078990 0x7fc60c07ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.960+0000 7fc61d7fa640 1 -- 172.21.15.28:0/1655364404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7fc6100b9b60 con 0x7fc6200cf780 2024-09-10T20:45:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.960+0000 7fc61ffff640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc60c078990 0x7fc60c07ae50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc614007fd0 tx=0x7fc614009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:56.964+0000 7fc61d7fa640 1 -- 172.21.15.28:0/1655364404 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc610083ef0 con 0x7fc6200cf780 2024-09-10T20:45:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:57.218+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fc6200d1400 con 0x7fc6200cf780 2024-09-10T20:45:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.205+0000 7fc61d7fa640 1 -- 172.21.15.28:0/1655364404 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]}]=0 v433) ==== 166+0+0 (secure 0 0 0) 0x7fc6100837f0 con 0x7fc6200cf780 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc60c078990 msgr2=0x7fc60c07ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc60c078990 0x7fc60c07ae50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc614007fd0 tx=0x7fc614009040 comp rx=0 tx=0).stop 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6200cf780 msgr2=0x7fc62015a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6200cf780 0x7fc62015a590 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fc610009850 tx=0x7fc6100039e0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 shutdown_connections 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc60c078990 0x7fc60c07ae50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc620162260 0x7fc620160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc62015de70 0x7fc62009e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 --2- 172.21.15.28:0/1655364404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6200cf780 0x7fc62015a590 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 >> 172.21.15.28:0/1655364404 conn(0x7fc6200d3fb0 msgr2=0x7fc62006eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 shutdown_connections 2024-09-10T20:45:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.209+0000 7fc627085640 1 -- 172.21.15.28:0/1655364404 wait complete. 2024-09-10T20:45:58.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-10T20:45:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 -- 172.21.15.28:0/1092149334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc40d2b90 msgr2=0x7f4fc4161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 --2- 172.21.15.28:0/1092149334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc40d2b90 0x7f4fc4161b50 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009830 tx=0x7f4fb802a510 comp rx=0 tx=0).stop 2024-09-10T20:45:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 -- 172.21.15.28:0/1092149334 shutdown_connections 2024-09-10T20:45:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 --2- 172.21.15.28:0/1092149334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fc4162090 0x7f4fc40b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 --2- 172.21.15.28:0/1092149334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc40d2b90 0x7f4fc4161b50 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 -- 172.21.15.28:0/1092149334 >> 172.21.15.28:0/1092149334 conn(0x7f4fc40d3fb0 msgr2=0x7f4fc40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 -- 172.21.15.28:0/1092149334 shutdown_connections 2024-09-10T20:45:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.335+0000 7f4fca272640 1 -- 172.21.15.28:0/1092149334 wait complete. 2024-09-10T20:45:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.336+0000 7f4fca272640 1 Processor -- start 2024-09-10T20:45:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.336+0000 7f4fca272640 1 -- start start 2024-09-10T20:45:58.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fca272640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fc40d2b90 0x7f4fc41f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fca272640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fc415de70 0x7f4fc4152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fca272640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc4162090 0x7f4fc41f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fca272640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc415e3d0 con 0x7f4fc4162090 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fca272640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc415e250 con 0x7f4fc415de70 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fca272640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc40b7340 con 0x7f4fc40d2b90 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fc880f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc4162090 0x7f4fc41f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fc880f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc4162090 0x7f4fc41f7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45092/0 (socket says 172.21.15.28:45092) 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.337+0000 7f4fc880f640 1 -- 172.21.15.28:0/968914613 learned_addr learned my addr 172.21.15.28:0/968914613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc3fff640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fc40d2b90 0x7f4fc41f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc37fe640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fc415de70 0x7f4fc4152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc880f640 1 -- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fc40d2b90 msgr2=0x7f4fc41f5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc880f640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fc40d2b90 0x7f4fc41f5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc880f640 1 -- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fc415de70 msgr2=0x7f4fc4152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc880f640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fc415de70 0x7f4fc4152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc880f640 1 -- 172.21.15.28:0/968914613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc41f8b00 con 0x7f4fc4162090 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc37fe640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fc415de70 0x7f4fc4152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc3fff640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fc40d2b90 0x7f4fc41f5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc880f640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc4162090 0x7f4fc41f7f40 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00e960 tx=0x7f4fbc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc0ff9640 1 -- 172.21.15.28:0/968914613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc004d20 con 0x7f4fc4162090 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc0ff9640 1 -- 172.21.15.28:0/968914613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fbc00cb20 con 0x7f4fc4162090 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc41f8de0 con 0x7f4fc4162090 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fc0ff9640 1 -- 172.21.15.28:0/968914613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc005020 con 0x7f4fc4162090 2024-09-10T20:45:58.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.338+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc41f4d40 con 0x7f4fc4162090 2024-09-10T20:45:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.339+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc40d38e0 con 0x7f4fc4162090 2024-09-10T20:45:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.340+0000 7f4fc0ff9640 1 -- 172.21.15.28:0/968914613 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f4fbc00f040 con 0x7f4fc4162090 2024-09-10T20:45:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.340+0000 7f4fc0ff9640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fb0078990 0x7f4fb007ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.341+0000 7f4fc0ff9640 1 -- 172.21.15.28:0/968914613 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7f4fbc098e90 con 0x7f4fc4162090 2024-09-10T20:45:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.341+0000 7f4fc3fff640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fb0078990 0x7f4fb007ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.341+0000 7f4fc3fff640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fb0078990 0x7f4fb007ae50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4007fd0 tx=0x7f4fb4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:58.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.345+0000 7f4fc0ff9640 1 -- 172.21.15.28:0/968914613 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4fc40d38e0 con 0x7f4fc4162090 2024-09-10T20:45:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.531+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f4fc40b9f30 con 0x7f4fc4162090 2024-09-10T20:45:58.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.532+0000 7f4fc0ff9640 1 -- 172.21.15.28:0/968914613 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v47) ==== 59+0+1998 (secure 0 0 0) 0x7f4fbc016030 con 0x7f4fc4162090 2024-09-10T20:45:58.536 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-10T20:45:58.536 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-10T20:45:58.536 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-10T20:45:58.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-10T20:45:58.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4474.681131 2024-09-10T20:45:58.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4237.515884 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4102.008703 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 3831.945533 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5130.648343 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6315.787001 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6472.271489 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4139.280508 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-10T20:45:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-10T20:45:58.538 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-10T20:45:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fb0078990 msgr2=0x7f4fb007ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fb0078990 0x7f4fb007ae50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4007fd0 tx=0x7f4fb4009040 comp rx=0 tx=0).stop 2024-09-10T20:45:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc4162090 msgr2=0x7f4fc41f7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc4162090 0x7f4fc41f7f40 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00e960 tx=0x7f4fbc00ee20 comp rx=0 tx=0).stop 2024-09-10T20:45:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 shutdown_connections 2024-09-10T20:45:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fb0078990 0x7f4fb007ae50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fc4162090 0x7f4fc41f7f40 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fc415de70 0x7f4fc4152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 --2- 172.21.15.28:0/968914613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fc40d2b90 0x7f4fc41f5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 >> 172.21.15.28:0/968914613 conn(0x7f4fc40d3fb0 msgr2=0x7f4fc40cdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 shutdown_connections 2024-09-10T20:45:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.534+0000 7f4fca272640 1 -- 172.21.15.28:0/968914613 wait complete. 2024-09-10T20:45:58.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:45:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.654+0000 7f749b55d640 1 -- 172.21.15.28:0/599709476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 msgr2=0x7f7480004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.654+0000 7f749b55d640 1 --2- 172.21.15.28:0/599709476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 0x7f7480004c80 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f748400c2e0 tx=0x7f7480005830 comp rx=0 tx=0).stop 2024-09-10T20:45:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.654+0000 7f7499afa640 1 -- 172.21.15.28:0/599709476 reap_dead start 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.654+0000 7f749b55d640 1 -- 172.21.15.28:0/599709476 shutdown_connections 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.654+0000 7f749b55d640 1 -- 172.21.15.28:0/599709476 >> 172.21.15.28:0/599709476 conn(0x7f74940d0d50 msgr2=0x7f74940d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.654+0000 7f749b55d640 1 -- 172.21.15.28:0/599709476 shutdown_connections 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.654+0000 7f749b55d640 1 -- 172.21.15.28:0/599709476 wait complete. 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.655+0000 7f749b55d640 1 Processor -- start 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.655+0000 7f749b55d640 1 -- start start 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f749b55d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 0x7f74940c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f749b55d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74941f6580 0x7f749414fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f749b55d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74941f69e0 0x7f74941500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f749b55d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74940cc8e0 con 0x7f7480004800 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f749b55d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f74940cc760 con 0x7f74941f69e0 2024-09-10T20:45:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f749b55d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f74940cca60 con 0x7f74941f6580 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f74992f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 0x7f74940c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f74992f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 0x7f74940c73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45108/0 (socket says 172.21.15.28:45108) 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.656+0000 7f74992f9640 1 -- 172.21.15.28:0/527534443 learned_addr learned my addr 172.21.15.28:0/527534443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f7498af8640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74941f6580 0x7f749414fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f7499afa640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74941f69e0 0x7f74941500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f74992f9640 1 -- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74941f6580 msgr2=0x7f749414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f74992f9640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74941f6580 0x7f749414fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f74992f9640 1 -- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74941f69e0 msgr2=0x7f74941500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f74992f9640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74941f69e0 0x7f74941500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f74992f9640 1 -- 172.21.15.28:0/527534443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74941f3710 con 0x7f7480004800 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f7498af8640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74941f6580 0x7f749414fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f7499afa640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74941f69e0 0x7f74941500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f74992f9640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 0x7f74940c73e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f7490011ab0 tx=0x7f7490011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f7489ffb640 1 -- 172.21.15.28:0/527534443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749001e070 con 0x7f7480004800 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f7489ffb640 1 -- 172.21.15.28:0/527534443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f749000fc70 con 0x7f7480004800 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74940c6620 con 0x7f7480004800 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f7489ffb640 1 -- 172.21.15.28:0/527534443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490019670 con 0x7f7480004800 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.657+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74940c6be0 con 0x7f7480004800 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.658+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74940ccff0 con 0x7f7480004800 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.659+0000 7f7489ffb640 1 -- 172.21.15.28:0/527534443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f7490019810 con 0x7f7480004800 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.659+0000 7f7489ffb640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74780789c0 0x7f747807ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.660+0000 7f7489ffb640 1 -- 172.21.15.28:0/527534443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7f74900993c0 con 0x7f7480004800 2024-09-10T20:45:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.660+0000 7f7498af8640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74780789c0 0x7f747807ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.660+0000 7f7498af8640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74780789c0 0x7f747807ae80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7484033000 tx=0x7f748402d000 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.664+0000 7f7489ffb640 1 -- 172.21.15.28:0/527534443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f749000fe60 con 0x7f7480004800 2024-09-10T20:45:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.851+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f74940b6cd0 con 0x7f7480004800 2024-09-10T20:45:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.854+0000 7f7489ffb640 1 -- 172.21.15.28:0/527534443 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7f74900630a0 con 0x7f7480004800 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74780789c0 msgr2=0x7f747807ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74780789c0 0x7f747807ae80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7484033000 tx=0x7f748402d000 comp rx=0 tx=0).stop 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 msgr2=0x7f74940c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 0x7f74940c73e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f7490011ab0 tx=0x7f7490011f70 comp rx=0 tx=0).stop 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 shutdown_connections 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f74780789c0 0x7f747807ae80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f74941f69e0 0x7f74941500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f74941f6580 0x7f749414fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 --2- 172.21.15.28:0/527534443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004800 0x7f74940c73e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:58.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 >> 172.21.15.28:0/527534443 conn(0x7f74940d0d50 msgr2=0x7f74940cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:58.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 shutdown_connections 2024-09-10T20:45:58.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:58.857+0000 7f749b55d640 1 -- 172.21.15.28:0/527534443 wait complete. 2024-09-10T20:45:58.871 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:45:58.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:58.911 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:45:58.976 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:45:58.976 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:58.976 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:58.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:45:58.993 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:58.993 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:59.011 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:45:59.011 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:45:59.011 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:59.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:45:59.050 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:45:59.050 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:45:59.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:45:59.072 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-10T20:45:59.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:45:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 -- 172.21.15.28:0/1454613828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680cf780 msgr2=0x7fa7680cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 --2- 172.21.15.28:0/1454613828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680cf780 0x7fa7680cfbc0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fa750005fd0 tx=0x7fa75002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 -- 172.21.15.28:0/1454613828 shutdown_connections 2024-09-10T20:45:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 --2- 172.21.15.28:0/1454613828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680cf780 0x7fa7680cfbc0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 --2- 172.21.15.28:0/1454613828 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa76815de70 0x7fa7680d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 -- 172.21.15.28:0/1454613828 >> 172.21.15.28:0/1454613828 conn(0x7fa7680d3fb0 msgr2=0x7fa7680d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 -- 172.21.15.28:0/1454613828 shutdown_connections 2024-09-10T20:45:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.190+0000 7fa76c860640 1 -- 172.21.15.28:0/1454613828 wait complete. 2024-09-10T20:45:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 Processor -- start 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 -- start start 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7680cf780 0x7fa768157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76815de70 0x7fa768159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa768161b50 0x7fa7680cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa768162350 con 0x7fa76815de70 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681621d0 con 0x7fa7680cf780 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.192+0000 7fa76c860640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa7681624d0 con 0x7fa768161b50 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76815de70 0x7fa768159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76815de70 0x7fa768159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45124/0 (socket says 172.21.15.28:45124) 2024-09-10T20:45:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 -- 172.21.15.28:0/1902785178 learned_addr learned my addr 172.21.15.28:0/1902785178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa766d9d640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa768161b50 0x7fa7680cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa76659c640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7680cf780 0x7fa768157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 -- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa768161b50 msgr2=0x7fa7680cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa768161b50 0x7fa7680cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 -- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7680cf780 msgr2=0x7fa768157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7680cf780 0x7fa768157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 -- 172.21.15.28:0/1902785178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7681f6010 con 0x7fa76815de70 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa766d9d640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa768161b50 0x7fa7680cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa765d9b640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76815de70 0x7fa768159300 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099f0 tx=0x7fa7500038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa75effd640 1 -- 172.21.15.28:0/1902785178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75003f070 con 0x7fa76815de70 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa75effd640 1 -- 172.21.15.28:0/1902785178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7500047a0 con 0x7fa76815de70 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7681f8690 con 0x7fa76815de70 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa75effd640 1 -- 172.21.15.28:0/1902785178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750035030 con 0x7fa76815de70 2024-09-10T20:45:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.193+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7681f8c50 con 0x7fa76815de70 2024-09-10T20:45:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.194+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7680cfbc0 con 0x7fa76815de70 2024-09-10T20:45:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.195+0000 7fa75effd640 1 -- 172.21.15.28:0/1902785178 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fa750004bf0 con 0x7fa76815de70 2024-09-10T20:45:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.195+0000 7fa75effd640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa748078990 0x7fa74807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.196+0000 7fa75effd640 1 -- 172.21.15.28:0/1902785178 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7fa7500b9890 con 0x7fa76815de70 2024-09-10T20:45:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.196+0000 7fa76659c640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa748078990 0x7fa74807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.196+0000 7fa76659c640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa748078990 0x7fa74807ae50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa758005c40 tx=0x7fa75800a780 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.200+0000 7fa75effd640 1 -- 172.21.15.28:0/1902785178 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa750083c20 con 0x7fa76815de70 2024-09-10T20:45:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.423+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7680b4f70 con 0x7fa76815de70 2024-09-10T20:45:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.424+0000 7fa75effd640 1 -- 172.21.15.28:0/1902785178 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+3919 (secure 0 0 0) 0x7fa750083520 con 0x7fa76815de70 2024-09-10T20:45:59.428 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:45:59.428 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":109,"btime":"2024-09-10T20:45:53:093930+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3007122398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3007122398},{"type":"v1","addr":"172.21.15.28:6834","nonce":3007122398}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":7031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3280811834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3280811834},{"type":"v1","addr":"172.21.15.159:6835","nonce":3280811834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":7630,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/632114757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":632114757},{"type":"v1","addr":"172.21.15.159:6834","nonce":632114757}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":8985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2668895899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2668895899},{"type":"v1","addr":"172.21.15.28:6835","nonce":2668895899}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":109}],"filesystems":[]} 2024-09-10T20:45:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 109 2024-09-10T20:45:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa748078990 msgr2=0x7fa74807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa748078990 0x7fa74807ae50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa758005c40 tx=0x7fa75800a780 comp rx=0 tx=0).stop 2024-09-10T20:45:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76815de70 msgr2=0x7fa768159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76815de70 0x7fa768159300 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099f0 tx=0x7fa7500038a0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 shutdown_connections 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa748078990 0x7fa74807ae50 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa768161b50 0x7fa7680cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76815de70 0x7fa768159300 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 --2- 172.21.15.28:0/1902785178 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa7680cf780 0x7fa768157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 >> 172.21.15.28:0/1902785178 conn(0x7fa7680d3fb0 msgr2=0x7fa7680b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 shutdown_connections 2024-09-10T20:45:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.426+0000 7fa76c860640 1 -- 172.21.15.28:0/1902785178 wait complete. 2024-09-10T20:45:59.442 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-10T20:45:59.442 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-10T20:45:59.443 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:45:59.443 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:59.443 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-10T20:45:59.443 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-10T20:45:59.443 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:45:59.444 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:59.444 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-10T20:45:59.444 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-10T20:45:59.444 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:45:59.444 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:59.444 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-10T20:45:59.444 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-10T20:45:59.445 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:45:59.445 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:45:59.453 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:45:59.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-10T20:45:59.454 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:45:59.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-10T20:45:59.455 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:45:59.456 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-10T20:45:59.456 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:45:59.456 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-10T20:45:59.495 INFO:tasks.ceph.mds.a:Started 2024-09-10T20:45:59.495 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:59.496 INFO:tasks.ceph.mds.c:Started 2024-09-10T20:45:59.496 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:59.498 INFO:tasks.ceph.mds.b:Started 2024-09-10T20:45:59.498 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:59.499 INFO:tasks.ceph.mds.d:Started 2024-09-10T20:45:59.499 DEBUG:teuthology.parallel:result is None 2024-09-10T20:45:59.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-10T20:45:59.573 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-10T20:45:59.574 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:45:59.579 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-09-10T20:45:59.579 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-09-10T20:45:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 -- 172.21.15.28:0/4029210631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 msgr2=0x7fa3980b6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/4029210631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa3980b6e00 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fa390005fd0 tx=0x7fa39002aa90 comp rx=0 tx=0).stop 2024-09-10T20:45:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 -- 172.21.15.28:0/4029210631 shutdown_connections 2024-09-10T20:45:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/4029210631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa3980b6e00 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/4029210631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3980d2b90 0x7fa398161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 -- 172.21.15.28:0/4029210631 >> 172.21.15.28:0/4029210631 conn(0x7fa3980d3fb0 msgr2=0x7fa3980d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:45:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 -- 172.21.15.28:0/4029210631 shutdown_connections 2024-09-10T20:45:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.606+0000 7fa39f9e8640 1 -- 172.21.15.28:0/4029210631 wait complete. 2024-09-10T20:45:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.607+0000 7fa39f9e8640 1 Processor -- start 2024-09-10T20:45:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.607+0000 7fa39f9e8640 1 -- start start 2024-09-10T20:45:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39f9e8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3980d2b90 0x7fa3980b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39f9e8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa39815de70 0x7fa3981f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39f9e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa39809ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39f9e8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa39815e3d0 con 0x7fa398162090 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39f9e8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa39815e250 con 0x7fa3980d2b90 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39f9e8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa3980b7340 con 0x7fa39815de70 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39df85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa39809ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39df85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa39809ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45164/0 (socket says 172.21.15.28:45164) 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39df85640 1 -- 172.21.15.28:0/1491058688 learned_addr learned my addr 172.21.15.28:0/1491058688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39d784640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3980d2b90 0x7fa3980b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.608+0000 7fa39cf83640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa39815de70 0x7fa3981f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39df85640 1 -- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa39815de70 msgr2=0x7fa3981f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39df85640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa39815de70 0x7fa3981f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39df85640 1 -- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3980d2b90 msgr2=0x7fa3980b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39df85640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3980d2b90 0x7fa3980b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39df85640 1 -- 172.21.15.28:0/1491058688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3981f4fc0 con 0x7fa398162090 2024-09-10T20:45:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39df85640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa39809ff60 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fa390009bf0 tx=0x7fa390003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa38dffb640 1 -- 172.21.15.28:0/1491058688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390044070 con 0x7fa398162090 2024-09-10T20:45:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa38dffb640 1 -- 172.21.15.28:0/1491058688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa390004250 con 0x7fa398162090 2024-09-10T20:45:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3981f52a0 con 0x7fa398162090 2024-09-10T20:45:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa38dffb640 1 -- 172.21.15.28:0/1491058688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390004660 con 0x7fa398162090 2024-09-10T20:45:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.609+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3980b6020 con 0x7fa398162090 2024-09-10T20:45:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.610+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3980b7fc0 con 0x7fa398162090 2024-09-10T20:45:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.611+0000 7fa38dffb640 1 -- 172.21.15.28:0/1491058688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fa39002e950 con 0x7fa398162090 2024-09-10T20:45:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.611+0000 7fa38dffb640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa37c078990 0x7fa37c07ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:45:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.611+0000 7fa38dffb640 1 -- 172.21.15.28:0/1491058688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 4971+0+0 (secure 0 0 0) 0x7fa3900b9980 con 0x7fa398162090 2024-09-10T20:45:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.612+0000 7fa39d784640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa37c078990 0x7fa37c07ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:45:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.612+0000 7fa39d784640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa37c078990 0x7fa37c07ae50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa394000f60 tx=0x7fa394009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:45:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.616+0000 7fa38dffb640 1 -- 172.21.15.28:0/1491058688 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa390032030 con 0x7fa398162090 2024-09-10T20:45:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:45:59.812+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa3980b9f30 con 0x7fa398162090 2024-09-10T20:46:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.228+0000 7fa38dffb640 1 -- 172.21.15.28:0/1491058688 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v156) ==== 97+0+0 (secure 0 0 0) 0x7fa3900836c0 con 0x7fa398162090 2024-09-10T20:46:00.232 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa37c078990 msgr2=0x7fa37c07ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa37c078990 0x7fa37c07ae50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa394000f60 tx=0x7fa394009040 comp rx=0 tx=0).stop 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 msgr2=0x7fa39809ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa39809ff60 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fa390009bf0 tx=0x7fa390003ce0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 shutdown_connections 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa37c078990 0x7fa37c07ae50 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa398162090 0x7fa39809ff60 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa39815de70 0x7fa3981f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 --2- 172.21.15.28:0/1491058688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3980d2b90 0x7fa3980b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.231+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 >> 172.21.15.28:0/1491058688 conn(0x7fa3980d3fb0 msgr2=0x7fa3980b95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.232+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 shutdown_connections 2024-09-10T20:46:00.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.232+0000 7fa39f9e8640 1 -- 172.21.15.28:0/1491058688 wait complete. 2024-09-10T20:46:00.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-10T20:46:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.352+0000 7fad1f240640 1 -- 172.21.15.28:0/2929640371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08004d40 msgr2=0x7fad080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.352+0000 7fad1f240640 1 --2- 172.21.15.28:0/2929640371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08004d40 0x7fad080051c0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fad10002230 tx=0x7fad08005830 comp rx=0 tx=0).stop 2024-09-10T20:46:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.352+0000 7fad1d7dd640 1 -- 172.21.15.28:0/2929640371 reap_dead start 2024-09-10T20:46:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.352+0000 7fad1f240640 1 -- 172.21.15.28:0/2929640371 shutdown_connections 2024-09-10T20:46:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.352+0000 7fad1f240640 1 -- 172.21.15.28:0/2929640371 >> 172.21.15.28:0/2929640371 conn(0x7fad180d3fb0 msgr2=0x7fad180d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.352+0000 7fad1f240640 1 -- 172.21.15.28:0/2929640371 shutdown_connections 2024-09-10T20:46:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.352+0000 7fad1f240640 1 -- 172.21.15.28:0/2929640371 wait complete. 2024-09-10T20:46:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.353+0000 7fad1f240640 1 Processor -- start 2024-09-10T20:46:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.353+0000 7fad1f240640 1 -- start start 2024-09-10T20:46:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1f240640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad08004d40 0x7fad1814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1f240640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad181f7b70 0x7fad181f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1f240640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180b6de0 0x7fad180b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1f240640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad1815e3d0 con 0x7fad180b6de0 2024-09-10T20:46:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1f240640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fad1815e250 con 0x7fad08004d40 2024-09-10T20:46:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1f240640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fad18161b50 con 0x7fad181f7b70 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1d7dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180b6de0 0x7fad180b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1d7dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180b6de0 0x7fad180b7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32918/0 (socket says 172.21.15.28:32918) 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1d7dd640 1 -- 172.21.15.28:0/447520363 learned_addr learned my addr 172.21.15.28:0/447520363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad1cfdc640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad08004d40 0x7fad1814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.354+0000 7fad17fff640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad181f7b70 0x7fad181f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1d7dd640 1 -- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad181f7b70 msgr2=0x7fad181f7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1d7dd640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad181f7b70 0x7fad181f7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1d7dd640 1 -- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad08004d40 msgr2=0x7fad1814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1d7dd640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad08004d40 0x7fad1814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1d7dd640 1 -- 172.21.15.28:0/447520363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1814f9b0 con 0x7fad180b6de0 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1d7dd640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180b6de0 0x7fad180b7260 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fad1000c700 tx=0x7fad10001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad157fa640 1 -- 172.21.15.28:0/447520363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad10039070 con 0x7fad180b6de0 2024-09-10T20:46:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad157fa640 1 -- 172.21.15.28:0/447520363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad100066a0 con 0x7fad180b6de0 2024-09-10T20:46:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad181f8710 con 0x7fad180b6de0 2024-09-10T20:46:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad157fa640 1 -- 172.21.15.28:0/447520363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad10002060 con 0x7fad180b6de0 2024-09-10T20:46:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.355+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1809e000 con 0x7fad180b6de0 2024-09-10T20:46:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.356+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad180cf780 con 0x7fad180b6de0 2024-09-10T20:46:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.357+0000 7fad157fa640 1 -- 172.21.15.28:0/447520363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fad10046030 con 0x7fad180b6de0 2024-09-10T20:46:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.357+0000 7fad157fa640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7facfc0789c0 0x7facfc07ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.357+0000 7fad157fa640 1 -- 172.21.15.28:0/447520363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4885+0+0 (secure 0 0 0) 0x7fad100b83b0 con 0x7fad180b6de0 2024-09-10T20:46:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.358+0000 7fad1cfdc640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7facfc0789c0 0x7facfc07ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.358+0000 7fad1cfdc640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7facfc0789c0 0x7facfc07ae80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00da30 tx=0x7fad0c006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.362+0000 7fad157fa640 1 -- 172.21.15.28:0/447520363 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fad10082200 con 0x7fad180b6de0 2024-09-10T20:46:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.621+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fad180b9f30 con 0x7fad180b6de0 2024-09-10T20:46:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.621+0000 7fad157fa640 1 -- 172.21.15.28:0/447520363 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) ==== 80+0+5304 (secure 0 0 0) 0x7fad10086c30 con 0x7fad180b6de0 2024-09-10T20:46:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:00.625 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmMS3eNBAA7cuy5Su2Oi6oODcs4vxK0A==", 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmxEE3BRAAf/AebZ0A2yrjN/2p7qa96w==", 2024-09-10T20:46:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmYZt1ABAAYQwJERKGmk5qqjCmbZzNRA==", 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:46:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmO51JDBAA7sQrs4rS1rTe0DWBIFfalg==", 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDwreBmXwjJMBAA4inQSnheYZtMrXk2+HPwCg==", 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDzreBmULwsMxAAKJ+FjmVYdHYkrIU5NFxQmw==", 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD3reBmqRo1CRAA3V/nenBOJ8MXRjkbG+9dvQ==", 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD6reBm7kd1CxAAcvhOjYtHExUW3B6X8Bbzug==", 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAAruBmSihiJhAAMDqhaBIrRXqwPB2mIu2KqA==", 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQADruBma46yNhAA6lzItcYMsu1hycr8aGmcyw==", 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAHruBmJZLNNxAAyKyBTfalRaErbyUPAtgO8w==", 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQALruBmBhtODBAAbBigCOIbK5ldjUT5i2/JRA==", 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-10T20:46:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ==", 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-10T20:46:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmtK52GRAAqT8tQfrGP13/ReFpj/exSw==", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBmya0wFhAAyBKrNCz4exc5gLkERTcibw==", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:46:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDvreBm2nAjHxAAOQCA/yA2OsT/kgE+6Q13BQ==", 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmRcEHGRAA63KjqK8poXjWecepkasEgg==", 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmKo2LMRAAJdlkts7ZPBhqYirOQx+2/A==", 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDureBmQVoaLhAAI4x6JIxJ4VzBJKiG5+uIyQ==", 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T20:46:00.638 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7facfc0789c0 msgr2=0x7facfc07ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7facfc0789c0 0x7facfc07ae80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00da30 tx=0x7fad0c006b70 comp rx=0 tx=0).stop 2024-09-10T20:46:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180b6de0 msgr2=0x7fad180b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180b6de0 0x7fad180b7260 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fad1000c700 tx=0x7fad10001aa0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 shutdown_connections 2024-09-10T20:46:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7facfc0789c0 0x7facfc07ae80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180b6de0 0x7fad180b7260 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad181f7b70 0x7fad181f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 --2- 172.21.15.28:0/447520363 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad08004d40 0x7fad1814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 >> 172.21.15.28:0/447520363 conn(0x7fad180d3fb0 msgr2=0x7fad180b80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:00.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 shutdown_connections 2024-09-10T20:46:00.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.624+0000 7fad1f240640 1 -- 172.21.15.28:0/447520363 wait complete. 2024-09-10T20:46:00.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-10T20:46:00.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/1970161055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680d0090 msgr2=0x7f1a680d04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/1970161055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680d0090 0x7f1a680d04e0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f1a680d0c80 tx=0x7f1a6002aa50 comp rx=0 tx=0).stop 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/1970161055 shutdown_connections 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/1970161055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680d0090 0x7f1a680d04e0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/1970161055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a680b4340 0x7f1a680b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/1970161055 >> v2:172.21.15.159:3300/0 conn(0x7f1a680cd3d0 0x7f1a680cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/1970161055 >> 172.21.15.28:0/1970161055 conn(0x7f1a6806cf00 msgr2=0x7f1a6806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.742+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/1970161055 shutdown_connections 2024-09-10T20:46:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.743+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/1970161055 wait complete. 2024-09-10T20:46:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.744+0000 7f1a6d1bc640 1 Processor -- start 2024-09-10T20:46:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.744+0000 7f1a6d1bc640 1 -- start start 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6d1bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680b4340 0x7f1a680cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6d1bc640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a680cd3d0 0x7f1a681581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6d1bc640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a680d0090 0x7f1a68158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6d1bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a680b3210 con 0x7f1a680b4340 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6d1bc640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1a680b3090 con 0x7f1a680d0090 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6d1bc640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1a680b3390 con 0x7f1a680cd3d0 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680b4340 0x7f1a680cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680b4340 0x7f1a680cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32944/0 (socket says 172.21.15.28:32944) 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 -- 172.21.15.28:0/3114161224 learned_addr learned my addr 172.21.15.28:0/3114161224 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6659c640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a680cd3d0 0x7f1a681581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a6759e640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a680d0090 0x7f1a68158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 -- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a680cd3d0 msgr2=0x7f1a681581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a680cd3d0 0x7f1a681581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 -- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a680d0090 msgr2=0x7f1a68158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a680d0090 0x7f1a68158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 -- 172.21.15.28:0/3114161224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a681f37c0 con 0x7f1a680b4340 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.745+0000 7f1a66d9d640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680b4340 0x7f1a680cbac0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800c920 tx=0x7f1a5800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.746+0000 7f1a4f7fe640 1 -- 172.21.15.28:0/3114161224 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58007bb0 con 0x7f1a680b4340 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.746+0000 7f1a4f7fe640 1 -- 172.21.15.28:0/3114161224 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a58007e40 con 0x7f1a680b4340 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.746+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a68147470 con 0x7f1a680b4340 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.746+0000 7f1a4f7fe640 1 -- 172.21.15.28:0/3114161224 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5800f6b0 con 0x7f1a680b4340 2024-09-10T20:46:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.746+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a68147a30 con 0x7f1a680b4340 2024-09-10T20:46:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.747+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a680b38b0 con 0x7f1a680b4340 2024-09-10T20:46:00.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.747+0000 7f1a4f7fe640 1 -- 172.21.15.28:0/3114161224 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f1a5800f850 con 0x7f1a680b4340 2024-09-10T20:46:00.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.748+0000 7f1a4f7fe640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a54078990 0x7f1a5407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:00.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.748+0000 7f1a6659c640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a54078990 0x7f1a5407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:00.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.748+0000 7f1a4f7fe640 1 -- 172.21.15.28:0/3114161224 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4885+0+0 (secure 0 0 0) 0x7f1a58098fa0 con 0x7f1a680b4340 2024-09-10T20:46:00.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.748+0000 7f1a6659c640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a54078990 0x7f1a5407ae50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1a680cf200 tx=0x7f1a5c009110 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:00.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.752+0000 7f1a4f7fe640 1 -- 172.21.15.28:0/3114161224 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1a58014030 con 0x7f1a680b4340 2024-09-10T20:46:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.938+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1a6806d530 con 0x7f1a680b4340 2024-09-10T20:46:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.938+0000 7f1a4f7fe640 1 -- 172.21.15.28:0/3114161224 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v158) ==== 84+0+65 (secure 0 0 0) 0x7f1a580632b0 con 0x7f1a680b4340 2024-09-10T20:46:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:00.942 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:46:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-10T20:46:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-10T20:46:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:00.943 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a54078990 msgr2=0x7f1a5407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a54078990 0x7f1a5407ae50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1a680cf200 tx=0x7f1a5c009110 comp rx=0 tx=0).stop 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680b4340 msgr2=0x7f1a680cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680b4340 0x7f1a680cbac0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800c920 tx=0x7f1a5800cd20 comp rx=0 tx=0).stop 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 shutdown_connections 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1a54078990 0x7f1a5407ae50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1a680d0090 0x7f1a68158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1a680cd3d0 0x7f1a681581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 --2- 172.21.15.28:0/3114161224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a680b4340 0x7f1a680cbac0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 >> 172.21.15.28:0/3114161224 conn(0x7f1a6806cf00 msgr2=0x7f1a6809cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 shutdown_connections 2024-09-10T20:46:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:00.940+0000 7f1a6d1bc640 1 -- 172.21.15.28:0/3114161224 wait complete. 2024-09-10T20:46:00.955 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-10T20:46:00.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-10T20:46:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.060+0000 7fdcc68cd640 1 -- 172.21.15.28:0/2966517016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0162000 msgr2=0x7fdcc00d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.060+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/2966517016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0162000 0x7fdcc00d4a80 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009830 tx=0x7fdcb002a5a0 comp rx=0 tx=0).stop 2024-09-10T20:46:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.061+0000 7fdcc68cd640 1 -- 172.21.15.28:0/2966517016 shutdown_connections 2024-09-10T20:46:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.061+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/2966517016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0162000 0x7fdcc00d4a80 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.061+0000 7fdcc68cd640 1 -- 172.21.15.28:0/2966517016 >> 172.21.15.28:0/2966517016 conn(0x7fdcc00d3f90 msgr2=0x7fdcc00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.061+0000 7fdcc68cd640 1 -- 172.21.15.28:0/2966517016 shutdown_connections 2024-09-10T20:46:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.061+0000 7fdcc68cd640 1 -- 172.21.15.28:0/2966517016 wait complete. 2024-09-10T20:46:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 Processor -- start 2024-09-10T20:46:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 -- start start 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc00b4c20 0x7fdcc01f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0161850 0x7fdcc01f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc0162000 0x7fdcc00cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc00d2370 con 0x7fdcc0161850 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc00d24f0 con 0x7fdcc0162000 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.062+0000 7fdcc68cd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc00d2670 con 0x7fdcc00b4c20 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0161850 0x7fdcc01f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0161850 0x7fdcc01f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32964/0 (socket says 172.21.15.28:32964) 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 -- 172.21.15.28:0/773244435 learned_addr learned my addr 172.21.15.28:0/773244435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcc4e6a640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc0162000 0x7fdcc00cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcbffff640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc00b4c20 0x7fdcc01f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 -- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc00b4c20 msgr2=0x7fdcc01f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc00b4c20 0x7fdcc01f4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 -- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc0162000 msgr2=0x7fdcc00cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc0162000 0x7fdcc00cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 -- 172.21.15.28:0/773244435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc015da80 con 0x7fdcc0161850 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcc4e6a640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc0162000 0x7fdcc00cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcbffff640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc00b4c20 0x7fdcc01f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcb7fff640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0161850 0x7fdcc01f21d0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fdca800c430 tx=0x7fdca800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcbd7fa640 1 -- 172.21.15.28:0/773244435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca8011790 con 0x7fdcc0161850 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcbd7fa640 1 -- 172.21.15.28:0/773244435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdca8017070 con 0x7fdcc0161850 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc015dd60 con 0x7fdcc0161850 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcbd7fa640 1 -- 172.21.15.28:0/773244435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca8011d00 con 0x7fdcc0161850 2024-09-10T20:46:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.063+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc015e320 con 0x7fdcc0161850 2024-09-10T20:46:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.064+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc00b9f10 con 0x7fdcc0161850 2024-09-10T20:46:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.065+0000 7fdcbd7fa640 1 -- 172.21.15.28:0/773244435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fdca801c460 con 0x7fdcc0161850 2024-09-10T20:46:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.066+0000 7fdcbd7fa640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdca0078990 0x7fdca007ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.066+0000 7fdcbffff640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdca0078990 0x7fdca007ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:01.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.066+0000 7fdcbd7fa640 1 -- 172.21.15.28:0/773244435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4885+0+0 (secure 0 0 0) 0x7fdca8098800 con 0x7fdcc0161850 2024-09-10T20:46:01.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.066+0000 7fdcbffff640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdca0078990 0x7fdca007ae50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0034000 tx=0x7fdcb0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:01.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.070+0000 7fdcbd7fa640 1 -- 172.21.15.28:0/773244435 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdca8062b30 con 0x7fdcc0161850 2024-09-10T20:46:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:01.257+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fdcc00cedf0 con 0x7fdcc0161850 2024-09-10T20:46:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.248+0000 7fdcbd7fa640 1 -- 172.21.15.28:0/773244435 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v159) ==== 138+0+0 (secure 0 0 0) 0x7fdca806b250 con 0x7fdcc0161850 2024-09-10T20:46:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-10T20:46:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.252+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdca0078990 msgr2=0x7fdca007ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdca0078990 0x7fdca007ae50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0034000 tx=0x7fdcb0004730 comp rx=0 tx=0).stop 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0161850 msgr2=0x7fdcc01f21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0161850 0x7fdcc01f21d0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fdca800c430 tx=0x7fdca800c8f0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 shutdown_connections 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdca0078990 0x7fdca007ae50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdcc0162000 0x7fdcc00cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc0161850 0x7fdcc01f21d0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 --2- 172.21.15.28:0/773244435 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdcc00b4c20 0x7fdcc01f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 >> 172.21.15.28:0/773244435 conn(0x7fdcc00d3f90 msgr2=0x7fdcc0156020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 shutdown_connections 2024-09-10T20:46:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.253+0000 7fdcc68cd640 1 -- 172.21.15.28:0/773244435 wait complete. 2024-09-10T20:46:02.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-10T20:46:02.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 -- 172.21.15.28:0/1316544593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 msgr2=0x7fcab40afe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:02.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 --2- 172.21.15.28:0/1316544593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab40afe30 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8005fd0 tx=0x7fcaa802a540 comp rx=0 tx=0).stop 2024-09-10T20:46:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 -- 172.21.15.28:0/1316544593 shutdown_connections 2024-09-10T20:46:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 --2- 172.21.15.28:0/1316544593 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcab40b0370 0x7fcab40cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 --2- 172.21.15.28:0/1316544593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab40afe30 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 -- 172.21.15.28:0/1316544593 >> 172.21.15.28:0/1316544593 conn(0x7fcab40b2920 msgr2=0x7fcab40b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 -- 172.21.15.28:0/1316544593 shutdown_connections 2024-09-10T20:46:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.389+0000 7fcabaf64640 1 -- 172.21.15.28:0/1316544593 wait complete. 2024-09-10T20:46:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.390+0000 7fcabaf64640 1 Processor -- start 2024-09-10T20:46:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.391+0000 7fcabaf64640 1 -- start start 2024-09-10T20:46:02.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.391+0000 7fcabaf64640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcab40b0370 0x7fcab41f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:02.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.391+0000 7fcabaf64640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcab40cf0f0 0x7fcab409ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:02.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.391+0000 7fcabaf64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab409b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:02.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.391+0000 7fcabaf64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcab415c690 con 0x7fcab4156640 2024-09-10T20:46:02.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.391+0000 7fcabaf64640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcab415c510 con 0x7fcab40cf0f0 2024-09-10T20:46:02.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.391+0000 7fcabaf64640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcab415c810 con 0x7fcab40b0370 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab409b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab409b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32984/0 (socket says 172.21.15.28:32984) 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 -- 172.21.15.28:0/2386021671 learned_addr learned my addr 172.21.15.28:0/2386021671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab8d00640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcab40b0370 0x7fcab41f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab3fff640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcab40cf0f0 0x7fcab409ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 -- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcab40b0370 msgr2=0x7fcab41f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcab40b0370 0x7fcab41f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 -- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcab40cf0f0 msgr2=0x7fcab409ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcab40cf0f0 0x7fcab409ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 -- 172.21.15.28:0/2386021671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab41f2950 con 0x7fcab4156640 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab9501640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab409b050 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00c7a0 tx=0x7fcaac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab17fa640 1 -- 172.21.15.28:0/2386021671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac003b70 con 0x7fcab4156640 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab17fa640 1 -- 172.21.15.28:0/2386021671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcaac003d10 con 0x7fcab4156640 2024-09-10T20:46:02.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab41f2c30 con 0x7fcab4156640 2024-09-10T20:46:02.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcab17fa640 1 -- 172.21.15.28:0/2386021671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac00aab0 con 0x7fcab4156640 2024-09-10T20:46:02.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.392+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab41f31f0 con 0x7fcab4156640 2024-09-10T20:46:02.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.394+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab415cda0 con 0x7fcab4156640 2024-09-10T20:46:02.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.394+0000 7fcab17fa640 1 -- 172.21.15.28:0/2386021671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fcaac00ac50 con 0x7fcab4156640 2024-09-10T20:46:02.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.395+0000 7fcab17fa640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca98078990 0x7fca9807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:02.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.395+0000 7fcab17fa640 1 -- 172.21.15.28:0/2386021671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 5255+0+0 (secure 0 0 0) 0x7fcaac0986f0 con 0x7fcab4156640 2024-09-10T20:46:02.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.395+0000 7fcab8d00640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca98078990 0x7fca9807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:02.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.395+0000 7fcab8d00640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca98078990 0x7fca9807ae50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8033000 tx=0x7fcaa8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.399+0000 7fcab17fa640 1 -- 172.21.15.28:0/2386021671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcaac062a80 con 0x7fcab4156640 2024-09-10T20:46:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:02.586+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fcab40cecd0 con 0x7fcab4156640 2024-09-10T20:46:02.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:02.735 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:03.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.255+0000 7fcab17fa640 1 -- 172.21.15.28:0/2386021671 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v160) ==== 170+0+0 (secure 0 0 0) 0x7fcaac06b1a0 con 0x7fcab4156640 2024-09-10T20:46:03.260 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca98078990 msgr2=0x7fca9807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca98078990 0x7fca9807ae50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8033000 tx=0x7fcaa8004730 comp rx=0 tx=0).stop 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 msgr2=0x7fcab409b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab409b050 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00c7a0 tx=0x7fcaac00cc60 comp rx=0 tx=0).stop 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 shutdown_connections 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fca98078990 0x7fca9807ae50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4156640 0x7fcab409b050 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcab40cf0f0 0x7fcab409ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 --2- 172.21.15.28:0/2386021671 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcab40b0370 0x7fcab41f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 >> 172.21.15.28:0/2386021671 conn(0x7fcab40b2920 msgr2=0x7fcab41553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 shutdown_connections 2024-09-10T20:46:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.259+0000 7fcabaf64640 1 -- 172.21.15.28:0/2386021671 wait complete. 2024-09-10T20:46:03.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-10T20:46:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.397+0000 7f4a3a67f640 1 -- 172.21.15.28:0/1728460114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1c004800 msgr2=0x7f4a1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.397+0000 7f4a3a67f640 1 --2- 172.21.15.28:0/1728460114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1c004800 0x7f4a1c004c80 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009b20 tx=0x7f4a1c005830 comp rx=0 tx=0).stop 2024-09-10T20:46:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.397+0000 7f4a38c1c640 1 -- 172.21.15.28:0/1728460114 reap_dead start 2024-09-10T20:46:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.397+0000 7f4a3a67f640 1 -- 172.21.15.28:0/1728460114 shutdown_connections 2024-09-10T20:46:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.397+0000 7f4a3a67f640 1 -- 172.21.15.28:0/1728460114 >> 172.21.15.28:0/1728460114 conn(0x7f4a340d0d50 msgr2=0x7f4a340d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.397+0000 7f4a3a67f640 1 -- 172.21.15.28:0/1728460114 shutdown_connections 2024-09-10T20:46:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.397+0000 7f4a3a67f640 1 -- 172.21.15.28:0/1728460114 wait complete. 2024-09-10T20:46:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.398+0000 7f4a3a67f640 1 Processor -- start 2024-09-10T20:46:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.398+0000 7f4a3a67f640 1 -- start start 2024-09-10T20:46:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a3a67f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a1c004800 0x7f4a34146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a3a67f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a340cfdc0 0x7f4a341f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a3a67f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3415d3d0 0x7f4a3415d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a3a67f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a34070a70 con 0x7f4a340cfdc0 2024-09-10T20:46:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a3a67f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4a340708f0 con 0x7f4a1c004800 2024-09-10T20:46:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a3a67f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3415f000 con 0x7f4a3415d3d0 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a340cfdc0 0x7f4a341f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a340cfdc0 0x7f4a341f3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33000/0 (socket says 172.21.15.28:33000) 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 -- 172.21.15.28:0/2305147391 learned_addr learned my addr 172.21.15.28:0/2305147391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a33fff640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a1c004800 0x7f4a34146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a38c1c640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3415d3d0 0x7f4a3415d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 -- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3415d3d0 msgr2=0x7f4a3415d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3415d3d0 0x7f4a3415d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 -- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a1c004800 msgr2=0x7f4a34146610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a1c004800 0x7f4a34146610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.399+0000 7f4a337fe640 1 -- 172.21.15.28:0/2305147391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a340c8a50 con 0x7f4a340cfdc0 2024-09-10T20:46:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.400+0000 7f4a337fe640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a340cfdc0 0x7f4a341f3340 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009c40 tx=0x7f4a28033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.400+0000 7f4a30ff9640 1 -- 172.21.15.28:0/2305147391 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28038040 con 0x7f4a340cfdc0 2024-09-10T20:46:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.400+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a341f3c20 con 0x7f4a340cfdc0 2024-09-10T20:46:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.400+0000 7f4a30ff9640 1 -- 172.21.15.28:0/2305147391 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a2802b7b0 con 0x7f4a340cfdc0 2024-09-10T20:46:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.400+0000 7f4a30ff9640 1 -- 172.21.15.28:0/2305147391 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28004280 con 0x7f4a340cfdc0 2024-09-10T20:46:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.400+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a341f41e0 con 0x7f4a340cfdc0 2024-09-10T20:46:03.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.401+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3415dd90 con 0x7f4a340cfdc0 2024-09-10T20:46:03.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.402+0000 7f4a30ff9640 1 -- 172.21.15.28:0/2305147391 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f4a2803b030 con 0x7f4a340cfdc0 2024-09-10T20:46:03.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.402+0000 7f4a30ff9640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a20078990 0x7f4a2007ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:03.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.402+0000 7f4a30ff9640 1 -- 172.21.15.28:0/2305147391 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5637+0+0 (secure 0 0 0) 0x7f4a280b99c0 con 0x7f4a340cfdc0 2024-09-10T20:46:03.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.402+0000 7f4a33fff640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a20078990 0x7f4a2007ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:03.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.403+0000 7f4a33fff640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a20078990 0x7f4a2007ae50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009b70 tx=0x7f4a24005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.406+0000 7f4a30ff9640 1 -- 172.21.15.28:0/2305147391 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4a2802b980 con 0x7f4a340cfdc0 2024-09-10T20:46:03.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:03.627+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4a340c9ec0 con 0x7f4a340cfdc0 2024-09-10T20:46:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.257+0000 7f4a30ff9640 1 -- 172.21.15.28:0/2305147391 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v112) ==== 174+0+0 (secure 0 0 0) 0x7f4a340c9ec0 con 0x7f4a340cfdc0 2024-09-10T20:46:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 26 and data pool 27 2024-09-10T20:46:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.266+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a20078990 msgr2=0x7f4a2007ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.266+0000 7f4a3a67f640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a20078990 0x7f4a2007ae50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009b70 tx=0x7f4a24005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.266+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a340cfdc0 msgr2=0x7f4a341f3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.266+0000 7f4a3a67f640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a340cfdc0 0x7f4a341f3340 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009c40 tx=0x7f4a28033670 comp rx=0 tx=0).stop 2024-09-10T20:46:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.270+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 shutdown_connections 2024-09-10T20:46:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.271+0000 7f4a3a67f640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a20078990 0x7f4a2007ae50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.271+0000 7f4a3a67f640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a3415d3d0 0x7f4a3415d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.271+0000 7f4a3a67f640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a340cfdc0 0x7f4a341f3340 secure :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009c40 tx=0x7f4a28033670 comp rx=0 tx=0).stop 2024-09-10T20:46:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.271+0000 7f4a3a67f640 1 --2- 172.21.15.28:0/2305147391 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a1c004800 0x7f4a34146610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.271+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 >> 172.21.15.28:0/2305147391 conn(0x7f4a340d0d50 msgr2=0x7f4a340ccad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.273+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 shutdown_connections 2024-09-10T20:46:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.274+0000 7f4a3a67f640 1 -- 172.21.15.28:0/2305147391 wait complete. 2024-09-10T20:46:04.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:46:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.411+0000 7fc841f16640 1 -- 172.21.15.28:0/3208693376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc824003a50 msgr2=0x7fc824003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.411+0000 7fc841f16640 1 --2- 172.21.15.28:0/3208693376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc824003a50 0x7fc824003e90 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c310 tx=0x7fc824005830 comp rx=0 tx=0).stop 2024-09-10T20:46:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.411+0000 7fc841f16640 1 -- 172.21.15.28:0/3208693376 shutdown_connections 2024-09-10T20:46:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.411+0000 7fc841f16640 1 --2- 172.21.15.28:0/3208693376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc824003a50 0x7fc824003e90 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.411+0000 7fc841f16640 1 -- 172.21.15.28:0/3208693376 >> 172.21.15.28:0/3208693376 conn(0x7fc83c0d0420 msgr2=0x7fc83c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.412+0000 7fc841f16640 1 -- 172.21.15.28:0/3208693376 shutdown_connections 2024-09-10T20:46:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.412+0000 7fc841f16640 1 -- 172.21.15.28:0/3208693376 wait complete. 2024-09-10T20:46:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.413+0000 7fc841f16640 1 Processor -- start 2024-09-10T20:46:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.413+0000 7fc841f16640 1 -- start start 2024-09-10T20:46:04.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc841f16640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc824003a50 0x7fc83c148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc841f16640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83c0b0290 0x7fc83c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc841f16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc83c0b0710 0x7fc83c0c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc841f16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c15f430 con 0x7fc83c0b0710 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc841f16640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c152150 con 0x7fc83c0b0290 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc841f16640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc83c15f5b0 con 0x7fc824003a50 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc83c0b0710 0x7fc83c0c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc83c0b0710 0x7fc83c0c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33016/0 (socket says 172.21.15.28:33016) 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 -- 172.21.15.28:0/376083674 learned_addr learned my addr 172.21.15.28:0/376083674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83b7fe640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc824003a50 0x7fc83c148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83affd640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83c0b0290 0x7fc83c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 -- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc824003a50 msgr2=0x7fc83c148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc824003a50 0x7fc83c148250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 -- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83c0b0290 msgr2=0x7fc83c0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83c0b0290 0x7fc83c0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 -- 172.21.15.28:0/376083674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c1f5b90 con 0x7fc83c0b0710 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.414+0000 7fc83bfff640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc83c0b0710 0x7fc83c0c9cb0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fc8340068d0 tx=0x7fc8340019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.415+0000 7fc823fff640 1 -- 172.21.15.28:0/376083674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834015020 con 0x7fc83c0b0710 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.415+0000 7fc823fff640 1 -- 172.21.15.28:0/376083674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc834011960 con 0x7fc83c0b0710 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.415+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c0b3640 con 0x7fc83c0b0710 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.415+0000 7fc823fff640 1 -- 172.21.15.28:0/376083674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834011d90 con 0x7fc83c0b0710 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.415+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c0b3c00 con 0x7fc83c0b0710 2024-09-10T20:46:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.416+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83c15ea60 con 0x7fc83c0b0710 2024-09-10T20:46:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.416+0000 7fc823fff640 1 -- 172.21.15.28:0/376083674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fc83401a400 con 0x7fc83c0b0710 2024-09-10T20:46:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.417+0000 7fc823fff640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc828078990 0x7fc82807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.417+0000 7fc823fff640 1 -- 172.21.15.28:0/376083674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 5737+0+0 (secure 0 0 0) 0x7fc834099410 con 0x7fc83c0b0710 2024-09-10T20:46:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.417+0000 7fc83b7fe640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc828078990 0x7fc82807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.417+0000 7fc83b7fe640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc828078990 0x7fc82807ae50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc82c033000 tx=0x7fc82c038040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:04.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.421+0000 7fc823fff640 1 -- 172.21.15.28:0/376083674 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc83401f020 con 0x7fc83c0b0710 2024-09-10T20:46:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.610+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc83c068d10 con 0x7fc83c0b0710 2024-09-10T20:46:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.611+0000 7fc823fff640 1 -- 172.21.15.28:0/376083674 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) ==== 74+0+16732 (secure 0 0 0) 0x7fc834063150 con 0x7fc83c0b0710 2024-09-10T20:46:04.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:04.616 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":161,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:46:03.628130+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:46:01.262178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-10T20:46:02.591573+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.613+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc828078990 msgr2=0x7fc82807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.613+0000 7fc841f16640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc828078990 0x7fc82807ae50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc82c033000 tx=0x7fc82c038040 comp rx=0 tx=0).stop 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.613+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc83c0b0710 msgr2=0x7fc83c0c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.613+0000 7fc841f16640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc83c0b0710 0x7fc83c0c9cb0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fc8340068d0 tx=0x7fc8340019b0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 shutdown_connections 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc828078990 0x7fc82807ae50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc83c0b0710 0x7fc83c0c9cb0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc83c0b0290 0x7fc83c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 --2- 172.21.15.28:0/376083674 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc824003a50 0x7fc83c148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 >> 172.21.15.28:0/376083674 conn(0x7fc83c0d0420 msgr2=0x7fc83c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:04.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 shutdown_connections 2024-09-10T20:46:04.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.614+0000 7fc841f16640 1 -- 172.21.15.28:0/376083674 wait complete. 2024-09-10T20:46:04.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:46:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 -- 172.21.15.28:0/172210627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 msgr2=0x7fb16015e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 --2- 172.21.15.28:0/172210627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 0x7fb16015e930 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb154009830 tx=0x7fb15402a510 comp rx=0 tx=0).stop 2024-09-10T20:46:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 -- 172.21.15.28:0/172210627 shutdown_connections 2024-09-10T20:46:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 --2- 172.21.15.28:0/172210627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 0x7fb16015e930 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 --2- 172.21.15.28:0/172210627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb160155100 0x7fb16015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 -- 172.21.15.28:0/172210627 >> 172.21.15.28:0/172210627 conn(0x7fb16006cf00 msgr2=0x7fb16006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 -- 172.21.15.28:0/172210627 shutdown_connections 2024-09-10T20:46:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.730+0000 7fb166865640 1 -- 172.21.15.28:0/172210627 wait complete. 2024-09-10T20:46:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.731+0000 7fb166865640 1 Processor -- start 2024-09-10T20:46:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb166865640 1 -- start start 2024-09-10T20:46:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb166865640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160155100 0x7fb1601f1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb166865640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 0x7fb1600afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb166865640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb16015ee70 0x7fb160155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb166865640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb160072f60 con 0x7fb160155100 2024-09-10T20:46:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb166865640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb160072de0 con 0x7fb1601554e0 2024-09-10T20:46:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb166865640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb1600730e0 con 0x7fb16015ee70 2024-09-10T20:46:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb15ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160155100 0x7fb1601f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb15ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160155100 0x7fb1601f1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33020/0 (socket says 172.21.15.28:33020) 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.732+0000 7fb15ffff640 1 -- 172.21.15.28:0/1693361823 learned_addr learned my addr 172.21.15.28:0/1693361823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15f7fe640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 0x7fb1600afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb164e02640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb16015ee70 0x7fb160155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15ffff640 1 -- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb16015ee70 msgr2=0x7fb160155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15ffff640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb16015ee70 0x7fb160155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15ffff640 1 -- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 msgr2=0x7fb1600afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15ffff640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 0x7fb1600afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15ffff640 1 -- 172.21.15.28:0/1693361823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1600b01f0 con 0x7fb160155100 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15f7fe640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 0x7fb1600afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb164e02640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb16015ee70 0x7fb160155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15ffff640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160155100 0x7fb1601f1600 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb15000e790 tx=0x7fb15000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15cff9640 1 -- 172.21.15.28:0/1693361823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15000eea0 con 0x7fb160155100 2024-09-10T20:46:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15cff9640 1 -- 172.21.15.28:0/1693361823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb15000cdb0 con 0x7fb160155100 2024-09-10T20:46:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1600b04d0 con 0x7fb160155100 2024-09-10T20:46:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb15cff9640 1 -- 172.21.15.28:0/1693361823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb150010690 con 0x7fb160155100 2024-09-10T20:46:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.733+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1600b0a90 con 0x7fb160155100 2024-09-10T20:46:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.734+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb160073d50 con 0x7fb160155100 2024-09-10T20:46:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.735+0000 7fb15cff9640 1 -- 172.21.15.28:0/1693361823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fb150010830 con 0x7fb160155100 2024-09-10T20:46:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.735+0000 7fb15cff9640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb14c078990 0x7fb14c07ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.736+0000 7fb15cff9640 1 -- 172.21.15.28:0/1693361823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 5737+0+0 (secure 0 0 0) 0x7fb150099300 con 0x7fb160155100 2024-09-10T20:46:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.736+0000 7fb15f7fe640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb14c078990 0x7fb14c07ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.736+0000 7fb15f7fe640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb14c078990 0x7fb14c07ae50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb154034000 tx=0x7fb154004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.740+0000 7fb15cff9640 1 -- 172.21.15.28:0/1693361823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb150063690 con 0x7fb160155100 2024-09-10T20:46:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.927+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb160070880 con 0x7fb160155100 2024-09-10T20:46:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.928+0000 7fb15cff9640 1 -- 172.21.15.28:0/1693361823 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) ==== 74+0+16732 (secure 0 0 0) 0x7fb150062f90 con 0x7fb160155100 2024-09-10T20:46:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:04.933 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":161,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:46:03.628130+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:46:01.262178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-10T20:46:02.591573+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:46:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb14c078990 msgr2=0x7fb14c07ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb14c078990 0x7fb14c07ae50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb154034000 tx=0x7fb154004730 comp rx=0 tx=0).stop 2024-09-10T20:46:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160155100 msgr2=0x7fb1601f1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160155100 0x7fb1601f1600 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb15000e790 tx=0x7fb15000ec50 comp rx=0 tx=0).stop 2024-09-10T20:46:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 shutdown_connections 2024-09-10T20:46:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb14c078990 0x7fb14c07ae50 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb16015ee70 0x7fb160155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb1601554e0 0x7fb1600afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 --2- 172.21.15.28:0/1693361823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160155100 0x7fb1601f1600 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 >> 172.21.15.28:0/1693361823 conn(0x7fb16006cf00 msgr2=0x7fb16006ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 shutdown_connections 2024-09-10T20:46:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:04.931+0000 7fb166865640 1 -- 172.21.15.28:0/1693361823 wait complete. 2024-09-10T20:46:04.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-10T20:46:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.052+0000 7f120262a640 1 -- 172.21.15.28:0/753861059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e4003a80 msgr2=0x7f11e4003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.052+0000 7f120262a640 1 --2- 172.21.15.28:0/753861059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e4003a80 0x7f11e4003ea0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f11ec00bb50 tx=0x7f11e4005830 comp rx=0 tx=0).stop 2024-09-10T20:46:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.052+0000 7f1200bc7640 1 -- 172.21.15.28:0/753861059 reap_dead start 2024-09-10T20:46:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.052+0000 7f120262a640 1 -- 172.21.15.28:0/753861059 shutdown_connections 2024-09-10T20:46:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.052+0000 7f120262a640 1 -- 172.21.15.28:0/753861059 >> 172.21.15.28:0/753861059 conn(0x7f11fc0d4070 msgr2=0x7f11fc0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.052+0000 7f120262a640 1 -- 172.21.15.28:0/753861059 shutdown_connections 2024-09-10T20:46:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.052+0000 7f120262a640 1 -- 172.21.15.28:0/753861059 wait complete. 2024-09-10T20:46:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.053+0000 7f120262a640 1 Processor -- start 2024-09-10T20:46:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.053+0000 7f120262a640 1 -- start start 2024-09-10T20:46:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f120262a640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11e4003a80 0x7f11fc0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f120262a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc06e610 0x7f11fc1576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f120262a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11fc0c8a70 0x7f11fc0b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f120262a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc0b2dd0 con 0x7f11fc06e610 2024-09-10T20:46:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f120262a640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc0b2a50 con 0x7f11fc0c8a70 2024-09-10T20:46:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f120262a640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f11fc0b2f50 con 0x7f11e4003a80 2024-09-10T20:46:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f11fb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc06e610 0x7f11fc1576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f11fb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc06e610 0x7f11fc1576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33034/0 (socket says 172.21.15.28:33034) 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f11fb7fe640 1 -- 172.21.15.28:0/1235954270 learned_addr learned my addr 172.21.15.28:0/1235954270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f1200bc7640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11fc0c8a70 0x7f11fc0b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f11fbfff640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11e4003a80 0x7f11fc0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.054+0000 7f11fb7fe640 1 -- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11e4003a80 msgr2=0x7f11fc0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11fb7fe640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11e4003a80 0x7f11fc0d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11fb7fe640 1 -- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11fc0c8a70 msgr2=0x7f11fc0b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11fb7fe640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11fc0c8a70 0x7f11fc0b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11fb7fe640 1 -- 172.21.15.28:0/1235954270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc0cd5d0 con 0x7f11fc06e610 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11fbfff640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11e4003a80 0x7f11fc0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11fb7fe640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc06e610 0x7f11fc1576d0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f11f0000fb0 tx=0x7f11f000ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11f8ff9640 1 -- 172.21.15.28:0/1235954270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0015070 con 0x7f11fc06e610 2024-09-10T20:46:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11f8ff9640 1 -- 172.21.15.28:0/1235954270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11f0010980 con 0x7f11fc06e610 2024-09-10T20:46:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11fc0ca840 con 0x7f11fc06e610 2024-09-10T20:46:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f11f8ff9640 1 -- 172.21.15.28:0/1235954270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0010db0 con 0x7f11fc06e610 2024-09-10T20:46:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.055+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11fc0cadc0 con 0x7f11fc06e610 2024-09-10T20:46:05.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.056+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11fc0d21c0 con 0x7f11fc06e610 2024-09-10T20:46:05.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.057+0000 7f11f8ff9640 1 -- 172.21.15.28:0/1235954270 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f11f0011460 con 0x7f11fc06e610 2024-09-10T20:46:05.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.057+0000 7f11f8ff9640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f11e80789c0 0x7f11e807ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:05.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.058+0000 7f11fbfff640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f11e80789c0 0x7f11e807ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:05.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.058+0000 7f11f8ff9640 1 -- 172.21.15.28:0/1235954270 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 5737+0+0 (secure 0 0 0) 0x7f11f0098e30 con 0x7f11fc06e610 2024-09-10T20:46:05.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.058+0000 7f11fbfff640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f11e80789c0 0x7f11e807ae80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f11ec036000 tx=0x7f11ec0046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:05.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.062+0000 7f11f8ff9640 1 -- 172.21.15.28:0/1235954270 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f11fc0cd5d0 con 0x7f11fc06e610 2024-09-10T20:46:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:05.284+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f11fc15ff10 con 0x7f11fc06e610 2024-09-10T20:46:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.288+0000 7f11f8ff9640 1 -- 172.21.15.28:0/1235954270 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v115) ==== 118+0+0 (secure 0 0 0) 0x7f11f0062b70 con 0x7f11fc06e610 2024-09-10T20:46:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f11e80789c0 msgr2=0x7f11e807ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f11e80789c0 0x7f11e807ae80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f11ec036000 tx=0x7f11ec0046c0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc06e610 msgr2=0x7f11fc1576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc06e610 0x7f11fc1576d0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f11f0000fb0 tx=0x7f11f000ea00 comp rx=0 tx=0).stop 2024-09-10T20:46:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 shutdown_connections 2024-09-10T20:46:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f11e80789c0 0x7f11e807ae80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f11fc0c8a70 0x7f11fc0b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc06e610 0x7f11fc1576d0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 --2- 172.21.15.28:0/1235954270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11e4003a80 0x7f11fc0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 >> 172.21.15.28:0/1235954270 conn(0x7f11fc0d4070 msgr2=0x7f11fc0b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 shutdown_connections 2024-09-10T20:46:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.291+0000 7f120262a640 1 -- 172.21.15.28:0/1235954270 wait complete. 2024-09-10T20:46:06.309 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-10T20:46:06.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-10T20:46:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.422+0000 7fadd0d5d640 1 -- 172.21.15.28:0/3772022558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 msgr2=0x7fadb8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.422+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/3772022558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadb8003e70 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fadc400c640 tx=0x7fadb8005830 comp rx=0 tx=0).stop 2024-09-10T20:46:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.422+0000 7fadd0d5d640 1 -- 172.21.15.28:0/3772022558 shutdown_connections 2024-09-10T20:46:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.422+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/3772022558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadb8003e70 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.422+0000 7fadd0d5d640 1 -- 172.21.15.28:0/3772022558 >> 172.21.15.28:0/3772022558 conn(0x7fadcc0d4070 msgr2=0x7fadcc0d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.422+0000 7fadd0d5d640 1 -- 172.21.15.28:0/3772022558 shutdown_connections 2024-09-10T20:46:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.422+0000 7fadd0d5d640 1 -- 172.21.15.28:0/3772022558 wait complete. 2024-09-10T20:46:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 Processor -- start 2024-09-10T20:46:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 -- start start 2024-09-10T20:46:06.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadcc0b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:06.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadcc15da80 0x7fadcc0c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:06.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadcc0c8af0 0x7fadcc15f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:06.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc0d4e00 con 0x7fadb8003a50 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc0b2fe0 con 0x7fadcc15da80 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadd0d5d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fadcc0d20e0 con 0x7fadcc0c8af0 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadcc0b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.424+0000 7fadca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadcc0b7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33058/0 (socket says 172.21.15.28:33058) 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadca59c640 1 -- 172.21.15.28:0/2046261717 learned_addr learned my addr 172.21.15.28:0/2046261717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadca59c640 1 -- 172.21.15.28:0/2046261717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadcc0c8af0 msgr2=0x7fadcc15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadcad9d640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadcc0c8af0 0x7fadcc15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadca59c640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadcc0c8af0 0x7fadcc15f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadca59c640 1 -- 172.21.15.28:0/2046261717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadcc15da80 msgr2=0x7fadcc0c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadca59c640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadcc15da80 0x7fadcc0c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadca59c640 1 -- 172.21.15.28:0/2046261717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadcc0b8410 con 0x7fadb8003a50 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadca59c640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadcc0b7ed0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fadc4000f80 tx=0x7fadc4001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:06.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadc2ffd640 1 -- 172.21.15.28:0/2046261717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc402bbb0 con 0x7fadb8003a50 2024-09-10T20:46:06.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadc2ffd640 1 -- 172.21.15.28:0/2046261717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadc403a070 con 0x7fadb8003a50 2024-09-10T20:46:06.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadc2ffd640 1 -- 172.21.15.28:0/2046261717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc40062d0 con 0x7fadb8003a50 2024-09-10T20:46:06.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadcc1f8460 con 0x7fadb8003a50 2024-09-10T20:46:06.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.425+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadcc1f8a20 con 0x7fadb8003a50 2024-09-10T20:46:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.426+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadcc0b3870 con 0x7fadb8003a50 2024-09-10T20:46:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.427+0000 7fadc2ffd640 1 -- 172.21.15.28:0/2046261717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fadc4006470 con 0x7fadb8003a50 2024-09-10T20:46:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.427+0000 7fadc2ffd640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fadac0789c0 0x7fadac07ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.427+0000 7fadc2ffd640 1 -- 172.21.15.28:0/2046261717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7fadc40b88e0 con 0x7fadb8003a50 2024-09-10T20:46:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.428+0000 7fadc9d9b640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fadac0789c0 0x7fadac07ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:06.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.428+0000 7fadc9d9b640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fadac0789c0 0x7fadac07ae80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fadb4005fd0 tx=0x7fadb4005ae0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.432+0000 7fadc2ffd640 1 -- 172.21.15.28:0/2046261717 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fadc4082c70 con 0x7fadb8003a50 2024-09-10T20:46:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:06.655+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fadcc0b18d0 con 0x7fadb8003a50 2024-09-10T20:46:07.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.294+0000 7fadc2ffd640 1 -- 172.21.15.28:0/2046261717 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v116) ==== 122+0+0 (secure 0 0 0) 0x7fadc408b390 con 0x7fadb8003a50 2024-09-10T20:46:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fadac0789c0 msgr2=0x7fadac07ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fadac0789c0 0x7fadac07ae80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fadb4005fd0 tx=0x7fadb4005ae0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 msgr2=0x7fadcc0b7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadcc0b7ed0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fadc4000f80 tx=0x7fadc4001b20 comp rx=0 tx=0).stop 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 shutdown_connections 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fadac0789c0 0x7fadac07ae80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadcc0c8af0 0x7fadcc15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadcc15da80 0x7fadcc0c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 --2- 172.21.15.28:0/2046261717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8003a50 0x7fadcc0b7ed0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 >> 172.21.15.28:0/2046261717 conn(0x7fadcc0d4070 msgr2=0x7fadcc0b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 shutdown_connections 2024-09-10T20:46:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.297+0000 7fadd0d5d640 1 -- 172.21.15.28:0/2046261717 wait complete. 2024-09-10T20:46:07.315 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-10T20:46:07.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:46:07.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 -- 172.21.15.28:0/3395935118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 msgr2=0x7f41880cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 --2- 172.21.15.28:0/3395935118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f41880cf0f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4178005fd0 tx=0x7f417802a520 comp rx=0 tx=0).stop 2024-09-10T20:46:07.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 -- 172.21.15.28:0/3395935118 shutdown_connections 2024-09-10T20:46:07.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 --2- 172.21.15.28:0/3395935118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f41880738b0 0x7f418815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 --2- 172.21.15.28:0/3395935118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f41880cf0f0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 -- 172.21.15.28:0/3395935118 >> 172.21.15.28:0/3395935118 conn(0x7f41880d0420 msgr2=0x7f41880d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 -- 172.21.15.28:0/3395935118 shutdown_connections 2024-09-10T20:46:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.423+0000 7f418e672640 1 -- 172.21.15.28:0/3395935118 wait complete. 2024-09-10T20:46:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.424+0000 7f418e672640 1 Processor -- start 2024-09-10T20:46:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.424+0000 7f418e672640 1 -- start start 2024-09-10T20:46:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418e672640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41880738b0 0x7f4188148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418e672640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4188073cf0 0x7f41881573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418e672640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f418806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418e672640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41880cb920 con 0x7f4188151cc0 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418e672640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f418815f640 con 0x7f41880738b0 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418e672640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f41880cbaa0 con 0x7f4188073cf0 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418cc0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f418806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418cc0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f418806ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33072/0 (socket says 172.21.15.28:33072) 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f418cc0f640 1 -- 172.21.15.28:0/3651969737 learned_addr learned my addr 172.21.15.28:0/3651969737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f4187fff640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41880738b0 0x7f4188148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.425+0000 7f417ffff640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4188073cf0 0x7f41881573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418cc0f640 1 -- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4188073cf0 msgr2=0x7f41881573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418cc0f640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4188073cf0 0x7f41881573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418cc0f640 1 -- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41880738b0 msgr2=0x7f4188148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418cc0f640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41880738b0 0x7f4188148270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418cc0f640 1 -- 172.21.15.28:0/3651969737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41881487b0 con 0x7f4188151cc0 2024-09-10T20:46:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418cc0f640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f418806ff80 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f418000c7b0 tx=0x7f418000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f41857fa640 1 -- 172.21.15.28:0/3651969737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180017070 con 0x7f4188151cc0 2024-09-10T20:46:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f41857fa640 1 -- 172.21.15.28:0/3651969737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4180005070 con 0x7f4188151cc0 2024-09-10T20:46:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4188157db0 con 0x7f4188151cc0 2024-09-10T20:46:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f41857fa640 1 -- 172.21.15.28:0/3651969737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41800054c0 con 0x7f4188151cc0 2024-09-10T20:46:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.426+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41880b0590 con 0x7f4188151cc0 2024-09-10T20:46:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.427+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f418815ee00 con 0x7f4188151cc0 2024-09-10T20:46:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.428+0000 7f41857fa640 1 -- 172.21.15.28:0/3651969737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f4180005660 con 0x7f4188151cc0 2024-09-10T20:46:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.428+0000 7f41857fa640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4168078990 0x7f416807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.428+0000 7f41857fa640 1 -- 172.21.15.28:0/3651969737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f4180098200 con 0x7f4188151cc0 2024-09-10T20:46:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.428+0000 7f4187fff640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4168078990 0x7f416807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.429+0000 7f4187fff640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4168078990 0x7f416807ae50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4178033000 tx=0x7f4178004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.432+0000 7f41857fa640 1 -- 172.21.15.28:0/3651969737 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4180062590 con 0x7f4188151cc0 2024-09-10T20:46:07.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.654+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41880cc450 con 0x7f4188151cc0 2024-09-10T20:46:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.654+0000 7f41857fa640 1 -- 172.21.15.28:0/3651969737 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5239 (secure 0 0 0) 0x7f4180061e90 con 0x7f4188151cc0 2024-09-10T20:46:07.658 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:07.658 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":116,"btime":"2024-09-10T20:46:07:289311+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4088436065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4088436065},{"type":"v1","addr":"172.21.15.28:6834","nonce":4088436065}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2443719102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2443719102},{"type":"v1","addr":"172.21.15.159:6801","nonce":2443719102}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":7250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3153361013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3153361013},{"type":"v1","addr":"172.21.15.159:6835","nonce":3153361013}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:46:03.628094+0000","modified":"2024-09-10T20:46:06.656668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9078":{"gid":9078,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1516352866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1516352866},{"type":"v1","addr":"172.21.15.28:6835","nonce":1516352866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9078,"qdb_cluster":[9078]},"id":13}]} 2024-09-10T20:46:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 116 2024-09-10T20:46:07.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4168078990 msgr2=0x7f416807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4168078990 0x7f416807ae50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4178033000 tx=0x7f4178004730 comp rx=0 tx=0).stop 2024-09-10T20:46:07.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 msgr2=0x7f418806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f418806ff80 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f418000c7b0 tx=0x7f418000cc70 comp rx=0 tx=0).stop 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 shutdown_connections 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4168078990 0x7f416807ae50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4188151cc0 0x7f418806ff80 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4188073cf0 0x7f41881573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 --2- 172.21.15.28:0/3651969737 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f41880738b0 0x7f4188148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 >> 172.21.15.28:0/3651969737 conn(0x7f41880d0420 msgr2=0x7f41880cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 shutdown_connections 2024-09-10T20:46:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.656+0000 7f418e672640 1 -- 172.21.15.28:0/3651969737 wait complete. 2024-09-10T20:46:07.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:46:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 -- 172.21.15.28:0/887368978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d07a0 msgr2=0x7f3b800d0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 --2- 172.21.15.28:0/887368978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d07a0 0x7f3b800d0be0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009830 tx=0x7f3b7402a510 comp rx=0 tx=0).stop 2024-09-10T20:46:07.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 -- 172.21.15.28:0/887368978 shutdown_connections 2024-09-10T20:46:07.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 --2- 172.21.15.28:0/887368978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d07a0 0x7f3b800d0be0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 --2- 172.21.15.28:0/887368978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b80161640 0x7f3b800b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 -- 172.21.15.28:0/887368978 >> 172.21.15.28:0/887368978 conn(0x7f3b8009d270 msgr2=0x7f3b800d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:07.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 -- 172.21.15.28:0/887368978 shutdown_connections 2024-09-10T20:46:07.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.773+0000 7f3b85add640 1 -- 172.21.15.28:0/887368978 wait complete. 2024-09-10T20:46:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.774+0000 7f3b85add640 1 Processor -- start 2024-09-10T20:46:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.774+0000 7f3b85add640 1 -- start start 2024-09-10T20:46:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b85add640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b800d07a0 0x7f3b801f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b85add640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d1120 0x7f3b80154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b85add640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b80161640 0x7f3b801f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b85add640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b800c8d40 con 0x7f3b800d1120 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b85add640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3b800c8bc0 con 0x7f3b80161640 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b85add640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3b800d2340 con 0x7f3b800d07a0 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d1120 0x7f3b80154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d1120 0x7f3b80154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33078/0 (socket says 172.21.15.28:33078) 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 -- 172.21.15.28:0/2251305647 learned_addr learned my addr 172.21.15.28:0/2251305647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 -- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b800d07a0 msgr2=0x7f3b801f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7f7fe640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b800d07a0 0x7f3b801f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7ffff640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b80161640 0x7f3b801f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b800d07a0 0x7f3b801f5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 -- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b80161640 msgr2=0x7f3b801f84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b80161640 0x7f3b801f84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.775+0000 7f3b7effd640 1 -- 172.21.15.28:0/2251305647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b80154ad0 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.776+0000 7f3b7effd640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d1120 0x7f3b80154590 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009950 tx=0x7f3b74032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.776+0000 7f3b67fff640 1 -- 172.21.15.28:0/2251305647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74040070 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.776+0000 7f3b67fff640 1 -- 172.21.15.28:0/2251305647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b74005330 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.776+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b801f4ec0 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.776+0000 7f3b67fff640 1 -- 172.21.15.28:0/2251305647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74003800 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.776+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b801f9210 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.777+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b800b95d0 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.778+0000 7f3b67fff640 1 -- 172.21.15.28:0/2251305647 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f3b740039a0 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.778+0000 7f3b67fff640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b6c078990 0x7f3b6c07ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.778+0000 7f3b67fff640 1 -- 172.21.15.28:0/2251305647 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f3b740b9e50 con 0x7f3b800d1120 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.778+0000 7f3b7f7fe640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b6c078990 0x7f3b6c07ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.779+0000 7f3b7f7fe640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b6c078990 0x7f3b6c07ae50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3b70005fd0 tx=0x7f3b70005f40 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.782+0000 7f3b67fff640 1 -- 172.21.15.28:0/2251305647 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3b800b95d0 con 0x7f3b800d1120 2024-09-10T20:46:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.966+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b800d31a0 con 0x7f3b800d1120 2024-09-10T20:46:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.968+0000 7f3b67fff640 1 -- 172.21.15.28:0/2251305647 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) ==== 74+0+16719 (secure 0 0 0) 0x7f3b800d31a0 con 0x7f3b800d1120 2024-09-10T20:46:07.972 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:07.972 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":162,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:46:05.271026+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:46:01.262178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-10T20:46:02.591573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:46:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b6c078990 msgr2=0x7f3b6c07ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b6c078990 0x7f3b6c07ae50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3b70005fd0 tx=0x7f3b70005f40 comp rx=0 tx=0).stop 2024-09-10T20:46:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d1120 msgr2=0x7f3b80154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d1120 0x7f3b80154590 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009950 tx=0x7f3b74032d50 comp rx=0 tx=0).stop 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 shutdown_connections 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3b6c078990 0x7f3b6c07ae50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b80161640 0x7f3b801f84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b800d1120 0x7f3b80154590 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 --2- 172.21.15.28:0/2251305647 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b800d07a0 0x7f3b801f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 >> 172.21.15.28:0/2251305647 conn(0x7f3b8009d270 msgr2=0x7f3b800b8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 shutdown_connections 2024-09-10T20:46:07.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:07.970+0000 7f3b85add640 1 -- 172.21.15.28:0/2251305647 wait complete. 2024-09-10T20:46:07.986 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:46:07.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:46:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.088+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/3986950806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0003a00 msgr2=0x7f4cf0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.088+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/3986950806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0003a00 0x7f4cf0003e00 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800c260 tx=0x7f4cf0005830 comp rx=0 tx=0).stop 2024-09-10T20:46:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.088+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/3986950806 shutdown_connections 2024-09-10T20:46:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.088+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/3986950806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0003a00 0x7f4cf0003e00 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.088+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/3986950806 >> 172.21.15.28:0/3986950806 conn(0x7f4d080d1e40 msgr2=0x7f4d080d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.088+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/3986950806 shutdown_connections 2024-09-10T20:46:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.088+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/3986950806 wait complete. 2024-09-10T20:46:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.090+0000 7f4d0e6dd640 1 Processor -- start 2024-09-10T20:46:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.090+0000 7f4d0e6dd640 1 -- start start 2024-09-10T20:46:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.090+0000 7f4d0e6dd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf0003a00 0x7f4d081515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0e6dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d08071990 0x7f4d08071e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0e6dd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d0806dc60 0x7f4d0806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0e6dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d080d2960 con 0x7f4d08071990 2024-09-10T20:46:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0e6dd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4d080d0d50 con 0x7f4cf0003a00 2024-09-10T20:46:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0e6dd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4d080d2ae0 con 0x7f4d0806dc60 2024-09-10T20:46:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d08071990 0x7f4d08071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d08071990 0x7f4d08071e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33098/0 (socket says 172.21.15.28:33098) 2024-09-10T20:46:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 -- 172.21.15.28:0/748835140 learned_addr learned my addr 172.21.15.28:0/748835140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0cc7a640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d0806dc60 0x7f4d0806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d07fff640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf0003a00 0x7f4d081515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 -- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d0806dc60 msgr2=0x7f4d0806e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d0806dc60 0x7f4d0806e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 -- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf0003a00 msgr2=0x7f4d081515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf0003a00 0x7f4d081515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 -- 172.21.15.28:0/748835140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d081f8910 con 0x7f4d08071990 2024-09-10T20:46:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d077fe640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d08071990 0x7f4d08071e10 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00fa00 tx=0x7f4cfc00fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d04ff9640 1 -- 172.21.15.28:0/748835140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc015070 con 0x7f4d08071990 2024-09-10T20:46:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d04ff9640 1 -- 172.21.15.28:0/748835140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cfc004020 con 0x7f4d08071990 2024-09-10T20:46:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d080ca130 con 0x7f4d08071990 2024-09-10T20:46:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d04ff9640 1 -- 172.21.15.28:0/748835140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00be60 con 0x7f4d08071990 2024-09-10T20:46:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.091+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d081f7b50 con 0x7f4d08071990 2024-09-10T20:46:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.092+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d080cffa0 con 0x7f4d08071990 2024-09-10T20:46:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.093+0000 7f4d04ff9640 1 -- 172.21.15.28:0/748835140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f4cfc021030 con 0x7f4d08071990 2024-09-10T20:46:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.094+0000 7f4d04ff9640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cf4078990 0x7f4cf407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.094+0000 7f4d04ff9640 1 -- 172.21.15.28:0/748835140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f4cfc099270 con 0x7f4d08071990 2024-09-10T20:46:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.094+0000 7f4d07fff640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cf4078990 0x7f4cf407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.094+0000 7f4d07fff640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cf4078990 0x7f4cf407ae50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8033000 tx=0x7f4d080cda10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:08.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.098+0000 7f4d04ff9640 1 -- 172.21.15.28:0/748835140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4cfc0635a0 con 0x7f4d08071990 2024-09-10T20:46:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.281+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d080c89f0 con 0x7f4cf4078990 2024-09-10T20:46:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.285+0000 7f4d04ff9640 1 -- 172.21.15.28:0/748835140 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7f4cf8081a50 con 0x7f4cf4078990 2024-09-10T20:46:08.292 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:46:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cf4078990 msgr2=0x7f4cf407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cf4078990 0x7f4cf407ae50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8033000 tx=0x7f4d080cda10 comp rx=0 tx=0).stop 2024-09-10T20:46:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d08071990 msgr2=0x7f4d08071e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d08071990 0x7f4d08071e10 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00fa00 tx=0x7f4cfc00fec0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 shutdown_connections 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4cf4078990 0x7f4cf407ae50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d0806dc60 0x7f4d0806e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d08071990 0x7f4d08071e10 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 --2- 172.21.15.28:0/748835140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4cf0003a00 0x7f4d081515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 >> 172.21.15.28:0/748835140 conn(0x7f4d080d1e40 msgr2=0x7f4d080b6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 shutdown_connections 2024-09-10T20:46:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.288+0000 7f4d0e6dd640 1 -- 172.21.15.28:0/748835140 wait complete. 2024-09-10T20:46:08.304 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-10T20:46:06.339140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":69,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283068,"kb_used_data":16916,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754691652,"statfs":{"total":773094113280,"available":772804251648,"internally_reserved":0,"allocated":17321984,"data_stored":8304531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.089037"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280963+0000","last_change":"2024-09-10T20:46:04.279885+0000","last_active":"2024-09-10T20:46:04.280963+0000","last_peered":"2024-09-10T20:46:04.280963+0000","last_clean":"2024-09-10T20:46:04.280963+0000","last_became_active":"2024-09-10T20:46:04.279611+0000","last_became_peered":"2024-09-10T20:46:04.279611+0000","last_unstale":"2024-09-10T20:46:04.280963+0000","last_undegraded":"2024-09-10T20:46:04.280963+0000","last_fullsized":"2024-09-10T20:46:04.280963+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:31:12.314359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281443+0000","last_change":"2024-09-10T20:46:04.280187+0000","last_active":"2024-09-10T20:46:04.281443+0000","last_peered":"2024-09-10T20:46:04.281443+0000","last_clean":"2024-09-10T20:46:04.281443+0000","last_became_active":"2024-09-10T20:46:04.279917+0000","last_became_peered":"2024-09-10T20:46:04.279917+0000","last_unstale":"2024-09-10T20:46:04.281443+0000","last_undegraded":"2024-09-10T20:46:04.281443+0000","last_fullsized":"2024-09-10T20:46:04.281443+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:32.852331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282759+0000","last_change":"2024-09-10T20:46:04.276576+0000","last_active":"2024-09-10T20:46:04.282759+0000","last_peered":"2024-09-10T20:46:04.282759+0000","last_clean":"2024-09-10T20:46:04.282759+0000","last_became_active":"2024-09-10T20:46:04.276332+0000","last_became_peered":"2024-09-10T20:46:04.276332+0000","last_unstale":"2024-09-10T20:46:04.282759+0000","last_undegraded":"2024-09-10T20:46:04.282759+0000","last_fullsized":"2024-09-10T20:46:04.282759+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:14:13.013609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281948+0000","last_change":"2024-09-10T20:46:04.277710+0000","last_active":"2024-09-10T20:46:04.281948+0000","last_peered":"2024-09-10T20:46:04.281948+0000","last_clean":"2024-09-10T20:46:04.281948+0000","last_became_active":"2024-09-10T20:46:04.277439+0000","last_became_peered":"2024-09-10T20:46:04.277439+0000","last_unstale":"2024-09-10T20:46:04.281948+0000","last_undegraded":"2024-09-10T20:46:04.281948+0000","last_fullsized":"2024-09-10T20:46:04.281948+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:31.534648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286076+0000","last_change":"2024-09-10T20:46:04.281561+0000","last_active":"2024-09-10T20:46:04.286076+0000","last_peered":"2024-09-10T20:46:04.286076+0000","last_clean":"2024-09-10T20:46:04.286076+0000","last_became_active":"2024-09-10T20:46:04.280692+0000","last_became_peered":"2024-09-10T20:46:04.280692+0000","last_unstale":"2024-09-10T20:46:04.286076+0000","last_undegraded":"2024-09-10T20:46:04.286076+0000","last_fullsized":"2024-09-10T20:46:04.286076+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:25.597728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280045+0000","last_change":"2024-09-10T20:46:04.275602+0000","last_active":"2024-09-10T20:46:04.280045+0000","last_peered":"2024-09-10T20:46:04.280045+0000","last_clean":"2024-09-10T20:46:04.280045+0000","last_became_active":"2024-09-10T20:46:04.273728+0000","last_became_peered":"2024-09-10T20:46:04.273728+0000","last_unstale":"2024-09-10T20:46:04.280045+0000","last_undegraded":"2024-09-10T20:46:04.280045+0000","last_fullsized":"2024-09-10T20:46:04.280045+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:59:53.115147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280766+0000","last_change":"2024-09-10T20:46:04.272890+0000","last_active":"2024-09-10T20:46:04.280766+0000","last_peered":"2024-09-10T20:46:04.280766+0000","last_clean":"2024-09-10T20:46:04.280766+0000","last_became_active":"2024-09-10T20:46:04.272443+0000","last_became_peered":"2024-09-10T20:46:04.272443+0000","last_unstale":"2024-09-10T20:46:04.280766+0000","last_undegraded":"2024-09-10T20:46:04.280766+0000","last_fullsized":"2024-09-10T20:46:04.280766+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:28.463022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279871+0000","last_change":"2024-09-10T20:46:04.276965+0000","last_active":"2024-09-10T20:46:04.279871+0000","last_peered":"2024-09-10T20:46:04.279871+0000","last_clean":"2024-09-10T20:46:04.279871+0000","last_became_active":"2024-09-10T20:46:04.276695+0000","last_became_peered":"2024-09-10T20:46:04.276695+0000","last_unstale":"2024-09-10T20:46:04.279871+0000","last_undegraded":"2024-09-10T20:46:04.279871+0000","last_fullsized":"2024-09-10T20:46:04.279871+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:56:37.707590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288945+0000","last_change":"2024-09-10T20:46:04.277826+0000","last_active":"2024-09-10T20:46:04.288945+0000","last_peered":"2024-09-10T20:46:04.288945+0000","last_clean":"2024-09-10T20:46:04.288945+0000","last_became_active":"2024-09-10T20:46:04.277518+0000","last_became_peered":"2024-09-10T20:46:04.277518+0000","last_unstale":"2024-09-10T20:46:04.288945+0000","last_undegraded":"2024-09-10T20:46:04.288945+0000","last_fullsized":"2024-09-10T20:46:04.288945+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:27.648314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281874+0000","last_change":"2024-09-10T20:46:04.275876+0000","last_active":"2024-09-10T20:46:04.281874+0000","last_peered":"2024-09-10T20:46:04.281874+0000","last_clean":"2024-09-10T20:46:04.281874+0000","last_became_active":"2024-09-10T20:46:04.275642+0000","last_became_peered":"2024-09-10T20:46:04.275642+0000","last_unstale":"2024-09-10T20:46:04.281874+0000","last_undegraded":"2024-09-10T20:46:04.281874+0000","last_fullsized":"2024-09-10T20:46:04.281874+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:54:44.726107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280212+0000","last_change":"2024-09-10T20:46:04.271961+0000","last_active":"2024-09-10T20:46:04.280212+0000","last_peered":"2024-09-10T20:46:04.280212+0000","last_clean":"2024-09-10T20:46:04.280212+0000","last_became_active":"2024-09-10T20:46:04.271474+0000","last_became_peered":"2024-09-10T20:46:04.271474+0000","last_unstale":"2024-09-10T20:46:04.280212+0000","last_undegraded":"2024-09-10T20:46:04.280212+0000","last_fullsized":"2024-09-10T20:46:04.280212+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:11.071023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280192+0000","last_change":"2024-09-10T20:46:04.276913+0000","last_active":"2024-09-10T20:46:04.280192+0000","last_peered":"2024-09-10T20:46:04.280192+0000","last_clean":"2024-09-10T20:46:04.280192+0000","last_became_active":"2024-09-10T20:46:04.273161+0000","last_became_peered":"2024-09-10T20:46:04.273161+0000","last_unstale":"2024-09-10T20:46:04.280192+0000","last_undegraded":"2024-09-10T20:46:04.280192+0000","last_fullsized":"2024-09-10T20:46:04.280192+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:27:31.029297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281962+0000","last_change":"2024-09-10T20:46:04.277023+0000","last_active":"2024-09-10T20:46:04.281962+0000","last_peered":"2024-09-10T20:46:04.281962+0000","last_clean":"2024-09-10T20:46:04.281962+0000","last_became_active":"2024-09-10T20:46:04.276428+0000","last_became_peered":"2024-09-10T20:46:04.276428+0000","last_unstale":"2024-09-10T20:46:04.281962+0000","last_undegraded":"2024-09-10T20:46:04.281962+0000","last_fullsized":"2024-09-10T20:46:04.281962+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:38.579162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280024+0000","last_change":"2024-09-10T20:46:04.276640+0000","last_active":"2024-09-10T20:46:04.280024+0000","last_peered":"2024-09-10T20:46:04.280024+0000","last_clean":"2024-09-10T20:46:04.280024+0000","last_became_active":"2024-09-10T20:46:04.276376+0000","last_became_peered":"2024-09-10T20:46:04.276376+0000","last_unstale":"2024-09-10T20:46:04.280024+0000","last_undegraded":"2024-09-10T20:46:04.280024+0000","last_fullsized":"2024-09-10T20:46:04.280024+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:44.158075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280464+0000","last_change":"2024-09-10T20:46:04.270610+0000","last_active":"2024-09-10T20:46:04.280464+0000","last_peered":"2024-09-10T20:46:04.280464+0000","last_clean":"2024-09-10T20:46:04.280464+0000","last_became_active":"2024-09-10T20:46:04.270327+0000","last_became_peered":"2024-09-10T20:46:04.270327+0000","last_unstale":"2024-09-10T20:46:04.280464+0000","last_undegraded":"2024-09-10T20:46:04.280464+0000","last_fullsized":"2024-09-10T20:46:04.280464+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:29:34.347595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280709+0000","last_change":"2024-09-10T20:46:04.279389+0000","last_active":"2024-09-10T20:46:04.280709+0000","last_peered":"2024-09-10T20:46:04.280709+0000","last_clean":"2024-09-10T20:46:04.280709+0000","last_became_active":"2024-09-10T20:46:04.279120+0000","last_became_peered":"2024-09-10T20:46:04.279120+0000","last_unstale":"2024-09-10T20:46:04.280709+0000","last_undegraded":"2024-09-10T20:46:04.280709+0000","last_fullsized":"2024-09-10T20:46:04.280709+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:28:36.257714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277540+0000","last_change":"2024-09-10T20:46:04.267487+0000","last_active":"2024-09-10T20:46:04.277540+0000","last_peered":"2024-09-10T20:46:04.277540+0000","last_clean":"2024-09-10T20:46:04.277540+0000","last_became_active":"2024-09-10T20:46:04.267223+0000","last_became_peered":"2024-09-10T20:46:04.267223+0000","last_unstale":"2024-09-10T20:46:04.277540+0000","last_undegraded":"2024-09-10T20:46:04.277540+0000","last_fullsized":"2024-09-10T20:46:04.277540+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:26.343404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286059+0000","last_change":"2024-09-10T20:46:04.281874+0000","last_active":"2024-09-10T20:46:04.286059+0000","last_peered":"2024-09-10T20:46:04.286059+0000","last_clean":"2024-09-10T20:46:04.286059+0000","last_became_active":"2024-09-10T20:46:04.281571+0000","last_became_peered":"2024-09-10T20:46:04.281571+0000","last_unstale":"2024-09-10T20:46:04.286059+0000","last_undegraded":"2024-09-10T20:46:04.286059+0000","last_fullsized":"2024-09-10T20:46:04.286059+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:51.485750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.283002+0000","last_change":"2024-09-10T20:46:04.277238+0000","last_active":"2024-09-10T20:46:04.283002+0000","last_peered":"2024-09-10T20:46:04.283002+0000","last_clean":"2024-09-10T20:46:04.283002+0000","last_became_active":"2024-09-10T20:46:04.276998+0000","last_became_peered":"2024-09-10T20:46:04.276998+0000","last_unstale":"2024-09-10T20:46:04.283002+0000","last_undegraded":"2024-09-10T20:46:04.283002+0000","last_fullsized":"2024-09-10T20:46:04.283002+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:39:11.756993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286096+0000","last_change":"2024-09-10T20:46:04.278210+0000","last_active":"2024-09-10T20:46:04.286096+0000","last_peered":"2024-09-10T20:46:04.286096+0000","last_clean":"2024-09-10T20:46:04.286096+0000","last_became_active":"2024-09-10T20:46:04.277902+0000","last_became_peered":"2024-09-10T20:46:04.277902+0000","last_unstale":"2024-09-10T20:46:04.286096+0000","last_undegraded":"2024-09-10T20:46:04.286096+0000","last_fullsized":"2024-09-10T20:46:04.286096+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:25:54.268492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280648+0000","last_change":"2024-09-10T20:46:04.276441+0000","last_active":"2024-09-10T20:46:04.280648+0000","last_peered":"2024-09-10T20:46:04.280648+0000","last_clean":"2024-09-10T20:46:04.280648+0000","last_became_active":"2024-09-10T20:46:04.276146+0000","last_became_peered":"2024-09-10T20:46:04.276146+0000","last_unstale":"2024-09-10T20:46:04.280648+0000","last_undegraded":"2024-09-10T20:46:04.280648+0000","last_fullsized":"2024-09-10T20:46:04.280648+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:12.063571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.291432+0000","last_change":"2024-09-10T20:46:04.290769+0000","last_active":"2024-09-10T20:46:04.291432+0000","last_peered":"2024-09-10T20:46:04.291432+0000","last_clean":"2024-09-10T20:46:04.291432+0000","last_became_active":"2024-09-10T20:46:04.290465+0000","last_became_peered":"2024-09-10T20:46:04.290465+0000","last_unstale":"2024-09-10T20:46:04.291432+0000","last_undegraded":"2024-09-10T20:46:04.291432+0000","last_fullsized":"2024-09-10T20:46:04.291432+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:52.540303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280601+0000","last_change":"2024-09-10T20:46:04.275709+0000","last_active":"2024-09-10T20:46:04.280601+0000","last_peered":"2024-09-10T20:46:04.280601+0000","last_clean":"2024-09-10T20:46:04.280601+0000","last_became_active":"2024-09-10T20:46:04.275377+0000","last_became_peered":"2024-09-10T20:46:04.275377+0000","last_unstale":"2024-09-10T20:46:04.280601+0000","last_undegraded":"2024-09-10T20:46:04.280601+0000","last_fullsized":"2024-09-10T20:46:04.280601+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:37:35.784037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289738+0000","last_change":"2024-09-10T20:46:04.283717+0000","last_active":"2024-09-10T20:46:04.289738+0000","last_peered":"2024-09-10T20:46:04.289738+0000","last_clean":"2024-09-10T20:46:04.289738+0000","last_became_active":"2024-09-10T20:46:04.283405+0000","last_became_peered":"2024-09-10T20:46:04.283405+0000","last_unstale":"2024-09-10T20:46:04.289738+0000","last_undegraded":"2024-09-10T20:46:04.289738+0000","last_fullsized":"2024-09-10T20:46:04.289738+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:44.158075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285315+0000","last_change":"2024-09-10T20:46:04.283900+0000","last_active":"2024-09-10T20:46:04.285315+0000","last_peered":"2024-09-10T20:46:04.285315+0000","last_clean":"2024-09-10T20:46:04.285315+0000","last_became_active":"2024-09-10T20:46:04.283572+0000","last_became_peered":"2024-09-10T20:46:04.283572+0000","last_unstale":"2024-09-10T20:46:04.285315+0000","last_undegraded":"2024-09-10T20:46:04.285315+0000","last_fullsized":"2024-09-10T20:46:04.285315+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:31.861693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277630+0000","last_change":"2024-09-10T20:46:04.275179+0000","last_active":"2024-09-10T20:46:04.277630+0000","last_peered":"2024-09-10T20:46:04.277630+0000","last_clean":"2024-09-10T20:46:04.277630+0000","last_became_active":"2024-09-10T20:46:04.274677+0000","last_became_peered":"2024-09-10T20:46:04.274677+0000","last_unstale":"2024-09-10T20:46:04.277630+0000","last_undegraded":"2024-09-10T20:46:04.277630+0000","last_fullsized":"2024-09-10T20:46:04.277630+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:13:08.696500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279076+0000","last_change":"2024-09-10T20:46:04.271560+0000","last_active":"2024-09-10T20:46:04.279076+0000","last_peered":"2024-09-10T20:46:04.279076+0000","last_clean":"2024-09-10T20:46:04.279076+0000","last_became_active":"2024-09-10T20:46:04.270156+0000","last_became_peered":"2024-09-10T20:46:04.270156+0000","last_unstale":"2024-09-10T20:46:04.279076+0000","last_undegraded":"2024-09-10T20:46:04.279076+0000","last_fullsized":"2024-09-10T20:46:04.279076+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:44.158075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.275471+0000","last_change":"2024-09-10T20:46:04.269676+0000","last_active":"2024-09-10T20:46:04.275471+0000","last_peered":"2024-09-10T20:46:04.275471+0000","last_clean":"2024-09-10T20:46:04.275471+0000","last_became_active":"2024-09-10T20:46:04.269419+0000","last_became_peered":"2024-09-10T20:46:04.269419+0000","last_unstale":"2024-09-10T20:46:04.275471+0000","last_undegraded":"2024-09-10T20:46:04.275471+0000","last_fullsized":"2024-09-10T20:46:04.275471+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:47.892370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286289+0000","last_change":"2024-09-10T20:46:04.284706+0000","last_active":"2024-09-10T20:46:04.286289+0000","last_peered":"2024-09-10T20:46:04.286289+0000","last_clean":"2024-09-10T20:46:04.286289+0000","last_became_active":"2024-09-10T20:46:04.284415+0000","last_became_peered":"2024-09-10T20:46:04.284415+0000","last_unstale":"2024-09-10T20:46:04.286289+0000","last_undegraded":"2024-09-10T20:46:04.286289+0000","last_fullsized":"2024-09-10T20:46:04.286289+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:19.634725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285371+0000","last_change":"2024-09-10T20:46:04.283601+0000","last_active":"2024-09-10T20:46:04.285371+0000","last_peered":"2024-09-10T20:46:04.285371+0000","last_clean":"2024-09-10T20:46:04.285371+0000","last_became_active":"2024-09-10T20:46:04.283309+0000","last_became_peered":"2024-09-10T20:46:04.283309+0000","last_unstale":"2024-09-10T20:46:04.285371+0000","last_undegraded":"2024-09-10T20:46:04.285371+0000","last_fullsized":"2024-09-10T20:46:04.285371+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:35.021345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285902+0000","last_change":"2024-09-10T20:46:04.279885+0000","last_active":"2024-09-10T20:46:04.285902+0000","last_peered":"2024-09-10T20:46:04.285902+0000","last_clean":"2024-09-10T20:46:04.285902+0000","last_became_active":"2024-09-10T20:46:04.279536+0000","last_became_peered":"2024-09-10T20:46:04.279536+0000","last_unstale":"2024-09-10T20:46:04.285902+0000","last_undegraded":"2024-09-10T20:46:04.285902+0000","last_fullsized":"2024-09-10T20:46:04.285902+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:16:07.271915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280650+0000","last_change":"2024-09-10T20:46:04.279527+0000","last_active":"2024-09-10T20:46:04.280650+0000","last_peered":"2024-09-10T20:46:04.280650+0000","last_clean":"2024-09-10T20:46:04.280650+0000","last_became_active":"2024-09-10T20:46:04.279213+0000","last_became_peered":"2024-09-10T20:46:04.279213+0000","last_unstale":"2024-09-10T20:46:04.280650+0000","last_undegraded":"2024-09-10T20:46:04.280650+0000","last_fullsized":"2024-09-10T20:46:04.280650+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:12.063571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281059+0000","last_change":"2024-09-10T20:46:04.279785+0000","last_active":"2024-09-10T20:46:04.281059+0000","last_peered":"2024-09-10T20:46:04.281059+0000","last_clean":"2024-09-10T20:46:04.281059+0000","last_became_active":"2024-09-10T20:46:04.279512+0000","last_became_peered":"2024-09-10T20:46:04.279512+0000","last_unstale":"2024-09-10T20:46:04.281059+0000","last_undegraded":"2024-09-10T20:46:04.281059+0000","last_fullsized":"2024-09-10T20:46:04.281059+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:51.576889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"161'4","reported_seq":17,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.298899+0000","last_change":"2024-09-10T20:46:03.273997+0000","last_active":"2024-09-10T20:46:04.298899+0000","last_peered":"2024-09-10T20:46:04.298899+0000","last_clean":"2024-09-10T20:46:04.298899+0000","last_became_active":"2024-09-10T20:46:03.273655+0000","last_became_peered":"2024-09-10T20:46:03.273655+0000","last_unstale":"2024-09-10T20:46:04.298899+0000","last_undegraded":"2024-09-10T20:46:04.298899+0000","last_fullsized":"2024-09-10T20:46:04.298899+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:26.645816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282107+0000","last_change":"2024-09-10T20:46:04.278324+0000","last_active":"2024-09-10T20:46:04.282107+0000","last_peered":"2024-09-10T20:46:04.282107+0000","last_clean":"2024-09-10T20:46:04.282107+0000","last_became_active":"2024-09-10T20:46:04.278015+0000","last_became_peered":"2024-09-10T20:46:04.278015+0000","last_unstale":"2024-09-10T20:46:04.282107+0000","last_undegraded":"2024-09-10T20:46:04.282107+0000","last_fullsized":"2024-09-10T20:46:04.282107+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:15.684579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1e","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.298146+0000","last_change":"2024-09-10T20:46:03.271129+0000","last_active":"2024-09-10T20:46:04.298146+0000","last_peered":"2024-09-10T20:46:04.298146+0000","last_clean":"2024-09-10T20:46:04.298146+0000","last_became_active":"2024-09-10T20:46:03.270774+0000","last_became_peered":"2024-09-10T20:46:03.270774+0000","last_unstale":"2024-09-10T20:46:04.298146+0000","last_undegraded":"2024-09-10T20:46:04.298146+0000","last_fullsized":"2024-09-10T20:46:04.298146+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:28:55.267711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286926+0000","last_change":"2024-09-10T20:46:04.284060+0000","last_active":"2024-09-10T20:46:04.286926+0000","last_peered":"2024-09-10T20:46:04.286926+0000","last_clean":"2024-09-10T20:46:04.286926+0000","last_became_active":"2024-09-10T20:46:04.283394+0000","last_became_peered":"2024-09-10T20:46:04.283394+0000","last_unstale":"2024-09-10T20:46:04.286926+0000","last_undegraded":"2024-09-10T20:46:04.286926+0000","last_fullsized":"2024-09-10T20:46:04.286926+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:30:46.427622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.291049+0000","last_change":"2024-09-10T20:46:03.270778+0000","last_active":"2024-09-10T20:46:04.291049+0000","last_peered":"2024-09-10T20:46:04.291049+0000","last_clean":"2024-09-10T20:46:04.291049+0000","last_became_active":"2024-09-10T20:46:03.270464+0000","last_became_peered":"2024-09-10T20:46:03.270464+0000","last_unstale":"2024-09-10T20:46:04.291049+0000","last_undegraded":"2024-09-10T20:46:04.291049+0000","last_fullsized":"2024-09-10T20:46:04.291049+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:40:06.806533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286875+0000","last_change":"2024-09-10T20:46:04.283714+0000","last_active":"2024-09-10T20:46:04.286875+0000","last_peered":"2024-09-10T20:46:04.286875+0000","last_clean":"2024-09-10T20:46:04.286875+0000","last_became_active":"2024-09-10T20:46:04.282547+0000","last_became_peered":"2024-09-10T20:46:04.282547+0000","last_unstale":"2024-09-10T20:46:04.286875+0000","last_undegraded":"2024-09-10T20:46:04.286875+0000","last_fullsized":"2024-09-10T20:46:04.286875+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:40:47.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1c","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.284201+0000","last_change":"2024-09-10T20:46:03.274749+0000","last_active":"2024-09-10T20:46:04.284201+0000","last_peered":"2024-09-10T20:46:04.284201+0000","last_clean":"2024-09-10T20:46:04.284201+0000","last_became_active":"2024-09-10T20:46:03.274459+0000","last_became_peered":"2024-09-10T20:46:03.274459+0000","last_unstale":"2024-09-10T20:46:04.284201+0000","last_undegraded":"2024-09-10T20:46:04.284201+0000","last_fullsized":"2024-09-10T20:46:04.284201+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:30.573119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281932+0000","last_change":"2024-09-10T20:46:04.276690+0000","last_active":"2024-09-10T20:46:04.281932+0000","last_peered":"2024-09-10T20:46:04.281932+0000","last_clean":"2024-09-10T20:46:04.281932+0000","last_became_active":"2024-09-10T20:46:04.276071+0000","last_became_peered":"2024-09-10T20:46:04.276071+0000","last_unstale":"2024-09-10T20:46:04.281932+0000","last_undegraded":"2024-09-10T20:46:04.281932+0000","last_fullsized":"2024-09-10T20:46:04.281932+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:45.429064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281883+0000","last_change":"2024-09-10T20:46:03.278981+0000","last_active":"2024-09-10T20:46:04.281883+0000","last_peered":"2024-09-10T20:46:04.281883+0000","last_clean":"2024-09-10T20:46:04.281883+0000","last_became_active":"2024-09-10T20:46:03.278683+0000","last_became_peered":"2024-09-10T20:46:03.278683+0000","last_unstale":"2024-09-10T20:46:04.281883+0000","last_undegraded":"2024-09-10T20:46:04.281883+0000","last_fullsized":"2024-09-10T20:46:04.281883+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:23.314995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279914+0000","last_change":"2024-09-10T20:46:04.272638+0000","last_active":"2024-09-10T20:46:04.279914+0000","last_peered":"2024-09-10T20:46:04.279914+0000","last_clean":"2024-09-10T20:46:04.279914+0000","last_became_active":"2024-09-10T20:46:04.272306+0000","last_became_peered":"2024-09-10T20:46:04.272306+0000","last_unstale":"2024-09-10T20:46:04.279914+0000","last_undegraded":"2024-09-10T20:46:04.279914+0000","last_fullsized":"2024-09-10T20:46:04.279914+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:58:56.369039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.262589+0000","last_change":"2024-09-10T20:46:03.276268+0000","last_active":"2024-09-10T20:46:04.262589+0000","last_peered":"2024-09-10T20:46:04.262589+0000","last_clean":"2024-09-10T20:46:04.262589+0000","last_became_active":"2024-09-10T20:46:03.275996+0000","last_became_peered":"2024-09-10T20:46:03.275996+0000","last_unstale":"2024-09-10T20:46:04.262589+0000","last_undegraded":"2024-09-10T20:46:04.262589+0000","last_fullsized":"2024-09-10T20:46:04.262589+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:20:31.459126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289079+0000","last_change":"2024-09-10T20:46:04.281271+0000","last_active":"2024-09-10T20:46:04.289079+0000","last_peered":"2024-09-10T20:46:04.289079+0000","last_clean":"2024-09-10T20:46:04.289079+0000","last_became_active":"2024-09-10T20:46:04.280965+0000","last_became_peered":"2024-09-10T20:46:04.280965+0000","last_unstale":"2024-09-10T20:46:04.289079+0000","last_undegraded":"2024-09-10T20:46:04.289079+0000","last_fullsized":"2024-09-10T20:46:04.289079+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268703+0000","last_change":"2024-09-10T20:46:03.278497+0000","last_active":"2024-09-10T20:46:04.268703+0000","last_peered":"2024-09-10T20:46:04.268703+0000","last_clean":"2024-09-10T20:46:04.268703+0000","last_became_active":"2024-09-10T20:46:03.278214+0000","last_became_peered":"2024-09-10T20:46:03.278214+0000","last_unstale":"2024-09-10T20:46:04.268703+0000","last_undegraded":"2024-09-10T20:46:04.268703+0000","last_fullsized":"2024-09-10T20:46:04.268703+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:52:28.812773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288544+0000","last_change":"2024-09-10T20:46:04.285255+0000","last_active":"2024-09-10T20:46:04.288544+0000","last_peered":"2024-09-10T20:46:04.288544+0000","last_clean":"2024-09-10T20:46:04.288544+0000","last_became_active":"2024-09-10T20:46:04.284968+0000","last_became_peered":"2024-09-10T20:46:04.284968+0000","last_unstale":"2024-09-10T20:46:04.288544+0000","last_undegraded":"2024-09-10T20:46:04.288544+0000","last_fullsized":"2024-09-10T20:46:04.288544+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:23.764962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268397+0000","last_change":"2024-09-10T20:46:03.278568+0000","last_active":"2024-09-10T20:46:04.268397+0000","last_peered":"2024-09-10T20:46:04.268397+0000","last_clean":"2024-09-10T20:46:04.268397+0000","last_became_active":"2024-09-10T20:46:03.278321+0000","last_became_peered":"2024-09-10T20:46:03.278321+0000","last_unstale":"2024-09-10T20:46:04.268397+0000","last_undegraded":"2024-09-10T20:46:04.268397+0000","last_fullsized":"2024-09-10T20:46:04.268397+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:16:09.409090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.276820+0000","last_change":"2024-09-10T20:46:04.274450+0000","last_active":"2024-09-10T20:46:04.276820+0000","last_peered":"2024-09-10T20:46:04.276820+0000","last_clean":"2024-09-10T20:46:04.276820+0000","last_became_active":"2024-09-10T20:46:04.274121+0000","last_became_peered":"2024-09-10T20:46:04.274121+0000","last_unstale":"2024-09-10T20:46:04.276820+0000","last_undegraded":"2024-09-10T20:46:04.276820+0000","last_fullsized":"2024-09-10T20:46:04.276820+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268479+0000","last_change":"2024-09-10T20:46:03.277426+0000","last_active":"2024-09-10T20:46:04.268479+0000","last_peered":"2024-09-10T20:46:04.268479+0000","last_clean":"2024-09-10T20:46:04.268479+0000","last_became_active":"2024-09-10T20:46:03.277116+0000","last_became_peered":"2024-09-10T20:46:03.277116+0000","last_unstale":"2024-09-10T20:46:04.268479+0000","last_undegraded":"2024-09-10T20:46:04.268479+0000","last_fullsized":"2024-09-10T20:46:04.268479+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:34.298329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.284384+0000","last_change":"2024-09-10T20:46:04.276669+0000","last_active":"2024-09-10T20:46:04.284384+0000","last_peered":"2024-09-10T20:46:04.284384+0000","last_clean":"2024-09-10T20:46:04.284384+0000","last_became_active":"2024-09-10T20:46:04.276268+0000","last_became_peered":"2024-09-10T20:46:04.276268+0000","last_unstale":"2024-09-10T20:46:04.284384+0000","last_undegraded":"2024-09-10T20:46:04.284384+0000","last_fullsized":"2024-09-10T20:46:04.284384+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:45.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"161'3","reported_seq":16,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288402+0000","last_change":"2024-09-10T20:46:03.266586+0000","last_active":"2024-09-10T20:46:04.288402+0000","last_peered":"2024-09-10T20:46:04.288402+0000","last_clean":"2024-09-10T20:46:04.288402+0000","last_became_active":"2024-09-10T20:46:03.265811+0000","last_became_peered":"2024-09-10T20:46:03.265811+0000","last_unstale":"2024-09-10T20:46:04.288402+0000","last_undegraded":"2024-09-10T20:46:04.288402+0000","last_fullsized":"2024-09-10T20:46:04.288402+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:58.295090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277728+0000","last_change":"2024-09-10T20:46:04.273448+0000","last_active":"2024-09-10T20:46:04.277728+0000","last_peered":"2024-09-10T20:46:04.277728+0000","last_clean":"2024-09-10T20:46:04.277728+0000","last_became_active":"2024-09-10T20:46:04.273151+0000","last_became_peered":"2024-09-10T20:46:04.273151+0000","last_unstale":"2024-09-10T20:46:04.277728+0000","last_undegraded":"2024-09-10T20:46:04.277728+0000","last_fullsized":"2024-09-10T20:46:04.277728+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:28.425757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"161'6","reported_seq":19,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.293162+0000","last_change":"2024-09-10T20:46:03.274482+0000","last_active":"2024-09-10T20:46:04.293162+0000","last_peered":"2024-09-10T20:46:04.293162+0000","last_clean":"2024-09-10T20:46:04.293162+0000","last_became_active":"2024-09-10T20:46:03.274218+0000","last_became_peered":"2024-09-10T20:46:03.274218+0000","last_unstale":"2024-09-10T20:46:04.293162+0000","last_undegraded":"2024-09-10T20:46:04.293162+0000","last_fullsized":"2024-09-10T20:46:04.293162+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:49.338885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280620+0000","last_change":"2024-09-10T20:46:04.278805+0000","last_active":"2024-09-10T20:46:04.280620+0000","last_peered":"2024-09-10T20:46:04.280620+0000","last_clean":"2024-09-10T20:46:04.280620+0000","last_became_active":"2024-09-10T20:46:04.278555+0000","last_became_peered":"2024-09-10T20:46:04.278555+0000","last_unstale":"2024-09-10T20:46:04.280620+0000","last_undegraded":"2024-09-10T20:46:04.280620+0000","last_fullsized":"2024-09-10T20:46:04.280620+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:59.637959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.13","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289196+0000","last_change":"2024-09-10T20:46:03.274947+0000","last_active":"2024-09-10T20:46:04.289196+0000","last_peered":"2024-09-10T20:46:04.289196+0000","last_clean":"2024-09-10T20:46:04.289196+0000","last_became_active":"2024-09-10T20:46:03.274586+0000","last_became_peered":"2024-09-10T20:46:03.274586+0000","last_unstale":"2024-09-10T20:46:04.289196+0000","last_undegraded":"2024-09-10T20:46:04.289196+0000","last_fullsized":"2024-09-10T20:46:04.289196+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:54.836074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286832+0000","last_change":"2024-09-10T20:46:04.282738+0000","last_active":"2024-09-10T20:46:04.286832+0000","last_peered":"2024-09-10T20:46:04.286832+0000","last_clean":"2024-09-10T20:46:04.286832+0000","last_became_active":"2024-09-10T20:46:04.282010+0000","last_became_peered":"2024-09-10T20:46:04.282010+0000","last_unstale":"2024-09-10T20:46:04.286832+0000","last_undegraded":"2024-09-10T20:46:04.286832+0000","last_fullsized":"2024-09-10T20:46:04.286832+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:40.199489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.271087+0000","last_change":"2024-09-10T20:46:03.275019+0000","last_active":"2024-09-10T20:46:04.271087+0000","last_peered":"2024-09-10T20:46:04.271087+0000","last_clean":"2024-09-10T20:46:04.271087+0000","last_became_active":"2024-09-10T20:46:03.274720+0000","last_became_peered":"2024-09-10T20:46:03.274720+0000","last_unstale":"2024-09-10T20:46:04.271087+0000","last_undegraded":"2024-09-10T20:46:04.271087+0000","last_fullsized":"2024-09-10T20:46:04.271087+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:28:18.029072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282102+0000","last_change":"2024-09-10T20:46:04.275561+0000","last_active":"2024-09-10T20:46:04.282102+0000","last_peered":"2024-09-10T20:46:04.282102+0000","last_clean":"2024-09-10T20:46:04.282102+0000","last_became_active":"2024-09-10T20:46:04.275254+0000","last_became_peered":"2024-09-10T20:46:04.275254+0000","last_unstale":"2024-09-10T20:46:04.282102+0000","last_undegraded":"2024-09-10T20:46:04.282102+0000","last_fullsized":"2024-09-10T20:46:04.282102+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:39:44.733159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.262802+0000","last_change":"2024-09-10T20:46:03.274340+0000","last_active":"2024-09-10T20:46:04.262802+0000","last_peered":"2024-09-10T20:46:04.262802+0000","last_clean":"2024-09-10T20:46:04.262802+0000","last_became_active":"2024-09-10T20:46:03.273971+0000","last_became_peered":"2024-09-10T20:46:03.273971+0000","last_unstale":"2024-09-10T20:46:04.262802+0000","last_undegraded":"2024-09-10T20:46:04.262802+0000","last_fullsized":"2024-09-10T20:46:04.262802+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:28.715180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.278207+0000","last_change":"2024-09-10T20:46:04.269118+0000","last_active":"2024-09-10T20:46:04.278207+0000","last_peered":"2024-09-10T20:46:04.278207+0000","last_clean":"2024-09-10T20:46:04.278207+0000","last_became_active":"2024-09-10T20:46:04.268207+0000","last_became_peered":"2024-09-10T20:46:04.268207+0000","last_unstale":"2024-09-10T20:46:04.278207+0000","last_undegraded":"2024-09-10T20:46:04.278207+0000","last_fullsized":"2024-09-10T20:46:04.278207+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:13.694647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.10","version":"161'8","reported_seq":21,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.299621+0000","last_change":"2024-09-10T20:46:03.276775+0000","last_active":"2024-09-10T20:46:04.299621+0000","last_peered":"2024-09-10T20:46:04.299621+0000","last_clean":"2024-09-10T20:46:04.299621+0000","last_became_active":"2024-09-10T20:46:03.276494+0000","last_became_peered":"2024-09-10T20:46:03.276494+0000","last_unstale":"2024-09-10T20:46:04.299621+0000","last_undegraded":"2024-09-10T20:46:04.299621+0000","last_fullsized":"2024-09-10T20:46:04.299621+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:32:34.260860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280145+0000","last_change":"2024-09-10T20:46:04.277168+0000","last_active":"2024-09-10T20:46:04.280145+0000","last_peered":"2024-09-10T20:46:04.280145+0000","last_clean":"2024-09-10T20:46:04.280145+0000","last_became_active":"2024-09-10T20:46:04.276889+0000","last_became_peered":"2024-09-10T20:46:04.276889+0000","last_unstale":"2024-09-10T20:46:04.280145+0000","last_undegraded":"2024-09-10T20:46:04.280145+0000","last_fullsized":"2024-09-10T20:46:04.280145+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:47:54.871210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.f","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286748+0000","last_change":"2024-09-10T20:46:03.279856+0000","last_active":"2024-09-10T20:46:04.286748+0000","last_peered":"2024-09-10T20:46:04.286748+0000","last_clean":"2024-09-10T20:46:04.286748+0000","last_became_active":"2024-09-10T20:46:03.279423+0000","last_became_peered":"2024-09-10T20:46:03.279423+0000","last_unstale":"2024-09-10T20:46:04.286748+0000","last_undegraded":"2024-09-10T20:46:04.286748+0000","last_fullsized":"2024-09-10T20:46:04.286748+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:04.303700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285293+0000","last_change":"2024-09-10T20:46:04.282675+0000","last_active":"2024-09-10T20:46:04.285293+0000","last_peered":"2024-09-10T20:46:04.285293+0000","last_clean":"2024-09-10T20:46:04.285293+0000","last_became_active":"2024-09-10T20:46:04.282404+0000","last_became_peered":"2024-09-10T20:46:04.282404+0000","last_unstale":"2024-09-10T20:46:04.285293+0000","last_undegraded":"2024-09-10T20:46:04.285293+0000","last_fullsized":"2024-09-10T20:46:04.285293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:26.244419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289134+0000","last_change":"2024-09-10T20:46:03.276034+0000","last_active":"2024-09-10T20:46:04.289134+0000","last_peered":"2024-09-10T20:46:04.289134+0000","last_clean":"2024-09-10T20:46:04.289134+0000","last_became_active":"2024-09-10T20:46:03.275716+0000","last_became_peered":"2024-09-10T20:46:03.275716+0000","last_unstale":"2024-09-10T20:46:04.289134+0000","last_undegraded":"2024-09-10T20:46:04.289134+0000","last_fullsized":"2024-09-10T20:46:04.289134+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:26:13.681116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286905+0000","last_change":"2024-09-10T20:46:04.283712+0000","last_active":"2024-09-10T20:46:04.286905+0000","last_peered":"2024-09-10T20:46:04.286905+0000","last_clean":"2024-09-10T20:46:04.286905+0000","last_became_active":"2024-09-10T20:46:04.281187+0000","last_became_peered":"2024-09-10T20:46:04.281187+0000","last_unstale":"2024-09-10T20:46:04.286905+0000","last_undegraded":"2024-09-10T20:46:04.286905+0000","last_fullsized":"2024-09-10T20:46:04.286905+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:15.590015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.d","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.291241+0000","last_change":"2024-09-10T20:46:03.278444+0000","last_active":"2024-09-10T20:46:04.291241+0000","last_peered":"2024-09-10T20:46:04.291241+0000","last_clean":"2024-09-10T20:46:04.291241+0000","last_became_active":"2024-09-10T20:46:03.278140+0000","last_became_peered":"2024-09-10T20:46:03.278140+0000","last_unstale":"2024-09-10T20:46:04.291241+0000","last_undegraded":"2024-09-10T20:46:04.291241+0000","last_fullsized":"2024-09-10T20:46:04.291241+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:26.645816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.287052+0000","last_change":"2024-09-10T20:46:04.282878+0000","last_active":"2024-09-10T20:46:04.287052+0000","last_peered":"2024-09-10T20:46:04.287052+0000","last_clean":"2024-09-10T20:46:04.287052+0000","last_became_active":"2024-09-10T20:46:04.282445+0000","last_became_peered":"2024-09-10T20:46:04.282445+0000","last_unstale":"2024-09-10T20:46:04.287052+0000","last_undegraded":"2024-09-10T20:46:04.287052+0000","last_fullsized":"2024-09-10T20:46:04.287052+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:38:56.374744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265156+0000","last_change":"2024-09-10T20:46:03.273431+0000","last_active":"2024-09-10T20:46:04.265156+0000","last_peered":"2024-09-10T20:46:04.265156+0000","last_clean":"2024-09-10T20:46:04.265156+0000","last_became_active":"2024-09-10T20:46:03.273123+0000","last_became_peered":"2024-09-10T20:46:03.273123+0000","last_unstale":"2024-09-10T20:46:04.265156+0000","last_undegraded":"2024-09-10T20:46:04.265156+0000","last_fullsized":"2024-09-10T20:46:04.265156+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:42:26.887811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286086+0000","last_change":"2024-09-10T20:46:04.278660+0000","last_active":"2024-09-10T20:46:04.286086+0000","last_peered":"2024-09-10T20:46:04.286086+0000","last_clean":"2024-09-10T20:46:04.286086+0000","last_became_active":"2024-09-10T20:46:04.278411+0000","last_became_peered":"2024-09-10T20:46:04.278411+0000","last_unstale":"2024-09-10T20:46:04.286086+0000","last_undegraded":"2024-09-10T20:46:04.286086+0000","last_fullsized":"2024-09-10T20:46:04.286086+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:45:53.175786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.b","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.287776+0000","last_change":"2024-09-10T20:46:03.276639+0000","last_active":"2024-09-10T20:46:04.287776+0000","last_peered":"2024-09-10T20:46:04.287776+0000","last_clean":"2024-09-10T20:46:04.287776+0000","last_became_active":"2024-09-10T20:46:03.275504+0000","last_became_peered":"2024-09-10T20:46:03.275504+0000","last_unstale":"2024-09-10T20:46:04.287776+0000","last_undegraded":"2024-09-10T20:46:04.287776+0000","last_fullsized":"2024-09-10T20:46:04.287776+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:44.666428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280140+0000","last_change":"2024-09-10T20:46:04.275702+0000","last_active":"2024-09-10T20:46:04.280140+0000","last_peered":"2024-09-10T20:46:04.280140+0000","last_clean":"2024-09-10T20:46:04.280140+0000","last_became_active":"2024-09-10T20:46:04.275446+0000","last_became_peered":"2024-09-10T20:46:04.275446+0000","last_unstale":"2024-09-10T20:46:04.280140+0000","last_undegraded":"2024-09-10T20:46:04.280140+0000","last_fullsized":"2024-09-10T20:46:04.280140+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:24.675307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.262356+0000","last_change":"2024-09-10T20:46:03.266872+0000","last_active":"2024-09-10T20:46:04.262356+0000","last_peered":"2024-09-10T20:46:04.262356+0000","last_clean":"2024-09-10T20:46:04.262356+0000","last_became_active":"2024-09-10T20:46:03.266558+0000","last_became_peered":"2024-09-10T20:46:03.266558+0000","last_unstale":"2024-09-10T20:46:04.262356+0000","last_undegraded":"2024-09-10T20:46:04.262356+0000","last_fullsized":"2024-09-10T20:46:04.262356+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:52:05.614953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282681+0000","last_change":"2024-09-10T20:46:04.280133+0000","last_active":"2024-09-10T20:46:04.282681+0000","last_peered":"2024-09-10T20:46:04.282681+0000","last_clean":"2024-09-10T20:46:04.282681+0000","last_became_active":"2024-09-10T20:46:04.279869+0000","last_became_peered":"2024-09-10T20:46:04.279869+0000","last_unstale":"2024-09-10T20:46:04.282681+0000","last_undegraded":"2024-09-10T20:46:04.282681+0000","last_fullsized":"2024-09-10T20:46:04.282681+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:36:35.045228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.269597+0000","last_change":"2024-09-10T20:46:03.275155+0000","last_active":"2024-09-10T20:46:04.269597+0000","last_peered":"2024-09-10T20:46:04.269597+0000","last_clean":"2024-09-10T20:46:04.269597+0000","last_became_active":"2024-09-10T20:46:03.274874+0000","last_became_peered":"2024-09-10T20:46:03.274874+0000","last_unstale":"2024-09-10T20:46:04.269597+0000","last_undegraded":"2024-09-10T20:46:04.269597+0000","last_fullsized":"2024-09-10T20:46:04.269597+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:05.472273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285668+0000","last_change":"2024-09-10T20:46:04.284753+0000","last_active":"2024-09-10T20:46:04.285668+0000","last_peered":"2024-09-10T20:46:04.285668+0000","last_clean":"2024-09-10T20:46:04.285668+0000","last_became_active":"2024-09-10T20:46:04.284473+0000","last_became_peered":"2024-09-10T20:46:04.284473+0000","last_unstale":"2024-09-10T20:46:04.285668+0000","last_undegraded":"2024-09-10T20:46:04.285668+0000","last_fullsized":"2024-09-10T20:46:04.285668+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:05:52.386213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.264930+0000","last_change":"2024-09-10T20:46:03.267428+0000","last_active":"2024-09-10T20:46:04.264930+0000","last_peered":"2024-09-10T20:46:04.264930+0000","last_clean":"2024-09-10T20:46:04.264930+0000","last_became_active":"2024-09-10T20:46:03.267061+0000","last_became_peered":"2024-09-10T20:46:03.267061+0000","last_unstale":"2024-09-10T20:46:04.264930+0000","last_undegraded":"2024-09-10T20:46:04.264930+0000","last_fullsized":"2024-09-10T20:46:04.264930+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:59:46.489389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279142+0000","last_change":"2024-09-10T20:46:04.272580+0000","last_active":"2024-09-10T20:46:04.279142+0000","last_peered":"2024-09-10T20:46:04.279142+0000","last_clean":"2024-09-10T20:46:04.279142+0000","last_became_active":"2024-09-10T20:46:04.270671+0000","last_became_peered":"2024-09-10T20:46:04.270671+0000","last_unstale":"2024-09-10T20:46:04.279142+0000","last_undegraded":"2024-09-10T20:46:04.279142+0000","last_fullsized":"2024-09-10T20:46:04.279142+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:59.637959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290515+0000","last_change":"2024-09-10T20:46:03.274514+0000","last_active":"2024-09-10T20:46:04.290515+0000","last_peered":"2024-09-10T20:46:04.290515+0000","last_clean":"2024-09-10T20:46:04.290515+0000","last_became_active":"2024-09-10T20:46:03.274203+0000","last_became_peered":"2024-09-10T20:46:03.274203+0000","last_unstale":"2024-09-10T20:46:04.290515+0000","last_undegraded":"2024-09-10T20:46:04.290515+0000","last_fullsized":"2024-09-10T20:46:04.290515+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:14:51.274955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277568+0000","last_change":"2024-09-10T20:46:04.268051+0000","last_active":"2024-09-10T20:46:04.277568+0000","last_peered":"2024-09-10T20:46:04.277568+0000","last_clean":"2024-09-10T20:46:04.277568+0000","last_became_active":"2024-09-10T20:46:04.266370+0000","last_became_peered":"2024-09-10T20:46:04.266370+0000","last_unstale":"2024-09-10T20:46:04.277568+0000","last_undegraded":"2024-09-10T20:46:04.277568+0000","last_fullsized":"2024-09-10T20:46:04.277568+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:03:15.698687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.269826+0000","last_change":"2024-09-10T20:46:03.277697+0000","last_active":"2024-09-10T20:46:04.269826+0000","last_peered":"2024-09-10T20:46:04.269826+0000","last_clean":"2024-09-10T20:46:04.269826+0000","last_became_active":"2024-09-10T20:46:03.277324+0000","last_became_peered":"2024-09-10T20:46:03.277324+0000","last_unstale":"2024-09-10T20:46:04.269826+0000","last_undegraded":"2024-09-10T20:46:04.269826+0000","last_fullsized":"2024-09-10T20:46:04.269826+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:41.094488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290364+0000","last_change":"2024-09-10T20:46:04.286805+0000","last_active":"2024-09-10T20:46:04.290364+0000","last_peered":"2024-09-10T20:46:04.290364+0000","last_clean":"2024-09-10T20:46:04.290364+0000","last_became_active":"2024-09-10T20:46:04.285914+0000","last_became_peered":"2024-09-10T20:46:04.285914+0000","last_unstale":"2024-09-10T20:46:04.290364+0000","last_undegraded":"2024-09-10T20:46:04.290364+0000","last_fullsized":"2024-09-10T20:46:04.290364+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:12:05.418352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265853+0000","last_change":"2024-09-10T20:46:03.274127+0000","last_active":"2024-09-10T20:46:04.265853+0000","last_peered":"2024-09-10T20:46:04.265853+0000","last_clean":"2024-09-10T20:46:04.265853+0000","last_became_active":"2024-09-10T20:46:03.273829+0000","last_became_peered":"2024-09-10T20:46:03.273829+0000","last_unstale":"2024-09-10T20:46:04.265853+0000","last_undegraded":"2024-09-10T20:46:04.265853+0000","last_fullsized":"2024-09-10T20:46:04.265853+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:30:53.294709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.273277+0000","last_change":"2024-09-10T20:46:04.270438+0000","last_active":"2024-09-10T20:46:04.273277+0000","last_peered":"2024-09-10T20:46:04.273277+0000","last_clean":"2024-09-10T20:46:04.273277+0000","last_became_active":"2024-09-10T20:46:04.269809+0000","last_became_peered":"2024-09-10T20:46:04.269809+0000","last_unstale":"2024-09-10T20:46:04.273277+0000","last_undegraded":"2024-09-10T20:46:04.273277+0000","last_fullsized":"2024-09-10T20:46:04.273277+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265483+0000","last_change":"2024-09-10T20:46:03.280294+0000","last_active":"2024-09-10T20:46:04.265483+0000","last_peered":"2024-09-10T20:46:04.265483+0000","last_clean":"2024-09-10T20:46:04.265483+0000","last_became_active":"2024-09-10T20:46:03.279886+0000","last_became_peered":"2024-09-10T20:46:03.279886+0000","last_unstale":"2024-09-10T20:46:04.265483+0000","last_undegraded":"2024-09-10T20:46:04.265483+0000","last_fullsized":"2024-09-10T20:46:04.265483+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:37:37.780197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280009+0000","last_change":"2024-09-10T20:46:04.273174+0000","last_active":"2024-09-10T20:46:04.280009+0000","last_peered":"2024-09-10T20:46:04.280009+0000","last_clean":"2024-09-10T20:46:04.280009+0000","last_became_active":"2024-09-10T20:46:04.272719+0000","last_became_peered":"2024-09-10T20:46:04.272719+0000","last_unstale":"2024-09-10T20:46:04.280009+0000","last_undegraded":"2024-09-10T20:46:04.280009+0000","last_fullsized":"2024-09-10T20:46:04.280009+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:21.717293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265953+0000","last_change":"2024-09-10T20:46:03.273174+0000","last_active":"2024-09-10T20:46:04.265953+0000","last_peered":"2024-09-10T20:46:04.265953+0000","last_clean":"2024-09-10T20:46:04.265953+0000","last_became_active":"2024-09-10T20:46:03.272892+0000","last_became_peered":"2024-09-10T20:46:03.272892+0000","last_unstale":"2024-09-10T20:46:04.265953+0000","last_undegraded":"2024-09-10T20:46:04.265953+0000","last_fullsized":"2024-09-10T20:46:04.265953+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:16.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290299+0000","last_change":"2024-09-10T20:46:04.286396+0000","last_active":"2024-09-10T20:46:04.290299+0000","last_peered":"2024-09-10T20:46:04.290299+0000","last_clean":"2024-09-10T20:46:04.290299+0000","last_became_active":"2024-09-10T20:46:04.286057+0000","last_became_peered":"2024-09-10T20:46:04.286057+0000","last_unstale":"2024-09-10T20:46:04.290299+0000","last_undegraded":"2024-09-10T20:46:04.290299+0000","last_fullsized":"2024-09-10T20:46:04.290299+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:32.786945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268228+0000","last_change":"2024-09-10T20:46:03.277670+0000","last_active":"2024-09-10T20:46:04.268228+0000","last_peered":"2024-09-10T20:46:04.268228+0000","last_clean":"2024-09-10T20:46:04.268228+0000","last_became_active":"2024-09-10T20:46:03.277344+0000","last_became_peered":"2024-09-10T20:46:03.277344+0000","last_unstale":"2024-09-10T20:46:04.268228+0000","last_undegraded":"2024-09-10T20:46:04.268228+0000","last_fullsized":"2024-09-10T20:46:04.268228+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:29.934871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.274994+0000","last_change":"2024-09-10T20:46:04.269145+0000","last_active":"2024-09-10T20:46:04.274994+0000","last_peered":"2024-09-10T20:46:04.274994+0000","last_clean":"2024-09-10T20:46:04.274994+0000","last_became_active":"2024-09-10T20:46:04.268851+0000","last_became_peered":"2024-09-10T20:46:04.268851+0000","last_unstale":"2024-09-10T20:46:04.274994+0000","last_undegraded":"2024-09-10T20:46:04.274994+0000","last_fullsized":"2024-09-10T20:46:04.274994+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:45.449761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.261322+0000","last_change":"2024-09-10T20:46:03.275747+0000","last_active":"2024-09-10T20:46:04.261322+0000","last_peered":"2024-09-10T20:46:04.261322+0000","last_clean":"2024-09-10T20:46:04.261322+0000","last_became_active":"2024-09-10T20:46:03.275397+0000","last_became_peered":"2024-09-10T20:46:03.275397+0000","last_unstale":"2024-09-10T20:46:04.261322+0000","last_undegraded":"2024-09-10T20:46:04.261322+0000","last_fullsized":"2024-09-10T20:46:04.261322+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:44.666428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288192+0000","last_change":"2024-09-10T20:46:04.285394+0000","last_active":"2024-09-10T20:46:04.288192+0000","last_peered":"2024-09-10T20:46:04.288192+0000","last_clean":"2024-09-10T20:46:04.288192+0000","last_became_active":"2024-09-10T20:46:04.285025+0000","last_became_peered":"2024-09-10T20:46:04.285025+0000","last_unstale":"2024-09-10T20:46:04.288192+0000","last_undegraded":"2024-09-10T20:46:04.288192+0000","last_fullsized":"2024-09-10T20:46:04.288192+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:37.446329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268932+0000","last_change":"2024-09-10T20:46:03.276263+0000","last_active":"2024-09-10T20:46:04.268932+0000","last_peered":"2024-09-10T20:46:04.268932+0000","last_clean":"2024-09-10T20:46:04.268932+0000","last_became_active":"2024-09-10T20:46:03.275982+0000","last_became_peered":"2024-09-10T20:46:03.275982+0000","last_unstale":"2024-09-10T20:46:04.268932+0000","last_undegraded":"2024-09-10T20:46:04.268932+0000","last_fullsized":"2024-09-10T20:46:04.268932+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:34.665955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.287099+0000","last_change":"2024-09-10T20:46:04.283741+0000","last_active":"2024-09-10T20:46:04.287099+0000","last_peered":"2024-09-10T20:46:04.287099+0000","last_clean":"2024-09-10T20:46:04.287099+0000","last_became_active":"2024-09-10T20:46:04.280770+0000","last_became_peered":"2024-09-10T20:46:04.280770+0000","last_unstale":"2024-09-10T20:46:04.287099+0000","last_undegraded":"2024-09-10T20:46:04.287099+0000","last_fullsized":"2024-09-10T20:46:04.287099+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:08:42.236370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290739+0000","last_change":"2024-09-10T20:46:03.278267+0000","last_active":"2024-09-10T20:46:04.290739+0000","last_peered":"2024-09-10T20:46:04.290739+0000","last_clean":"2024-09-10T20:46:04.290739+0000","last_became_active":"2024-09-10T20:46:03.277979+0000","last_became_peered":"2024-09-10T20:46:03.277979+0000","last_unstale":"2024-09-10T20:46:04.290739+0000","last_undegraded":"2024-09-10T20:46:04.290739+0000","last_fullsized":"2024-09-10T20:46:04.290739+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:48:38.838819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":398,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268939+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:46:04.268939+0000","last_peered":"2024-09-10T20:46:04.268939+0000","last_clean":"2024-09-10T20:46:04.268939+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:46:04.268939+0000","last_undegraded":"2024-09-10T20:46:04.268939+0000","last_fullsized":"2024-09-10T20:46:04.268939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542243,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2609152,"data_stored":1479992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542243,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2027520,"data_stored":896427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509540,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37320,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334520,"statfs":{"total":96636764160,"available":96598548480,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509540,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37896,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333944,"statfs":{"total":96636764160,"available":96597958656,"internally_reserved":0,"allocated":2609152,"data_stored":1479992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:46:08.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:46:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.447+0000 7f6ffb00f640 1 -- 172.21.15.28:0/3350259855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe40051c0 msgr2=0x7f6fe400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.447+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/3350259855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe40051c0 0x7f6fe400b9c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40ce120 tx=0x7f6fe4005830 comp rx=0 tx=0).stop 2024-09-10T20:46:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.447+0000 7f6ffb00f640 1 -- 172.21.15.28:0/3350259855 shutdown_connections 2024-09-10T20:46:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.447+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/3350259855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe40051c0 0x7f6fe400b9c0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.447+0000 7f6ffb00f640 1 -- 172.21.15.28:0/3350259855 >> 172.21.15.28:0/3350259855 conn(0x7f6ff406cf00 msgr2=0x7f6ff406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.447+0000 7f6ffb00f640 1 -- 172.21.15.28:0/3350259855 shutdown_connections 2024-09-10T20:46:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.447+0000 7f6ffb00f640 1 -- 172.21.15.28:0/3350259855 wait complete. 2024-09-10T20:46:08.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.448+0000 7f6ffb00f640 1 Processor -- start 2024-09-10T20:46:08.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.448+0000 7f6ffb00f640 1 -- start start 2024-09-10T20:46:08.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ffb00f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fe40051c0 0x7f6ff414e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ffb00f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff415e430 0x7f6ff415e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ffb00f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ff40cac50 0x7f6ff40cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ffb00f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff40cc570 con 0x7f6ff415e430 2024-09-10T20:46:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ffb00f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff40b4a90 con 0x7f6ff40cac50 2024-09-10T20:46:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ffb00f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff40cc6f0 con 0x7f6fe40051c0 2024-09-10T20:46:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff415e430 0x7f6ff415e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff415e430 0x7f6ff415e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33130/0 (socket says 172.21.15.28:33130) 2024-09-10T20:46:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ff3fff640 1 -- 172.21.15.28:0/1900787269 learned_addr learned my addr 172.21.15.28:0/1900787269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ff95ac640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ff40cac50 0x7f6ff40cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.449+0000 7f6ff8dab640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fe40051c0 0x7f6ff414e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff3fff640 1 -- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fe40051c0 msgr2=0x7f6ff414e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff3fff640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fe40051c0 0x7f6ff414e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff3fff640 1 -- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ff40cac50 msgr2=0x7f6ff40cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff3fff640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ff40cac50 0x7f6ff40cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff3fff640 1 -- 172.21.15.28:0/1900787269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff41f4810 con 0x7f6ff415e430 2024-09-10T20:46:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff95ac640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ff40cac50 0x7f6ff40cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff3fff640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff415e430 0x7f6ff415e8b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0003f00 tx=0x7f6fe0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff17fa640 1 -- 172.21.15.28:0/1900787269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0011640 con 0x7f6ff415e430 2024-09-10T20:46:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff17fa640 1 -- 172.21.15.28:0/1900787269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fe0016020 con 0x7f6ff415e430 2024-09-10T20:46:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff41f4ab0 con 0x7f6ff415e430 2024-09-10T20:46:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ff17fa640 1 -- 172.21.15.28:0/1900787269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0011a80 con 0x7f6ff415e430 2024-09-10T20:46:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.450+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff41f5070 con 0x7f6ff415e430 2024-09-10T20:46:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.451+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff415edf0 con 0x7f6ff415e430 2024-09-10T20:46:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.452+0000 7f6ff17fa640 1 -- 172.21.15.28:0/1900787269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f6fe0011c20 con 0x7f6ff415e430 2024-09-10T20:46:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.452+0000 7f6ff17fa640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fd80789c0 0x7f6fd807ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.452+0000 7f6ff17fa640 1 -- 172.21.15.28:0/1900787269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f6fe00993a0 con 0x7f6ff415e430 2024-09-10T20:46:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.452+0000 7f6ff8dab640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fd80789c0 0x7f6fd807ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.453+0000 7f6ff8dab640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fd80789c0 0x7f6fd807ae80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009b10 tx=0x7f6fe8005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.456+0000 7f6ff17fa640 1 -- 172.21.15.28:0/1900787269 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6fe0020020 con 0x7f6ff415e430 2024-09-10T20:46:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.638+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6ff40d0170 con 0x7f6fd80789c0 2024-09-10T20:46:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.642+0000 7f6ff17fa640 1 -- 172.21.15.28:0/1900787269 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7f6ff40d0170 con 0x7f6fd80789c0 2024-09-10T20:46:08.648 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:46:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fd80789c0 msgr2=0x7f6fd807ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fd80789c0 0x7f6fd807ae80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009b10 tx=0x7f6fe8005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff415e430 msgr2=0x7f6ff415e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff415e430 0x7f6ff415e8b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0003f00 tx=0x7f6fe0001630 comp rx=0 tx=0).stop 2024-09-10T20:46:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 shutdown_connections 2024-09-10T20:46:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f6fd80789c0 0x7f6fd807ae80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6ff40cac50 0x7f6ff40cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff415e430 0x7f6ff415e8b0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 --2- 172.21.15.28:0/1900787269 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6fe40051c0 0x7f6ff414e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 >> 172.21.15.28:0/1900787269 conn(0x7f6ff406cf00 msgr2=0x7f6ff40cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 shutdown_connections 2024-09-10T20:46:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.645+0000 7f6ffb00f640 1 -- 172.21.15.28:0/1900787269 wait complete. 2024-09-10T20:46:08.660 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-10T20:46:08.339464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":69,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283068,"kb_used_data":16916,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754691652,"statfs":{"total":773094113280,"available":772804251648,"internally_reserved":0,"allocated":17321984,"data_stored":8304531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001150"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280963+0000","last_change":"2024-09-10T20:46:04.279885+0000","last_active":"2024-09-10T20:46:04.280963+0000","last_peered":"2024-09-10T20:46:04.280963+0000","last_clean":"2024-09-10T20:46:04.280963+0000","last_became_active":"2024-09-10T20:46:04.279611+0000","last_became_peered":"2024-09-10T20:46:04.279611+0000","last_unstale":"2024-09-10T20:46:04.280963+0000","last_undegraded":"2024-09-10T20:46:04.280963+0000","last_fullsized":"2024-09-10T20:46:04.280963+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:31:12.314359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281443+0000","last_change":"2024-09-10T20:46:04.280187+0000","last_active":"2024-09-10T20:46:04.281443+0000","last_peered":"2024-09-10T20:46:04.281443+0000","last_clean":"2024-09-10T20:46:04.281443+0000","last_became_active":"2024-09-10T20:46:04.279917+0000","last_became_peered":"2024-09-10T20:46:04.279917+0000","last_unstale":"2024-09-10T20:46:04.281443+0000","last_undegraded":"2024-09-10T20:46:04.281443+0000","last_fullsized":"2024-09-10T20:46:04.281443+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:55:32.852331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282759+0000","last_change":"2024-09-10T20:46:04.276576+0000","last_active":"2024-09-10T20:46:04.282759+0000","last_peered":"2024-09-10T20:46:04.282759+0000","last_clean":"2024-09-10T20:46:04.282759+0000","last_became_active":"2024-09-10T20:46:04.276332+0000","last_became_peered":"2024-09-10T20:46:04.276332+0000","last_unstale":"2024-09-10T20:46:04.282759+0000","last_undegraded":"2024-09-10T20:46:04.282759+0000","last_fullsized":"2024-09-10T20:46:04.282759+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:14:13.013609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281948+0000","last_change":"2024-09-10T20:46:04.277710+0000","last_active":"2024-09-10T20:46:04.281948+0000","last_peered":"2024-09-10T20:46:04.281948+0000","last_clean":"2024-09-10T20:46:04.281948+0000","last_became_active":"2024-09-10T20:46:04.277439+0000","last_became_peered":"2024-09-10T20:46:04.277439+0000","last_unstale":"2024-09-10T20:46:04.281948+0000","last_undegraded":"2024-09-10T20:46:04.281948+0000","last_fullsized":"2024-09-10T20:46:04.281948+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:31.534648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286076+0000","last_change":"2024-09-10T20:46:04.281561+0000","last_active":"2024-09-10T20:46:04.286076+0000","last_peered":"2024-09-10T20:46:04.286076+0000","last_clean":"2024-09-10T20:46:04.286076+0000","last_became_active":"2024-09-10T20:46:04.280692+0000","last_became_peered":"2024-09-10T20:46:04.280692+0000","last_unstale":"2024-09-10T20:46:04.286076+0000","last_undegraded":"2024-09-10T20:46:04.286076+0000","last_fullsized":"2024-09-10T20:46:04.286076+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:25.597728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280045+0000","last_change":"2024-09-10T20:46:04.275602+0000","last_active":"2024-09-10T20:46:04.280045+0000","last_peered":"2024-09-10T20:46:04.280045+0000","last_clean":"2024-09-10T20:46:04.280045+0000","last_became_active":"2024-09-10T20:46:04.273728+0000","last_became_peered":"2024-09-10T20:46:04.273728+0000","last_unstale":"2024-09-10T20:46:04.280045+0000","last_undegraded":"2024-09-10T20:46:04.280045+0000","last_fullsized":"2024-09-10T20:46:04.280045+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:59:53.115147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280766+0000","last_change":"2024-09-10T20:46:04.272890+0000","last_active":"2024-09-10T20:46:04.280766+0000","last_peered":"2024-09-10T20:46:04.280766+0000","last_clean":"2024-09-10T20:46:04.280766+0000","last_became_active":"2024-09-10T20:46:04.272443+0000","last_became_peered":"2024-09-10T20:46:04.272443+0000","last_unstale":"2024-09-10T20:46:04.280766+0000","last_undegraded":"2024-09-10T20:46:04.280766+0000","last_fullsized":"2024-09-10T20:46:04.280766+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:28.463022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279871+0000","last_change":"2024-09-10T20:46:04.276965+0000","last_active":"2024-09-10T20:46:04.279871+0000","last_peered":"2024-09-10T20:46:04.279871+0000","last_clean":"2024-09-10T20:46:04.279871+0000","last_became_active":"2024-09-10T20:46:04.276695+0000","last_became_peered":"2024-09-10T20:46:04.276695+0000","last_unstale":"2024-09-10T20:46:04.279871+0000","last_undegraded":"2024-09-10T20:46:04.279871+0000","last_fullsized":"2024-09-10T20:46:04.279871+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:56:37.707590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288945+0000","last_change":"2024-09-10T20:46:04.277826+0000","last_active":"2024-09-10T20:46:04.288945+0000","last_peered":"2024-09-10T20:46:04.288945+0000","last_clean":"2024-09-10T20:46:04.288945+0000","last_became_active":"2024-09-10T20:46:04.277518+0000","last_became_peered":"2024-09-10T20:46:04.277518+0000","last_unstale":"2024-09-10T20:46:04.288945+0000","last_undegraded":"2024-09-10T20:46:04.288945+0000","last_fullsized":"2024-09-10T20:46:04.288945+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:27.648314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281874+0000","last_change":"2024-09-10T20:46:04.275876+0000","last_active":"2024-09-10T20:46:04.281874+0000","last_peered":"2024-09-10T20:46:04.281874+0000","last_clean":"2024-09-10T20:46:04.281874+0000","last_became_active":"2024-09-10T20:46:04.275642+0000","last_became_peered":"2024-09-10T20:46:04.275642+0000","last_unstale":"2024-09-10T20:46:04.281874+0000","last_undegraded":"2024-09-10T20:46:04.281874+0000","last_fullsized":"2024-09-10T20:46:04.281874+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:54:44.726107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280212+0000","last_change":"2024-09-10T20:46:04.271961+0000","last_active":"2024-09-10T20:46:04.280212+0000","last_peered":"2024-09-10T20:46:04.280212+0000","last_clean":"2024-09-10T20:46:04.280212+0000","last_became_active":"2024-09-10T20:46:04.271474+0000","last_became_peered":"2024-09-10T20:46:04.271474+0000","last_unstale":"2024-09-10T20:46:04.280212+0000","last_undegraded":"2024-09-10T20:46:04.280212+0000","last_fullsized":"2024-09-10T20:46:04.280212+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:11.071023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280192+0000","last_change":"2024-09-10T20:46:04.276913+0000","last_active":"2024-09-10T20:46:04.280192+0000","last_peered":"2024-09-10T20:46:04.280192+0000","last_clean":"2024-09-10T20:46:04.280192+0000","last_became_active":"2024-09-10T20:46:04.273161+0000","last_became_peered":"2024-09-10T20:46:04.273161+0000","last_unstale":"2024-09-10T20:46:04.280192+0000","last_undegraded":"2024-09-10T20:46:04.280192+0000","last_fullsized":"2024-09-10T20:46:04.280192+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:27:31.029297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281962+0000","last_change":"2024-09-10T20:46:04.277023+0000","last_active":"2024-09-10T20:46:04.281962+0000","last_peered":"2024-09-10T20:46:04.281962+0000","last_clean":"2024-09-10T20:46:04.281962+0000","last_became_active":"2024-09-10T20:46:04.276428+0000","last_became_peered":"2024-09-10T20:46:04.276428+0000","last_unstale":"2024-09-10T20:46:04.281962+0000","last_undegraded":"2024-09-10T20:46:04.281962+0000","last_fullsized":"2024-09-10T20:46:04.281962+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:38.579162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280024+0000","last_change":"2024-09-10T20:46:04.276640+0000","last_active":"2024-09-10T20:46:04.280024+0000","last_peered":"2024-09-10T20:46:04.280024+0000","last_clean":"2024-09-10T20:46:04.280024+0000","last_became_active":"2024-09-10T20:46:04.276376+0000","last_became_peered":"2024-09-10T20:46:04.276376+0000","last_unstale":"2024-09-10T20:46:04.280024+0000","last_undegraded":"2024-09-10T20:46:04.280024+0000","last_fullsized":"2024-09-10T20:46:04.280024+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:44.158075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280464+0000","last_change":"2024-09-10T20:46:04.270610+0000","last_active":"2024-09-10T20:46:04.280464+0000","last_peered":"2024-09-10T20:46:04.280464+0000","last_clean":"2024-09-10T20:46:04.280464+0000","last_became_active":"2024-09-10T20:46:04.270327+0000","last_became_peered":"2024-09-10T20:46:04.270327+0000","last_unstale":"2024-09-10T20:46:04.280464+0000","last_undegraded":"2024-09-10T20:46:04.280464+0000","last_fullsized":"2024-09-10T20:46:04.280464+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:29:34.347595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280709+0000","last_change":"2024-09-10T20:46:04.279389+0000","last_active":"2024-09-10T20:46:04.280709+0000","last_peered":"2024-09-10T20:46:04.280709+0000","last_clean":"2024-09-10T20:46:04.280709+0000","last_became_active":"2024-09-10T20:46:04.279120+0000","last_became_peered":"2024-09-10T20:46:04.279120+0000","last_unstale":"2024-09-10T20:46:04.280709+0000","last_undegraded":"2024-09-10T20:46:04.280709+0000","last_fullsized":"2024-09-10T20:46:04.280709+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:28:36.257714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277540+0000","last_change":"2024-09-10T20:46:04.267487+0000","last_active":"2024-09-10T20:46:04.277540+0000","last_peered":"2024-09-10T20:46:04.277540+0000","last_clean":"2024-09-10T20:46:04.277540+0000","last_became_active":"2024-09-10T20:46:04.267223+0000","last_became_peered":"2024-09-10T20:46:04.267223+0000","last_unstale":"2024-09-10T20:46:04.277540+0000","last_undegraded":"2024-09-10T20:46:04.277540+0000","last_fullsized":"2024-09-10T20:46:04.277540+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:26.343404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286059+0000","last_change":"2024-09-10T20:46:04.281874+0000","last_active":"2024-09-10T20:46:04.286059+0000","last_peered":"2024-09-10T20:46:04.286059+0000","last_clean":"2024-09-10T20:46:04.286059+0000","last_became_active":"2024-09-10T20:46:04.281571+0000","last_became_peered":"2024-09-10T20:46:04.281571+0000","last_unstale":"2024-09-10T20:46:04.286059+0000","last_undegraded":"2024-09-10T20:46:04.286059+0000","last_fullsized":"2024-09-10T20:46:04.286059+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:51.485750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.283002+0000","last_change":"2024-09-10T20:46:04.277238+0000","last_active":"2024-09-10T20:46:04.283002+0000","last_peered":"2024-09-10T20:46:04.283002+0000","last_clean":"2024-09-10T20:46:04.283002+0000","last_became_active":"2024-09-10T20:46:04.276998+0000","last_became_peered":"2024-09-10T20:46:04.276998+0000","last_unstale":"2024-09-10T20:46:04.283002+0000","last_undegraded":"2024-09-10T20:46:04.283002+0000","last_fullsized":"2024-09-10T20:46:04.283002+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:39:11.756993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286096+0000","last_change":"2024-09-10T20:46:04.278210+0000","last_active":"2024-09-10T20:46:04.286096+0000","last_peered":"2024-09-10T20:46:04.286096+0000","last_clean":"2024-09-10T20:46:04.286096+0000","last_became_active":"2024-09-10T20:46:04.277902+0000","last_became_peered":"2024-09-10T20:46:04.277902+0000","last_unstale":"2024-09-10T20:46:04.286096+0000","last_undegraded":"2024-09-10T20:46:04.286096+0000","last_fullsized":"2024-09-10T20:46:04.286096+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:25:54.268492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280648+0000","last_change":"2024-09-10T20:46:04.276441+0000","last_active":"2024-09-10T20:46:04.280648+0000","last_peered":"2024-09-10T20:46:04.280648+0000","last_clean":"2024-09-10T20:46:04.280648+0000","last_became_active":"2024-09-10T20:46:04.276146+0000","last_became_peered":"2024-09-10T20:46:04.276146+0000","last_unstale":"2024-09-10T20:46:04.280648+0000","last_undegraded":"2024-09-10T20:46:04.280648+0000","last_fullsized":"2024-09-10T20:46:04.280648+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:12.063571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.291432+0000","last_change":"2024-09-10T20:46:04.290769+0000","last_active":"2024-09-10T20:46:04.291432+0000","last_peered":"2024-09-10T20:46:04.291432+0000","last_clean":"2024-09-10T20:46:04.291432+0000","last_became_active":"2024-09-10T20:46:04.290465+0000","last_became_peered":"2024-09-10T20:46:04.290465+0000","last_unstale":"2024-09-10T20:46:04.291432+0000","last_undegraded":"2024-09-10T20:46:04.291432+0000","last_fullsized":"2024-09-10T20:46:04.291432+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:52.540303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280601+0000","last_change":"2024-09-10T20:46:04.275709+0000","last_active":"2024-09-10T20:46:04.280601+0000","last_peered":"2024-09-10T20:46:04.280601+0000","last_clean":"2024-09-10T20:46:04.280601+0000","last_became_active":"2024-09-10T20:46:04.275377+0000","last_became_peered":"2024-09-10T20:46:04.275377+0000","last_unstale":"2024-09-10T20:46:04.280601+0000","last_undegraded":"2024-09-10T20:46:04.280601+0000","last_fullsized":"2024-09-10T20:46:04.280601+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:37:35.784037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289738+0000","last_change":"2024-09-10T20:46:04.283717+0000","last_active":"2024-09-10T20:46:04.289738+0000","last_peered":"2024-09-10T20:46:04.289738+0000","last_clean":"2024-09-10T20:46:04.289738+0000","last_became_active":"2024-09-10T20:46:04.283405+0000","last_became_peered":"2024-09-10T20:46:04.283405+0000","last_unstale":"2024-09-10T20:46:04.289738+0000","last_undegraded":"2024-09-10T20:46:04.289738+0000","last_fullsized":"2024-09-10T20:46:04.289738+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:44.158075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285315+0000","last_change":"2024-09-10T20:46:04.283900+0000","last_active":"2024-09-10T20:46:04.285315+0000","last_peered":"2024-09-10T20:46:04.285315+0000","last_clean":"2024-09-10T20:46:04.285315+0000","last_became_active":"2024-09-10T20:46:04.283572+0000","last_became_peered":"2024-09-10T20:46:04.283572+0000","last_unstale":"2024-09-10T20:46:04.285315+0000","last_undegraded":"2024-09-10T20:46:04.285315+0000","last_fullsized":"2024-09-10T20:46:04.285315+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:24:31.861693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277630+0000","last_change":"2024-09-10T20:46:04.275179+0000","last_active":"2024-09-10T20:46:04.277630+0000","last_peered":"2024-09-10T20:46:04.277630+0000","last_clean":"2024-09-10T20:46:04.277630+0000","last_became_active":"2024-09-10T20:46:04.274677+0000","last_became_peered":"2024-09-10T20:46:04.274677+0000","last_unstale":"2024-09-10T20:46:04.277630+0000","last_undegraded":"2024-09-10T20:46:04.277630+0000","last_fullsized":"2024-09-10T20:46:04.277630+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:13:08.696500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279076+0000","last_change":"2024-09-10T20:46:04.271560+0000","last_active":"2024-09-10T20:46:04.279076+0000","last_peered":"2024-09-10T20:46:04.279076+0000","last_clean":"2024-09-10T20:46:04.279076+0000","last_became_active":"2024-09-10T20:46:04.270156+0000","last_became_peered":"2024-09-10T20:46:04.270156+0000","last_unstale":"2024-09-10T20:46:04.279076+0000","last_undegraded":"2024-09-10T20:46:04.279076+0000","last_fullsized":"2024-09-10T20:46:04.279076+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:44.158075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.275471+0000","last_change":"2024-09-10T20:46:04.269676+0000","last_active":"2024-09-10T20:46:04.275471+0000","last_peered":"2024-09-10T20:46:04.275471+0000","last_clean":"2024-09-10T20:46:04.275471+0000","last_became_active":"2024-09-10T20:46:04.269419+0000","last_became_peered":"2024-09-10T20:46:04.269419+0000","last_unstale":"2024-09-10T20:46:04.275471+0000","last_undegraded":"2024-09-10T20:46:04.275471+0000","last_fullsized":"2024-09-10T20:46:04.275471+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:38:47.892370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286289+0000","last_change":"2024-09-10T20:46:04.284706+0000","last_active":"2024-09-10T20:46:04.286289+0000","last_peered":"2024-09-10T20:46:04.286289+0000","last_clean":"2024-09-10T20:46:04.286289+0000","last_became_active":"2024-09-10T20:46:04.284415+0000","last_became_peered":"2024-09-10T20:46:04.284415+0000","last_unstale":"2024-09-10T20:46:04.286289+0000","last_undegraded":"2024-09-10T20:46:04.286289+0000","last_fullsized":"2024-09-10T20:46:04.286289+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:19.634725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285371+0000","last_change":"2024-09-10T20:46:04.283601+0000","last_active":"2024-09-10T20:46:04.285371+0000","last_peered":"2024-09-10T20:46:04.285371+0000","last_clean":"2024-09-10T20:46:04.285371+0000","last_became_active":"2024-09-10T20:46:04.283309+0000","last_became_peered":"2024-09-10T20:46:04.283309+0000","last_unstale":"2024-09-10T20:46:04.285371+0000","last_undegraded":"2024-09-10T20:46:04.285371+0000","last_fullsized":"2024-09-10T20:46:04.285371+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:35.021345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285902+0000","last_change":"2024-09-10T20:46:04.279885+0000","last_active":"2024-09-10T20:46:04.285902+0000","last_peered":"2024-09-10T20:46:04.285902+0000","last_clean":"2024-09-10T20:46:04.285902+0000","last_became_active":"2024-09-10T20:46:04.279536+0000","last_became_peered":"2024-09-10T20:46:04.279536+0000","last_unstale":"2024-09-10T20:46:04.285902+0000","last_undegraded":"2024-09-10T20:46:04.285902+0000","last_fullsized":"2024-09-10T20:46:04.285902+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:16:07.271915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280650+0000","last_change":"2024-09-10T20:46:04.279527+0000","last_active":"2024-09-10T20:46:04.280650+0000","last_peered":"2024-09-10T20:46:04.280650+0000","last_clean":"2024-09-10T20:46:04.280650+0000","last_became_active":"2024-09-10T20:46:04.279213+0000","last_became_peered":"2024-09-10T20:46:04.279213+0000","last_unstale":"2024-09-10T20:46:04.280650+0000","last_undegraded":"2024-09-10T20:46:04.280650+0000","last_fullsized":"2024-09-10T20:46:04.280650+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:12.063571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281059+0000","last_change":"2024-09-10T20:46:04.279785+0000","last_active":"2024-09-10T20:46:04.281059+0000","last_peered":"2024-09-10T20:46:04.281059+0000","last_clean":"2024-09-10T20:46:04.281059+0000","last_became_active":"2024-09-10T20:46:04.279512+0000","last_became_peered":"2024-09-10T20:46:04.279512+0000","last_unstale":"2024-09-10T20:46:04.281059+0000","last_undegraded":"2024-09-10T20:46:04.281059+0000","last_fullsized":"2024-09-10T20:46:04.281059+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:51.576889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"161'4","reported_seq":17,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.298899+0000","last_change":"2024-09-10T20:46:03.273997+0000","last_active":"2024-09-10T20:46:04.298899+0000","last_peered":"2024-09-10T20:46:04.298899+0000","last_clean":"2024-09-10T20:46:04.298899+0000","last_became_active":"2024-09-10T20:46:03.273655+0000","last_became_peered":"2024-09-10T20:46:03.273655+0000","last_unstale":"2024-09-10T20:46:04.298899+0000","last_undegraded":"2024-09-10T20:46:04.298899+0000","last_fullsized":"2024-09-10T20:46:04.298899+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:26.645816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282107+0000","last_change":"2024-09-10T20:46:04.278324+0000","last_active":"2024-09-10T20:46:04.282107+0000","last_peered":"2024-09-10T20:46:04.282107+0000","last_clean":"2024-09-10T20:46:04.282107+0000","last_became_active":"2024-09-10T20:46:04.278015+0000","last_became_peered":"2024-09-10T20:46:04.278015+0000","last_unstale":"2024-09-10T20:46:04.282107+0000","last_undegraded":"2024-09-10T20:46:04.282107+0000","last_fullsized":"2024-09-10T20:46:04.282107+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:15.684579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1e","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.298146+0000","last_change":"2024-09-10T20:46:03.271129+0000","last_active":"2024-09-10T20:46:04.298146+0000","last_peered":"2024-09-10T20:46:04.298146+0000","last_clean":"2024-09-10T20:46:04.298146+0000","last_became_active":"2024-09-10T20:46:03.270774+0000","last_became_peered":"2024-09-10T20:46:03.270774+0000","last_unstale":"2024-09-10T20:46:04.298146+0000","last_undegraded":"2024-09-10T20:46:04.298146+0000","last_fullsized":"2024-09-10T20:46:04.298146+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:28:55.267711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286926+0000","last_change":"2024-09-10T20:46:04.284060+0000","last_active":"2024-09-10T20:46:04.286926+0000","last_peered":"2024-09-10T20:46:04.286926+0000","last_clean":"2024-09-10T20:46:04.286926+0000","last_became_active":"2024-09-10T20:46:04.283394+0000","last_became_peered":"2024-09-10T20:46:04.283394+0000","last_unstale":"2024-09-10T20:46:04.286926+0000","last_undegraded":"2024-09-10T20:46:04.286926+0000","last_fullsized":"2024-09-10T20:46:04.286926+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:30:46.427622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.291049+0000","last_change":"2024-09-10T20:46:03.270778+0000","last_active":"2024-09-10T20:46:04.291049+0000","last_peered":"2024-09-10T20:46:04.291049+0000","last_clean":"2024-09-10T20:46:04.291049+0000","last_became_active":"2024-09-10T20:46:03.270464+0000","last_became_peered":"2024-09-10T20:46:03.270464+0000","last_unstale":"2024-09-10T20:46:04.291049+0000","last_undegraded":"2024-09-10T20:46:04.291049+0000","last_fullsized":"2024-09-10T20:46:04.291049+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:40:06.806533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286875+0000","last_change":"2024-09-10T20:46:04.283714+0000","last_active":"2024-09-10T20:46:04.286875+0000","last_peered":"2024-09-10T20:46:04.286875+0000","last_clean":"2024-09-10T20:46:04.286875+0000","last_became_active":"2024-09-10T20:46:04.282547+0000","last_became_peered":"2024-09-10T20:46:04.282547+0000","last_unstale":"2024-09-10T20:46:04.286875+0000","last_undegraded":"2024-09-10T20:46:04.286875+0000","last_fullsized":"2024-09-10T20:46:04.286875+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:40:47.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1c","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.284201+0000","last_change":"2024-09-10T20:46:03.274749+0000","last_active":"2024-09-10T20:46:04.284201+0000","last_peered":"2024-09-10T20:46:04.284201+0000","last_clean":"2024-09-10T20:46:04.284201+0000","last_became_active":"2024-09-10T20:46:03.274459+0000","last_became_peered":"2024-09-10T20:46:03.274459+0000","last_unstale":"2024-09-10T20:46:04.284201+0000","last_undegraded":"2024-09-10T20:46:04.284201+0000","last_fullsized":"2024-09-10T20:46:04.284201+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:30.573119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281932+0000","last_change":"2024-09-10T20:46:04.276690+0000","last_active":"2024-09-10T20:46:04.281932+0000","last_peered":"2024-09-10T20:46:04.281932+0000","last_clean":"2024-09-10T20:46:04.281932+0000","last_became_active":"2024-09-10T20:46:04.276071+0000","last_became_peered":"2024-09-10T20:46:04.276071+0000","last_unstale":"2024-09-10T20:46:04.281932+0000","last_undegraded":"2024-09-10T20:46:04.281932+0000","last_fullsized":"2024-09-10T20:46:04.281932+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:45.429064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.281883+0000","last_change":"2024-09-10T20:46:03.278981+0000","last_active":"2024-09-10T20:46:04.281883+0000","last_peered":"2024-09-10T20:46:04.281883+0000","last_clean":"2024-09-10T20:46:04.281883+0000","last_became_active":"2024-09-10T20:46:03.278683+0000","last_became_peered":"2024-09-10T20:46:03.278683+0000","last_unstale":"2024-09-10T20:46:04.281883+0000","last_undegraded":"2024-09-10T20:46:04.281883+0000","last_fullsized":"2024-09-10T20:46:04.281883+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:23.314995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279914+0000","last_change":"2024-09-10T20:46:04.272638+0000","last_active":"2024-09-10T20:46:04.279914+0000","last_peered":"2024-09-10T20:46:04.279914+0000","last_clean":"2024-09-10T20:46:04.279914+0000","last_became_active":"2024-09-10T20:46:04.272306+0000","last_became_peered":"2024-09-10T20:46:04.272306+0000","last_unstale":"2024-09-10T20:46:04.279914+0000","last_undegraded":"2024-09-10T20:46:04.279914+0000","last_fullsized":"2024-09-10T20:46:04.279914+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:58:56.369039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.262589+0000","last_change":"2024-09-10T20:46:03.276268+0000","last_active":"2024-09-10T20:46:04.262589+0000","last_peered":"2024-09-10T20:46:04.262589+0000","last_clean":"2024-09-10T20:46:04.262589+0000","last_became_active":"2024-09-10T20:46:03.275996+0000","last_became_peered":"2024-09-10T20:46:03.275996+0000","last_unstale":"2024-09-10T20:46:04.262589+0000","last_undegraded":"2024-09-10T20:46:04.262589+0000","last_fullsized":"2024-09-10T20:46:04.262589+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:20:31.459126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289079+0000","last_change":"2024-09-10T20:46:04.281271+0000","last_active":"2024-09-10T20:46:04.289079+0000","last_peered":"2024-09-10T20:46:04.289079+0000","last_clean":"2024-09-10T20:46:04.289079+0000","last_became_active":"2024-09-10T20:46:04.280965+0000","last_became_peered":"2024-09-10T20:46:04.280965+0000","last_unstale":"2024-09-10T20:46:04.289079+0000","last_undegraded":"2024-09-10T20:46:04.289079+0000","last_fullsized":"2024-09-10T20:46:04.289079+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268703+0000","last_change":"2024-09-10T20:46:03.278497+0000","last_active":"2024-09-10T20:46:04.268703+0000","last_peered":"2024-09-10T20:46:04.268703+0000","last_clean":"2024-09-10T20:46:04.268703+0000","last_became_active":"2024-09-10T20:46:03.278214+0000","last_became_peered":"2024-09-10T20:46:03.278214+0000","last_unstale":"2024-09-10T20:46:04.268703+0000","last_undegraded":"2024-09-10T20:46:04.268703+0000","last_fullsized":"2024-09-10T20:46:04.268703+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:52:28.812773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288544+0000","last_change":"2024-09-10T20:46:04.285255+0000","last_active":"2024-09-10T20:46:04.288544+0000","last_peered":"2024-09-10T20:46:04.288544+0000","last_clean":"2024-09-10T20:46:04.288544+0000","last_became_active":"2024-09-10T20:46:04.284968+0000","last_became_peered":"2024-09-10T20:46:04.284968+0000","last_unstale":"2024-09-10T20:46:04.288544+0000","last_undegraded":"2024-09-10T20:46:04.288544+0000","last_fullsized":"2024-09-10T20:46:04.288544+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:23.764962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268397+0000","last_change":"2024-09-10T20:46:03.278568+0000","last_active":"2024-09-10T20:46:04.268397+0000","last_peered":"2024-09-10T20:46:04.268397+0000","last_clean":"2024-09-10T20:46:04.268397+0000","last_became_active":"2024-09-10T20:46:03.278321+0000","last_became_peered":"2024-09-10T20:46:03.278321+0000","last_unstale":"2024-09-10T20:46:04.268397+0000","last_undegraded":"2024-09-10T20:46:04.268397+0000","last_fullsized":"2024-09-10T20:46:04.268397+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:16:09.409090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.276820+0000","last_change":"2024-09-10T20:46:04.274450+0000","last_active":"2024-09-10T20:46:04.276820+0000","last_peered":"2024-09-10T20:46:04.276820+0000","last_clean":"2024-09-10T20:46:04.276820+0000","last_became_active":"2024-09-10T20:46:04.274121+0000","last_became_peered":"2024-09-10T20:46:04.274121+0000","last_unstale":"2024-09-10T20:46:04.276820+0000","last_undegraded":"2024-09-10T20:46:04.276820+0000","last_fullsized":"2024-09-10T20:46:04.276820+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268479+0000","last_change":"2024-09-10T20:46:03.277426+0000","last_active":"2024-09-10T20:46:04.268479+0000","last_peered":"2024-09-10T20:46:04.268479+0000","last_clean":"2024-09-10T20:46:04.268479+0000","last_became_active":"2024-09-10T20:46:03.277116+0000","last_became_peered":"2024-09-10T20:46:03.277116+0000","last_unstale":"2024-09-10T20:46:04.268479+0000","last_undegraded":"2024-09-10T20:46:04.268479+0000","last_fullsized":"2024-09-10T20:46:04.268479+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:15:34.298329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.284384+0000","last_change":"2024-09-10T20:46:04.276669+0000","last_active":"2024-09-10T20:46:04.284384+0000","last_peered":"2024-09-10T20:46:04.284384+0000","last_clean":"2024-09-10T20:46:04.284384+0000","last_became_active":"2024-09-10T20:46:04.276268+0000","last_became_peered":"2024-09-10T20:46:04.276268+0000","last_unstale":"2024-09-10T20:46:04.284384+0000","last_undegraded":"2024-09-10T20:46:04.284384+0000","last_fullsized":"2024-09-10T20:46:04.284384+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:45.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"161'3","reported_seq":16,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288402+0000","last_change":"2024-09-10T20:46:03.266586+0000","last_active":"2024-09-10T20:46:04.288402+0000","last_peered":"2024-09-10T20:46:04.288402+0000","last_clean":"2024-09-10T20:46:04.288402+0000","last_became_active":"2024-09-10T20:46:03.265811+0000","last_became_peered":"2024-09-10T20:46:03.265811+0000","last_unstale":"2024-09-10T20:46:04.288402+0000","last_undegraded":"2024-09-10T20:46:04.288402+0000","last_fullsized":"2024-09-10T20:46:04.288402+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:58.295090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277728+0000","last_change":"2024-09-10T20:46:04.273448+0000","last_active":"2024-09-10T20:46:04.277728+0000","last_peered":"2024-09-10T20:46:04.277728+0000","last_clean":"2024-09-10T20:46:04.277728+0000","last_became_active":"2024-09-10T20:46:04.273151+0000","last_became_peered":"2024-09-10T20:46:04.273151+0000","last_unstale":"2024-09-10T20:46:04.277728+0000","last_undegraded":"2024-09-10T20:46:04.277728+0000","last_fullsized":"2024-09-10T20:46:04.277728+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:28.425757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"161'6","reported_seq":19,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.293162+0000","last_change":"2024-09-10T20:46:03.274482+0000","last_active":"2024-09-10T20:46:04.293162+0000","last_peered":"2024-09-10T20:46:04.293162+0000","last_clean":"2024-09-10T20:46:04.293162+0000","last_became_active":"2024-09-10T20:46:03.274218+0000","last_became_peered":"2024-09-10T20:46:03.274218+0000","last_unstale":"2024-09-10T20:46:04.293162+0000","last_undegraded":"2024-09-10T20:46:04.293162+0000","last_fullsized":"2024-09-10T20:46:04.293162+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:45:49.338885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280620+0000","last_change":"2024-09-10T20:46:04.278805+0000","last_active":"2024-09-10T20:46:04.280620+0000","last_peered":"2024-09-10T20:46:04.280620+0000","last_clean":"2024-09-10T20:46:04.280620+0000","last_became_active":"2024-09-10T20:46:04.278555+0000","last_became_peered":"2024-09-10T20:46:04.278555+0000","last_unstale":"2024-09-10T20:46:04.280620+0000","last_undegraded":"2024-09-10T20:46:04.280620+0000","last_fullsized":"2024-09-10T20:46:04.280620+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:59.637959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.13","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289196+0000","last_change":"2024-09-10T20:46:03.274947+0000","last_active":"2024-09-10T20:46:04.289196+0000","last_peered":"2024-09-10T20:46:04.289196+0000","last_clean":"2024-09-10T20:46:04.289196+0000","last_became_active":"2024-09-10T20:46:03.274586+0000","last_became_peered":"2024-09-10T20:46:03.274586+0000","last_unstale":"2024-09-10T20:46:04.289196+0000","last_undegraded":"2024-09-10T20:46:04.289196+0000","last_fullsized":"2024-09-10T20:46:04.289196+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:54.836074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286832+0000","last_change":"2024-09-10T20:46:04.282738+0000","last_active":"2024-09-10T20:46:04.286832+0000","last_peered":"2024-09-10T20:46:04.286832+0000","last_clean":"2024-09-10T20:46:04.286832+0000","last_became_active":"2024-09-10T20:46:04.282010+0000","last_became_peered":"2024-09-10T20:46:04.282010+0000","last_unstale":"2024-09-10T20:46:04.286832+0000","last_undegraded":"2024-09-10T20:46:04.286832+0000","last_fullsized":"2024-09-10T20:46:04.286832+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:40.199489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.271087+0000","last_change":"2024-09-10T20:46:03.275019+0000","last_active":"2024-09-10T20:46:04.271087+0000","last_peered":"2024-09-10T20:46:04.271087+0000","last_clean":"2024-09-10T20:46:04.271087+0000","last_became_active":"2024-09-10T20:46:03.274720+0000","last_became_peered":"2024-09-10T20:46:03.274720+0000","last_unstale":"2024-09-10T20:46:04.271087+0000","last_undegraded":"2024-09-10T20:46:04.271087+0000","last_fullsized":"2024-09-10T20:46:04.271087+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:28:18.029072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282102+0000","last_change":"2024-09-10T20:46:04.275561+0000","last_active":"2024-09-10T20:46:04.282102+0000","last_peered":"2024-09-10T20:46:04.282102+0000","last_clean":"2024-09-10T20:46:04.282102+0000","last_became_active":"2024-09-10T20:46:04.275254+0000","last_became_peered":"2024-09-10T20:46:04.275254+0000","last_unstale":"2024-09-10T20:46:04.282102+0000","last_undegraded":"2024-09-10T20:46:04.282102+0000","last_fullsized":"2024-09-10T20:46:04.282102+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:39:44.733159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.262802+0000","last_change":"2024-09-10T20:46:03.274340+0000","last_active":"2024-09-10T20:46:04.262802+0000","last_peered":"2024-09-10T20:46:04.262802+0000","last_clean":"2024-09-10T20:46:04.262802+0000","last_became_active":"2024-09-10T20:46:03.273971+0000","last_became_peered":"2024-09-10T20:46:03.273971+0000","last_unstale":"2024-09-10T20:46:04.262802+0000","last_undegraded":"2024-09-10T20:46:04.262802+0000","last_fullsized":"2024-09-10T20:46:04.262802+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:38:28.715180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.278207+0000","last_change":"2024-09-10T20:46:04.269118+0000","last_active":"2024-09-10T20:46:04.278207+0000","last_peered":"2024-09-10T20:46:04.278207+0000","last_clean":"2024-09-10T20:46:04.278207+0000","last_became_active":"2024-09-10T20:46:04.268207+0000","last_became_peered":"2024-09-10T20:46:04.268207+0000","last_unstale":"2024-09-10T20:46:04.278207+0000","last_undegraded":"2024-09-10T20:46:04.278207+0000","last_fullsized":"2024-09-10T20:46:04.278207+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:13.694647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.10","version":"161'8","reported_seq":21,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.299621+0000","last_change":"2024-09-10T20:46:03.276775+0000","last_active":"2024-09-10T20:46:04.299621+0000","last_peered":"2024-09-10T20:46:04.299621+0000","last_clean":"2024-09-10T20:46:04.299621+0000","last_became_active":"2024-09-10T20:46:03.276494+0000","last_became_peered":"2024-09-10T20:46:03.276494+0000","last_unstale":"2024-09-10T20:46:04.299621+0000","last_undegraded":"2024-09-10T20:46:04.299621+0000","last_fullsized":"2024-09-10T20:46:04.299621+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:32:34.260860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280145+0000","last_change":"2024-09-10T20:46:04.277168+0000","last_active":"2024-09-10T20:46:04.280145+0000","last_peered":"2024-09-10T20:46:04.280145+0000","last_clean":"2024-09-10T20:46:04.280145+0000","last_became_active":"2024-09-10T20:46:04.276889+0000","last_became_peered":"2024-09-10T20:46:04.276889+0000","last_unstale":"2024-09-10T20:46:04.280145+0000","last_undegraded":"2024-09-10T20:46:04.280145+0000","last_fullsized":"2024-09-10T20:46:04.280145+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:47:54.871210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.f","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286748+0000","last_change":"2024-09-10T20:46:03.279856+0000","last_active":"2024-09-10T20:46:04.286748+0000","last_peered":"2024-09-10T20:46:04.286748+0000","last_clean":"2024-09-10T20:46:04.286748+0000","last_became_active":"2024-09-10T20:46:03.279423+0000","last_became_peered":"2024-09-10T20:46:03.279423+0000","last_unstale":"2024-09-10T20:46:04.286748+0000","last_undegraded":"2024-09-10T20:46:04.286748+0000","last_fullsized":"2024-09-10T20:46:04.286748+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:04.303700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285293+0000","last_change":"2024-09-10T20:46:04.282675+0000","last_active":"2024-09-10T20:46:04.285293+0000","last_peered":"2024-09-10T20:46:04.285293+0000","last_clean":"2024-09-10T20:46:04.285293+0000","last_became_active":"2024-09-10T20:46:04.282404+0000","last_became_peered":"2024-09-10T20:46:04.282404+0000","last_unstale":"2024-09-10T20:46:04.285293+0000","last_undegraded":"2024-09-10T20:46:04.285293+0000","last_fullsized":"2024-09-10T20:46:04.285293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:26.244419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.289134+0000","last_change":"2024-09-10T20:46:03.276034+0000","last_active":"2024-09-10T20:46:04.289134+0000","last_peered":"2024-09-10T20:46:04.289134+0000","last_clean":"2024-09-10T20:46:04.289134+0000","last_became_active":"2024-09-10T20:46:03.275716+0000","last_became_peered":"2024-09-10T20:46:03.275716+0000","last_unstale":"2024-09-10T20:46:04.289134+0000","last_undegraded":"2024-09-10T20:46:04.289134+0000","last_fullsized":"2024-09-10T20:46:04.289134+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:26:13.681116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286905+0000","last_change":"2024-09-10T20:46:04.283712+0000","last_active":"2024-09-10T20:46:04.286905+0000","last_peered":"2024-09-10T20:46:04.286905+0000","last_clean":"2024-09-10T20:46:04.286905+0000","last_became_active":"2024-09-10T20:46:04.281187+0000","last_became_peered":"2024-09-10T20:46:04.281187+0000","last_unstale":"2024-09-10T20:46:04.286905+0000","last_undegraded":"2024-09-10T20:46:04.286905+0000","last_fullsized":"2024-09-10T20:46:04.286905+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:25:15.590015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.d","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.291241+0000","last_change":"2024-09-10T20:46:03.278444+0000","last_active":"2024-09-10T20:46:04.291241+0000","last_peered":"2024-09-10T20:46:04.291241+0000","last_clean":"2024-09-10T20:46:04.291241+0000","last_became_active":"2024-09-10T20:46:03.278140+0000","last_became_peered":"2024-09-10T20:46:03.278140+0000","last_unstale":"2024-09-10T20:46:04.291241+0000","last_undegraded":"2024-09-10T20:46:04.291241+0000","last_fullsized":"2024-09-10T20:46:04.291241+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:02:26.645816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.287052+0000","last_change":"2024-09-10T20:46:04.282878+0000","last_active":"2024-09-10T20:46:04.287052+0000","last_peered":"2024-09-10T20:46:04.287052+0000","last_clean":"2024-09-10T20:46:04.287052+0000","last_became_active":"2024-09-10T20:46:04.282445+0000","last_became_peered":"2024-09-10T20:46:04.282445+0000","last_unstale":"2024-09-10T20:46:04.287052+0000","last_undegraded":"2024-09-10T20:46:04.287052+0000","last_fullsized":"2024-09-10T20:46:04.287052+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:38:56.374744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265156+0000","last_change":"2024-09-10T20:46:03.273431+0000","last_active":"2024-09-10T20:46:04.265156+0000","last_peered":"2024-09-10T20:46:04.265156+0000","last_clean":"2024-09-10T20:46:04.265156+0000","last_became_active":"2024-09-10T20:46:03.273123+0000","last_became_peered":"2024-09-10T20:46:03.273123+0000","last_unstale":"2024-09-10T20:46:04.265156+0000","last_undegraded":"2024-09-10T20:46:04.265156+0000","last_fullsized":"2024-09-10T20:46:04.265156+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:42:26.887811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.286086+0000","last_change":"2024-09-10T20:46:04.278660+0000","last_active":"2024-09-10T20:46:04.286086+0000","last_peered":"2024-09-10T20:46:04.286086+0000","last_clean":"2024-09-10T20:46:04.286086+0000","last_became_active":"2024-09-10T20:46:04.278411+0000","last_became_peered":"2024-09-10T20:46:04.278411+0000","last_unstale":"2024-09-10T20:46:04.286086+0000","last_undegraded":"2024-09-10T20:46:04.286086+0000","last_fullsized":"2024-09-10T20:46:04.286086+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:45:53.175786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.b","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.287776+0000","last_change":"2024-09-10T20:46:03.276639+0000","last_active":"2024-09-10T20:46:04.287776+0000","last_peered":"2024-09-10T20:46:04.287776+0000","last_clean":"2024-09-10T20:46:04.287776+0000","last_became_active":"2024-09-10T20:46:03.275504+0000","last_became_peered":"2024-09-10T20:46:03.275504+0000","last_unstale":"2024-09-10T20:46:04.287776+0000","last_undegraded":"2024-09-10T20:46:04.287776+0000","last_fullsized":"2024-09-10T20:46:04.287776+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:44.666428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280140+0000","last_change":"2024-09-10T20:46:04.275702+0000","last_active":"2024-09-10T20:46:04.280140+0000","last_peered":"2024-09-10T20:46:04.280140+0000","last_clean":"2024-09-10T20:46:04.280140+0000","last_became_active":"2024-09-10T20:46:04.275446+0000","last_became_peered":"2024-09-10T20:46:04.275446+0000","last_unstale":"2024-09-10T20:46:04.280140+0000","last_undegraded":"2024-09-10T20:46:04.280140+0000","last_fullsized":"2024-09-10T20:46:04.280140+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:36:24.675307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.262356+0000","last_change":"2024-09-10T20:46:03.266872+0000","last_active":"2024-09-10T20:46:04.262356+0000","last_peered":"2024-09-10T20:46:04.262356+0000","last_clean":"2024-09-10T20:46:04.262356+0000","last_became_active":"2024-09-10T20:46:03.266558+0000","last_became_peered":"2024-09-10T20:46:03.266558+0000","last_unstale":"2024-09-10T20:46:04.262356+0000","last_undegraded":"2024-09-10T20:46:04.262356+0000","last_fullsized":"2024-09-10T20:46:04.262356+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:52:05.614953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.282681+0000","last_change":"2024-09-10T20:46:04.280133+0000","last_active":"2024-09-10T20:46:04.282681+0000","last_peered":"2024-09-10T20:46:04.282681+0000","last_clean":"2024-09-10T20:46:04.282681+0000","last_became_active":"2024-09-10T20:46:04.279869+0000","last_became_peered":"2024-09-10T20:46:04.279869+0000","last_unstale":"2024-09-10T20:46:04.282681+0000","last_undegraded":"2024-09-10T20:46:04.282681+0000","last_fullsized":"2024-09-10T20:46:04.282681+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:36:35.045228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.269597+0000","last_change":"2024-09-10T20:46:03.275155+0000","last_active":"2024-09-10T20:46:04.269597+0000","last_peered":"2024-09-10T20:46:04.269597+0000","last_clean":"2024-09-10T20:46:04.269597+0000","last_became_active":"2024-09-10T20:46:03.274874+0000","last_became_peered":"2024-09-10T20:46:03.274874+0000","last_unstale":"2024-09-10T20:46:04.269597+0000","last_undegraded":"2024-09-10T20:46:04.269597+0000","last_fullsized":"2024-09-10T20:46:04.269597+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:05.472273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.285668+0000","last_change":"2024-09-10T20:46:04.284753+0000","last_active":"2024-09-10T20:46:04.285668+0000","last_peered":"2024-09-10T20:46:04.285668+0000","last_clean":"2024-09-10T20:46:04.285668+0000","last_became_active":"2024-09-10T20:46:04.284473+0000","last_became_peered":"2024-09-10T20:46:04.284473+0000","last_unstale":"2024-09-10T20:46:04.285668+0000","last_undegraded":"2024-09-10T20:46:04.285668+0000","last_fullsized":"2024-09-10T20:46:04.285668+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:05:52.386213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.264930+0000","last_change":"2024-09-10T20:46:03.267428+0000","last_active":"2024-09-10T20:46:04.264930+0000","last_peered":"2024-09-10T20:46:04.264930+0000","last_clean":"2024-09-10T20:46:04.264930+0000","last_became_active":"2024-09-10T20:46:03.267061+0000","last_became_peered":"2024-09-10T20:46:03.267061+0000","last_unstale":"2024-09-10T20:46:04.264930+0000","last_undegraded":"2024-09-10T20:46:04.264930+0000","last_fullsized":"2024-09-10T20:46:04.264930+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:59:46.489389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.279142+0000","last_change":"2024-09-10T20:46:04.272580+0000","last_active":"2024-09-10T20:46:04.279142+0000","last_peered":"2024-09-10T20:46:04.279142+0000","last_clean":"2024-09-10T20:46:04.279142+0000","last_became_active":"2024-09-10T20:46:04.270671+0000","last_became_peered":"2024-09-10T20:46:04.270671+0000","last_unstale":"2024-09-10T20:46:04.279142+0000","last_undegraded":"2024-09-10T20:46:04.279142+0000","last_fullsized":"2024-09-10T20:46:04.279142+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:59.637959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"161'2","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290515+0000","last_change":"2024-09-10T20:46:03.274514+0000","last_active":"2024-09-10T20:46:04.290515+0000","last_peered":"2024-09-10T20:46:04.290515+0000","last_clean":"2024-09-10T20:46:04.290515+0000","last_became_active":"2024-09-10T20:46:03.274203+0000","last_became_peered":"2024-09-10T20:46:03.274203+0000","last_unstale":"2024-09-10T20:46:04.290515+0000","last_undegraded":"2024-09-10T20:46:04.290515+0000","last_fullsized":"2024-09-10T20:46:04.290515+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:14:51.274955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.277568+0000","last_change":"2024-09-10T20:46:04.268051+0000","last_active":"2024-09-10T20:46:04.277568+0000","last_peered":"2024-09-10T20:46:04.277568+0000","last_clean":"2024-09-10T20:46:04.277568+0000","last_became_active":"2024-09-10T20:46:04.266370+0000","last_became_peered":"2024-09-10T20:46:04.266370+0000","last_unstale":"2024-09-10T20:46:04.277568+0000","last_undegraded":"2024-09-10T20:46:04.277568+0000","last_fullsized":"2024-09-10T20:46:04.277568+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:03:15.698687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.269826+0000","last_change":"2024-09-10T20:46:03.277697+0000","last_active":"2024-09-10T20:46:04.269826+0000","last_peered":"2024-09-10T20:46:04.269826+0000","last_clean":"2024-09-10T20:46:04.269826+0000","last_became_active":"2024-09-10T20:46:03.277324+0000","last_became_peered":"2024-09-10T20:46:03.277324+0000","last_unstale":"2024-09-10T20:46:04.269826+0000","last_undegraded":"2024-09-10T20:46:04.269826+0000","last_fullsized":"2024-09-10T20:46:04.269826+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:41.094488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290364+0000","last_change":"2024-09-10T20:46:04.286805+0000","last_active":"2024-09-10T20:46:04.290364+0000","last_peered":"2024-09-10T20:46:04.290364+0000","last_clean":"2024-09-10T20:46:04.290364+0000","last_became_active":"2024-09-10T20:46:04.285914+0000","last_became_peered":"2024-09-10T20:46:04.285914+0000","last_unstale":"2024-09-10T20:46:04.290364+0000","last_undegraded":"2024-09-10T20:46:04.290364+0000","last_fullsized":"2024-09-10T20:46:04.290364+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:12:05.418352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265853+0000","last_change":"2024-09-10T20:46:03.274127+0000","last_active":"2024-09-10T20:46:04.265853+0000","last_peered":"2024-09-10T20:46:04.265853+0000","last_clean":"2024-09-10T20:46:04.265853+0000","last_became_active":"2024-09-10T20:46:03.273829+0000","last_became_peered":"2024-09-10T20:46:03.273829+0000","last_unstale":"2024-09-10T20:46:04.265853+0000","last_undegraded":"2024-09-10T20:46:04.265853+0000","last_fullsized":"2024-09-10T20:46:04.265853+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:30:53.294709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.273277+0000","last_change":"2024-09-10T20:46:04.270438+0000","last_active":"2024-09-10T20:46:04.273277+0000","last_peered":"2024-09-10T20:46:04.273277+0000","last_clean":"2024-09-10T20:46:04.273277+0000","last_became_active":"2024-09-10T20:46:04.269809+0000","last_became_peered":"2024-09-10T20:46:04.269809+0000","last_unstale":"2024-09-10T20:46:04.273277+0000","last_undegraded":"2024-09-10T20:46:04.273277+0000","last_fullsized":"2024-09-10T20:46:04.273277+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:07:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265483+0000","last_change":"2024-09-10T20:46:03.280294+0000","last_active":"2024-09-10T20:46:04.265483+0000","last_peered":"2024-09-10T20:46:04.265483+0000","last_clean":"2024-09-10T20:46:04.265483+0000","last_became_active":"2024-09-10T20:46:03.279886+0000","last_became_peered":"2024-09-10T20:46:03.279886+0000","last_unstale":"2024-09-10T20:46:04.265483+0000","last_undegraded":"2024-09-10T20:46:04.265483+0000","last_fullsized":"2024-09-10T20:46:04.265483+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:37:37.780197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.280009+0000","last_change":"2024-09-10T20:46:04.273174+0000","last_active":"2024-09-10T20:46:04.280009+0000","last_peered":"2024-09-10T20:46:04.280009+0000","last_clean":"2024-09-10T20:46:04.280009+0000","last_became_active":"2024-09-10T20:46:04.272719+0000","last_became_peered":"2024-09-10T20:46:04.272719+0000","last_unstale":"2024-09-10T20:46:04.280009+0000","last_undegraded":"2024-09-10T20:46:04.280009+0000","last_fullsized":"2024-09-10T20:46:04.280009+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:21.717293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.265953+0000","last_change":"2024-09-10T20:46:03.273174+0000","last_active":"2024-09-10T20:46:04.265953+0000","last_peered":"2024-09-10T20:46:04.265953+0000","last_clean":"2024-09-10T20:46:04.265953+0000","last_became_active":"2024-09-10T20:46:03.272892+0000","last_became_peered":"2024-09-10T20:46:03.272892+0000","last_unstale":"2024-09-10T20:46:04.265953+0000","last_undegraded":"2024-09-10T20:46:04.265953+0000","last_fullsized":"2024-09-10T20:46:04.265953+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:30:16.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290299+0000","last_change":"2024-09-10T20:46:04.286396+0000","last_active":"2024-09-10T20:46:04.290299+0000","last_peered":"2024-09-10T20:46:04.290299+0000","last_clean":"2024-09-10T20:46:04.290299+0000","last_became_active":"2024-09-10T20:46:04.286057+0000","last_became_peered":"2024-09-10T20:46:04.286057+0000","last_unstale":"2024-09-10T20:46:04.290299+0000","last_undegraded":"2024-09-10T20:46:04.290299+0000","last_fullsized":"2024-09-10T20:46:04.290299+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:32.786945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268228+0000","last_change":"2024-09-10T20:46:03.277670+0000","last_active":"2024-09-10T20:46:04.268228+0000","last_peered":"2024-09-10T20:46:04.268228+0000","last_clean":"2024-09-10T20:46:04.268228+0000","last_became_active":"2024-09-10T20:46:03.277344+0000","last_became_peered":"2024-09-10T20:46:03.277344+0000","last_unstale":"2024-09-10T20:46:04.268228+0000","last_undegraded":"2024-09-10T20:46:04.268228+0000","last_fullsized":"2024-09-10T20:46:04.268228+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:29.934871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.274994+0000","last_change":"2024-09-10T20:46:04.269145+0000","last_active":"2024-09-10T20:46:04.274994+0000","last_peered":"2024-09-10T20:46:04.274994+0000","last_clean":"2024-09-10T20:46:04.274994+0000","last_became_active":"2024-09-10T20:46:04.268851+0000","last_became_peered":"2024-09-10T20:46:04.268851+0000","last_unstale":"2024-09-10T20:46:04.274994+0000","last_undegraded":"2024-09-10T20:46:04.274994+0000","last_fullsized":"2024-09-10T20:46:04.274994+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:45.449761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.261322+0000","last_change":"2024-09-10T20:46:03.275747+0000","last_active":"2024-09-10T20:46:04.261322+0000","last_peered":"2024-09-10T20:46:04.261322+0000","last_clean":"2024-09-10T20:46:04.261322+0000","last_became_active":"2024-09-10T20:46:03.275397+0000","last_became_peered":"2024-09-10T20:46:03.275397+0000","last_unstale":"2024-09-10T20:46:04.261322+0000","last_undegraded":"2024-09-10T20:46:04.261322+0000","last_fullsized":"2024-09-10T20:46:04.261322+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:44.666428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.288192+0000","last_change":"2024-09-10T20:46:04.285394+0000","last_active":"2024-09-10T20:46:04.288192+0000","last_peered":"2024-09-10T20:46:04.288192+0000","last_clean":"2024-09-10T20:46:04.288192+0000","last_became_active":"2024-09-10T20:46:04.285025+0000","last_became_peered":"2024-09-10T20:46:04.285025+0000","last_unstale":"2024-09-10T20:46:04.288192+0000","last_undegraded":"2024-09-10T20:46:04.288192+0000","last_fullsized":"2024-09-10T20:46:04.288192+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:37.446329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268932+0000","last_change":"2024-09-10T20:46:03.276263+0000","last_active":"2024-09-10T20:46:04.268932+0000","last_peered":"2024-09-10T20:46:04.268932+0000","last_clean":"2024-09-10T20:46:04.268932+0000","last_became_active":"2024-09-10T20:46:03.275982+0000","last_became_peered":"2024-09-10T20:46:03.275982+0000","last_unstale":"2024-09-10T20:46:04.268932+0000","last_undegraded":"2024-09-10T20:46:04.268932+0000","last_fullsized":"2024-09-10T20:46:04.268932+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:10:34.665955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.287099+0000","last_change":"2024-09-10T20:46:04.283741+0000","last_active":"2024-09-10T20:46:04.287099+0000","last_peered":"2024-09-10T20:46:04.287099+0000","last_clean":"2024-09-10T20:46:04.287099+0000","last_became_active":"2024-09-10T20:46:04.280770+0000","last_became_peered":"2024-09-10T20:46:04.280770+0000","last_unstale":"2024-09-10T20:46:04.287099+0000","last_undegraded":"2024-09-10T20:46:04.287099+0000","last_fullsized":"2024-09-10T20:46:04.287099+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:03.250036+0000","last_clean_scrub_stamp":"2024-09-10T20:46:03.250036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:08:42.236370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"161'1","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.290739+0000","last_change":"2024-09-10T20:46:03.278267+0000","last_active":"2024-09-10T20:46:04.290739+0000","last_peered":"2024-09-10T20:46:04.290739+0000","last_clean":"2024-09-10T20:46:04.290739+0000","last_became_active":"2024-09-10T20:46:03.277979+0000","last_became_peered":"2024-09-10T20:46:03.277979+0000","last_unstale":"2024-09-10T20:46:04.290739+0000","last_undegraded":"2024-09-10T20:46:04.290739+0000","last_fullsized":"2024-09-10T20:46:04.290739+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:46:02.247538+0000","last_clean_scrub_stamp":"2024-09-10T20:46:02.247538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:48:38.838819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"69'79","reported_seq":398,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T20:46:04.268939+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:46:04.268939+0000","last_peered":"2024-09-10T20:46:04.268939+0000","last_clean":"2024-09-10T20:46:04.268939+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:46:04.268939+0000","last_undegraded":"2024-09-10T20:46:04.268939+0000","last_fullsized":"2024-09-10T20:46:04.268939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542243,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2609152,"data_stored":1479992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542243,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2027520,"data_stored":896427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509540,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37320,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334520,"statfs":{"total":96636764160,"available":96598548480,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509540,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37896,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333944,"statfs":{"total":96636764160,"available":96597958656,"internally_reserved":0,"allocated":2609152,"data_stored":1479992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2015232,"data_stored":889624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:46:08.664 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:46:08.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-10T20:46:08.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.808+0000 7faa7b2ef640 1 -- 172.21.15.28:0/2210729658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa7415e130 msgr2=0x7faa740d4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.808+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/2210729658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa7415e130 0x7faa740d4b60 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7faa740d50a0 tx=0x7faa7002a4a0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.808+0000 7faa7b2ef640 1 -- 172.21.15.28:0/2210729658 shutdown_connections 2024-09-10T20:46:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.808+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/2210729658 >> v2:172.21.15.159:3300/0 conn(0x7faa740cfc50 0x7faa74161c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.808+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/2210729658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa7415e130 0x7faa740d4b60 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.808+0000 7faa7b2ef640 1 -- 172.21.15.28:0/2210729658 >> 172.21.15.28:0/2210729658 conn(0x7faa740d4040 msgr2=0x7faa740d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.808+0000 7faa7b2ef640 1 -- 172.21.15.28:0/2210729658 shutdown_connections 2024-09-10T20:46:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.809+0000 7faa7b2ef640 1 -- 172.21.15.28:0/2210729658 wait complete. 2024-09-10T20:46:08.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 Processor -- start 2024-09-10T20:46:08.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 -- start start 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa740b0c90 0x7faa741f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa740cfc50 0x7faa741f8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa7415e130 0x7faa740b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa741622d0 con 0x7faa740cfc50 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faa74162150 con 0x7faa7415e130 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.810+0000 7faa7b2ef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faa74162450 con 0x7faa740b0c90 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa740cfc50 0x7faa741f8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa740cfc50 0x7faa741f8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33148/0 (socket says 172.21.15.28:33148) 2024-09-10T20:46:08.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 -- 172.21.15.28:0/3214319869 learned_addr learned my addr 172.21.15.28:0/3214319869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7908b640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa740b0c90 0x7faa741f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7988c640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa7415e130 0x7faa740b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 -- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa740b0c90 msgr2=0x7faa741f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa740b0c90 0x7faa741f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 -- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa7415e130 msgr2=0x7faa740b8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa7415e130 0x7faa740b8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 -- 172.21.15.28:0/3214319869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa741f7d60 con 0x7faa740cfc50 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7888a640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa740cfc50 0x7faa741f8800 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c530 tx=0x7faa6400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa69ffb640 1 -- 172.21.15.28:0/3214319869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64016020 con 0x7faa740cfc50 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa69ffb640 1 -- 172.21.15.28:0/3214319869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa64012920 con 0x7faa740cfc50 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa69ffb640 1 -- 172.21.15.28:0/3214319869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64012d30 con 0x7faa740cfc50 2024-09-10T20:46:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.811+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa740b9390 con 0x7faa740cfc50 2024-09-10T20:46:08.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.812+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa741f82e0 con 0x7faa740cfc50 2024-09-10T20:46:08.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.812+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa741629e0 con 0x7faa740cfc50 2024-09-10T20:46:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.813+0000 7faa69ffb640 1 -- 172.21.15.28:0/3214319869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7faa6401a3b0 con 0x7faa740cfc50 2024-09-10T20:46:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.813+0000 7faa69ffb640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa58078990 0x7faa5807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.814+0000 7faa69ffb640 1 -- 172.21.15.28:0/3214319869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7faa64098780 con 0x7faa740cfc50 2024-09-10T20:46:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.814+0000 7faa7908b640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa58078990 0x7faa5807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.814+0000 7faa7908b640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa58078990 0x7faa5807ae50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa70034000 tx=0x7faa70004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:08.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:08.818+0000 7faa69ffb640 1 -- 172.21.15.28:0/3214319869 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faa64020030 con 0x7faa740cfc50 2024-09-10T20:46:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.073+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7faa7415db00 con 0x7faa740cfc50 2024-09-10T20:46:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.075+0000 7faa69ffb640 1 -- 172.21.15.28:0/3214319869 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v38) ==== 190+0+147 (secure 0 0 0) 0x7faa64062b70 con 0x7faa740cfc50 2024-09-10T20:46:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T20:46:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvreBmpmB2EBAAkOO3VlxR4Z9JBsSJmswddQ== 2024-09-10T20:46:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-09-10T20:46:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-09-10T20:46:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-10T20:46:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-10T20:46:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa58078990 msgr2=0x7faa5807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa58078990 0x7faa5807ae50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa70034000 tx=0x7faa70004730 comp rx=0 tx=0).stop 2024-09-10T20:46:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa740cfc50 msgr2=0x7faa741f8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa740cfc50 0x7faa741f8800 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c530 tx=0x7faa6400c9f0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 shutdown_connections 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7faa58078990 0x7faa5807ae50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faa7415e130 0x7faa740b8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa740cfc50 0x7faa741f8800 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 --2- 172.21.15.28:0/3214319869 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faa740b0c90 0x7faa741f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 >> 172.21.15.28:0/3214319869 conn(0x7faa740d4040 msgr2=0x7faa740b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 shutdown_connections 2024-09-10T20:46:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.077+0000 7faa7b2ef640 1 -- 172.21.15.28:0/3214319869 wait complete. 2024-09-10T20:46:09.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 -- 172.21.15.28:0/2622748267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54155100 msgr2=0x7f4a5415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/2622748267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54155100 0x7f4a5415bd50 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f4a54072af0 tx=0x7f4a4402a4a0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 -- 172.21.15.28:0/2622748267 shutdown_connections 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/2622748267 >> v2:172.21.15.159:3300/0 conn(0x7f4a540d0140 0x7f4a540d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/2622748267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54155100 0x7f4a5415bd50 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 -- 172.21.15.28:0/2622748267 >> 172.21.15.28:0/2622748267 conn(0x7f4a5406cf00 msgr2=0x7f4a5406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 -- 172.21.15.28:0/2622748267 shutdown_connections 2024-09-10T20:46:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.192+0000 7f4a59c7f640 1 -- 172.21.15.28:0/2622748267 wait complete. 2024-09-10T20:46:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.193+0000 7f4a59c7f640 1 Processor -- start 2024-09-10T20:46:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a59c7f640 1 -- start start 2024-09-10T20:46:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a59c7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a540d0140 0x7f4a54153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a59c7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a54155100 0x7f4a541f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a59c7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a541554e0 0x7f4a541f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a59c7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54072f60 con 0x7f4a541554e0 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a59c7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54072de0 con 0x7f4a540d0140 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a59c7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4a540730e0 con 0x7f4a54155100 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a541554e0 0x7f4a541f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.194+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a541554e0 0x7f4a541f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33174/0 (socket says 172.21.15.28:33174) 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a53fff640 1 -- 172.21.15.28:0/72505017 learned_addr learned my addr 172.21.15.28:0/72505017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a53fff640 1 -- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a54155100 msgr2=0x7f4a541f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a537fe640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a540d0140 0x7f4a54153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a4affd640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a54155100 0x7f4a541f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a53fff640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a54155100 0x7f4a541f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a53fff640 1 -- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a540d0140 msgr2=0x7f4a54153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a53fff640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a540d0140 0x7f4a54153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a53fff640 1 -- 172.21.15.28:0/72505017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a54158240 con 0x7f4a541554e0 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a53fff640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a541554e0 0x7f4a541f2ab0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00b970 tx=0x7f4a4c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a50ff9640 1 -- 172.21.15.28:0/72505017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c019070 con 0x7f4a541554e0 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a50ff9640 1 -- 172.21.15.28:0/72505017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a4c00cc50 con 0x7f4a541554e0 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a540cf9f0 con 0x7f4a541554e0 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a50ff9640 1 -- 172.21.15.28:0/72505017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c0146a0 con 0x7f4a541554e0 2024-09-10T20:46:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.195+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a540b1250 con 0x7f4a541554e0 2024-09-10T20:46:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.196+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a54158410 con 0x7f4a541554e0 2024-09-10T20:46:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.197+0000 7f4a50ff9640 1 -- 172.21.15.28:0/72505017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f4a4c01d3b0 con 0x7f4a541554e0 2024-09-10T20:46:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.197+0000 7f4a50ff9640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a34078990 0x7f4a3407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.198+0000 7f4a50ff9640 1 -- 172.21.15.28:0/72505017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f4a4c098930 con 0x7f4a541554e0 2024-09-10T20:46:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.198+0000 7f4a537fe640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a34078990 0x7f4a3407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.198+0000 7f4a537fe640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a34078990 0x7f4a3407ae50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4a44034000 tx=0x7f4a44004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:09.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.202+0000 7f4a50ff9640 1 -- 172.21.15.28:0/72505017 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4a4c062cc0 con 0x7f4a541554e0 2024-09-10T20:46:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.428+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a540735b0 con 0x7f4a541554e0 2024-09-10T20:46:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.428+0000 7f4a50ff9640 1 -- 172.21.15.28:0/72505017 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5239 (secure 0 0 0) 0x7f4a4c0625c0 con 0x7f4a541554e0 2024-09-10T20:46:09.432 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:09.433 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":116,"btime":"2024-09-10T20:46:07:289311+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4088436065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4088436065},{"type":"v1","addr":"172.21.15.28:6834","nonce":4088436065}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2443719102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2443719102},{"type":"v1","addr":"172.21.15.159:6801","nonce":2443719102}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":7250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3153361013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3153361013},{"type":"v1","addr":"172.21.15.159:6835","nonce":3153361013}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:46:03.628094+0000","modified":"2024-09-10T20:46:06.656668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9078":{"gid":9078,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1516352866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1516352866},{"type":"v1","addr":"172.21.15.28:6835","nonce":1516352866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9078,"qdb_cluster":[9078]},"id":13}]} 2024-09-10T20:46:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 116 2024-09-10T20:46:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a34078990 msgr2=0x7f4a3407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a34078990 0x7f4a3407ae50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4a44034000 tx=0x7f4a44004730 comp rx=0 tx=0).stop 2024-09-10T20:46:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a541554e0 msgr2=0x7f4a541f2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a541554e0 0x7f4a541f2ab0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00b970 tx=0x7f4a4c00be30 comp rx=0 tx=0).stop 2024-09-10T20:46:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 shutdown_connections 2024-09-10T20:46:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4a34078990 0x7f4a3407ae50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a541554e0 0x7f4a541f2ab0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4a54155100 0x7f4a541f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 --2- 172.21.15.28:0/72505017 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4a540d0140 0x7f4a54153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 >> 172.21.15.28:0/72505017 conn(0x7f4a5406cf00 msgr2=0x7f4a540737c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 shutdown_connections 2024-09-10T20:46:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.431+0000 7f4a59c7f640 1 -- 172.21.15.28:0/72505017 wait complete. 2024-09-10T20:46:09.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-10T20:46:03.628094+0000', 'modified': '2024-09-10T20:46:06.656668+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9078}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9078': {'gid': 9078, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1516352866', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1516352866}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1516352866}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9078, 'qdb_cluster': [9078]} 2024-09-10T20:46:09.849 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-10T20:46:09.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-10T20:46:09.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1340500709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00155020 msgr2=0x7f9f000b4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1340500709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00155020 0x7f9f000b4c40 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f9f00158c70 tx=0x7f9ef002aa30 comp rx=0 tx=0).stop 2024-09-10T20:46:09.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1340500709 shutdown_connections 2024-09-10T20:46:09.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1340500709 >> v2:172.21.15.159:3300/0 conn(0x7f9f000cf370 0x7f9f000d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1340500709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00155020 0x7f9f000b4c40 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1340500709 >> 172.21.15.28:0/1340500709 conn(0x7f9f0006cf00 msgr2=0x7f9f0006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1340500709 shutdown_connections 2024-09-10T20:46:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.954+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1340500709 wait complete. 2024-09-10T20:46:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.955+0000 7f9f07cbf640 1 Processor -- start 2024-09-10T20:46:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.955+0000 7f9f07cbf640 1 -- start start 2024-09-10T20:46:09.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f07cbf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f000cf370 0x7f9f0014d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f07cbf640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f000d00f0 0x7f9f000b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f07cbf640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f00155020 0x7f9f0006f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f07cbf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f000ad0f0 con 0x7f9f000cf370 2024-09-10T20:46:09.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f07cbf640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9f000acf70 con 0x7f9f000d00f0 2024-09-10T20:46:09.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f07cbf640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9f000ad270 con 0x7f9f00155020 2024-09-10T20:46:09.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f05a5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f000cf370 0x7f9f0014d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f05a5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f000cf370 0x7f9f0014d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33192/0 (socket says 172.21.15.28:33192) 2024-09-10T20:46:09.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f05a5b640 1 -- 172.21.15.28:0/1177123776 learned_addr learned my addr 172.21.15.28:0/1177123776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:09.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f0525a640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f000d00f0 0x7f9f000b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.956+0000 7f9f0625c640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f00155020 0x7f9f0006f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f05a5b640 1 -- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f00155020 msgr2=0x7f9f0006f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f00155020 0x7f9f0006f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f05a5b640 1 -- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f000d00f0 msgr2=0x7f9f000b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f000d00f0 0x7f9f000b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f05a5b640 1 -- 172.21.15.28:0/1177123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f000b05f0 con 0x7f9f000cf370 2024-09-10T20:46:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f0625c640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f00155020 0x7f9f0006f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f000cf370 0x7f9f0014d580 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009bf0 tx=0x7f9ef0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9efe7fc640 1 -- 172.21.15.28:0/1177123776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0033050 con 0x7f9f000cf370 2024-09-10T20:46:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9efe7fc640 1 -- 172.21.15.28:0/1177123776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ef0004550 con 0x7f9f000cf370 2024-09-10T20:46:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f000b08d0 con 0x7f9f000cf370 2024-09-10T20:46:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9efe7fc640 1 -- 172.21.15.28:0/1177123776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0038450 con 0x7f9f000cf370 2024-09-10T20:46:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.957+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0014dac0 con 0x7f9f000cf370 2024-09-10T20:46:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.959+0000 7f9efe7fc640 1 -- 172.21.15.28:0/1177123776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f9ef0040070 con 0x7f9f000cf370 2024-09-10T20:46:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.960+0000 7f9efe7fc640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9eec0789c0 0x7f9eec07ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.960+0000 7f9efe7fc640 1 -- 172.21.15.28:0/1177123776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f9ef00b9a60 con 0x7f9f000cf370 2024-09-10T20:46:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.960+0000 7f9f0525a640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9eec0789c0 0x7f9eec07ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.960+0000 7f9f0525a640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9eec0789c0 0x7f9eec07ae80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9f000b2930 tx=0x7f9ef4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.960+0000 7f9f0aa8e740 1 Processor -- start 2024-09-10T20:46:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0aa8e740 1 -- start start 2024-09-10T20:46:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x5619690fab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x5619691431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619690f85f0 0x5619690f8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56196906f820 con 0x5619690f85f0 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x56196905ec50 con 0x561969142df0 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x561969062a60 con 0x5619691231b0 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0625c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619690f85f0 0x5619690f8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0625c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619690f85f0 0x5619690f8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33204/0 (socket says 172.21.15.28:33204) 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.961+0000 7f9f0625c640 1 -- 172.21.15.28:0/2754766143 learned_addr learned my addr 172.21.15.28:0/2754766143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x5619690fab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f0525a640 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x5619691431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f0625c640 1 -- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 msgr2=0x5619690fab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f0625c640 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x5619690fab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f0625c640 1 -- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 msgr2=0x5619691431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f0625c640 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x5619691431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f0625c640 1 -- 172.21.15.28:0/2754766143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561968dc0850 con 0x5619690f85f0 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x5619690fab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9f0625c640 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619690f85f0 0x5619690f8a40 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800cc30 tx=0x7f9ef800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c60c72dd9e7d4b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9ee77fe640 1 -- 172.21.15.28:0/2754766143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8014070 con 0x5619690f85f0 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9ee77fe640 1 -- 172.21.15.28:0/2754766143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ef8004330 con 0x5619690f85f0 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.962+0000 7f9ee77fe640 1 -- 172.21.15.28:0/2754766143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8003830 con 0x5619690f85f0 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2754766143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619690f85f0 msgr2=0x5619690f8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619690f85f0 0x5619690f8a40 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800cc30 tx=0x7f9ef800e720 comp rx=0 tx=0).stop 2024-09-10T20:46:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2754766143 shutdown_connections 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619690f85f0 0x5619690f8a40 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x5619691431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2754766143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x5619690fab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2754766143 >> 172.21.15.28:0/2754766143 conn(0x5619690f96a0 msgr2=0x5619690fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2754766143 shutdown_connections 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2754766143 wait complete. 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.963+0000 7f9f0aa8e740 1 -- start start 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.964+0000 7f9f0aa8e740 1 Processor -- start 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.964+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x56196913d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.964+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561969142df0 0x56196913dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.964+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969154f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561968dcf620 con 0x561969142df0 2024-09-10T20:46:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x5619690045f0 con 0x56196913b500 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x5619690f7b30 con 0x5619691231b0 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0aa8e740 10 client.0 _ll_register_callbacks cb 0x5619690080a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561969142df0 0x56196913dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561969142df0 0x56196913dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33220/0 (socket says 172.21.15.28:33220) 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 -- 172.21.15.28:0/3136504679 learned_addr learned my addr 172.21.15.28:0/3136504679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x56196913d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0625c640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 -- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 msgr2=0x56196913d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x56196913d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 -- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 msgr2=0x561969154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 -- 172.21.15.28:0/3136504679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561968e376b0 con 0x561969142df0 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0625c640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0525a640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561969142df0 0x56196913dcd0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4014700 tx=0x7f9ef4001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e2e5355ce3b6a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9ee4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3136504679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4013680 con 0x561969142df0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ef8004330 con 0x561969142df0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.965+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3136504679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f9ef4013820 con 0x561969142df0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3136504679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f9ef402d050 con 0x561969142df0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 10 client.9207 fetch_fsmap learned FSMap version 116 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x56196913c230 con 0x561969142df0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3136504679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ef8004330 con 0x561969142df0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3136504679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 116) ==== 3502+0+0 (secure 0 0 0) 0x7f9ef4030020 con 0x561969142df0 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 10 client.9207 fetch_fsmap finished waiting for FSMap version 116 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 10 client.9207 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:46:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 20 client.9207 populate_metadata read hostname 'smithi028' 2024-09-10T20:46:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691519e0 0x561969139920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 4 client.9207 mds_command: new command op to 9078 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:46:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 --> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561969071770 con 0x5619691519e0 2024-09-10T20:46:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691519e0 0x561969139920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9ee4ff9640 10 client.9207 ms_handle_connect on v2:172.21.15.28:6833/1516352866 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.966+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691519e0 0x561969139920 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a647897dac0bae87 server_cookie=9fe9e1ee683aca04 in_seq=0 out_seq=0 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3136504679 <== mds.0 v2:172.21.15.28:6833/1516352866 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561969071770 con 0x5619691519e0 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9ee4ff9640 10 client.9207 handle_command_reply: tid=0 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 client.9207 shutdown 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561969142df0 msgr2=0x56196913dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561969142df0 0x56196913dcd0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4014700 tx=0x7f9ef4001020 comp rx=0 tx=0).stop 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 shutdown_connections 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691519e0 0x561969139920 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561969142df0 0x56196913dcd0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9ee4ff9640 0 client.9207 ms_handle_reset on v2:172.21.15.28:6833/1516352866 2024-09-10T20:46:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3136504679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x5619691231b0 0x56196913d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 >> 172.21.15.28:0/3136504679 conn(0x5619690f85f0 msgr2=0x5619690fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 shutdown_connections 2024-09-10T20:46:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3136504679 wait complete. 2024-09-10T20:46:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.967+0000 7f9f0aa8e740 20 client.9207 trim_cache size 0 max 16384 2024-09-10T20:46:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 Processor -- start 2024-09-10T20:46:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 -- start start 2024-09-10T20:46:09.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x561969137e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56196913b500 0x561969138370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x561969165200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f002003d0 con 0x5619691231b0 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f9f000b03a0 con 0x561969142df0 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.986+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f9f001f5490 con 0x56196913b500 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x561969137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x561969137e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33232/0 (socket says 172.21.15.28:33232) 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 -- 172.21.15.28:0/2738488989 learned_addr learned my addr 172.21.15.28:0/2738488989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f0625c640 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x561969165200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f0525a640 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56196913b500 0x561969138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 -- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56196913b500 msgr2=0x561969138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56196913b500 0x561969138370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 -- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 msgr2=0x561969165200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x561969165200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 -- 172.21.15.28:0/2738488989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561969144f80 con 0x5619691231b0 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f0625c640 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x561969165200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f0525a640 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56196913b500 0x561969138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x561969137e30 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00838a0 tx=0x7f9ef00bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6d41cc0c42d2ea server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9ee6ffd640 1 -- 172.21.15.28:0/2738488989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0083910 con 0x5619691231b0 2024-09-10T20:46:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9ee6ffd640 1 -- 172.21.15.28:0/2738488989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ef00883f0 con 0x5619691231b0 2024-09-10T20:46:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.987+0000 7f9ee6ffd640 1 -- 172.21.15.28:0/2738488989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef008ce10 con 0x5619691231b0 2024-09-10T20:46:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2738488989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 msgr2=0x561969137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x561969137e30 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00838a0 tx=0x7f9ef00bc040 comp rx=0 tx=0).stop 2024-09-10T20:46:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2738488989 shutdown_connections 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x561969142df0 0x561969165200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56196913b500 0x561969138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/2738488989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x561969137e30 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2738488989 >> 172.21.15.28:0/2738488989 conn(0x5619690f85f0 msgr2=0x56196911a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2738488989 shutdown_connections 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/2738488989 wait complete. 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 -- start start 2024-09-10T20:46:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.988+0000 7f9f0aa8e740 1 Processor -- start 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.989+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x5619691bc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.989+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969137e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.989+0000 7f9f0aa8e740 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561969142df0 0x561969138370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.989+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561969151860 con 0x5619691231b0 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.989+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x561969155940 con 0x56196913b500 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.989+0000 7f9f0aa8e740 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x561969151e50 con 0x561969142df0 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.989+0000 7f9f0aa8e740 10 client.0 _ll_register_callbacks cb 0x7f9ef4017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f0625c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561969142df0 0x561969138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f0525a640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f05a5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x5619691bc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f0625c640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561969142df0 0x561969138370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:42308/0 (socket says 172.21.15.28:42308) 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f0625c640 1 -- 172.21.15.28:0/3927529556 learned_addr learned my addr 172.21.15.28:0/3927529556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f05a5b640 1 -- 172.21.15.28:0/3927529556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561969142df0 msgr2=0x561969138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561969142df0 0x561969138370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f05a5b640 1 -- 172.21.15.28:0/3927529556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 msgr2=0x561969137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969137e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f05a5b640 1 -- 172.21.15.28:0/3927529556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56196915b5b0 con 0x5619691231b0 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f0625c640 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561969142df0 0x561969138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f05a5b640 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x5619691bc410 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00b9a30 tx=0x7f9ef00831b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1348916fb6c53e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9ee4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3927529556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00b7060 con 0x5619691231b0 2024-09-10T20:46:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3927529556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f9ef007fb80 con 0x5619691231b0 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.990+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ef00883f0 con 0x5619691231b0 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3927529556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f9ef0083910 con 0x5619691231b0 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 10 client.9213 fetch_fsmap learned FSMap version 116 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x56196915b890 con 0x5619691231b0 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3927529556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ef00883f0 con 0x5619691231b0 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3927529556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 116) ==== 3502+0+0 (secure 0 0 0) 0x7f9ef006d030 con 0x5619691231b0 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 10 client.9213 fetch_fsmap finished waiting for FSMap version 116 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 10 client.9213 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 20 client.9213 populate_metadata read hostname 'smithi028' 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691c27e0 0x5619691c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 4 client.9213 mds_command: new command op to 9078 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 --> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] -- command(tid 0: {"prefix": "status"}) -- 0x56196906d220 con 0x5619691c27e0 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0525a640 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691c27e0 0x5619691c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9ee4ff9640 10 client.9213 ms_handle_connect on v2:172.21.15.28:6833/1516352866 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.991+0000 7f9f0525a640 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691c27e0 0x5619691c2bc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a0e6fc54b5f7081 server_cookie=ade1ffe569492864 in_seq=0 out_seq=0 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9ee4ff9640 1 -- 172.21.15.28:0/3927529556 <== mds.0 v2:172.21.15.28:6833/1516352866 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x56196906d220 con 0x5619691c27e0 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9ee4ff9640 10 client.9213 handle_command_reply: tid=0 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 client.9213 shutdown 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 msgr2=0x5619691bc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x5619691bc410 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00b9a30 tx=0x7f9ef00831b0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 shutdown_connections 2024-09-10T20:46:10.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x561969142df0 0x561969138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:6833/1516352866,v1:172.21.15.28:6835/1516352866] conn(0x5619691c27e0 0x5619691c2bc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56196913b500 0x561969137e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 --2- 172.21.15.28:0/3927529556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619691231b0 0x5619691bc410 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 >> 172.21.15.28:0/3927529556 conn(0x5619690f85f0 msgr2=0x5619690f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9ee4ff9640 0 client.9213 ms_handle_reset on v2:172.21.15.28:6833/1516352866 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 shutdown_connections 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 1 -- 172.21.15.28:0/3927529556 wait complete. 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f0aa8e740 20 client.9213 trim_cache size 0 max 16384 2024-09-10T20:46:10.004 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "943b2354-acb0-43ad-b06a-e7974842506f", 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 9078, 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.7333421729999996, 2024-09-10T20:46:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 116, 2024-09-10T20:46:10.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 162, 2024-09-10T20:46:10.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 162, 2024-09-10T20:46:10.006 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.419960250999999 2024-09-10T20:46:10.006 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9eec0789c0 msgr2=0x7f9eec07ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9eec0789c0 0x7f9eec07ae80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9f000b2930 tx=0x7f9ef4009040 comp rx=0 tx=0).stop 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f000cf370 msgr2=0x7f9f0014d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.992+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f000cf370 0x7f9f0014d580 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009bf0 tx=0x7f9ef0003f10 comp rx=0 tx=0).stop 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 shutdown_connections 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f9eec0789c0 0x7f9eec07ae80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f00155020 0x7f9f0006f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f000d00f0 0x7f9f000b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 --2- 172.21.15.28:0/1177123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f000cf370 0x7f9f0014d580 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 >> 172.21.15.28:0/1177123776 conn(0x7f9f0006cf00 msgr2=0x7f9f000737c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 shutdown_connections 2024-09-10T20:46:10.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:09.993+0000 7f9f07cbf640 1 -- 172.21.15.28:0/1177123776 wait complete. 2024-09-10T20:46:10.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:46:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/4167650944 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a640b4d10 msgr2=0x7f5a640ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/4167650944 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a640b4d10 0x7f5a640ce030 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c005fd0 tx=0x7f5a5c02a7a0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/4167650944 shutdown_connections 2024-09-10T20:46:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/4167650944 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a640b4d10 0x7f5a640ce030 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/4167650944 >> v2:172.21.15.159:3300/0 conn(0x7f5a6415a190 0x7f5a640b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/4167650944 >> 172.21.15.28:0/4167650944 conn(0x7f5a6406cf00 msgr2=0x7f5a6406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/4167650944 shutdown_connections 2024-09-10T20:46:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.110+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/4167650944 wait complete. 2024-09-10T20:46:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 Processor -- start 2024-09-10T20:46:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 -- start start 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a640b4d10 0x7f5a641467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a6415a190 0x7f5a641596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6415df70 0x7f5a6409cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6415a570 con 0x7f5a6415a190 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5a640ceae0 con 0x7f5a640b4d10 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.112+0000 7f5a6a3d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5a6415a6f0 con 0x7f5a6415df70 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a6415a190 0x7f5a641596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a6415a190 0x7f5a641596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33260/0 (socket says 172.21.15.28:33260) 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 -- 172.21.15.28:0/628593818 learned_addr learned my addr 172.21.15.28:0/628593818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 -- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6415df70 msgr2=0x7f5a6409cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a63fff640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a640b4d10 0x7f5a641467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a68970640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6415df70 0x7f5a6409cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6415df70 0x7f5a6409cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 -- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a640b4d10 msgr2=0x7f5a641467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a640b4d10 0x7f5a641467d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 -- 172.21.15.28:0/628593818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a640aff10 con 0x7f5a6415a190 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a63fff640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a640b4d10 0x7f5a641467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a637fe640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a6415a190 0x7f5a641596f0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800c500 tx=0x7f5a5800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a60ff9640 1 -- 172.21.15.28:0/628593818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58016020 con 0x7f5a6415a190 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a60ff9640 1 -- 172.21.15.28:0/628593818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a58012920 con 0x7f5a6415a190 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a640afc30 con 0x7f5a6415a190 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a60ff9640 1 -- 172.21.15.28:0/628593818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58012d70 con 0x7f5a6415a190 2024-09-10T20:46:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.113+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a6409c7d0 con 0x7f5a6415a190 2024-09-10T20:46:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.114+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a640ce030 con 0x7f5a6415a190 2024-09-10T20:46:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.115+0000 7f5a60ff9640 1 -- 172.21.15.28:0/628593818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f5a5801b400 con 0x7f5a6415a190 2024-09-10T20:46:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.116+0000 7f5a60ff9640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5a500789c0 0x7f5a5007ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.116+0000 7f5a60ff9640 1 -- 172.21.15.28:0/628593818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5737+0+0 (secure 0 0 0) 0x7f5a58098930 con 0x7f5a6415a190 2024-09-10T20:46:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.116+0000 7f5a63fff640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5a500789c0 0x7f5a5007ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.116+0000 7f5a63fff640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5a500789c0 0x7f5a5007ae80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009b50 tx=0x7f5a54005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.120+0000 7f5a60ff9640 1 -- 172.21.15.28:0/628593818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5a58020030 con 0x7f5a6415a190 2024-09-10T20:46:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.346+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5a640cee00 con 0x7f5a6415a190 2024-09-10T20:46:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.346+0000 7f5a60ff9640 1 -- 172.21.15.28:0/628593818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v116) ==== 53+0+73 (secure 0 0 0) 0x7f5a58062cc0 con 0x7f5a6415a190 2024-09-10T20:46:10.350 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:46:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5a500789c0 msgr2=0x7f5a5007ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5a500789c0 0x7f5a5007ae80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009b50 tx=0x7f5a54005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a6415a190 msgr2=0x7f5a641596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a6415a190 0x7f5a641596f0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800c500 tx=0x7f5a5800c9c0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 shutdown_connections 2024-09-10T20:46:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5a500789c0 0x7f5a5007ae80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6415df70 0x7f5a6409cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a6415a190 0x7f5a641596f0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 --2- 172.21.15.28:0/628593818 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a640b4d10 0x7f5a641467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.348+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 >> 172.21.15.28:0/628593818 conn(0x7f5a6406cf00 msgr2=0x7f5a6406d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.349+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 shutdown_connections 2024-09-10T20:46:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10.349+0000 7f5a6a3d3640 1 -- 172.21.15.28:0/628593818 wait complete. 2024-09-10T20:46:10.363 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:46:10.364 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:46:10.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:10.364 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-10T20:46:10.381 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:46:10.382 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:10.382 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-10T20:46:10.382 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:46:10.382 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:10.382 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-10T20:46:10.382 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-10T20:46:10.382 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42260sec preferred_lft 42260sec 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::800e:81ff:fe05:7b01/64 scope link 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 22:d8:55:24:1b:75 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:46:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::20d8:55ff:fe24:1b75/64 scope link 2024-09-10T20:46:10.384 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:10.384 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:46:10.384 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ce:b4:6f:be:9c:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:46:10.384 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ccb4:6fff:febe:9c70/64 scope link 2024-09-10T20:46:10.384 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:10.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:10.384 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:46:10.445 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-10T20:46:10.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:10.447 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-10T20:46:10.505 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-10T20:46:10.505 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:46:10.505 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-10T20:46:10.505 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-10T20:46:10.505 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:46:10.506 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:46:10.506 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-10T20:46:10.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-10T20:46:10.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-10T20:46:10.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-10T20:46:10.506 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-10T20:46:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:46:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:46:10.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:10.634 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:46:10.650 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:46:10.650 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:46:10.650 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:46:10.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:10.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-10T20:46:10.716 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:46:10.717 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T20:46:10.774 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T20:46:10.775 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:46:10.775 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:46:10.829 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:46:10.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:10.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:46:10.914 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:46:10.914 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:46:10.914 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:46:10.923 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:46:10.923 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:46:10.923 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:46:10.924 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:46:10.924 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:46:10.928 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T20:46:10.928 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T20:46:10.928 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-10T20:46:10.928 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:46:10.928 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:46:10.928 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:46:10.928 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:46:10.929 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:46:10.929 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T20:46:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:46:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:46:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:46:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:46:10.930 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:46:10.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:10.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:46:10.956 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:46:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:46:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:46:11.058 INFO:teuthology.orchestra.run.smithi028.stdout:client9222 2024-09-10T20:46:11.059 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:46:11.059 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:11.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-10T20:46:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 -- 172.21.15.28:0/1555637977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 msgr2=0x7f361c15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 --2- 172.21.15.28:0/1555637977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c15e930 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f3610009830 tx=0x7f361002a510 comp rx=0 tx=0).stop 2024-09-10T20:46:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 -- 172.21.15.28:0/1555637977 shutdown_connections 2024-09-10T20:46:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 --2- 172.21.15.28:0/1555637977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f361c15ee70 0x7f361c0d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 --2- 172.21.15.28:0/1555637977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c15e930 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 -- 172.21.15.28:0/1555637977 >> 172.21.15.28:0/1555637977 conn(0x7f361c06cf00 msgr2=0x7f361c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 -- 172.21.15.28:0/1555637977 shutdown_connections 2024-09-10T20:46:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.217+0000 7f362127f640 1 -- 172.21.15.28:0/1555637977 wait complete. 2024-09-10T20:46:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.218+0000 7f362127f640 1 Processor -- start 2024-09-10T20:46:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.218+0000 7f362127f640 1 -- start start 2024-09-10T20:46:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f362127f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f361c155100 0x7f361c152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f362127f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f362127f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f361c15ee70 0x7f361c0ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f362127f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f361c072de0 con 0x7f361c1554e0 2024-09-10T20:46:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f362127f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f361c0d0af0 con 0x7f361c155100 2024-09-10T20:46:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f362127f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f361c072f60 con 0x7f361c15ee70 2024-09-10T20:46:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f361a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f361a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c0cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40104/0 (socket says 172.21.15.28:40104) 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f361a7fc640 1 -- 172.21.15.28:0/2687447320 learned_addr learned my addr 172.21.15.28:0/2687447320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f361b7fe640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f361c15ee70 0x7f361c0ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.219+0000 7f361affd640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f361c155100 0x7f361c152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f361a7fc640 1 -- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f361c15ee70 msgr2=0x7f361c0ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f361a7fc640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f361c15ee70 0x7f361c0ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f361a7fc640 1 -- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f361c155100 msgr2=0x7f361c152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f361a7fc640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f361c155100 0x7f361c152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f361a7fc640 1 -- 172.21.15.28:0/2687447320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361c152c90 con 0x7f361c1554e0 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f361a7fc640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c0cc6d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f3610009950 tx=0x7f3610032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f36037fe640 1 -- 172.21.15.28:0/2687447320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610040070 con 0x7f361c1554e0 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f36037fe640 1 -- 172.21.15.28:0/2687447320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3610005310 con 0x7f361c1554e0 2024-09-10T20:46:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361c0adc00 con 0x7f361c1554e0 2024-09-10T20:46:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f36037fe640 1 -- 172.21.15.28:0/2687447320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36100047a0 con 0x7f361c1554e0 2024-09-10T20:46:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.220+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361c0afba0 con 0x7f361c1554e0 2024-09-10T20:46:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.221+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f361c15e930 con 0x7f361c1554e0 2024-09-10T20:46:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.222+0000 7f36037fe640 1 -- 172.21.15.28:0/2687447320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f3610004940 con 0x7f361c1554e0 2024-09-10T20:46:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.222+0000 7f36037fe640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3608078990 0x7f360807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.222+0000 7f36037fe640 1 -- 172.21.15.28:0/2687447320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5795+0+0 (secure 0 0 0) 0x7f36100b9ba0 con 0x7f361c1554e0 2024-09-10T20:46:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.222+0000 7f361affd640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3608078990 0x7f360807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.223+0000 7f361affd640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3608078990 0x7f360807ae50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f360c007fd0 tx=0x7f360c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.227+0000 7f36037fe640 1 -- 172.21.15.28:0/2687447320 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3610084150 con 0x7f361c1554e0 2024-09-10T20:46:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.453+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f361c0ae050 con 0x7f361c1554e0 2024-09-10T20:46:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.453+0000 7f36037fe640 1 -- 172.21.15.28:0/2687447320 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v116) ==== 53+0+73 (secure 0 0 0) 0x7f3610083a50 con 0x7f361c1554e0 2024-09-10T20:46:11.457 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-10T20:46:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3608078990 msgr2=0x7f360807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3608078990 0x7f360807ae50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f360c007fd0 tx=0x7f360c009040 comp rx=0 tx=0).stop 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 msgr2=0x7f361c0cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c0cc6d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f3610009950 tx=0x7f3610032c10 comp rx=0 tx=0).stop 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 shutdown_connections 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3608078990 0x7f360807ae50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f361c15ee70 0x7f361c0ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c1554e0 0x7f361c0cc6d0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 --2- 172.21.15.28:0/2687447320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f361c155100 0x7f361c152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 >> 172.21.15.28:0/2687447320 conn(0x7f361c06cf00 msgr2=0x7f361c074250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 shutdown_connections 2024-09-10T20:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:11.456+0000 7f362127f640 1 -- 172.21.15.28:0/2687447320 wait complete. 2024-09-10T20:46:11.471 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-10T20:46:11.471 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T20:46:11.471 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-09-10T20:46:11.471 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T20:46:11.471 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T20:46:11.471 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-10T20:46:11.472 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-10T20:46:11.472 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T20:46:11.472 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:46:11.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:11.472 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-09-10T20:46:11.492 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T20:46:11.492 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T20:46:11.492 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T20:46:11.492 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:11.492 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-09-10T20:46:11.492 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:11.492 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42277sec preferred_lft 42277sec 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::ec4:7aff:fe88:724f/64 scope link noprefixroute 2024-09-10T20:46:11.493 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::e026:55ff:fe74:2419/64 scope link 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 26:90:87:ba:17:0f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::2490:87ff:feba:170f/64 scope link 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether ca:94:b2:c1:85:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:46:11.494 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::c894:b2ff:fec1:85ca/64 scope link 2024-09-10T20:46:11.495 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-09-10T20:46:11.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:11.495 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T20:46:11.560 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-10T20:46:11.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:11.562 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-09-10T20:46:11.620 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-09-10T20:46:11.620 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T20:46:11.620 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T20:46:11.620 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T20:46:11.621 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T20:46:11.621 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T20:46:11.621 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-09-10T20:46:11.621 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T20:46:11.621 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T20:46:11.621 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T20:46:11.621 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-09-10T20:46:11.715 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:46:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T20:46:11.749 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:46:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:46:11.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:11.757 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:46:11.773 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:46:11.773 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:46:11.773 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:46:11.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:11.774 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-09-10T20:46:11.839 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:46:11.840 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T20:46:11.895 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T20:46:11.895 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T20:46:11.895 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:46:11.951 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T20:46:11.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:11.951 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:46:12.034 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:46:12.034 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:46:12.034 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:46:12.043 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:46:12.043 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:46:12.043 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:46:12.043 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:46:12.043 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:invalid new device string format 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-09-10T20:46:12.054 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=943b2354-acb0-43ad-b06a-e7974842506f" will pass to kernel 2024-09-10T20:46:12.055 INFO:teuthology.orchestra.run.smithi159.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T20:46:12.055 INFO:teuthology.orchestra.run.smithi159.stdout: You just mounted a file system that supports labels which does not 2024-09-10T20:46:12.055 INFO:teuthology.orchestra.run.smithi159.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T20:46:12.055 INFO:teuthology.orchestra.run.smithi159.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T20:46:12.055 INFO:teuthology.orchestra.run.smithi159.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T20:46:12.056 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T20:46:12.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:12.057 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T20:46:12.121 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T20:46:12.211 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:46:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:46:12.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-10T20:46:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:46:12.218 INFO:teuthology.orchestra.run.smithi159.stdout:client9234 2024-09-10T20:46:12.219 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:46:12.219 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:12.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-10T20:46:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 -- 172.21.15.28:0/2167352124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 msgr2=0x7efc8c06fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 --2- 172.21.15.28:0/2167352124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c06fd30 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7efc8c06db30 tx=0x7efc8002a5f0 comp rx=0 tx=0).stop 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 -- 172.21.15.28:0/2167352124 shutdown_connections 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 --2- 172.21.15.28:0/2167352124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c06fd30 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 --2- 172.21.15.28:0/2167352124 >> v2:172.21.15.159:3300/0 conn(0x7efc8c151cf0 0x7efc8c15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 -- 172.21.15.28:0/2167352124 >> 172.21.15.28:0/2167352124 conn(0x7efc8c0cfc80 msgr2=0x7efc8c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 -- 172.21.15.28:0/2167352124 shutdown_connections 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.345+0000 7efc929ec640 1 -- 172.21.15.28:0/2167352124 wait complete. 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.346+0000 7efc929ec640 1 Processor -- start 2024-09-10T20:46:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.347+0000 7efc929ec640 1 -- start start 2024-09-10T20:46:12.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.347+0000 7efc929ec640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc8c070270 0x7efc8c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:12.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.347+0000 7efc929ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c1f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:12.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.347+0000 7efc929ec640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc8c151cf0 0x7efc8c0c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:12.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.347+0000 7efc929ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c15eed0 con 0x7efc8c072db0 2024-09-10T20:46:12.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.347+0000 7efc929ec640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c0cb840 con 0x7efc8c151cf0 2024-09-10T20:46:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.347+0000 7efc929ec640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7efc8c15f050 con 0x7efc8c070270 2024-09-10T20:46:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c1f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c1f1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40120/0 (socket says 172.21.15.28:40120) 2024-09-10T20:46:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2977776201 learned_addr learned my addr 172.21.15.28:0/2977776201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc8c070270 0x7efc8c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc90f89640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc8c151cf0 0x7efc8c0c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc8c070270 msgr2=0x7efc8c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc8c070270 0x7efc8c0b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc8c151cf0 msgr2=0x7efc8c0c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc8c151cf0 0x7efc8c0c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2977776201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c1508c0 con 0x7efc8c072db0 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc8b7fe640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c1f1b70 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7efc80009850 tx=0x7efc80003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc88ff9640 1 -- 172.21.15.28:0/2977776201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8003f070 con 0x7efc8c072db0 2024-09-10T20:46:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc88ff9640 1 -- 172.21.15.28:0/2977776201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc800047a0 con 0x7efc8c072db0 2024-09-10T20:46:12.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc88ff9640 1 -- 172.21.15.28:0/2977776201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80035030 con 0x7efc8c072db0 2024-09-10T20:46:12.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8c150ba0 con 0x7efc8c072db0 2024-09-10T20:46:12.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.348+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc8c14dc90 con 0x7efc8c072db0 2024-09-10T20:46:12.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.349+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc8c14e070 con 0x7efc8c072db0 2024-09-10T20:46:12.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.350+0000 7efc88ff9640 1 -- 172.21.15.28:0/2977776201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7efc80004bb0 con 0x7efc8c072db0 2024-09-10T20:46:12.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.351+0000 7efc88ff9640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efc78078990 0x7efc7807ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:12.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.351+0000 7efc88ff9640 1 -- 172.21.15.28:0/2977776201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7efc800b98a0 con 0x7efc8c072db0 2024-09-10T20:46:12.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.351+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efc78078990 0x7efc7807ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:12.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.351+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efc78078990 0x7efc7807ae50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009b30 tx=0x7efc7c005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:12.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.355+0000 7efc88ff9640 1 -- 172.21.15.28:0/2977776201 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efc800835e0 con 0x7efc8c072db0 2024-09-10T20:46:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:12.578+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7efc8c15f500 con 0x7efc8c072db0 2024-09-10T20:46:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.321+0000 7efc88ff9640 1 -- 172.21.15.28:0/2977776201 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v117) ==== 136+0+0 (secure 0 0 0) 0x7efc80088010 con 0x7efc8c072db0 2024-09-10T20:46:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efc78078990 msgr2=0x7efc7807ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efc78078990 0x7efc7807ae50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009b30 tx=0x7efc7c005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 msgr2=0x7efc8c1f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c1f1b70 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7efc80009850 tx=0x7efc80003820 comp rx=0 tx=0).stop 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 shutdown_connections 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7efc78078990 0x7efc7807ae50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efc8c151cf0 0x7efc8c0c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c072db0 0x7efc8c1f1b70 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 --2- 172.21.15.28:0/2977776201 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efc8c070270 0x7efc8c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.324+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 >> 172.21.15.28:0/2977776201 conn(0x7efc8c0cfc80 msgr2=0x7efc8c0d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.325+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 shutdown_connections 2024-09-10T20:46:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.325+0000 7efc929ec640 1 -- 172.21.15.28:0/2977776201 wait complete. 2024-09-10T20:46:13.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-10T20:46:13.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.451+0000 7f45c9752640 1 -- 172.21.15.28:0/4093965264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40cc570 msgr2=0x7f45c40d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.451+0000 7f45c9752640 1 --2- 172.21.15.28:0/4093965264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40cc570 0x7f45c40d0090 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009860 tx=0x7f45b802a540 comp rx=0 tx=0).stop 2024-09-10T20:46:13.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.451+0000 7f45c9752640 1 -- 172.21.15.28:0/4093965264 shutdown_connections 2024-09-10T20:46:13.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.451+0000 7f45c9752640 1 --2- 172.21.15.28:0/4093965264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40cc570 0x7f45c40d0090 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.451+0000 7f45c9752640 1 -- 172.21.15.28:0/4093965264 >> 172.21.15.28:0/4093965264 conn(0x7f45c406cf00 msgr2=0x7f45c406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:13.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.451+0000 7f45c9752640 1 -- 172.21.15.28:0/4093965264 shutdown_connections 2024-09-10T20:46:13.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.451+0000 7f45c9752640 1 -- 172.21.15.28:0/4093965264 wait complete. 2024-09-10T20:46:13.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.452+0000 7f45c9752640 1 Processor -- start 2024-09-10T20:46:13.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.452+0000 7f45c9752640 1 -- start start 2024-09-10T20:46:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c9752640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45c40cc570 0x7f45c41e8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c9752640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45c40cd3d0 0x7f45c4147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c9752640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40d05d0 0x7f45c4147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c9752640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45c40b3590 con 0x7f45c40d05d0 2024-09-10T20:46:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c9752640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f45c40b4b60 con 0x7f45c40cc570 2024-09-10T20:46:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c9752640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f45c40b3710 con 0x7f45c40cd3d0 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40d05d0 0x7f45c4147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40d05d0 0x7f45c4147880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40148/0 (socket says 172.21.15.28:40148) 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 -- 172.21.15.28:0/2776117407 learned_addr learned my addr 172.21.15.28:0/2776117407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c27fc640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45c40cd3d0 0x7f45c4147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c2ffd640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45c40cc570 0x7f45c41e8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 -- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45c40cd3d0 msgr2=0x7f45c4147340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45c40cd3d0 0x7f45c4147340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 -- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45c40cc570 msgr2=0x7f45c41e8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45c40cc570 0x7f45c41e8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.453+0000 7f45c37fe640 1 -- 172.21.15.28:0/2776117407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c41f4cc0 con 0x7f45c40d05d0 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.454+0000 7f45c37fe640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40d05d0 0x7f45c4147880 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00b970 tx=0x7f45bc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:13.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.454+0000 7f45ab7fe640 1 -- 172.21.15.28:0/2776117407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc019070 con 0x7f45c40d05d0 2024-09-10T20:46:13.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.454+0000 7f45ab7fe640 1 -- 172.21.15.28:0/2776117407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45bc00cc30 con 0x7f45c40d05d0 2024-09-10T20:46:13.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.454+0000 7f45ab7fe640 1 -- 172.21.15.28:0/2776117407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0146a0 con 0x7f45c40d05d0 2024-09-10T20:46:13.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.454+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c41f4f60 con 0x7f45c40d05d0 2024-09-10T20:46:13.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.454+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c41f5520 con 0x7f45c40d05d0 2024-09-10T20:46:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.455+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c40b3b70 con 0x7f45c40d05d0 2024-09-10T20:46:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.456+0000 7f45ab7fe640 1 -- 172.21.15.28:0/2776117407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f45bc01e440 con 0x7f45c40d05d0 2024-09-10T20:46:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.456+0000 7f45ab7fe640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f45b0078990 0x7f45b007ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.456+0000 7f45ab7fe640 1 -- 172.21.15.28:0/2776117407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f45bc098770 con 0x7f45c40d05d0 2024-09-10T20:46:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.456+0000 7f45c2ffd640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f45b0078990 0x7f45b007ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.457+0000 7f45c2ffd640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f45b0078990 0x7f45b007ae50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f45c40b28f0 tx=0x7f45b4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:13.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.460+0000 7f45ab7fe640 1 -- 172.21.15.28:0/2776117407 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f45bc062b00 con 0x7f45c40d05d0 2024-09-10T20:46:13.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.670+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f45c409de20 con 0x7f45b0078990 2024-09-10T20:46:13.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.671+0000 7f45ab7fe640 1 -- 172.21.15.28:0/2776117407 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f45c409de20 con 0x7f45b0078990 2024-09-10T20:46:13.675 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-10T20:46:13.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T20:46:13.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephfs" 2024-09-10T20:46:13.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T20:46:13.675 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f45b0078990 msgr2=0x7f45b007ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f45b0078990 0x7f45b007ae50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f45c40b28f0 tx=0x7f45b4009040 comp rx=0 tx=0).stop 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40d05d0 msgr2=0x7f45c4147880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40d05d0 0x7f45c4147880 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00b970 tx=0x7f45bc00be30 comp rx=0 tx=0).stop 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 shutdown_connections 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f45b0078990 0x7f45b007ae50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45c40d05d0 0x7f45c4147880 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45c40cd3d0 0x7f45c4147340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 --2- 172.21.15.28:0/2776117407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45c40cc570 0x7f45c41e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 >> 172.21.15.28:0/2776117407 conn(0x7f45c406cf00 msgr2=0x7f45c415ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 shutdown_connections 2024-09-10T20:46:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.673+0000 7f45c9752640 1 -- 172.21.15.28:0/2776117407 wait complete. 2024-09-10T20:46:13.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-10T20:46:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 -- 172.21.15.28:0/436837399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8161770 msgr2=0x7fc0c80d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 --2- 172.21.15.28:0/436837399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8161770 0x7fc0c80d20e0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80d5170 tx=0x7fc0b802a4a0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 -- 172.21.15.28:0/436837399 shutdown_connections 2024-09-10T20:46:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 --2- 172.21.15.28:0/436837399 >> v2:172.21.15.159:3300/0 conn(0x7fc0c8161b50 0x7fc0c80d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 --2- 172.21.15.28:0/436837399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8161770 0x7fc0c80d20e0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 -- 172.21.15.28:0/436837399 >> 172.21.15.28:0/436837399 conn(0x7fc0c80d4070 msgr2=0x7fc0c80d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 -- 172.21.15.28:0/436837399 shutdown_connections 2024-09-10T20:46:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.793+0000 7fc0cde08640 1 -- 172.21.15.28:0/436837399 wait complete. 2024-09-10T20:46:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.794+0000 7fc0cde08640 1 Processor -- start 2024-09-10T20:46:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0cde08640 1 -- start start 2024-09-10T20:46:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0cde08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c80d4b90 0x7fc0c815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0cde08640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c8161770 0x7fc0c815d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0cde08640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c8161b50 0x7fc0c81f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0cde08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c80b2f50 con 0x7fc0c80d4b90 2024-09-10T20:46:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0cde08640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c80b2dd0 con 0x7fc0c8161770 2024-09-10T20:46:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0cde08640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c80b30d0 con 0x7fc0c8161b50 2024-09-10T20:46:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.795+0000 7fc0c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c80d4b90 0x7fc0c815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c80d4b90 0x7fc0c815dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40166/0 (socket says 172.21.15.28:40166) 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 -- 172.21.15.28:0/2281018623 learned_addr learned my addr 172.21.15.28:0/2281018623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c6ffd640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c8161770 0x7fc0c815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c7fff640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c8161b50 0x7fc0c81f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 -- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c8161b50 msgr2=0x7fc0c81f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c8161b50 0x7fc0c81f5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 -- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c8161770 msgr2=0x7fc0c815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c8161770 0x7fc0c815d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 -- 172.21.15.28:0/2281018623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c815d9b0 con 0x7fc0c80d4b90 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c6ffd640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c8161770 0x7fc0c815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0c77fe640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c80d4b90 0x7fc0c815dba0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80098f0 tx=0x7fc0b8032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0affff640 1 -- 172.21.15.28:0/2281018623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8040070 con 0x7fc0c80d4b90 2024-09-10T20:46:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0affff640 1 -- 172.21.15.28:0/2281018623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0b8005310 con 0x7fc0c80d4b90 2024-09-10T20:46:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c80c8160 con 0x7fc0c80d4b90 2024-09-10T20:46:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0affff640 1 -- 172.21.15.28:0/2281018623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b80047a0 con 0x7fc0c80d4b90 2024-09-10T20:46:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.796+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c80c8720 con 0x7fc0c80d4b90 2024-09-10T20:46:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.797+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c80b3870 con 0x7fc0c80d4b90 2024-09-10T20:46:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.798+0000 7fc0affff640 1 -- 172.21.15.28:0/2281018623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fc0b8004940 con 0x7fc0c80d4b90 2024-09-10T20:46:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.798+0000 7fc0affff640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078990 0x7fc0b407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.799+0000 7fc0affff640 1 -- 172.21.15.28:0/2281018623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7fc0b80b9c40 con 0x7fc0c80d4b90 2024-09-10T20:46:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.799+0000 7fc0c6ffd640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078990 0x7fc0b407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.799+0000 7fc0c6ffd640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078990 0x7fc0b407ae50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80d4630 tx=0x7fc0bc005db0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:13.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.803+0000 7fc0affff640 1 -- 172.21.15.28:0/2281018623 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc0b8084110 con 0x7fc0c80d4b90 2024-09-10T20:46:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.987+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc0c80b12f0 con 0x7fc0c80d4b90 2024-09-10T20:46:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.987+0000 7fc0affff640 1 -- 172.21.15.28:0/2281018623 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7fc0b8083a10 con 0x7fc0c80d4b90 2024-09-10T20:46:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078990 msgr2=0x7fc0b407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078990 0x7fc0b407ae50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80d4630 tx=0x7fc0bc005db0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c80d4b90 msgr2=0x7fc0c815dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c80d4b90 0x7fc0c815dba0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80098f0 tx=0x7fc0b8032c10 comp rx=0 tx=0).stop 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 shutdown_connections 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fc0b4078990 0x7fc0b407ae50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0c8161b50 0x7fc0c81f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0c8161770 0x7fc0c815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 --2- 172.21.15.28:0/2281018623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c80d4b90 0x7fc0c815dba0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 >> 172.21.15.28:0/2281018623 conn(0x7fc0c80d4070 msgr2=0x7fc0c80ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 shutdown_connections 2024-09-10T20:46:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:13.989+0000 7fc0cde08640 1 -- 172.21.15.28:0/2281018623 wait complete. 2024-09-10T20:46:14.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:14.004 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp95640/subvol20530) 2024-09-10T20:46:14.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:14.035 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-10T20:46:14.092 INFO:teuthology.orchestra.run.smithi028.stdout:cephfs_data 2024-09-10T20:46:14.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:14.093 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp95640/subvol20530) 2024-09-10T20:46:14.161 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:46:14.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp95640/subvol20530/.meta 2024-09-10T20:46:14.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol20530 subvol_grp95640 2024-09-10T20:46:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.379+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2001550520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91d4004d40 msgr2=0x7f91d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.379+0000 7f91f0c7f640 1 --2- 172.21.15.28:0/2001550520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91d4004d40 0x7f91d40051c0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c590 tx=0x7f91d4005830 comp rx=0 tx=0).stop 2024-09-10T20:46:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.379+0000 7f91ead9d640 1 -- 172.21.15.28:0/2001550520 reap_dead start 2024-09-10T20:46:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.379+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2001550520 shutdown_connections 2024-09-10T20:46:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.379+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2001550520 >> 172.21.15.28:0/2001550520 conn(0x7f91ec0d1da0 msgr2=0x7f91ec0d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.379+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2001550520 shutdown_connections 2024-09-10T20:46:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.379+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2001550520 wait complete. 2024-09-10T20:46:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.380+0000 7f91f0c7f640 1 Processor -- start 2024-09-10T20:46:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.380+0000 7f91f0c7f640 1 -- start start 2024-09-10T20:46:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91f0c7f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d4004d40 0x7f91ec1f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91f0c7f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91ec1f23d0 0x7f91ec146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91f0c7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec1473a0 0x7f91ec15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91f0c7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec0cfaf0 con 0x7f91ec1473a0 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91f0c7f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec0cf970 con 0x7f91d4004d40 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91f0c7f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f91ec0cfc70 con 0x7f91ec1f23d0 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec1473a0 0x7f91ec15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec1473a0 0x7f91ec15a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40186/0 (socket says 172.21.15.28:40186) 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 -- 172.21.15.28:0/2191731188 learned_addr learned my addr 172.21.15.28:0/2191731188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ea59c640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d4004d40 0x7f91ec1f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91e9d9b640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91ec1f23d0 0x7f91ec146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 -- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91ec1f23d0 msgr2=0x7f91ec146e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91ec1f23d0 0x7f91ec146e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 -- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d4004d40 msgr2=0x7f91ec1f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d4004d40 0x7f91ec1f2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 -- 172.21.15.28:0/2191731188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec159800 con 0x7f91ec1473a0 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ea59c640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d4004d40 0x7f91ec1f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.381+0000 7f91ead9d640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec1473a0 0x7f91ec15a280 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f91e4000f80 tx=0x7f91e4004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.382+0000 7f91daffd640 1 -- 172.21.15.28:0/2191731188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e402b930 con 0x7f91ec1473a0 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.382+0000 7f91daffd640 1 -- 172.21.15.28:0/2191731188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91e402bbc0 con 0x7f91ec1473a0 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.382+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec1f3190 con 0x7f91ec1473a0 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.382+0000 7f91daffd640 1 -- 172.21.15.28:0/2191731188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e40060a0 con 0x7f91ec1473a0 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.382+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec147b00 con 0x7f91ec1473a0 2024-09-10T20:46:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.383+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ec0b67c0 con 0x7f91ec1473a0 2024-09-10T20:46:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.383+0000 7f91daffd640 1 -- 172.21.15.28:0/2191731188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7f91e4006240 con 0x7f91ec1473a0 2024-09-10T20:46:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.384+0000 7f91daffd640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078990 0x7f91cc07ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.384+0000 7f91daffd640 1 -- 172.21.15.28:0/2191731188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f91e40b8780 con 0x7f91ec1473a0 2024-09-10T20:46:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.384+0000 7f91ea59c640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078990 0x7f91cc07ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.384+0000 7f91ea59c640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078990 0x7f91cc07ae50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009c50 tx=0x7f91dc005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.388+0000 7f91daffd640 1 -- 172.21.15.28:0/2191731188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f91ec159800 con 0x7f91ec1473a0 2024-09-10T20:46:14.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.602+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol20530", "group_name": "subvol_grp95640", "target": ["mon-mgr", ""]}) -- 0x7f91ec0b2ee0 con 0x7f91cc078990 2024-09-10T20:46:14.610 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:14.607+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:14.611 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:14.607+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:14.611 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:14.607+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:14.611 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:14.607+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:14.611 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:14.607+0000 7f93ce3fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.637+0000 7f91daffd640 1 -- 172.21.15.28:0/2191731188 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f91ec0b2ee0 con 0x7f91cc078990 2024-09-10T20:46:14.641 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp95640/subvol20530 2024-09-10T20:46:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078990 msgr2=0x7f91cc07ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078990 0x7f91cc07ae50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009c50 tx=0x7f91dc005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec1473a0 msgr2=0x7f91ec15a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec1473a0 0x7f91ec15a280 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f91e4000f80 tx=0x7f91e4004bb0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 shutdown_connections 2024-09-10T20:46:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f91cc078990 0x7f91cc07ae50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec1473a0 0x7f91ec15a280 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f91ec1f23d0 0x7f91ec146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 --2- 172.21.15.28:0/2191731188 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91d4004d40 0x7f91ec1f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 >> 172.21.15.28:0/2191731188 conn(0x7f91ec0d1da0 msgr2=0x7f91ec0d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 shutdown_connections 2024-09-10T20:46:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.639+0000 7f91f0c7f640 1 -- 172.21.15.28:0/2191731188 wait complete. 2024-09-10T20:46:14.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol20530 subvol_grp95640 2024-09-10T20:46:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 -- 172.21.15.28:0/1900466988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea0161850 msgr2=0x7faea00d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 --2- 172.21.15.28:0/1900466988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea0161850 0x7faea00d4a80 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7faea00ce570 tx=0x7fae9402a5d0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 -- 172.21.15.28:0/1900466988 shutdown_connections 2024-09-10T20:46:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 --2- 172.21.15.28:0/1900466988 >> v2:172.21.15.159:3300/0 conn(0x7faea00b4c20 0x7faea00d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 --2- 172.21.15.28:0/1900466988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea0161850 0x7faea00d4a80 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 -- 172.21.15.28:0/1900466988 >> 172.21.15.28:0/1900466988 conn(0x7faea00d3f90 msgr2=0x7faea00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 -- 172.21.15.28:0/1900466988 shutdown_connections 2024-09-10T20:46:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.759+0000 7faea74b1640 1 -- 172.21.15.28:0/1900466988 wait complete. 2024-09-10T20:46:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.760+0000 7faea74b1640 1 Processor -- start 2024-09-10T20:46:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.760+0000 7faea74b1640 1 -- start start 2024-09-10T20:46:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea74b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea00b4c20 0x7faea00cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea74b1640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faea0161850 0x7faea0156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea74b1640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faea0162000 0x7faea00cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea74b1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faea00d21a0 con 0x7faea00b4c20 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea74b1640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7faea00d2320 con 0x7faea0161850 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea74b1640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7faea00d24a0 con 0x7faea0162000 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea524d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea00b4c20 0x7faea00cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea524d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea00b4c20 0x7faea00cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40242/0 (socket says 172.21.15.28:40242) 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.761+0000 7faea524d640 1 -- 172.21.15.28:0/1984815367 learned_addr learned my addr 172.21.15.28:0/1984815367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea4a4c640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faea0161850 0x7faea0156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea5a4e640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faea0162000 0x7faea00cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea524d640 1 -- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faea0162000 msgr2=0x7faea00cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea524d640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faea0162000 0x7faea00cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea524d640 1 -- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faea0161850 msgr2=0x7faea0156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea524d640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faea0161850 0x7faea0156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea524d640 1 -- 172.21.15.28:0/1984815367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea00cc050 con 0x7faea00b4c20 2024-09-10T20:46:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea524d640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea00b4c20 0x7faea00cba70 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c3a0 tx=0x7fae9800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7fae9dffb640 1 -- 172.21.15.28:0/1984815367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98011790 con 0x7faea00b4c20 2024-09-10T20:46:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7fae9dffb640 1 -- 172.21.15.28:0/1984815367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae98017070 con 0x7faea00b4c20 2024-09-10T20:46:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea00cc330 con 0x7faea00b4c20 2024-09-10T20:46:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7fae9dffb640 1 -- 172.21.15.28:0/1984815367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98011cc0 con 0x7faea00b4c20 2024-09-10T20:46:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.762+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea00cdfa0 con 0x7faea00b4c20 2024-09-10T20:46:14.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.763+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea00d1c60 con 0x7faea00b4c20 2024-09-10T20:46:14.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.764+0000 7fae9dffb640 1 -- 172.21.15.28:0/1984815367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 101519+0+0 (secure 0 0 0) 0x7fae9801c460 con 0x7faea00b4c20 2024-09-10T20:46:14.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.765+0000 7fae9dffb640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae8c0789c0 0x7fae8c07ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:14.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.765+0000 7faea4a4c640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae8c0789c0 0x7fae8c07ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:14.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.765+0000 7fae9dffb640 1 -- 172.21.15.28:0/1984815367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7fae98098710 con 0x7faea00b4c20 2024-09-10T20:46:14.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.765+0000 7faea4a4c640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae8c0789c0 0x7fae8c07ae80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fae94034000 tx=0x7fae94004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.769+0000 7fae9dffb640 1 -- 172.21.15.28:0/1984815367 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fae980629b0 con 0x7faea00b4c20 2024-09-10T20:46:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.981+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol20530", "group_name": "subvol_grp95640", "target": ["mon-mgr", ""]}) -- 0x7faea00b9f10 con 0x7fae8c0789c0 2024-09-10T20:46:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.985+0000 7fae9dffb640 1 -- 172.21.15.28:0/1984815367 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faea00b9f10 con 0x7fae8c0789c0 2024-09-10T20:46:14.989 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/subvol_grp95640/subvol20530 2024-09-10T20:46:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae8c0789c0 msgr2=0x7fae8c07ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae8c0789c0 0x7fae8c07ae80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fae94034000 tx=0x7fae94004730 comp rx=0 tx=0).stop 2024-09-10T20:46:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea00b4c20 msgr2=0x7faea00cba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea00b4c20 0x7faea00cba70 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c3a0 tx=0x7fae9800c860 comp rx=0 tx=0).stop 2024-09-10T20:46:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 shutdown_connections 2024-09-10T20:46:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae8c0789c0 0x7fae8c07ae80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faea0162000 0x7faea00cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faea0161850 0x7faea0156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 --2- 172.21.15.28:0/1984815367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faea00b4c20 0x7faea00cba70 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 >> 172.21.15.28:0/1984815367 conn(0x7faea00d3f90 msgr2=0x7faea015ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 shutdown_connections 2024-09-10T20:46:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:14.988+0000 7faea74b1640 1 -- 172.21.15.28:0/1984815367 wait complete. 2024-09-10T20:46:15.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:15.003 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/5fa1c73555b7affcdedf640a0091b2b3.meta) 2024-09-10T20:46:19.584 INFO:teuthology.orchestra.run.smithi028.stdout:[GLOBAL] 2024-09-10T20:46:19.585 INFO:teuthology.orchestra.run.smithi028.stdout:version = 1 2024-09-10T20:46:19.585 INFO:teuthology.orchestra.run.smithi028.stdout:type = subvolume 2024-09-10T20:46:19.585 INFO:teuthology.orchestra.run.smithi028.stdout:path = /volumes/subvol_grp95640/subvol20530 2024-09-10T20:46:19.585 INFO:teuthology.orchestra.run.smithi028.stdout:state = complete 2024-09-10T20:46:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:19.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol20530 subvol_grp95640 2024-09-10T20:46:19.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 -- 172.21.15.28:0/3983419118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d80b4c20 msgr2=0x7f71d8161140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:19.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 --2- 172.21.15.28:0/3983419118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d80b4c20 0x7f71d8161140 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009830 tx=0x7f71c002a5a0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 -- 172.21.15.28:0/3983419118 shutdown_connections 2024-09-10T20:46:19.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 --2- 172.21.15.28:0/3983419118 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d80b4c20 0x7f71d8161140 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 --2- 172.21.15.28:0/3983419118 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d8162000 0x7f71d80d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 -- 172.21.15.28:0/3983419118 >> 172.21.15.28:0/3983419118 conn(0x7f71d80d3f90 msgr2=0x7f71d80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:19.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 -- 172.21.15.28:0/3983419118 shutdown_connections 2024-09-10T20:46:19.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.705+0000 7f71dd332640 1 -- 172.21.15.28:0/3983419118 wait complete. 2024-09-10T20:46:19.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.706+0000 7f71dd332640 1 Processor -- start 2024-09-10T20:46:19.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.706+0000 7f71dd332640 1 -- start start 2024-09-10T20:46:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71dd332640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d80b4c20 0x7f71d81f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71dd332640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d8162000 0x7f71d81eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71dd332640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d81623e0 0x7f71d80b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71dd332640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71d8161800 con 0x7f71d81623e0 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71dd332640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f71d8161680 con 0x7f71d8162000 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71dd332640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f71d8161980 con 0x7f71d80b4c20 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d81623e0 0x7f71d80b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d81623e0 0x7f71d80b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40252/0 (socket says 172.21.15.28:40252) 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71d77fe640 1 -- 172.21.15.28:0/3828075445 learned_addr learned my addr 172.21.15.28:0/3828075445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71cffff640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d8162000 0x7f71d81eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.707+0000 7f71d6ffd640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d80b4c20 0x7f71d81f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71d77fe640 1 -- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d80b4c20 msgr2=0x7f71d81f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71d77fe640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d80b4c20 0x7f71d81f5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71d77fe640 1 -- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d8162000 msgr2=0x7f71d81eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71d77fe640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d8162000 0x7f71d81eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71d77fe640 1 -- 172.21.15.28:0/3828075445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d80b2ea0 con 0x7f71d81623e0 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71d77fe640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d81623e0 0x7f71d80b0ac0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c700 tx=0x7f71d000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3828075445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d0003b70 con 0x7f71d81623e0 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3828075445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71d0003d10 con 0x7f71d81623e0 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d80b31e0 con 0x7f71d81623e0 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3828075445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d000aab0 con 0x7f71d81623e0 2024-09-10T20:46:19.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.708+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d80b3a20 con 0x7f71d81623e0 2024-09-10T20:46:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.709+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d806ea60 con 0x7f71d81623e0 2024-09-10T20:46:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.710+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3828075445 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 28) ==== 101570+0+0 (secure 0 0 0) 0x7f71d000ac50 con 0x7f71d81623e0 2024-09-10T20:46:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.710+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f71b8078a20 0x7f71b807aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.710+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3828075445 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f71d0098380 con 0x7f71d81623e0 2024-09-10T20:46:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.710+0000 7f71d6ffd640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f71b8078a20 0x7f71b807aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.711+0000 7f71d6ffd640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f71b8078a20 0x7f71b807aee0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009530 tx=0x7f71c8009260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:19.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.715+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3828075445 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f71d806ea60 con 0x7f71d81623e0 2024-09-10T20:46:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.925+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol20530", "group_name": "subvol_grp95640", "target": ["mon-mgr", ""]}) -- 0x7f71d80d16d0 con 0x7f71b8078a20 2024-09-10T20:46:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.938+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3828075445 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f71d80d16d0 con 0x7f71b8078a20 2024-09-10T20:46:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f71b8078a20 msgr2=0x7f71b807aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f71b8078a20 0x7f71b807aee0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009530 tx=0x7f71c8009260 comp rx=0 tx=0).stop 2024-09-10T20:46:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d81623e0 msgr2=0x7f71d80b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d81623e0 0x7f71d80b0ac0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c700 tx=0x7f71d000cbc0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 shutdown_connections 2024-09-10T20:46:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f71b8078a20 0x7f71b807aee0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d81623e0 0x7f71d80b0ac0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f71d8162000 0x7f71d81eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 --2- 172.21.15.28:0/3828075445 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f71d80b4c20 0x7f71d81f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 >> 172.21.15.28:0/3828075445 conn(0x7f71d80d3f90 msgr2=0x7f71d806df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 shutdown_connections 2024-09-10T20:46:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:19.941+0000 7f71dd332640 1 -- 172.21.15.28:0/3828075445 wait complete. 2024-09-10T20:46:19.948 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.943+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.948 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.943+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.948 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.943+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.948 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.943+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.948 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.943+0000 7f93d0401640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.960 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-10T20:46:19.963 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.957+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.963 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.957+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.963 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.957+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.963 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.957+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:19.963 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-10T20:46:19.957+0000 7f93cfc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-10T20:46:20.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T20:46:20.014 INFO:teuthology.orchestra.run.smithi028.stdout:2 2024-09-10T20:46:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T20:46:20.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp95640 2024-09-10T20:46:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 -- 172.21.15.28:0/2736533113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2b70 msgr2=0x7ff5dc074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2736533113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2b70 0x7ff5dc074340 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009830 tx=0x7ff5cc02a510 comp rx=0 tx=0).stop 2024-09-10T20:46:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 -- 172.21.15.28:0/2736533113 shutdown_connections 2024-09-10T20:46:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2736533113 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc06f8a0 0x7ff5dc163ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2736533113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2b70 0x7ff5dc074340 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 -- 172.21.15.28:0/2736533113 >> 172.21.15.28:0/2736533113 conn(0x7ff5dc06d110 msgr2=0x7ff5dc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 -- 172.21.15.28:0/2736533113 shutdown_connections 2024-09-10T20:46:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.124+0000 7ff5e1623640 1 -- 172.21.15.28:0/2736533113 wait complete. 2024-09-10T20:46:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.125+0000 7ff5e1623640 1 Processor -- start 2024-09-10T20:46:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.125+0000 7ff5e1623640 1 -- start start 2024-09-10T20:46:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5e1623640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc06f8a0 0x7ff5dc149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5e1623640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc0b2b70 0x7ff5dc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5e1623640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2f50 0x7ff5dc0cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5e1623640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc164590 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5e1623640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc164410 con 0x7ff5dc0b2b70 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5e1623640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff5dc164710 con 0x7ff5dc06f8a0 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2f50 0x7ff5dc0cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2f50 0x7ff5dc0cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40292/0 (socket says 172.21.15.28:40292) 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 -- 172.21.15.28:0/2077191698 learned_addr learned my addr 172.21.15.28:0/2077191698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5daffd640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc06f8a0 0x7ff5dc149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5d27fc640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc0b2b70 0x7ff5dc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 -- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc06f8a0 msgr2=0x7ff5dc149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc06f8a0 0x7ff5dc149360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 -- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc0b2b70 msgr2=0x7ff5dc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc0b2b70 0x7ff5dc155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5db7fe640 1 -- 172.21.15.28:0/2077191698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc1f2720 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.126+0000 7ff5d27fc640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc0b2b70 0x7ff5dc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.127+0000 7ff5db7fe640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2f50 0x7ff5dc0cd360 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400e960 tx=0x7ff5d400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.127+0000 7ff5d3fff640 1 -- 172.21.15.28:0/2077191698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4004d20 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.127+0000 7ff5d3fff640 1 -- 172.21.15.28:0/2077191698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5d400cb20 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.127+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5dc1f29e0 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.127+0000 7ff5d3fff640 1 -- 172.21.15.28:0/2077191698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4005020 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.127+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5dc1f2fa0 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.128+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5dc074340 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.129+0000 7ff5d3fff640 1 -- 172.21.15.28:0/2077191698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 28) ==== 101570+0+0 (secure 0 0 0) 0x7ff5d400f040 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.129+0000 7ff5d3fff640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff5bc078a20 0x7ff5bc07aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.129+0000 7ff5daffd640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff5bc078a20 0x7ff5bc07aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.129+0000 7ff5d3fff640 1 -- 172.21.15.28:0/2077191698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7ff5d4098b60 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.130+0000 7ff5daffd640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff5bc078a20 0x7ff5bc07aee0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc034000 tx=0x7ff5cc004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:20.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.134+0000 7ff5d3fff640 1 -- 172.21.15.28:0/2077191698 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff5d4016030 con 0x7ff5dc0b2f50 2024-09-10T20:46:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.346+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp95640", "target": ["mon-mgr", ""]}) -- 0x7ff5dc150f50 con 0x7ff5bc078a20 2024-09-10T20:46:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.350+0000 7ff5d3fff640 1 -- 172.21.15.28:0/2077191698 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff5dc150f50 con 0x7ff5bc078a20 2024-09-10T20:46:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff5bc078a20 msgr2=0x7ff5bc07aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff5bc078a20 0x7ff5bc07aee0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc034000 tx=0x7ff5cc004730 comp rx=0 tx=0).stop 2024-09-10T20:46:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2f50 msgr2=0x7ff5dc0cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2f50 0x7ff5dc0cd360 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400e960 tx=0x7ff5d400ee20 comp rx=0 tx=0).stop 2024-09-10T20:46:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 shutdown_connections 2024-09-10T20:46:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ff5bc078a20 0x7ff5bc07aee0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5dc0b2f50 0x7ff5dc0cd360 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff5dc0b2b70 0x7ff5dc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 --2- 172.21.15.28:0/2077191698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff5dc06f8a0 0x7ff5dc149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 >> 172.21.15.28:0/2077191698 conn(0x7ff5dc06d110 msgr2=0x7ff5dc14cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 shutdown_connections 2024-09-10T20:46:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.352+0000 7ff5e1623640 1 -- 172.21.15.28:0/2077191698 wait complete. 2024-09-10T20:46:20.368 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-10T20:46:20.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:46:20.406 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-10T20:46:20.451 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-10T20:46:20.451 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:46:20.451 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:46:20.451 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:20.467 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:46:20.467 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:20.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:46:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 -- 172.21.15.28:0/1470360924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd4151cc0 msgr2=0x7f4fd40cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/1470360924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd4151cc0 0x7f4fd40cf0f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4005fd0 tx=0x7f4fc402a5c0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 -- 172.21.15.28:0/1470360924 shutdown_connections 2024-09-10T20:46:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/1470360924 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fd415eef0 0x7f4fd40cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/1470360924 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd4151cc0 0x7f4fd40cf0f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 -- 172.21.15.28:0/1470360924 >> 172.21.15.28:0/1470360924 conn(0x7f4fd40d0420 msgr2=0x7f4fd40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 -- 172.21.15.28:0/1470360924 shutdown_connections 2024-09-10T20:46:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.666+0000 7f4fda2bd640 1 -- 172.21.15.28:0/1470360924 wait complete. 2024-09-10T20:46:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.667+0000 7f4fda2bd640 1 Processor -- start 2024-09-10T20:46:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fda2bd640 1 -- start start 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fda2bd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd40738b0 0x7f4fd4147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fda2bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd4151cc0 0x7f4fd41e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fda2bd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fd415eef0 0x7f4fd41e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fda2bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd4073e70 con 0x7f4fd4151cc0 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fda2bd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd4073cf0 con 0x7f4fd40738b0 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fda2bd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd4073ff0 con 0x7f4fd415eef0 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd4151cc0 0x7f4fd41e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd4151cc0 0x7f4fd41e8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50414/0 (socket says 172.21.15.28:50414) 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.668+0000 7f4fd37fe640 1 -- 172.21.15.28:0/3124998456 learned_addr learned my addr 172.21.15.28:0/3124998456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd3fff640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd40738b0 0x7f4fd4147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd885a640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fd415eef0 0x7f4fd41e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd37fe640 1 -- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fd415eef0 msgr2=0x7f4fd41e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd37fe640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fd415eef0 0x7f4fd41e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd37fe640 1 -- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd40738b0 msgr2=0x7f4fd4147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd37fe640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd40738b0 0x7f4fd4147840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd37fe640 1 -- 172.21.15.28:0/3124998456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd4070450 con 0x7f4fd4151cc0 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd885a640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fd415eef0 0x7f4fd41e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd37fe640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd4151cc0 0x7f4fd41e8a30 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800c500 tx=0x7f4fc800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd0ff9640 1 -- 172.21.15.28:0/3124998456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc80052d0 con 0x7f4fd4151cc0 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd0ff9640 1 -- 172.21.15.28:0/3124998456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fc8005470 con 0x7f4fd4151cc0 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd40b3f30 con 0x7f4fd4151cc0 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fd0ff9640 1 -- 172.21.15.28:0/3124998456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8012870 con 0x7f4fd4151cc0 2024-09-10T20:46:20.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.669+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd40b44f0 con 0x7f4fd4151cc0 2024-09-10T20:46:20.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.670+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd40b46f0 con 0x7f4fd4151cc0 2024-09-10T20:46:20.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.671+0000 7f4fd0ff9640 1 -- 172.21.15.28:0/3124998456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 28) ==== 101570+0+0 (secure 0 0 0) 0x7f4fc80035f0 con 0x7f4fd4151cc0 2024-09-10T20:46:20.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.671+0000 7f4fd0ff9640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fc0078a20 0x7f4fc007aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:20.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.672+0000 7f4fd3fff640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fc0078a20 0x7f4fc007aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:20.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.672+0000 7f4fd0ff9640 1 -- 172.21.15.28:0/3124998456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f4fc8098210 con 0x7f4fd4151cc0 2024-09-10T20:46:20.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.672+0000 7f4fd3fff640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fc0078a20 0x7f4fc007aee0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4033000 tx=0x7f4fc4004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:20.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.676+0000 7f4fd0ff9640 1 -- 172.21.15.28:0/3124998456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4fd40b46f0 con 0x7f4fd4151cc0 2024-09-10T20:46:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.873+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4fd40cc1b0 con 0x7f4fd4151cc0 2024-09-10T20:46:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.873+0000 7f4fd0ff9640 1 -- 172.21.15.28:0/3124998456 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v164) ==== 80+0+0 (secure 0 0 0) 0x7f4fc8062740 con 0x7f4fd4151cc0 2024-09-10T20:46:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:46:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fc0078a20 msgr2=0x7f4fc007aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fc0078a20 0x7f4fc007aee0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4033000 tx=0x7f4fc4004730 comp rx=0 tx=0).stop 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd4151cc0 msgr2=0x7f4fd41e8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd4151cc0 0x7f4fd41e8a30 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800c500 tx=0x7f4fc800c9c0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 shutdown_connections 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f4fc0078a20 0x7f4fc007aee0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4fd415eef0 0x7f4fd41e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd4151cc0 0x7f4fd41e8a30 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 --2- 172.21.15.28:0/3124998456 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4fd40738b0 0x7f4fd4147840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 >> 172.21.15.28:0/3124998456 conn(0x7f4fd40d0420 msgr2=0x7f4fd40cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 shutdown_connections 2024-09-10T20:46:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:20.875+0000 7f4fda2bd640 1 -- 172.21.15.28:0/3124998456 wait complete. 2024-09-10T20:46:20.891 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-10T20:46:20.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:20.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-10T20:46:20.941 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:46:20.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:20.941 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T20:46:20.965 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:46:20.965 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:46:20.965 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:20.982 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:46:20.982 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:21.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-10T20:46:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 -- 172.21.15.28:0/2180151259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d20151cc0 msgr2=0x7f7d200cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 --2- 172.21.15.28:0/2180151259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d20151cc0 0x7f7d200cf0f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009830 tx=0x7f7d1002a500 comp rx=0 tx=0).stop 2024-09-10T20:46:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 -- 172.21.15.28:0/2180151259 shutdown_connections 2024-09-10T20:46:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 --2- 172.21.15.28:0/2180151259 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d2015eef0 0x7f7d200cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 --2- 172.21.15.28:0/2180151259 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d20151cc0 0x7f7d200cf0f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 -- 172.21.15.28:0/2180151259 >> 172.21.15.28:0/2180151259 conn(0x7f7d200d0420 msgr2=0x7f7d200d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 -- 172.21.15.28:0/2180151259 shutdown_connections 2024-09-10T20:46:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.142+0000 7f7d24975640 1 -- 172.21.15.28:0/2180151259 wait complete. 2024-09-10T20:46:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.143+0000 7f7d24975640 1 Processor -- start 2024-09-10T20:46:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d24975640 1 -- start start 2024-09-10T20:46:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d24975640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d200738b0 0x7f7d20150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d24975640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20151cc0 0x7f7d200b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d24975640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d2015eef0 0x7f7d200c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d24975640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20073e70 con 0x7f7d20151cc0 2024-09-10T20:46:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d24975640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20073cf0 con 0x7f7d2015eef0 2024-09-10T20:46:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d24975640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7d20073ff0 con 0x7f7d200738b0 2024-09-10T20:46:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d15d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20151cc0 0x7f7d200b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d15d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20151cc0 0x7f7d200b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50424/0 (socket says 172.21.15.28:50424) 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.144+0000 7f7d15d9b640 1 -- 172.21.15.28:0/1958779590 learned_addr learned my addr 172.21.15.28:0/1958779590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d1ed9d640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d2015eef0 0x7f7d200c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d1e59c640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d200738b0 0x7f7d20150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d15d9b640 1 -- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d200738b0 msgr2=0x7f7d20150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d15d9b640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d200738b0 0x7f7d20150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d15d9b640 1 -- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d2015eef0 msgr2=0x7f7d200c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d15d9b640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d2015eef0 0x7f7d200c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d15d9b640 1 -- 172.21.15.28:0/1958779590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d200b1da0 con 0x7f7d20151cc0 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d1ed9d640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d2015eef0 0x7f7d200c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d15d9b640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20151cc0 0x7f7d200b4cd0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800c450 tx=0x7f7d0800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d080052d0 con 0x7f7d20151cc0 2024-09-10T20:46:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d08005470 con 0x7f7d20151cc0 2024-09-10T20:46:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d201f4fb0 con 0x7f7d20151cc0 2024-09-10T20:46:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08012870 con 0x7f7d20151cc0 2024-09-10T20:46:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.145+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d201f5570 con 0x7f7d20151cc0 2024-09-10T20:46:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.146+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d200cb9f0 con 0x7f7d20151cc0 2024-09-10T20:46:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.147+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 28) ==== 101570+0+0 (secure 0 0 0) 0x7f7d080035f0 con 0x7f7d20151cc0 2024-09-10T20:46:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.147+0000 7f7d177fe640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d00078a20 0x7f7d0007aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.148+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f7d08098250 con 0x7f7d20151cc0 2024-09-10T20:46:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.148+0000 7f7d1e59c640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d00078a20 0x7f7d0007aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.148+0000 7f7d1e59c640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d00078a20 0x7f7d0007aee0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7d10034000 tx=0x7f7d10004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.152+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7d08062780 con 0x7f7d20151cc0 2024-09-10T20:46:21.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.348+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7d20068d10 con 0x7f7d20151cc0 2024-09-10T20:46:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.351+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f7d0805e2f0 con 0x7f7d20151cc0 2024-09-10T20:46:21.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.351+0000 7f7d177fe640 1 -- 172.21.15.28:0/1958779590 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v164) ==== 80+0+0 (secure 0 0 0) 0x7f7d08096e70 con 0x7f7d20151cc0 2024-09-10T20:46:21.356 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d00078a20 msgr2=0x7f7d0007aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d00078a20 0x7f7d0007aee0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7d10034000 tx=0x7f7d10004730 comp rx=0 tx=0).stop 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20151cc0 msgr2=0x7f7d200b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20151cc0 0x7f7d200b4cd0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800c450 tx=0x7f7d0800c910 comp rx=0 tx=0).stop 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 shutdown_connections 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7d00078a20 0x7f7d0007aee0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d2015eef0 0x7f7d200c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20151cc0 0x7f7d200b4cd0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 --2- 172.21.15.28:0/1958779590 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d200738b0 0x7f7d20150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 >> 172.21.15.28:0/1958779590 conn(0x7f7d200d0420 msgr2=0x7f7d200cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 shutdown_connections 2024-09-10T20:46:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.354+0000 7f7d24975640 1 -- 172.21.15.28:0/1958779590 wait complete. 2024-09-10T20:46:21.370 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-10T20:46:21.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:21.370 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-09-10T20:46:21.399 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:46:21.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:21.400 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-10T20:46:21.456 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:46:21.457 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:46:21.457 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:21.474 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:46:21.475 INFO:tasks.cephfs.mount:Terminating background process 2024-09-10T20:46:21.475 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:46:21.475 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:21.514 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:46:21.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:46:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.616+0000 7f25d8b52640 1 -- 172.21.15.28:0/4042679202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 msgr2=0x7f25bc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.616+0000 7f25d8b52640 1 --2- 172.21.15.28:0/4042679202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 0x7f25bc003e70 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f25c4006fb0 tx=0x7f25bc005830 comp rx=0 tx=0).stop 2024-09-10T20:46:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.616+0000 7f25d2d9d640 1 -- 172.21.15.28:0/4042679202 reap_dead start 2024-09-10T20:46:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.616+0000 7f25d8b52640 1 -- 172.21.15.28:0/4042679202 shutdown_connections 2024-09-10T20:46:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.616+0000 7f25d8b52640 1 -- 172.21.15.28:0/4042679202 >> 172.21.15.28:0/4042679202 conn(0x7f25d40d1e40 msgr2=0x7f25d40d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.616+0000 7f25d8b52640 1 -- 172.21.15.28:0/4042679202 shutdown_connections 2024-09-10T20:46:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.616+0000 7f25d8b52640 1 -- 172.21.15.28:0/4042679202 wait complete. 2024-09-10T20:46:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.618+0000 7f25d8b52640 1 Processor -- start 2024-09-10T20:46:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.618+0000 7f25d8b52640 1 -- start start 2024-09-10T20:46:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 0x7f25d414fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25d409d210 0x7f25d40b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25d40c86e0 0x7f25d40ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25d40b13e0 con 0x7f25bc003a50 2024-09-10T20:46:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f25d40cf170 con 0x7f25d409d210 2024-09-10T20:46:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f25d40b1560 con 0x7f25d40c86e0 2024-09-10T20:46:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 0x7f25d414fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 0x7f25d414fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50444/0 (socket says 172.21.15.28:50444) 2024-09-10T20:46:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 -- 172.21.15.28:0/4157417814 learned_addr learned my addr 172.21.15.28:0/4157417814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d2d9d640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25d40c86e0 0x7f25d40ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 -- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25d40c86e0 msgr2=0x7f25d40ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d1d9b640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25d409d210 0x7f25d40b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25d40c86e0 0x7f25d40ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 -- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25d409d210 msgr2=0x7f25d40b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25d409d210 0x7f25d40b8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 -- 172.21.15.28:0/4157417814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d40b8850 con 0x7f25bc003a50 2024-09-10T20:46:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d259c640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 0x7f25d414fed0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f25c402b0b0 tx=0x7f25c4001ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25c2ffd640 1 -- 172.21.15.28:0/4157417814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4031040 con 0x7f25bc003a50 2024-09-10T20:46:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25c2ffd640 1 -- 172.21.15.28:0/4157417814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25c4003800 con 0x7f25bc003a50 2024-09-10T20:46:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25c2ffd640 1 -- 172.21.15.28:0/4157417814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4003c10 con 0x7f25bc003a50 2024-09-10T20:46:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d41f2bb0 con 0x7f25bc003a50 2024-09-10T20:46:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.619+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d41f3170 con 0x7f25bc003a50 2024-09-10T20:46:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.620+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d4071d90 con 0x7f25bc003a50 2024-09-10T20:46:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.621+0000 7f25c2ffd640 1 -- 172.21.15.28:0/4157417814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f25c4048030 con 0x7f25bc003a50 2024-09-10T20:46:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.621+0000 7f25c2ffd640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f25b4078b70 0x7f25b407b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.622+0000 7f25c2ffd640 1 -- 172.21.15.28:0/4157417814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f25c40b8680 con 0x7f25bc003a50 2024-09-10T20:46:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.622+0000 7f25d1d9b640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f25b4078b70 0x7f25b407b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.622+0000 7f25d1d9b640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f25b4078b70 0x7f25b407b030 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f25d4156510 tx=0x7f25c8005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.626+0000 7f25c2ffd640 1 -- 172.21.15.28:0/4157417814 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f25c4082bd0 con 0x7f25bc003a50 2024-09-10T20:46:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.847+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25d4071ab0 con 0x7f25bc003a50 2024-09-10T20:46:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.847+0000 7f25c2ffd640 1 -- 172.21.15.28:0/4157417814 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5239 (secure 0 0 0) 0x7f25c40824d0 con 0x7f25bc003a50 2024-09-10T20:46:21.852 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:21.852 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":117,"btime":"2024-09-10T20:46:13:316195+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4088436065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4088436065},{"type":"v1","addr":"172.21.15.28:6834","nonce":4088436065}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2443719102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2443719102},{"type":"v1","addr":"172.21.15.159:6801","nonce":2443719102}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":7250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3153361013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3153361013},{"type":"v1","addr":"172.21.15.159:6835","nonce":3153361013}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:46:03.628094+0000","modified":"2024-09-10T20:46:06.656668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9078":{"gid":9078,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1516352866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1516352866},{"type":"v1","addr":"172.21.15.28:6835","nonce":1516352866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9078,"qdb_cluster":[9078]},"id":13}]} 2024-09-10T20:46:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 117 2024-09-10T20:46:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f25b4078b70 msgr2=0x7f25b407b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f25b4078b70 0x7f25b407b030 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f25d4156510 tx=0x7f25c8005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 msgr2=0x7f25d414fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 0x7f25d414fed0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f25c402b0b0 tx=0x7f25c4001ce0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 shutdown_connections 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f25b4078b70 0x7f25b407b030 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f25d40c86e0 0x7f25d40ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f25d409d210 0x7f25d40b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 --2- 172.21.15.28:0/4157417814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc003a50 0x7f25d414fed0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 >> 172.21.15.28:0/4157417814 conn(0x7f25d40d1e40 msgr2=0x7f25d40aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 shutdown_connections 2024-09-10T20:46:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.849+0000 7f25d8b52640 1 -- 172.21.15.28:0/4157417814 wait complete. 2024-09-10T20:46:21.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:46:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 -- 172.21.15.28:0/1702043935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb00b4d10 msgr2=0x7f0eb00ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 --2- 172.21.15.28:0/1702043935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00ce030 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8005fd0 tx=0x7f0ea802aa90 comp rx=0 tx=0).stop 2024-09-10T20:46:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 -- 172.21.15.28:0/1702043935 shutdown_connections 2024-09-10T20:46:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 --2- 172.21.15.28:0/1702043935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00ce030 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 --2- 172.21.15.28:0/1702043935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0eb015a190 0x7f0eb00b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 -- 172.21.15.28:0/1702043935 >> 172.21.15.28:0/1702043935 conn(0x7f0eb006cf00 msgr2=0x7f0eb006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 -- 172.21.15.28:0/1702043935 shutdown_connections 2024-09-10T20:46:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.966+0000 7f0eb6623640 1 -- 172.21.15.28:0/1702043935 wait complete. 2024-09-10T20:46:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.967+0000 7f0eb6623640 1 Processor -- start 2024-09-10T20:46:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eb6623640 1 -- start start 2024-09-10T20:46:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eb6623640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eb6623640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb015a190 0x7f0eb014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eb6623640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0eb015df70 0x7f0eb014fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eb6623640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb00d00f0 con 0x7f0eb00b4d10 2024-09-10T20:46:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eb6623640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb00ceae0 con 0x7f0eb015a190 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eb6623640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb00d0270 con 0x7f0eb015df70 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00b0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50456/0 (socket says 172.21.15.28:50456) 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.968+0000 7f0eaffff640 1 -- 172.21.15.28:0/2925184681 learned_addr learned my addr 172.21.15.28:0/2925184681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eb4bc0640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0eb015df70 0x7f0eb014fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eaf7fe640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb015a190 0x7f0eb014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eaffff640 1 -- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0eb015df70 msgr2=0x7f0eb014fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eaffff640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0eb015df70 0x7f0eb014fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eaffff640 1 -- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb015a190 msgr2=0x7f0eb014f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eaffff640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb015a190 0x7f0eb014f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eaffff640 1 -- 172.21.15.28:0/2925184681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb0070120 con 0x7f0eb00b4d10 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eaffff640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00b0b00 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000ea60 tx=0x7f0ea000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eacff9640 1 -- 172.21.15.28:0/2925184681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea000cde0 con 0x7f0eb00b4d10 2024-09-10T20:46:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb014aa60 con 0x7f0eb00b4d10 2024-09-10T20:46:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eacff9640 1 -- 172.21.15.28:0/2925184681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ea0004020 con 0x7f0eb00b4d10 2024-09-10T20:46:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eacff9640 1 -- 172.21.15.28:0/2925184681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0004450 con 0x7f0eb00b4d10 2024-09-10T20:46:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.969+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb01503a0 con 0x7f0eb00b4d10 2024-09-10T20:46:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.970+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb00ce030 con 0x7f0eb00b4d10 2024-09-10T20:46:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.971+0000 7f0eacff9640 1 -- 172.21.15.28:0/2925184681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f0ea00035f0 con 0x7f0eb00b4d10 2024-09-10T20:46:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.971+0000 7f0eacff9640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c078b40 0x7f0e9c07b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.972+0000 7f0eacff9640 1 -- 172.21.15.28:0/2925184681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f0ea0099070 con 0x7f0eb00b4d10 2024-09-10T20:46:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.972+0000 7f0eaf7fe640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c078b40 0x7f0e9c07b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.972+0000 7f0eaf7fe640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c078b40 0x7f0e9c07b000 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0eb014a830 tx=0x7f0ea4009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:21.976+0000 7f0eacff9640 1 -- 172.21.15.28:0/2925184681 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0ea0062f50 con 0x7f0eb00b4d10 2024-09-10T20:46:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.196+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0eb009de90 con 0x7f0eb00b4d10 2024-09-10T20:46:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.197+0000 7f0eacff9640 1 -- 172.21.15.28:0/2925184681 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5239 (secure 0 0 0) 0x7f0ea0067980 con 0x7f0eb00b4d10 2024-09-10T20:46:22.202 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:22.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":117,"btime":"2024-09-10T20:46:13:316195+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4088436065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4088436065},{"type":"v1","addr":"172.21.15.28:6834","nonce":4088436065}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2443719102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2443719102},{"type":"v1","addr":"172.21.15.159:6801","nonce":2443719102}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":7250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3153361013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3153361013},{"type":"v1","addr":"172.21.15.159:6835","nonce":3153361013}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:46:03.628094+0000","modified":"2024-09-10T20:46:06.656668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9078":{"gid":9078,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1516352866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1516352866},{"type":"v1","addr":"172.21.15.28:6835","nonce":1516352866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9078,"qdb_cluster":[9078]},"id":13}]} 2024-09-10T20:46:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 117 2024-09-10T20:46:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c078b40 msgr2=0x7f0e9c07b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c078b40 0x7f0e9c07b000 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0eb014a830 tx=0x7f0ea4009040 comp rx=0 tx=0).stop 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb00b4d10 msgr2=0x7f0eb00b0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00b0b00 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000ea60 tx=0x7f0ea000ef20 comp rx=0 tx=0).stop 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 shutdown_connections 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f0e9c078b40 0x7f0e9c07b000 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0eb015df70 0x7f0eb014fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0eb015a190 0x7f0eb014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 --2- 172.21.15.28:0/2925184681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0eb00b4d10 0x7f0eb00b0b00 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 >> 172.21.15.28:0/2925184681 conn(0x7f0eb006cf00 msgr2=0x7f0eb0159780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 shutdown_connections 2024-09-10T20:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.199+0000 7f0eb6623640 1 -- 172.21.15.28:0/2925184681 wait complete. 2024-09-10T20:46:22.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:46:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 -- 172.21.15.28:0/3711344040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 msgr2=0x7f54bc0ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3711344040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc0ce030 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f54bc0b4140 tx=0x7f54b402aac0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 -- 172.21.15.28:0/3711344040 shutdown_connections 2024-09-10T20:46:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3711344040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc0ce030 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3711344040 >> v2:172.21.15.159:3300/0 conn(0x7f54bc15df70 0x7f54bc0b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 -- 172.21.15.28:0/3711344040 >> 172.21.15.28:0/3711344040 conn(0x7f54bc06cf00 msgr2=0x7f54bc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 -- 172.21.15.28:0/3711344040 shutdown_connections 2024-09-10T20:46:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.317+0000 7f54c14ef640 1 -- 172.21.15.28:0/3711344040 wait complete. 2024-09-10T20:46:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.318+0000 7f54c14ef640 1 Processor -- start 2024-09-10T20:46:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.318+0000 7f54c14ef640 1 -- start start 2024-09-10T20:46:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54c14ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc1f8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54c14ef640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc15a190 0x7f54bc14c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54c14ef640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc15df70 0x7f54bc15c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54c14ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc0d0270 con 0x7f54bc0b4d10 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54c14ef640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc0d00f0 con 0x7f54bc15df70 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54c14ef640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f54bc0d03f0 con 0x7f54bc15a190 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc1f8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc1f8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50470/0 (socket says 172.21.15.28:50470) 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 -- 172.21.15.28:0/3885136680 learned_addr learned my addr 172.21.15.28:0/3885136680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54bb7fe640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc15df70 0x7f54bc15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54b3fff640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc15a190 0x7f54bc14c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 -- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc15a190 msgr2=0x7f54bc14c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc15a190 0x7f54bc14c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 -- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc15df70 msgr2=0x7f54bc15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc15df70 0x7f54bc15c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54baffd640 1 -- 172.21.15.28:0/3885136680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54bc1f8fd0 con 0x7f54bc0b4d10 2024-09-10T20:46:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.319+0000 7f54bb7fe640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc15df70 0x7f54bc15c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.320+0000 7f54baffd640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc1f8a90 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00c420 tx=0x7f54ac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.320+0000 7f54b37fe640 1 -- 172.21.15.28:0/3885136680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0052d0 con 0x7f54bc0b4d10 2024-09-10T20:46:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.320+0000 7f54b37fe640 1 -- 172.21.15.28:0/3885136680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54ac007020 con 0x7f54bc0b4d10 2024-09-10T20:46:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.320+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54bc14c8b0 con 0x7f54bc0b4d10 2024-09-10T20:46:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.320+0000 7f54b37fe640 1 -- 172.21.15.28:0/3885136680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac007470 con 0x7f54bc0b4d10 2024-09-10T20:46:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.320+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54bc06f5c0 con 0x7f54bc0b4d10 2024-09-10T20:46:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.321+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54bc0ce030 con 0x7f54bc0b4d10 2024-09-10T20:46:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.322+0000 7f54b37fe640 1 -- 172.21.15.28:0/3885136680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f54ac0067d0 con 0x7f54bc0b4d10 2024-09-10T20:46:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.322+0000 7f54b37fe640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f549c078b40 0x7f549c07b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.322+0000 7f54b37fe640 1 -- 172.21.15.28:0/3885136680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f54ac097eb0 con 0x7f54bc0b4d10 2024-09-10T20:46:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.322+0000 7f54b3fff640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f549c078b40 0x7f549c07b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.323+0000 7f54b3fff640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f549c078b40 0x7f549c07b000 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f54a4005fd0 tx=0x7f54a4005850 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.326+0000 7f54b37fe640 1 -- 172.21.15.28:0/3885136680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f54bc1f8fd0 con 0x7f54bc0b4d10 2024-09-10T20:46:22.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.515+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54bc073c70 con 0x7f54bc0b4d10 2024-09-10T20:46:22.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.516+0000 7f54b37fe640 1 -- 172.21.15.28:0/3885136680 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) ==== 74+0+16816 (secure 0 0 0) 0x7f54ac062200 con 0x7f54bc0b4d10 2024-09-10T20:46:22.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:22.521 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":164,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:46:11.307286+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:46:01.262178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-10T20:46:02.591573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"27.3","mappings":[{"from":7,"to":6}]},{"pgid":"27.3b","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:46:22.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f549c078b40 msgr2=0x7f549c07b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f549c078b40 0x7f549c07b000 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f54a4005fd0 tx=0x7f54a4005850 comp rx=0 tx=0).stop 2024-09-10T20:46:22.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 msgr2=0x7f54bc1f8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc1f8a90 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00c420 tx=0x7f54ac00c8e0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 shutdown_connections 2024-09-10T20:46:22.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f549c078b40 0x7f549c07b000 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f54bc15df70 0x7f54bc15c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f54bc15a190 0x7f54bc14c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 --2- 172.21.15.28:0/3885136680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc0b4d10 0x7f54bc1f8a90 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 >> 172.21.15.28:0/3885136680 conn(0x7f54bc06cf00 msgr2=0x7f54bc06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:22.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 shutdown_connections 2024-09-10T20:46:22.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.518+0000 7f54c14ef640 1 -- 172.21.15.28:0/3885136680 wait complete. 2024-09-10T20:46:22.534 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:46:22.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:46:22.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.635+0000 7fe8220ed640 1 -- 172.21.15.28:0/1435581427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe804003a00 msgr2=0x7fe804003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.635+0000 7fe8220ed640 1 --2- 172.21.15.28:0/1435581427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe804003a00 0x7fe804003e20 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c260 tx=0x7fe804005830 comp rx=0 tx=0).stop 2024-09-10T20:46:22.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.635+0000 7fe81bfff640 1 -- 172.21.15.28:0/1435581427 reap_dead start 2024-09-10T20:46:22.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.635+0000 7fe8220ed640 1 -- 172.21.15.28:0/1435581427 shutdown_connections 2024-09-10T20:46:22.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.635+0000 7fe8220ed640 1 -- 172.21.15.28:0/1435581427 >> 172.21.15.28:0/1435581427 conn(0x7fe81c0d1e40 msgr2=0x7fe81c0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:22.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.635+0000 7fe8220ed640 1 -- 172.21.15.28:0/1435581427 shutdown_connections 2024-09-10T20:46:22.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.635+0000 7fe8220ed640 1 -- 172.21.15.28:0/1435581427 wait complete. 2024-09-10T20:46:22.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.636+0000 7fe8220ed640 1 Processor -- start 2024-09-10T20:46:22.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe8220ed640 1 -- start start 2024-09-10T20:46:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe8220ed640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe804003a00 0x7fe81c0cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe8220ed640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe81c1f47d0 0x7fe81c1f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe8220ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c15cfb0 0x7fe81c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe8220ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c0d2960 con 0x7fe81c15cfb0 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe8220ed640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c0d0d50 con 0x7fe804003a00 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe8220ed640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe81c0d2ae0 con 0x7fe81c1f47d0 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe81bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c15cfb0 0x7fe81c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe81bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c15cfb0 0x7fe81c15d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50488/0 (socket says 172.21.15.28:50488) 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.637+0000 7fe81bfff640 1 -- 172.21.15.28:0/1321426800 learned_addr learned my addr 172.21.15.28:0/1321426800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81b7fe640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe804003a00 0x7fe81c0cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81affd640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe81c1f47d0 0x7fe81c1f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81bfff640 1 -- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe81c1f47d0 msgr2=0x7fe81c1f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81bfff640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe81c1f47d0 0x7fe81c1f4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81bfff640 1 -- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe804003a00 msgr2=0x7fe81c0cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81bfff640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe804003a00 0x7fe81c0cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81bfff640 1 -- 172.21.15.28:0/1321426800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c0cbef0 con 0x7fe81c15cfb0 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81b7fe640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe804003a00 0x7fe81c0cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe81bfff640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c15cfb0 0x7fe81c15d430 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe81400e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe803fff640 1 -- 172.21.15.28:0/1321426800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814004b20 con 0x7fe81c15cfb0 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe803fff640 1 -- 172.21.15.28:0/1321426800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe814015030 con 0x7fe81c15cfb0 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c1f52a0 con 0x7fe81c15cfb0 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe803fff640 1 -- 172.21.15.28:0/1321426800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814003970 con 0x7fe81c15cfb0 2024-09-10T20:46:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.638+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c15c7b0 con 0x7fe81c15cfb0 2024-09-10T20:46:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.639+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81c15c5a0 con 0x7fe81c15cfb0 2024-09-10T20:46:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.640+0000 7fe803fff640 1 -- 172.21.15.28:0/1321426800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fe814003b10 con 0x7fe81c15cfb0 2024-09-10T20:46:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.640+0000 7fe803fff640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe808078b40 0x7fe80807b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.640+0000 7fe81b7fe640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe808078b40 0x7fe80807b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.640+0000 7fe803fff640 1 -- 172.21.15.28:0/1321426800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7fe814099100 con 0x7fe81c15cfb0 2024-09-10T20:46:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.641+0000 7fe81b7fe640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe808078b40 0x7fe80807b000 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe80c033000 tx=0x7fe81c0cda10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:22.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.644+0000 7fe803fff640 1 -- 172.21.15.28:0/1321426800 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe814063650 con 0x7fe81c15cfb0 2024-09-10T20:46:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.865+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe81c0cecb0 con 0x7fe81c15cfb0 2024-09-10T20:46:22.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.866+0000 7fe803fff640 1 -- 172.21.15.28:0/1321426800 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5239 (secure 0 0 0) 0x7fe814062f50 con 0x7fe81c15cfb0 2024-09-10T20:46:22.871 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:22.871 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":117,"btime":"2024-09-10T20:46:13:316195+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4088436065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4088436065},{"type":"v1","addr":"172.21.15.28:6834","nonce":4088436065}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2443719102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2443719102},{"type":"v1","addr":"172.21.15.159:6801","nonce":2443719102}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":7250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3153361013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3153361013},{"type":"v1","addr":"172.21.15.159:6835","nonce":3153361013}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:46:03.628094+0000","modified":"2024-09-10T20:46:06.656668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9078":{"gid":9078,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1516352866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1516352866},{"type":"v1","addr":"172.21.15.28:6835","nonce":1516352866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9078,"qdb_cluster":[9078]},"id":13}]} 2024-09-10T20:46:22.871 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 117 2024-09-10T20:46:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe808078b40 msgr2=0x7fe80807b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe808078b40 0x7fe80807b000 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe80c033000 tx=0x7fe81c0cda10 comp rx=0 tx=0).stop 2024-09-10T20:46:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c15cfb0 msgr2=0x7fe81c15d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c15cfb0 0x7fe81c15d430 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe81400e350 comp rx=0 tx=0).stop 2024-09-10T20:46:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 shutdown_connections 2024-09-10T20:46:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe808078b40 0x7fe80807b000 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c15cfb0 0x7fe81c15d430 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe81c1f47d0 0x7fe81c1f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 --2- 172.21.15.28:0/1321426800 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe804003a00 0x7fe81c0cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 >> 172.21.15.28:0/1321426800 conn(0x7fe81c0d1e40 msgr2=0x7fe81c15b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:22.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 shutdown_connections 2024-09-10T20:46:22.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.868+0000 7fe8220ed640 1 -- 172.21.15.28:0/1321426800 wait complete. 2024-09-10T20:46:22.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:46:22.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.985+0000 7f1f75565640 1 -- 172.21.15.28:0/3538079369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c003a00 msgr2=0x7f1f5c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.985+0000 7f1f75565640 1 --2- 172.21.15.28:0/3538079369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f5c003a00 0x7f1f5c003e20 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009f00 tx=0x7f1f5c005830 comp rx=0 tx=0).stop 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.985+0000 7f1f6f7fe640 1 -- 172.21.15.28:0/3538079369 reap_dead start 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.985+0000 7f1f75565640 1 -- 172.21.15.28:0/3538079369 shutdown_connections 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.985+0000 7f1f75565640 1 -- 172.21.15.28:0/3538079369 >> 172.21.15.28:0/3538079369 conn(0x7f1f700d0420 msgr2=0x7f1f700d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.985+0000 7f1f75565640 1 -- 172.21.15.28:0/3538079369 shutdown_connections 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.985+0000 7f1f75565640 1 -- 172.21.15.28:0/3538079369 wait complete. 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.987+0000 7f1f75565640 1 Processor -- start 2024-09-10T20:46:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.987+0000 7f1f75565640 1 -- start start 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.987+0000 7f1f75565640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c003a00 0x7f1f701f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.987+0000 7f1f75565640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7014e730 0x7f1f7014ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.987+0000 7f1f75565640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7014c740 0x7f1f7014cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f75565640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7006f2c0 con 0x7f1f7014e730 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f75565640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7006f140 con 0x7f1f5c003a00 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f75565640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1f700731f0 con 0x7f1f7014c740 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7014e730 0x7f1f7014ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7014e730 0x7f1f7014ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50500/0 (socket says 172.21.15.28:50500) 2024-09-10T20:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 -- 172.21.15.28:0/1164845746 learned_addr learned my addr 172.21.15.28:0/1164845746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6f7fe640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7014c740 0x7f1f7014cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6effd640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c003a00 0x7f1f701f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 -- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7014c740 msgr2=0x7f1f7014cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7014c740 0x7f1f7014cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 -- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c003a00 msgr2=0x7f1f701f3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c003a00 0x7f1f701f3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 -- 172.21.15.28:0/1164845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7014f260 con 0x7f1f7014e730 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f6e7fc640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7014e730 0x7f1f7014ebb0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800e590 tx=0x7f1f5800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f677fe640 1 -- 172.21.15.28:0/1164845746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58015070 con 0x7f1f7014e730 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f677fe640 1 -- 172.21.15.28:0/1164845746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f58010980 con 0x7f1f7014e730 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7014f540 con 0x7f1f7014e730 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f677fe640 1 -- 172.21.15.28:0/1164845746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58010dd0 con 0x7f1f7014e730 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.988+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f701f38a0 con 0x7f1f7014e730 2024-09-10T20:46:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.989+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7014fa60 con 0x7f1f7014e730 2024-09-10T20:46:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.990+0000 7f1f677fe640 1 -- 172.21.15.28:0/1164845746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f1f58011460 con 0x7f1f7014e730 2024-09-10T20:46:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.991+0000 7f1f677fe640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f50078b40 0x7f1f5007b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.991+0000 7f1f677fe640 1 -- 172.21.15.28:0/1164845746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f1f580990d0 con 0x7f1f7014e730 2024-09-10T20:46:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.991+0000 7f1f6effd640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f50078b40 0x7f1f5007b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.991+0000 7f1f6effd640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f50078b40 0x7f1f5007b000 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1f68035000 tx=0x7f1f6802d040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:22.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:22.995+0000 7f1f677fe640 1 -- 172.21.15.28:0/1164845746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1f7014fa60 con 0x7f1f7014e730 2024-09-10T20:46:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.216+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f700caa70 con 0x7f1f7014e730 2024-09-10T20:46:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.216+0000 7f1f677fe640 1 -- 172.21.15.28:0/1164845746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5239 (secure 0 0 0) 0x7f1f58063340 con 0x7f1f7014e730 2024-09-10T20:46:23.222 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:23.222 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":117,"btime":"2024-09-10T20:46:13:316195+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4088436065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4088436065},{"type":"v1","addr":"172.21.15.28:6834","nonce":4088436065}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2443719102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2443719102},{"type":"v1","addr":"172.21.15.159:6801","nonce":2443719102}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":7250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3153361013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3153361013},{"type":"v1","addr":"172.21.15.159:6835","nonce":3153361013}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T20:46:03.628094+0000","modified":"2024-09-10T20:46:06.656668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9078":{"gid":9078,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1516352866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1516352866},{"type":"v1","addr":"172.21.15.28:6835","nonce":1516352866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9078,"qdb_cluster":[9078]},"id":13}]} 2024-09-10T20:46:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 117 2024-09-10T20:46:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f50078b40 msgr2=0x7f1f5007b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f50078b40 0x7f1f5007b000 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1f68035000 tx=0x7f1f6802d040 comp rx=0 tx=0).stop 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7014e730 msgr2=0x7f1f7014ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7014e730 0x7f1f7014ebb0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800e590 tx=0x7f1f5800ea50 comp rx=0 tx=0).stop 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 shutdown_connections 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1f50078b40 0x7f1f5007b000 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f7014c740 0x7f1f7014cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f7014e730 0x7f1f7014ebb0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.218+0000 7f1f75565640 1 --2- 172.21.15.28:0/1164845746 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f5c003a00 0x7f1f701f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.219+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 >> 172.21.15.28:0/1164845746 conn(0x7f1f700d0420 msgr2=0x7f1f7006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.219+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 shutdown_connections 2024-09-10T20:46:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.219+0000 7f1f75565640 1 -- 172.21.15.28:0/1164845746 wait complete. 2024-09-10T20:46:23.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 -- 172.21.15.28:0/1015384487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 msgr2=0x7fadc80d05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/1015384487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc80d05d0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fadc0005fd0 tx=0x7fadc002a530 comp rx=0 tx=0).stop 2024-09-10T20:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 -- 172.21.15.28:0/1015384487 shutdown_connections 2024-09-10T20:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/1015384487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc80d05d0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/1015384487 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadc8155100 0x7fadc815bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 -- 172.21.15.28:0/1015384487 >> 172.21.15.28:0/1015384487 conn(0x7fadc806cf00 msgr2=0x7fadc806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 -- 172.21.15.28:0/1015384487 shutdown_connections 2024-09-10T20:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.336+0000 7fadcd4cd640 1 -- 172.21.15.28:0/1015384487 wait complete. 2024-09-10T20:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.337+0000 7fadcd4cd640 1 Processor -- start 2024-09-10T20:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.337+0000 7fadcd4cd640 1 -- start start 2024-09-10T20:46:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadcd4cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc81f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadcd4cd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadc8155100 0x7fadc80b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadcd4cd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadc81554e0 0x7fadc8159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadcd4cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8072f60 con 0x7fadc80d0140 2024-09-10T20:46:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadcd4cd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8072de0 con 0x7fadc81554e0 2024-09-10T20:46:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadcd4cd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fadc80730e0 con 0x7fadc8155100 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc81f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc81f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50526/0 (socket says 172.21.15.28:50526) 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 -- 172.21.15.28:0/65738162 learned_addr learned my addr 172.21.15.28:0/65738162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 -- 172.21.15.28:0/65738162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadc8155100 msgr2=0x7fadc80b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadc8155100 0x7fadc80b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 -- 172.21.15.28:0/65738162 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadc81554e0 msgr2=0x7fadc8159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadc81554e0 0x7fadc8159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.338+0000 7fadc6ffd640 1 -- 172.21.15.28:0/65738162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc8150f60 con 0x7fadc80d0140 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.339+0000 7fadc6ffd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc81f5870 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fadb800cd10 tx=0x7fadb800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.339+0000 7fadbf7fe640 1 -- 172.21.15.28:0/65738162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8004b20 con 0x7fadc80d0140 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.339+0000 7fadbf7fe640 1 -- 172.21.15.28:0/65738162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadb8004dd0 con 0x7fadc80d0140 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.339+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc81f1d70 con 0x7fadc80d0140 2024-09-10T20:46:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.339+0000 7fadbf7fe640 1 -- 172.21.15.28:0/65738162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8005280 con 0x7fadc80d0140 2024-09-10T20:46:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.339+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc81f2330 con 0x7fadc80d0140 2024-09-10T20:46:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.340+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc8073620 con 0x7fadc80d0140 2024-09-10T20:46:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.340+0000 7fadbf7fe640 1 -- 172.21.15.28:0/65738162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fadb8005420 con 0x7fadc80d0140 2024-09-10T20:46:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.341+0000 7fadbf7fe640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fada8078b40 0x7fada807b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.341+0000 7fadbf7fe640 1 -- 172.21.15.28:0/65738162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7fadb8098f80 con 0x7fadc80d0140 2024-09-10T20:46:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.341+0000 7fadc67fc640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fada8078b40 0x7fada807b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.341+0000 7fadc67fc640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fada8078b40 0x7fada807b000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fadc81e88d0 tx=0x7fadb00079b0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.345+0000 7fadbf7fe640 1 -- 172.21.15.28:0/65738162 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fadb8014030 con 0x7fadc80d0140 2024-09-10T20:46:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.531+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fadc80d05c0 con 0x7fadc80d0140 2024-09-10T20:46:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.532+0000 7fadbf7fe640 1 -- 172.21.15.28:0/65738162 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) ==== 74+0+16816 (secure 0 0 0) 0x7fadb8063530 con 0x7fadc80d0140 2024-09-10T20:46:23.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:23.537 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":164,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:46:11.307286+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-10T20:46:01.262178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-10T20:46:02.591573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"27.3","mappings":[{"from":7,"to":6}]},{"pgid":"27.3b","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:46:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fada8078b40 msgr2=0x7fada807b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fada8078b40 0x7fada807b000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fadc81e88d0 tx=0x7fadb00079b0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 msgr2=0x7fadc81f5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc81f5870 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fadb800cd10 tx=0x7fadb800a620 comp rx=0 tx=0).stop 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 shutdown_connections 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fada8078b40 0x7fada807b000 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fadc81554e0 0x7fadc8159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fadc8155100 0x7fadc80b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 --2- 172.21.15.28:0/65738162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc80d0140 0x7fadc81f5870 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 >> 172.21.15.28:0/65738162 conn(0x7fadc806cf00 msgr2=0x7fadc815dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 shutdown_connections 2024-09-10T20:46:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.534+0000 7fadcd4cd640 1 -- 172.21.15.28:0/65738162 wait complete. 2024-09-10T20:46:23.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-10T20:46:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 -- 172.21.15.28:0/2129091336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec0738b0 msgr2=0x7f13ec15e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 --2- 172.21.15.28:0/2129091336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec0738b0 0x7f13ec15e7e0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f13ec072ea0 tx=0x7f13dc02a530 comp rx=0 tx=0).stop 2024-09-10T20:46:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 -- 172.21.15.28:0/2129091336 shutdown_connections 2024-09-10T20:46:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 --2- 172.21.15.28:0/2129091336 >> v2:172.21.15.159:3300/0 conn(0x7f13ec073cf0 0x7f13ec15ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 --2- 172.21.15.28:0/2129091336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec0738b0 0x7f13ec15e7e0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 -- 172.21.15.28:0/2129091336 >> 172.21.15.28:0/2129091336 conn(0x7f13ec0d0420 msgr2=0x7f13ec0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 -- 172.21.15.28:0/2129091336 shutdown_connections 2024-09-10T20:46:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.652+0000 7f13f4582640 1 -- 172.21.15.28:0/2129091336 wait complete. 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.653+0000 7f13f4582640 1 Processor -- start 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.653+0000 7f13f4582640 1 -- start start 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f4582640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13ec0738b0 0x7f13ec15d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f4582640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13ec073cf0 0x7f13ec157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f4582640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec151e30 0x7f13ec0b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f4582640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec0cb390 con 0x7f13ec151e30 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f4582640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec0cb210 con 0x7f13ec073cf0 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f4582640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec0cb510 con 0x7f13ec0738b0 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec151e30 0x7f13ec0b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec151e30 0x7f13ec0b5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50540/0 (socket says 172.21.15.28:50540) 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 -- 172.21.15.28:0/783574965 learned_addr learned my addr 172.21.15.28:0/783574965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f231e640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13ec0738b0 0x7f13ec15d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f1b1d640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13ec073cf0 0x7f13ec157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 -- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13ec0738b0 msgr2=0x7f13ec15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13ec0738b0 0x7f13ec15d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 -- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13ec073cf0 msgr2=0x7f13ec157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13ec073cf0 0x7f13ec157bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 -- 172.21.15.28:0/783574965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec15c750 con 0x7f13ec151e30 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.654+0000 7f13f2b1f640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec151e30 0x7f13ec0b5a40 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f13e400db90 tx=0x7f13e400dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.655+0000 7f13e2ffd640 1 -- 172.21.15.28:0/783574965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e400e980 con 0x7f13ec151e30 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.655+0000 7f13e2ffd640 1 -- 172.21.15.28:0/783574965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13e400ec30 con 0x7f13ec151e30 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.655+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec15ca30 con 0x7f13ec151e30 2024-09-10T20:46:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.655+0000 7f13e2ffd640 1 -- 172.21.15.28:0/783574965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e40097e0 con 0x7f13ec151e30 2024-09-10T20:46:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.655+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec15cff0 con 0x7f13ec151e30 2024-09-10T20:46:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.656+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ec15dd30 con 0x7f13ec151e30 2024-09-10T20:46:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.657+0000 7f13e2ffd640 1 -- 172.21.15.28:0/783574965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f13e4009980 con 0x7f13ec151e30 2024-09-10T20:46:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.657+0000 7f13e2ffd640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13d0078b40 0x7f13d007b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.657+0000 7f13e2ffd640 1 -- 172.21.15.28:0/783574965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f13e409a5d0 con 0x7f13ec151e30 2024-09-10T20:46:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.657+0000 7f13f231e640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13d0078b40 0x7f13d007b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.658+0000 7f13f231e640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13d0078b40 0x7f13d007b000 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f13e8000f60 tx=0x7f13e8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.661+0000 7f13e2ffd640 1 -- 172.21.15.28:0/783574965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f13ec15dd30 con 0x7f13ec151e30 2024-09-10T20:46:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.882+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f13ec068d10 con 0x7f13ec151e30 2024-09-10T20:46:23.888 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-10T20:46:23.883+0000 7f27079ff640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T20:46:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.884+0000 7f13e2ffd640 1 -- 172.21.15.28:0/783574965 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) ==== 161+0+0 (secure 0 0 0) 0x7f13e400ee20 con 0x7f13ec151e30 2024-09-10T20:46:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-10T20:46:23.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13d0078b40 msgr2=0x7f13d007b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13d0078b40 0x7f13d007b000 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f13e8000f60 tx=0x7f13e8009040 comp rx=0 tx=0).stop 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec151e30 msgr2=0x7f13ec0b5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec151e30 0x7f13ec0b5a40 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f13e400db90 tx=0x7f13e400dc80 comp rx=0 tx=0).stop 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 shutdown_connections 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f13d0078b40 0x7f13d007b000 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec151e30 0x7f13ec0b5a40 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f13ec073cf0 0x7f13ec157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 --2- 172.21.15.28:0/783574965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f13ec0738b0 0x7f13ec15d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.887+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 >> 172.21.15.28:0/783574965 conn(0x7f13ec0d0420 msgr2=0x7f13ec15e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.888+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 shutdown_connections 2024-09-10T20:46:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:23.888+0000 7f13f4582640 1 -- 172.21.15.28:0/783574965 wait complete. 2024-09-10T20:46:23.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-10T20:46:23.930 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-10T20:46:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.008+0000 7f78add3d640 1 -- 172.21.15.28:0/3000248282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 msgr2=0x7f7894003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.008+0000 7f78add3d640 1 --2- 172.21.15.28:0/3000248282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 0x7f7894003e70 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f789800bb20 tx=0x7f7894005830 comp rx=0 tx=0).stop 2024-09-10T20:46:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.008+0000 7f78a7fff640 1 -- 172.21.15.28:0/3000248282 reap_dead start 2024-09-10T20:46:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.009+0000 7f78add3d640 1 -- 172.21.15.28:0/3000248282 shutdown_connections 2024-09-10T20:46:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.009+0000 7f78add3d640 1 -- 172.21.15.28:0/3000248282 >> 172.21.15.28:0/3000248282 conn(0x7f78a806dbf0 msgr2=0x7f78a806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.009+0000 7f78add3d640 1 -- 172.21.15.28:0/3000248282 shutdown_connections 2024-09-10T20:46:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.009+0000 7f78add3d640 1 -- 172.21.15.28:0/3000248282 wait complete. 2024-09-10T20:46:24.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.010+0000 7f78add3d640 1 Processor -- start 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.010+0000 7f78add3d640 1 -- start start 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78add3d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 0x7f78a8157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78add3d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78a80b0d30 0x7f78a81e71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78add3d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78a81568c0 0x7f78a8156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78add3d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78a815e960 con 0x7f7894003a50 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78add3d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f78a815eae0 con 0x7f78a81568c0 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78add3d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f78a80b4fd0 con 0x7f78a80b0d30 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 0x7f78a8157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 0x7f78a8157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50584/0 (socket says 172.21.15.28:50584) 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 -- 172.21.15.28:0/3690324053 learned_addr learned my addr 172.21.15.28:0/3690324053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a7fff640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78a81568c0 0x7f78a8156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 -- 172.21.15.28:0/3690324053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78a80b0d30 msgr2=0x7f78a81e71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78a80b0d30 0x7f78a81e71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 -- 172.21.15.28:0/3690324053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78a81568c0 msgr2=0x7f78a8156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78a81568c0 0x7f78a8156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 -- 172.21.15.28:0/3690324053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a8152820 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a77fe640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 0x7f78a8157ec0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f789800bb50 tx=0x7f789800b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.011+0000 7f78a4ff9640 1 -- 172.21.15.28:0/3690324053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7898042070 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.012+0000 7f78a4ff9640 1 -- 172.21.15.28:0/3690324053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f789803e860 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.012+0000 7f78a4ff9640 1 -- 172.21.15.28:0/3690324053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789803ecb0 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.012+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a80b12f0 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.012+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a81e69a0 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.012+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a80d0d00 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.013+0000 7f78a4ff9640 1 -- 172.21.15.28:0/3690324053 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f789803c420 con 0x7f7894003a50 2024-09-10T20:46:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.014+0000 7f78a4ff9640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7888078b40 0x7f788807b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.014+0000 7f78a4ff9640 1 -- 172.21.15.28:0/3690324053 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 5881+0+0 (secure 0 0 0) 0x7f78980b9d10 con 0x7f7894003a50 2024-09-10T20:46:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.014+0000 7f789ffff640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7888078b40 0x7f788807b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.014+0000 7f789ffff640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7888078b40 0x7f788807b000 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f78a8071250 tx=0x7f7890005f10 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.018+0000 7f78a4ff9640 1 -- 172.21.15.28:0/3690324053 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7898084120 con 0x7f7894003a50 2024-09-10T20:46:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.243+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f78a815f1f0 con 0x7f7894003a50 2024-09-10T20:46:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.890+0000 7f78a4ff9640 1 -- 172.21.15.28:0/3690324053 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v119) ==== 104+0+0 (secure 0 0 0) 0x7f7898083a20 con 0x7f7894003a50 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.892+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7888078b40 msgr2=0x7f788807b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.892+0000 7f78add3d640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7888078b40 0x7f788807b000 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f78a8071250 tx=0x7f7890005f10 comp rx=0 tx=0).stop 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 msgr2=0x7f78a8157ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 0x7f78a8157ec0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f789800bb50 tx=0x7f789800b6f0 comp rx=0 tx=0).stop 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 shutdown_connections 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7888078b40 0x7f788807b000 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f78a81568c0 0x7f78a8156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f78a80b0d30 0x7f78a81e71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 --2- 172.21.15.28:0/3690324053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894003a50 0x7f78a8157ec0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 >> 172.21.15.28:0/3690324053 conn(0x7f78a806dbf0 msgr2=0x7f78a80afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 shutdown_connections 2024-09-10T20:46:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:24.893+0000 7f78add3d640 1 -- 172.21.15.28:0/3690324053 wait complete. 2024-09-10T20:46:24.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-10T20:46:25.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 -- 172.21.15.28:0/184347335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 msgr2=0x7fccfc0715b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 --2- 172.21.15.28:0/184347335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc0715b0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fccfc070a70 tx=0x7fccf002a4d0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 -- 172.21.15.28:0/184347335 shutdown_connections 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 --2- 172.21.15.28:0/184347335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc0715b0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 --2- 172.21.15.28:0/184347335 >> v2:172.21.15.159:3300/0 conn(0x7fccfc071af0 0x7fccfc15c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 -- 172.21.15.28:0/184347335 >> 172.21.15.28:0/184347335 conn(0x7fccfc0b2920 msgr2=0x7fccfc0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 -- 172.21.15.28:0/184347335 shutdown_connections 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.061+0000 7fcd00c57640 1 -- 172.21.15.28:0/184347335 wait complete. 2024-09-10T20:46:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 Processor -- start 2024-09-10T20:46:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 -- start start 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc14bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccfc071af0 0x7fccfc09d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccfc0af930 0x7fccfc153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc15c990 con 0x7fccfc06d1f0 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc15c810 con 0x7fccfc071af0 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.063+0000 7fcd00c57640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fccfc15cb10 con 0x7fccfc0af930 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc14bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc14bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50598/0 (socket says 172.21.15.28:50598) 2024-09-10T20:46:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 -- 172.21.15.28:0/4085410486 learned_addr learned my addr 172.21.15.28:0/4085410486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccf9d9b640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccfc071af0 0x7fccfc09d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfad9d640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccfc0af930 0x7fccfc153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 -- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccfc0af930 msgr2=0x7fccfc153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccfc0af930 0x7fccfc153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 -- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccfc071af0 msgr2=0x7fccfc09d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccfc071af0 0x7fccfc09d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 -- 172.21.15.28:0/4085410486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc09d8d0 con 0x7fccfc06d1f0 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfad9d640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccfc0af930 0x7fccfc153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fccfa59c640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc14bc20 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fccec00b3e0 tx=0x7fccec00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fcceaffd640 1 -- 172.21.15.28:0/4085410486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec015020 con 0x7fccfc06d1f0 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fcceaffd640 1 -- 172.21.15.28:0/4085410486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fccec011940 con 0x7fccfc06d1f0 2024-09-10T20:46:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fcceaffd640 1 -- 172.21.15.28:0/4085410486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec011d90 con 0x7fccfc06d1f0 2024-09-10T20:46:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc09ab60 con 0x7fccfc06d1f0 2024-09-10T20:46:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.064+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc09b120 con 0x7fccfc06d1f0 2024-09-10T20:46:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.065+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc15d050 con 0x7fccfc06d1f0 2024-09-10T20:46:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.066+0000 7fcceaffd640 1 -- 172.21.15.28:0/4085410486 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fccec01a400 con 0x7fccfc06d1f0 2024-09-10T20:46:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.067+0000 7fcceaffd640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccdc078b70 0x7fccdc07b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.067+0000 7fcceaffd640 1 -- 172.21.15.28:0/4085410486 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 5881+0+0 (secure 0 0 0) 0x7fccec098b30 con 0x7fccfc06d1f0 2024-09-10T20:46:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.067+0000 7fccf9d9b640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccdc078b70 0x7fccdc07b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.067+0000 7fccf9d9b640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccdc078b70 0x7fccdc07b030 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fccf0033000 tx=0x7fccf0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.071+0000 7fcceaffd640 1 -- 172.21.15.28:0/4085410486 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fccec01f020 con 0x7fccfc06d1f0 2024-09-10T20:46:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.256+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fccfc0b0e50 con 0x7fccfc06d1f0 2024-09-10T20:46:25.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.893+0000 7fcceaffd640 1 -- 172.21.15.28:0/4085410486 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v166) ==== 181+0+0 (secure 0 0 0) 0x7fccec062e80 con 0x7fccfc06d1f0 2024-09-10T20:46:25.897 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-10T20:46:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccdc078b70 msgr2=0x7fccdc07b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccdc078b70 0x7fccdc07b030 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fccf0033000 tx=0x7fccf0004730 comp rx=0 tx=0).stop 2024-09-10T20:46:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 msgr2=0x7fccfc14bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc14bc20 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fccec00b3e0 tx=0x7fccec00b8a0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 shutdown_connections 2024-09-10T20:46:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fccdc078b70 0x7fccdc07b030 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fccfc0af930 0x7fccfc153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fccfc071af0 0x7fccfc09d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 --2- 172.21.15.28:0/4085410486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccfc06d1f0 0x7fccfc14bc20 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 >> 172.21.15.28:0/4085410486 conn(0x7fccfc0b2920 msgr2=0x7fccfc0b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 shutdown_connections 2024-09-10T20:46:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:25.897+0000 7fcd00c57640 1 -- 172.21.15.28:0/4085410486 wait complete. 2024-09-10T20:46:25.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-10T20:46:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.029+0000 7f5de19ad640 1 -- 172.21.15.28:0/2889865693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc80040b0 msgr2=0x7f5dc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.029+0000 7f5de19ad640 1 --2- 172.21.15.28:0/2889865693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc80040b0 0x7f5dc8004800 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c2e0 tx=0x7f5dc8005830 comp rx=0 tx=0).stop 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.029+0000 7f5ddb7fe640 1 -- 172.21.15.28:0/2889865693 reap_dead start 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.029+0000 7f5de19ad640 1 -- 172.21.15.28:0/2889865693 shutdown_connections 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.029+0000 7f5de19ad640 1 -- 172.21.15.28:0/2889865693 >> 172.21.15.28:0/2889865693 conn(0x7f5ddc0d3f90 msgr2=0x7f5ddc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.029+0000 7f5de19ad640 1 -- 172.21.15.28:0/2889865693 shutdown_connections 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.029+0000 7f5de19ad640 1 -- 172.21.15.28:0/2889865693 wait complete. 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.030+0000 7f5de19ad640 1 Processor -- start 2024-09-10T20:46:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.031+0000 7f5de19ad640 1 -- start start 2024-09-10T20:46:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.031+0000 7f5de19ad640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc80040b0 0x7f5ddc0b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.031+0000 7f5de19ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc0b0ac0 0x7f5ddc0b2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.031+0000 7f5de19ad640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ddc0b2a80 0x7f5ddc146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.031+0000 7f5de19ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc0d4c00 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.031+0000 7f5de19ad640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc0d4a80 con 0x7f5ddc0b2a80 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.031+0000 7f5de19ad640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f5ddc0d4d80 con 0x7f5dc80040b0 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc0b0ac0 0x7f5ddc0b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc0b0ac0 0x7f5ddc0b2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50606/0 (socket says 172.21.15.28:50606) 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 -- 172.21.15.28:0/1702572319 learned_addr learned my addr 172.21.15.28:0/1702572319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5ddb7fe640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ddc0b2a80 0x7f5ddc146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5ddaffd640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc80040b0 0x7f5ddc0b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 -- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc80040b0 msgr2=0x7f5ddc0b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc80040b0 0x7f5ddc0b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 -- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ddc0b2a80 msgr2=0x7f5ddc146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ddc0b2a80 0x7f5ddc146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 -- 172.21.15.28:0/1702572319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc145210 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5ddb7fe640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ddc0b2a80 0x7f5ddc146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5ddaffd640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc80040b0 0x7f5ddc0b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dda7fc640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc0b0ac0 0x7f5ddc0b2540 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402b130 tx=0x7f5dc4001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dd3fff640 1 -- 172.21.15.28:0/1702572319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4031040 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dd3fff640 1 -- 172.21.15.28:0/1702572319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dc40035f0 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5dd3fff640 1 -- 172.21.15.28:0/1702572319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4003a40 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ddc1454f0 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.032+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ddc145ab0 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.033+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ddc0d1c60 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.034+0000 7f5dd3fff640 1 -- 172.21.15.28:0/1702572319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f5dc4003be0 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.034+0000 7f5dd3fff640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5dbc078b40 0x7f5dbc07b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.035+0000 7f5dd3fff640 1 -- 172.21.15.28:0/1702572319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 5443+0+0 (secure 0 0 0) 0x7f5dc40b8ac0 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.035+0000 7f5ddaffd640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5dbc078b40 0x7f5dbc07b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.035+0000 7f5ddaffd640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5dbc078b40 0x7f5dbc07b000 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4000c00 tx=0x7f5dd4006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.039+0000 7f5dd3fff640 1 -- 172.21.15.28:0/1702572319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5dc4082d90 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.222+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5ddc0b9f10 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.898+0000 7f5dd3fff640 1 -- 172.21.15.28:0/1702572319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v167) ==== 169+0+0 (secure 0 0 0) 0x7f5dc4082690 con 0x7f5ddc0b0ac0 2024-09-10T20:46:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-10T20:46:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5dbc078b40 msgr2=0x7f5dbc07b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5dbc078b40 0x7f5dbc07b000 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4000c00 tx=0x7f5dd4006b70 comp rx=0 tx=0).stop 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc0b0ac0 msgr2=0x7f5ddc0b2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc0b0ac0 0x7f5ddc0b2540 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402b130 tx=0x7f5dc4001cf0 comp rx=0 tx=0).stop 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 shutdown_connections 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f5dbc078b40 0x7f5dbc07b000 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ddc0b2a80 0x7f5ddc146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.903+0000 7f5de19ad640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc0b0ac0 0x7f5ddc0b2540 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.904+0000 7f5de19ad640 1 --2- 172.21.15.28:0/1702572319 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5dc80040b0 0x7f5ddc0b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.904+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 >> 172.21.15.28:0/1702572319 conn(0x7f5ddc0d3f90 msgr2=0x7f5ddc0ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.904+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 shutdown_connections 2024-09-10T20:46:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:26.904+0000 7f5de19ad640 1 -- 172.21.15.28:0/1702572319 wait complete. 2024-09-10T20:46:26.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-10T20:46:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.036+0000 7fb799c47640 1 -- 172.21.15.28:0/662254381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb77c003f40 msgr2=0x7fb77c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.036+0000 7fb799c47640 1 --2- 172.21.15.28:0/662254381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb77c003f40 0x7fb77c004800 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fb794072b20 tx=0x7fb77c005830 comp rx=0 tx=0).stop 2024-09-10T20:46:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.036+0000 7fb793fff640 1 -- 172.21.15.28:0/662254381 reap_dead start 2024-09-10T20:46:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.036+0000 7fb799c47640 1 -- 172.21.15.28:0/662254381 shutdown_connections 2024-09-10T20:46:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.036+0000 7fb799c47640 1 -- 172.21.15.28:0/662254381 >> 172.21.15.28:0/662254381 conn(0x7fb79406cf00 msgr2=0x7fb79406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.036+0000 7fb799c47640 1 -- 172.21.15.28:0/662254381 shutdown_connections 2024-09-10T20:46:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.036+0000 7fb799c47640 1 -- 172.21.15.28:0/662254381 wait complete. 2024-09-10T20:46:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.037+0000 7fb799c47640 1 Processor -- start 2024-09-10T20:46:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.037+0000 7fb799c47640 1 -- start start 2024-09-10T20:46:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb799c47640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb77c003f40 0x7fb794143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb799c47640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7941f0260 0x7fb7941f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb799c47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79415cea0 0x7fb7940b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb799c47640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7940d0710 con 0x7fb79415cea0 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb799c47640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fb7940d0590 con 0x7fb7941f0260 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb799c47640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fb7940d0890 con 0x7fb77c003f40 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb793fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79415cea0 0x7fb7940b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb793fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79415cea0 0x7fb7940b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50620/0 (socket says 172.21.15.28:50620) 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.038+0000 7fb793fff640 1 -- 172.21.15.28:0/2523233104 learned_addr learned my addr 172.21.15.28:0/2523233104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb7937fe640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb77c003f40 0x7fb794143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb792ffd640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7941f0260 0x7fb7941f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb793fff640 1 -- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb77c003f40 msgr2=0x7fb794143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb793fff640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb77c003f40 0x7fb794143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb793fff640 1 -- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7941f0260 msgr2=0x7fb7941f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb793fff640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7941f0260 0x7fb7941f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb793fff640 1 -- 172.21.15.28:0/2523233104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7941f47a0 con 0x7fb79415cea0 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb793fff640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79415cea0 0x7fb7940b1c20 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb7940b4990 tx=0x7fb78c00fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb77bfff640 1 -- 172.21.15.28:0/2523233104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c003b80 con 0x7fb79415cea0 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb77bfff640 1 -- 172.21.15.28:0/2523233104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb78c003e10 con 0x7fb79415cea0 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7941f0720 con 0x7fb79415cea0 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb77bfff640 1 -- 172.21.15.28:0/2523233104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c004d80 con 0x7fb79415cea0 2024-09-10T20:46:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.039+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7941f0ce0 con 0x7fb79415cea0 2024-09-10T20:46:27.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.040+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb79415bd50 con 0x7fb79415cea0 2024-09-10T20:46:27.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.041+0000 7fb77bfff640 1 -- 172.21.15.28:0/2523233104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fb78c00c070 con 0x7fb79415cea0 2024-09-10T20:46:27.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.041+0000 7fb77bfff640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb780078b70 0x7fb78007b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.041+0000 7fb77bfff640 1 -- 172.21.15.28:0/2523233104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fb78c098900 con 0x7fb79415cea0 2024-09-10T20:46:27.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.041+0000 7fb7937fe640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb780078b70 0x7fb78007b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.042+0000 7fb7937fe640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb780078b70 0x7fb78007b030 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb784003700 tx=0x7fb78400b260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:27.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.046+0000 7fb77bfff640 1 -- 172.21.15.28:0/2523233104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb7941f47a0 con 0x7fb79415cea0 2024-09-10T20:46:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.231+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fb7940ad020 con 0x7fb79415cea0 2024-09-10T20:46:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.235+0000 7fb77bfff640 1 -- 172.21.15.28:0/2523233104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v49) ==== 71+0+0 (secure 0 0 0) 0x7fb78c062d90 con 0x7fb79415cea0 2024-09-10T20:46:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.237+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb780078b70 msgr2=0x7fb78007b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.237+0000 7fb799c47640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb780078b70 0x7fb78007b030 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb784003700 tx=0x7fb78400b260 comp rx=0 tx=0).stop 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.237+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79415cea0 msgr2=0x7fb7940b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.237+0000 7fb799c47640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79415cea0 0x7fb7940b1c20 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb7940b4990 tx=0x7fb78c00fd90 comp rx=0 tx=0).stop 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 shutdown_connections 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fb780078b70 0x7fb78007b030 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79415cea0 0x7fb7940b1c20 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb7941f0260 0x7fb7941f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 --2- 172.21.15.28:0/2523233104 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb77c003f40 0x7fb794143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 >> 172.21.15.28:0/2523233104 conn(0x7fb79406cf00 msgr2=0x7fb7940cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 shutdown_connections 2024-09-10T20:46:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.238+0000 7fb799c47640 1 -- 172.21.15.28:0/2523233104 wait complete. 2024-09-10T20:46:27.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade' 2024-09-10T20:46:27.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.356+0000 7ffab88c6640 1 -- 172.21.15.28:0/1062486076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa0004070 msgr2=0x7ffaa0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.356+0000 7ffab88c6640 1 --2- 172.21.15.28:0/1062486076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa0004070 0x7ffaa0004800 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009b20 tx=0x7ffaa0005830 comp rx=0 tx=0).stop 2024-09-10T20:46:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.356+0000 7ffab2d9d640 1 -- 172.21.15.28:0/1062486076 reap_dead start 2024-09-10T20:46:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.356+0000 7ffab88c6640 1 -- 172.21.15.28:0/1062486076 shutdown_connections 2024-09-10T20:46:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.356+0000 7ffab88c6640 1 -- 172.21.15.28:0/1062486076 >> 172.21.15.28:0/1062486076 conn(0x7ffab40d0420 msgr2=0x7ffab40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.356+0000 7ffab88c6640 1 -- 172.21.15.28:0/1062486076 shutdown_connections 2024-09-10T20:46:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.356+0000 7ffab88c6640 1 -- 172.21.15.28:0/1062486076 wait complete. 2024-09-10T20:46:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.358+0000 7ffab88c6640 1 Processor -- start 2024-09-10T20:46:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.358+0000 7ffab88c6640 1 -- start start 2024-09-10T20:46:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaa0004070 0x7ffab4149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab41f6b10 0x7ffab40cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab41f3570 0x7ffab41ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffab415f5b0 con 0x7ffab41f3570 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ffab415f430 con 0x7ffab41f6b10 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ffab415e7e0 con 0x7ffaa0004070 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab41f3570 0x7ffab41ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab41f3570 0x7ffab41ef900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50646/0 (socket says 172.21.15.28:50646) 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 -- 172.21.15.28:0/488903307 learned_addr learned my addr 172.21.15.28:0/488903307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffaabfff640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab41f6b10 0x7ffab40cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab259c640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaa0004070 0x7ffab4149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 -- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaa0004070 msgr2=0x7ffab4149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaa0004070 0x7ffab4149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 -- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab41f6b10 msgr2=0x7ffab40cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab41f6b10 0x7ffab40cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 -- 172.21.15.28:0/488903307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab414a890 con 0x7ffab41f3570 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffaabfff640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab41f6b10 0x7ffab40cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab2d9d640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab41f3570 0x7ffab41ef900 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7ffaac002230 tx=0x7ffaac00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffaaaffd640 1 -- 172.21.15.28:0/488903307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac015070 con 0x7ffab41f3570 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffaaaffd640 1 -- 172.21.15.28:0/488903307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffaac00cb00 con 0x7ffab41f3570 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffaaaffd640 1 -- 172.21.15.28:0/488903307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac011460 con 0x7ffab41f3570 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab414a5b0 con 0x7ffab41f3570 2024-09-10T20:46:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.359+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab41f2d70 con 0x7ffab41f3570 2024-09-10T20:46:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.360+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab40cfee0 con 0x7ffab41f3570 2024-09-10T20:46:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.361+0000 7ffaaaffd640 1 -- 172.21.15.28:0/488903307 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7ffaac011600 con 0x7ffab41f3570 2024-09-10T20:46:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.362+0000 7ffaaaffd640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa94078b70 0x7ffa9407b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.362+0000 7ffab259c640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa94078b70 0x7ffa9407b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.362+0000 7ffaaaffd640 1 -- 172.21.15.28:0/488903307 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7ffaac0990b0 con 0x7ffab41f3570 2024-09-10T20:46:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.362+0000 7ffab259c640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa94078b70 0x7ffa9407b030 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4000c00 tx=0x7ffaa4006b70 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.366+0000 7ffaaaffd640 1 -- 172.21.15.28:0/488903307 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffab40cfee0 con 0x7ffab41f3570 2024-09-10T20:46:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.621+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7ffab40738b0 con 0x7ffab41f3570 2024-09-10T20:46:27.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.900+0000 7ffaaaffd640 1 -- 172.21.15.28:0/488903307 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]}]=0 v461) ==== 163+0+0 (secure 0 0 0) 0x7ffab414a890 con 0x7ffab41f3570 2024-09-10T20:46:27.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa94078b70 msgr2=0x7ffa9407b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa94078b70 0x7ffa9407b030 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4000c00 tx=0x7ffaa4006b70 comp rx=0 tx=0).stop 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab41f3570 msgr2=0x7ffab41ef900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab41f3570 0x7ffab41ef900 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7ffaac002230 tx=0x7ffaac00eb20 comp rx=0 tx=0).stop 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 shutdown_connections 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7ffa94078b70 0x7ffa9407b030 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab41f3570 0x7ffab41ef900 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffab41f6b10 0x7ffab40cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 --2- 172.21.15.28:0/488903307 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffaa0004070 0x7ffab4149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.902+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 >> 172.21.15.28:0/488903307 conn(0x7ffab40d0420 msgr2=0x7ffab415c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.903+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 shutdown_connections 2024-09-10T20:46:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:27.903+0000 7ffab88c6640 1 -- 172.21.15.28:0/488903307 wait complete. 2024-09-10T20:46:27.918 INFO:tasks.cephfs_test_runner:Validate unparseable text .meta file on legacy subvol root doesn't break the system ... ok 2024-09-10T20:46:27.919 INFO:tasks.cephfs_test_runner: 2024-09-10T20:46:27.919 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-10T20:46:27.919 INFO:tasks.cephfs_test_runner:Ran 11 tests in 471.384s 2024-09-10T20:46:27.919 INFO:tasks.cephfs_test_runner: 2024-09-10T20:46:27.919 INFO:tasks.cephfs_test_runner:OK 2024-09-10T20:46:27.919 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-09-10T20:46:27.928 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-10T20:46:27.963 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-10T20:46:27.963 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:46:27.964 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:27.981 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:46:27.981 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T20:46:27.981 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:28.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16521 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-10T20:46:28.039 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:46:28.039 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:28.058 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:46:28.058 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-10T20:46:28.058 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T20:46:28.117 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19606 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-10T20:46:28.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.117 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:46:28.134 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-10T20:46:28.134 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-10T20:46:28.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-10T20:46:28.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T20:46:28.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-10T20:46:28.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.365 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-10T20:46:28.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-10T20:46:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-10T20:46:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:46:28.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.453 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:46:28.472 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-10T20:46:28.472 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-10T20:46:28.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-10T20:46:28.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.542 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T20:46:28.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.611 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-09-10T20:46:28.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.714 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-09-10T20:46:28.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T20:46:28.733 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-09-10T20:46:28.802 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-09-10T20:46:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:46:28.804 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-10T20:46:28.812 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:46:28.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 -- 172.21.15.28:0/3873848319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 msgr2=0x7f92c80d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 --2- 172.21.15.28:0/3873848319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c80d0140 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f92c809e0b0 tx=0x7f92c002aa30 comp rx=0 tx=0).stop 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 -- 172.21.15.28:0/3873848319 shutdown_connections 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 --2- 172.21.15.28:0/3873848319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c80d0140 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 --2- 172.21.15.28:0/3873848319 >> v2:172.21.15.159:3300/0 conn(0x7f92c81554e0 0x7f92c815e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 -- 172.21.15.28:0/3873848319 >> 172.21.15.28:0/3873848319 conn(0x7f92c806cf00 msgr2=0x7f92c806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 -- 172.21.15.28:0/3873848319 shutdown_connections 2024-09-10T20:46:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.920+0000 7f92ce32b640 1 -- 172.21.15.28:0/3873848319 wait complete. 2024-09-10T20:46:28.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.921+0000 7f92ce32b640 1 Processor -- start 2024-09-10T20:46:28.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.921+0000 7f92ce32b640 1 -- start start 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92ce32b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c8155100 0x7f92c81f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92ce32b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c81554e0 0x7f92c80b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92ce32b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c8157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92ce32b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92c8072d50 con 0x7f92c815ee70 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92ce32b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f92c8072bd0 con 0x7f92c81554e0 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92ce32b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f92c8072ed0 con 0x7f92c8155100 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c8157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c8157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50670/0 (socket says 172.21.15.28:50670) 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 -- 172.21.15.28:0/264135556 learned_addr learned my addr 172.21.15.28:0/264135556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92c77fe640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c81554e0 0x7f92c80b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92c7fff640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c8155100 0x7f92c81f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 -- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c8155100 msgr2=0x7f92c81f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c8155100 0x7f92c81f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 -- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c81554e0 msgr2=0x7f92c80b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c81554e0 0x7f92c80b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 -- 172.21.15.28:0/264135556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c81f6240 con 0x7f92c815ee70 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92c7fff640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c8155100 0x7f92c81f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.922+0000 7f92cc8c8640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c8157580 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f92c002b020 tx=0x7f92c0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.923+0000 7f92c4ff9640 1 -- 172.21.15.28:0/264135556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0046070 con 0x7f92c815ee70 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.923+0000 7f92c4ff9640 1 -- 172.21.15.28:0/264135556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92c0004250 con 0x7f92c815ee70 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.923+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c81f6520 con 0x7f92c815ee70 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.923+0000 7f92c4ff9640 1 -- 172.21.15.28:0/264135556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0033050 con 0x7f92c815ee70 2024-09-10T20:46:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.923+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c81f2710 con 0x7f92c815ee70 2024-09-10T20:46:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.924+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c815e930 con 0x7f92c815ee70 2024-09-10T20:46:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.924+0000 7f92c4ff9640 1 -- 172.21.15.28:0/264135556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f92c0004680 con 0x7f92c815ee70 2024-09-10T20:46:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.925+0000 7f92c4ff9640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92b4078b40 0x7f92b407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:28.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.925+0000 7f92c4ff9640 1 -- 172.21.15.28:0/264135556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f92c00b9b20 con 0x7f92c815ee70 2024-09-10T20:46:28.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.925+0000 7f92c7fff640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92b4078b40 0x7f92b407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:28.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.925+0000 7f92c7fff640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92b4078b40 0x7f92b407b000 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007fd0 tx=0x7f92b8009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:28.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:28.929+0000 7f92c4ff9640 1 -- 172.21.15.28:0/264135556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f92c0083820 con 0x7f92c815ee70 2024-09-10T20:46:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.111+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92c81f28e0 con 0x7f92b4078b40 2024-09-10T20:46:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.111+0000 7f92c4ff9640 1 -- 172.21.15.28:0/264135556 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13503 (secure 0 0 0) 0x7f92c81f28e0 con 0x7f92b4078b40 2024-09-10T20:46:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:29.116 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92b4078b40 msgr2=0x7f92b407b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92b4078b40 0x7f92b407b000 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007fd0 tx=0x7f92b8009040 comp rx=0 tx=0).stop 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 msgr2=0x7f92c8157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c8157580 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f92c002b020 tx=0x7f92c0003f10 comp rx=0 tx=0).stop 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 shutdown_connections 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f92b4078b40 0x7f92b407b000 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c815ee70 0x7f92c8157580 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c81554e0 0x7f92c80b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 --2- 172.21.15.28:0/264135556 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c8155100 0x7f92c81f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 >> 172.21.15.28:0/264135556 conn(0x7f92c806cf00 msgr2=0x7f92c809d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 shutdown_connections 2024-09-10T20:46:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.114+0000 7f92ce32b640 1 -- 172.21.15.28:0/264135556 wait complete. 2024-09-10T20:46:29.128 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-10T20:46:28.343197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283608,"kb_used_data":17492,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754691112,"statfs":{"total":773094113280,"available":772803698688,"internally_reserved":0,"allocated":17911808,"data_stored":8632021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29747,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-61,"num_write_kb":-51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001087"},"pg_stats":[{"pgid":"1.0","version":"69'79","reported_seq":404,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-10T20:46:11.316307+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:46:11.316307+0000","last_peered":"2024-09-10T20:46:11.316307+0000","last_clean":"2024-09-10T20:46:11.316307+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:46:11.316307+0000","last_undegraded":"2024-09-10T20:46:11.316307+0000","last_fullsized":"2024-09-10T20:46:11.316307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542248,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2068480,"data_stored":914841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542249,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2068480,"data_stored":914889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542247,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33796,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338044,"statfs":{"total":96636764160,"available":96602157056,"internally_reserved":0,"allocated":2674688,"data_stored":1505915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542247,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33220,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338620,"statfs":{"total":96636764160,"available":96602746880,"internally_reserved":0,"allocated":2080768,"data_stored":921076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509545,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37432,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334408,"statfs":{"total":96636764160,"available":96598433792,"internally_reserved":0,"allocated":2138112,"data_stored":977001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509544,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37440,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334400,"statfs":{"total":96636764160,"available":96598425600,"internally_reserved":0,"allocated":2146304,"data_stored":977105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509544,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37944,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333896,"statfs":{"total":96636764160,"available":96597909504,"internally_reserved":0,"allocated":2662400,"data_stored":1505490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509544,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37368,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334472,"statfs":{"total":96636764160,"available":96598499328,"internally_reserved":0,"allocated":2072576,"data_stored":915704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:46:29.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:46:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 -- 172.21.15.28:0/1470430776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 msgr2=0x7fd8c015bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 --2- 172.21.15.28:0/1470430776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c015bd50 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0072af0 tx=0x7fd8b002a560 comp rx=0 tx=0).stop 2024-09-10T20:46:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 -- 172.21.15.28:0/1470430776 shutdown_connections 2024-09-10T20:46:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 --2- 172.21.15.28:0/1470430776 >> v2:172.21.15.159:3300/0 conn(0x7fd8c00d0140 0x7fd8c00d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 --2- 172.21.15.28:0/1470430776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c015bd50 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 -- 172.21.15.28:0/1470430776 >> 172.21.15.28:0/1470430776 conn(0x7fd8c006cf00 msgr2=0x7fd8c006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 -- 172.21.15.28:0/1470430776 shutdown_connections 2024-09-10T20:46:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.231+0000 7fd8c5652640 1 -- 172.21.15.28:0/1470430776 wait complete. 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.232+0000 7fd8c5652640 1 Processor -- start 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.232+0000 7fd8c5652640 1 -- start start 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8c5652640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8c00d0140 0x7fd8c01f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8c5652640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c0158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8c5652640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8c01554e0 0x7fd8c01591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8c5652640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0072f60 con 0x7fd8c0155100 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8c5652640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0072de0 con 0x7fd8c00d0140 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8c5652640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c00730e0 con 0x7fd8c01554e0 2024-09-10T20:46:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c0158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c0158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50690/0 (socket says 172.21.15.28:50690) 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 -- 172.21.15.28:0/3145043143 learned_addr learned my addr 172.21.15.28:0/3145043143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8beffd640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8c00d0140 0x7fd8c01f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8bf7fe640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8c01554e0 0x7fd8c01591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 -- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8c01554e0 msgr2=0x7fd8c01591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8c01554e0 0x7fd8c01591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 -- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8c00d0140 msgr2=0x7fd8c01f5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8c00d0140 0x7fd8c01f5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 -- 172.21.15.28:0/3145043143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c0152e90 con 0x7fd8c0155100 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8bf7fe640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8c01554e0 0x7fd8c01591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.233+0000 7fd8b67fc640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c0158ca0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800c730 tx=0x7fd8a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.234+0000 7fd8b7fff640 1 -- 172.21.15.28:0/3145043143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8017070 con 0x7fd8c0155100 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.234+0000 7fd8b7fff640 1 -- 172.21.15.28:0/3145043143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8a8005180 con 0x7fd8c0155100 2024-09-10T20:46:29.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.234+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c0153170 con 0x7fd8c0155100 2024-09-10T20:46:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.234+0000 7fd8b7fff640 1 -- 172.21.15.28:0/3145043143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a800a640 con 0x7fd8c0155100 2024-09-10T20:46:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.234+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c0153730 con 0x7fd8c0155100 2024-09-10T20:46:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.235+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c0073590 con 0x7fd8c0155100 2024-09-10T20:46:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.235+0000 7fd8b7fff640 1 -- 172.21.15.28:0/3145043143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fd8a800a7e0 con 0x7fd8c0155100 2024-09-10T20:46:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.236+0000 7fd8b7fff640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8a0078b40 0x7fd8a007b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.236+0000 7fd8beffd640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8a0078b40 0x7fd8a007b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.236+0000 7fd8b7fff640 1 -- 172.21.15.28:0/3145043143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fd8a8098080 con 0x7fd8c0155100 2024-09-10T20:46:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.237+0000 7fd8beffd640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8a0078b40 0x7fd8a007b000 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0033000 tx=0x7fd8b0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:29.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.240+0000 7fd8b7fff640 1 -- 172.21.15.28:0/3145043143 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd8c015c290 con 0x7fd8c0155100 2024-09-10T20:46:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.422+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd8c0069a80 con 0x7fd8a0078b40 2024-09-10T20:46:29.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.423+0000 7fd8b7fff640 1 -- 172.21.15.28:0/3145043143 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13503 (secure 0 0 0) 0x7fd8c0069a80 con 0x7fd8a0078b40 2024-09-10T20:46:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:29.427 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:46:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8a0078b40 msgr2=0x7fd8a007b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8a0078b40 0x7fd8a007b000 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0033000 tx=0x7fd8b0004730 comp rx=0 tx=0).stop 2024-09-10T20:46:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 msgr2=0x7fd8c0158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c0158ca0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800c730 tx=0x7fd8a800cbf0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 shutdown_connections 2024-09-10T20:46:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd8a0078b40 0x7fd8a007b000 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd8c01554e0 0x7fd8c01591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0155100 0x7fd8c0158ca0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 --2- 172.21.15.28:0/3145043143 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd8c00d0140 0x7fd8c01f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 >> 172.21.15.28:0/3145043143 conn(0x7fd8c006cf00 msgr2=0x7fd8c00737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 shutdown_connections 2024-09-10T20:46:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.425+0000 7fd8c5652640 1 -- 172.21.15.28:0/3145043143 wait complete. 2024-09-10T20:46:29.439 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-10T20:46:28.343197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283608,"kb_used_data":17492,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754691112,"statfs":{"total":773094113280,"available":772803698688,"internally_reserved":0,"allocated":17911808,"data_stored":8632021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29747,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-61,"num_write_kb":-51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001087"},"pg_stats":[{"pgid":"1.0","version":"69'79","reported_seq":404,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-10T20:46:11.316307+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:46:11.316307+0000","last_peered":"2024-09-10T20:46:11.316307+0000","last_clean":"2024-09-10T20:46:11.316307+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:46:11.316307+0000","last_undegraded":"2024-09-10T20:46:11.316307+0000","last_fullsized":"2024-09-10T20:46:11.316307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542248,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2068480,"data_stored":914841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542249,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2068480,"data_stored":914889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542247,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33796,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338044,"statfs":{"total":96636764160,"available":96602157056,"internally_reserved":0,"allocated":2674688,"data_stored":1505915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542247,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33220,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338620,"statfs":{"total":96636764160,"available":96602746880,"internally_reserved":0,"allocated":2080768,"data_stored":921076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509545,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37432,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334408,"statfs":{"total":96636764160,"available":96598433792,"internally_reserved":0,"allocated":2138112,"data_stored":977001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509544,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37440,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334400,"statfs":{"total":96636764160,"available":96598425600,"internally_reserved":0,"allocated":2146304,"data_stored":977105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509544,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37944,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333896,"statfs":{"total":96636764160,"available":96597909504,"internally_reserved":0,"allocated":2662400,"data_stored":1505490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509544,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37368,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334472,"statfs":{"total":96636764160,"available":96598499328,"internally_reserved":0,"allocated":2072576,"data_stored":915704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:46:29.440 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:46:29.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:46:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 -- 172.21.15.28:0/3933278349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300d05d0 msgr2=0x7fdf300b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 --2- 172.21.15.28:0/3933278349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300d05d0 0x7fdf300b2e80 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009be0 tx=0x7fdf2802a8d0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 -- 172.21.15.28:0/3933278349 shutdown_connections 2024-09-10T20:46:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 --2- 172.21.15.28:0/3933278349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300d05d0 0x7fdf300b2e80 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 --2- 172.21.15.28:0/3933278349 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf300cc570 0x7fdf300d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 -- 172.21.15.28:0/3933278349 >> 172.21.15.28:0/3933278349 conn(0x7fdf3006cf00 msgr2=0x7fdf3006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 -- 172.21.15.28:0/3933278349 shutdown_connections 2024-09-10T20:46:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.542+0000 7fdf3727f640 1 -- 172.21.15.28:0/3933278349 wait complete. 2024-09-10T20:46:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.543+0000 7fdf3727f640 1 Processor -- start 2024-09-10T20:46:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.543+0000 7fdf3727f640 1 -- start start 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3727f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300cc570 0x7fdf301f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3727f640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf300cd3d0 0x7fdf301f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3727f640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf300d05d0 0x7fdf301f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3727f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf300b33c0 con 0x7fdf300cc570 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3727f640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fdf300b4b60 con 0x7fdf300cd3d0 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3727f640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fdf300b3540 con 0x7fdf300d05d0 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300cc570 0x7fdf301f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300cc570 0x7fdf301f40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50722/0 (socket says 172.21.15.28:50722) 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 -- 172.21.15.28:0/4106154920 learned_addr learned my addr 172.21.15.28:0/4106154920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3581c640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf300d05d0 0x7fdf301f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3481a640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf300cd3d0 0x7fdf301f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 -- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf300d05d0 msgr2=0x7fdf301f6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf300d05d0 0x7fdf301f6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 -- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf300cd3d0 msgr2=0x7fdf301f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf300cd3d0 0x7fdf301f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 -- 172.21.15.28:0/4106154920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf301f49c0 con 0x7fdf300cc570 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.544+0000 7fdf3501b640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300cc570 0x7fdf301f40d0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00c490 tx=0x7fdf2c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.545+0000 7fdf25ffb640 1 -- 172.21.15.28:0/4106154920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c016020 con 0x7fdf300cc570 2024-09-10T20:46:29.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.545+0000 7fdf25ffb640 1 -- 172.21.15.28:0/4106154920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf2c012940 con 0x7fdf300cc570 2024-09-10T20:46:29.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.545+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf301f4ca0 con 0x7fdf300cc570 2024-09-10T20:46:29.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.545+0000 7fdf25ffb640 1 -- 172.21.15.28:0/4106154920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c012d90 con 0x7fdf300cc570 2024-09-10T20:46:29.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.545+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf301f5260 con 0x7fdf300cc570 2024-09-10T20:46:29.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.546+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf300b39d0 con 0x7fdf300cc570 2024-09-10T20:46:29.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.547+0000 7fdf25ffb640 1 -- 172.21.15.28:0/4106154920 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fdf2c01b400 con 0x7fdf300cc570 2024-09-10T20:46:29.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.547+0000 7fdf25ffb640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdf14078b40 0x7fdf1407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.547+0000 7fdf25ffb640 1 -- 172.21.15.28:0/4106154920 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fdf2c098ad0 con 0x7fdf300cc570 2024-09-10T20:46:29.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.547+0000 7fdf3481a640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdf14078b40 0x7fdf1407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.548+0000 7fdf3481a640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdf14078b40 0x7fdf1407b000 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdf3015bd30 tx=0x7fdf20005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:29.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.552+0000 7fdf25ffb640 1 -- 172.21.15.28:0/4106154920 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdf2c020020 con 0x7fdf300cc570 2024-09-10T20:46:29.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.734+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf3015ba80 con 0x7fdf14078b40 2024-09-10T20:46:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.734+0000 7fdf25ffb640 1 -- 172.21.15.28:0/4106154920 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13503 (secure 0 0 0) 0x7fdf3015ba80 con 0x7fdf14078b40 2024-09-10T20:46:29.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdf14078b40 msgr2=0x7fdf1407b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdf14078b40 0x7fdf1407b000 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdf3015bd30 tx=0x7fdf20005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300cc570 msgr2=0x7fdf301f40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300cc570 0x7fdf301f40d0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00c490 tx=0x7fdf2c00c950 comp rx=0 tx=0).stop 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 shutdown_connections 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fdf14078b40 0x7fdf1407b000 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdf300d05d0 0x7fdf301f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdf300cd3d0 0x7fdf301f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 --2- 172.21.15.28:0/4106154920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf300cc570 0x7fdf301f40d0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 >> 172.21.15.28:0/4106154920 conn(0x7fdf3006cf00 msgr2=0x7fdf3009ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 shutdown_connections 2024-09-10T20:46:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.737+0000 7fdf3727f640 1 -- 172.21.15.28:0/4106154920 wait complete. 2024-09-10T20:46:29.751 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-10T20:46:28.343197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283608,"kb_used_data":17492,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754691112,"statfs":{"total":773094113280,"available":772803698688,"internally_reserved":0,"allocated":17911808,"data_stored":8632021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29747,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-61,"num_write_kb":-51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001087"},"pg_stats":[{"pgid":"1.0","version":"69'79","reported_seq":404,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-10T20:46:11.316307+0000","last_change":"2024-09-10T20:38:01.346713+0000","last_active":"2024-09-10T20:46:11.316307+0000","last_peered":"2024-09-10T20:46:11.316307+0000","last_clean":"2024-09-10T20:46:11.316307+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:46:11.316307+0000","last_undegraded":"2024-09-10T20:46:11.316307+0000","last_fullsized":"2024-09-10T20:46:11.316307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:38:00.333783+0000","last_clean_scrub_stamp":"2024-09-10T20:38:00.333783+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:36.702466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542248,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2068480,"data_stored":914841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542249,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2068480,"data_stored":914889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542247,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33796,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338044,"statfs":{"total":96636764160,"available":96602157056,"internally_reserved":0,"allocated":2674688,"data_stored":1505915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542247,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33220,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338620,"statfs":{"total":96636764160,"available":96602746880,"internally_reserved":0,"allocated":2080768,"data_stored":921076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509545,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37432,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334408,"statfs":{"total":96636764160,"available":96598433792,"internally_reserved":0,"allocated":2138112,"data_stored":977001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509544,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37440,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334400,"statfs":{"total":96636764160,"available":96598425600,"internally_reserved":0,"allocated":2146304,"data_stored":977105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509544,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37944,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333896,"statfs":{"total":96636764160,"available":96597909504,"internally_reserved":0,"allocated":2662400,"data_stored":1505490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509544,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37368,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334472,"statfs":{"total":96636764160,"available":96598499328,"internally_reserved":0,"allocated":2072576,"data_stored":915704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:46:29.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:46:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 -- 172.21.15.28:0/271756598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a300d2b90 msgr2=0x7f3a30161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 --2- 172.21.15.28:0/271756598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a300d2b90 0x7f3a30161b50 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f3a300d01d0 tx=0x7f3a1802a5f0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 -- 172.21.15.28:0/271756598 shutdown_connections 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 --2- 172.21.15.28:0/271756598 >> v2:172.21.15.159:3300/0 conn(0x7f3a30162090 0x7f3a300b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 --2- 172.21.15.28:0/271756598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a300d2b90 0x7f3a30161b50 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 -- 172.21.15.28:0/271756598 >> 172.21.15.28:0/271756598 conn(0x7f3a300d3fb0 msgr2=0x7f3a300d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 -- 172.21.15.28:0/271756598 shutdown_connections 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.853+0000 7f3a34d32640 1 -- 172.21.15.28:0/271756598 wait complete. 2024-09-10T20:46:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.855+0000 7f3a34d32640 1 Processor -- start 2024-09-10T20:46:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.855+0000 7f3a34d32640 1 -- start start 2024-09-10T20:46:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a34d32640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a300d2b90 0x7f3a301f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a34d32640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a3015de70 0x7f3a300cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a34d32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30162090 0x7f3a300cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a34d32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3015e3d0 con 0x7f3a30162090 2024-09-10T20:46:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a34d32640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3015e250 con 0x7f3a300d2b90 2024-09-10T20:46:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a34d32640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f3a300b7340 con 0x7f3a3015de70 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30162090 0x7f3a300cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30162090 0x7f3a300cb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50730/0 (socket says 172.21.15.28:50730) 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 -- 172.21.15.28:0/26606282 learned_addr learned my addr 172.21.15.28:0/26606282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2e59c640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a300d2b90 0x7f3a301f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2dd9b640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a3015de70 0x7f3a300cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 -- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a3015de70 msgr2=0x7f3a300cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a3015de70 0x7f3a300cb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 -- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a300d2b90 msgr2=0x7f3a301f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a300d2b90 0x7f3a301f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 -- 172.21.15.28:0/26606282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a300cc130 con 0x7f3a30162090 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2e59c640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a300d2b90 0x7f3a301f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a2ed9d640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30162090 0x7f3a300cb5c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000d660 tx=0x7f3a2000db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a26ffd640 1 -- 172.21.15.28:0/26606282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20017070 con 0x7f3a30162090 2024-09-10T20:46:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a26ffd640 1 -- 172.21.15.28:0/26606282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a20005310 con 0x7f3a30162090 2024-09-10T20:46:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.856+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a300cddb0 con 0x7f3a30162090 2024-09-10T20:46:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.857+0000 7f3a26ffd640 1 -- 172.21.15.28:0/26606282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20009820 con 0x7f3a30162090 2024-09-10T20:46:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.857+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a300cc680 con 0x7f3a30162090 2024-09-10T20:46:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.857+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a300ce150 con 0x7f3a30162090 2024-09-10T20:46:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.858+0000 7f3a26ffd640 1 -- 172.21.15.28:0/26606282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f3a200099c0 con 0x7f3a30162090 2024-09-10T20:46:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.859+0000 7f3a26ffd640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3a10078b40 0x7f3a1007b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.859+0000 7f3a26ffd640 1 -- 172.21.15.28:0/26606282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f3a20098a30 con 0x7f3a30162090 2024-09-10T20:46:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.859+0000 7f3a2e59c640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3a10078b40 0x7f3a1007b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.859+0000 7f3a2e59c640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3a10078b40 0x7f3a1007b000 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3a28001db0 tx=0x7f3a28009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:29.863+0000 7f3a26ffd640 1 -- 172.21.15.28:0/26606282 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3a20062d80 con 0x7f3a30162090 2024-09-10T20:46:30.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.052+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a300b9f30 con 0x7f3a30162090 2024-09-10T20:46:30.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.053+0000 7f3a26ffd640 1 -- 172.21.15.28:0/26606282 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) ==== 74+0+13086 (secure 0 0 0) 0x7f3a20062680 con 0x7f3a30162090 2024-09-10T20:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":167,"fsid":"943b2354-acb0-43ad-b06a-e7974842506f","created":"2024-09-10T20:37:47.616376+0000","modified":"2024-09-10T20:46:26.894187+0000","last_up_change":"2024-09-10T20:38:00.333783+0000","last_in_change":"2024-09-10T20:37:50.526381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:37:59.990929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"547ada29-9f8c-465c-a0ce-218d9a56bff4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6809","nonce":2488809808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6811","nonce":2488809808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6815","nonce":2488809808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2488809808},{"type":"v1","addr":"172.21.15.28:6813","nonce":2488809808}]},"public_addr":"172.21.15.28:6809/2488809808","cluster_addr":"172.21.15.28:6811/2488809808","heartbeat_back_addr":"172.21.15.28:6815/2488809808","heartbeat_front_addr":"172.21.15.28:6813/2488809808","state":["exists","up"]},{"osd":1,"uuid":"c8bd37ec-f775-41d8-a6bd-517095803a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6801","nonce":1047588896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6803","nonce":1047588896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6807","nonce":1047588896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1047588896},{"type":"v1","addr":"172.21.15.28:6805","nonce":1047588896}]},"public_addr":"172.21.15.28:6801/1047588896","cluster_addr":"172.21.15.28:6803/1047588896","heartbeat_back_addr":"172.21.15.28:6807/1047588896","heartbeat_front_addr":"172.21.15.28:6805/1047588896","state":["exists","up"]},{"osd":2,"uuid":"082d2815-a747-4bf3-bdb2-02af942ae051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6817","nonce":4001706199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6819","nonce":4001706199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6823","nonce":4001706199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4001706199},{"type":"v1","addr":"172.21.15.28:6821","nonce":4001706199}]},"public_addr":"172.21.15.28:6817/4001706199","cluster_addr":"172.21.15.28:6819/4001706199","heartbeat_back_addr":"172.21.15.28:6823/4001706199","heartbeat_front_addr":"172.21.15.28:6821/4001706199","state":["exists","up"]},{"osd":3,"uuid":"d222baa7-bb15-4bc0-9aca-a3a95301665d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6825","nonce":1557654590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6827","nonce":1557654590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6831","nonce":1557654590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1557654590},{"type":"v1","addr":"172.21.15.28:6829","nonce":1557654590}]},"public_addr":"172.21.15.28:6825/1557654590","cluster_addr":"172.21.15.28:6827/1557654590","heartbeat_back_addr":"172.21.15.28:6831/1557654590","heartbeat_front_addr":"172.21.15.28:6829/1557654590","state":["exists","up"]},{"osd":4,"uuid":"c143c33b-86b2-466c-9cff-cd0494dd6675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6803","nonce":1548693517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6805","nonce":1548693517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6809","nonce":1548693517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":1548693517},{"type":"v1","addr":"172.21.15.159:6807","nonce":1548693517}]},"public_addr":"172.21.15.159:6803/1548693517","cluster_addr":"172.21.15.159:6805/1548693517","heartbeat_back_addr":"172.21.15.159:6809/1548693517","heartbeat_front_addr":"172.21.15.159:6807/1548693517","state":["exists","up"]},{"osd":5,"uuid":"4ec51bc6-6e6d-4aaf-b889-9f47e66698b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6827","nonce":2363244923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6829","nonce":2363244923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6833","nonce":2363244923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":2363244923},{"type":"v1","addr":"172.21.15.159:6831","nonce":2363244923}]},"public_addr":"172.21.15.159:6827/2363244923","cluster_addr":"172.21.15.159:6829/2363244923","heartbeat_back_addr":"172.21.15.159:6833/2363244923","heartbeat_front_addr":"172.21.15.159:6831/2363244923","state":["exists","up"]},{"osd":6,"uuid":"26884790-c3f6-4a07-8df5-9ddb7f0cbfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6811","nonce":3584510546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6813","nonce":3584510546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6817","nonce":3584510546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3584510546},{"type":"v1","addr":"172.21.15.159:6815","nonce":3584510546}]},"public_addr":"172.21.15.159:6811/3584510546","cluster_addr":"172.21.15.159:6813/3584510546","heartbeat_back_addr":"172.21.15.159:6817/3584510546","heartbeat_front_addr":"172.21.15.159:6815/3584510546","state":["exists","up"]},{"osd":7,"uuid":"2318f19e-1d8f-4383-97c5-e10e7331c8b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6819","nonce":88942910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6821","nonce":88942910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6825","nonce":88942910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":88942910},{"type":"v1","addr":"172.21.15.159:6823","nonce":88942910}]},"public_addr":"172.21.15.159:6819/88942910","cluster_addr":"172.21.15.159:6821/88942910","heartbeat_back_addr":"172.21.15.159:6825/88942910","heartbeat_front_addr":"172.21.15.159:6823/88942910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.118306+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.311825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.346176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.864379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.537925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.609551+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:57.425310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:37:56.780846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1516352866":"2024-09-11T20:46:23.884640+0000","172.21.15.28:6833/1516352866":"2024-09-11T20:46:23.884640+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:46:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3a10078b40 msgr2=0x7f3a1007b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3a10078b40 0x7f3a1007b000 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3a28001db0 tx=0x7f3a28009040 comp rx=0 tx=0).stop 2024-09-10T20:46:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30162090 msgr2=0x7f3a300cb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30162090 0x7f3a300cb5c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000d660 tx=0x7f3a2000db20 comp rx=0 tx=0).stop 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 shutdown_connections 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3a10078b40 0x7f3a1007b000 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30162090 0x7f3a300cb5c0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3a3015de70 0x7f3a300cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 --2- 172.21.15.28:0/26606282 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3a300d2b90 0x7f3a301f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 >> 172.21.15.28:0/26606282 conn(0x7f3a300d3fb0 msgr2=0x7f3a3006eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 shutdown_connections 2024-09-10T20:46:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:30.055+0000 7f3a34d32640 1 -- 172.21.15.28:0/26606282 wait complete. 2024-09-10T20:46:31.071 INFO:tasks.ceph:Scrubbing osd.0 2024-09-10T20:46:31.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:31.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/1910507855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b415f540 msgr2=0x7fd0b40cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 --2- 172.21.15.28:0/1910507855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b415f540 0x7fd0b40cc950 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009900 tx=0x7fd0b002a560 comp rx=0 tx=0).stop 2024-09-10T20:46:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/1910507855 shutdown_connections 2024-09-10T20:46:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 --2- 172.21.15.28:0/1910507855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0b40cce90 0x7fd0b40cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 --2- 172.21.15.28:0/1910507855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b415f540 0x7fd0b40cc950 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/1910507855 >> 172.21.15.28:0/1910507855 conn(0x7fd0b40d1e40 msgr2=0x7fd0b40d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/1910507855 shutdown_connections 2024-09-10T20:46:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.190+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/1910507855 wait complete. 2024-09-10T20:46:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.192+0000 7fd0bbe9b640 1 Processor -- start 2024-09-10T20:46:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.192+0000 7fd0bbe9b640 1 -- start start 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.192+0000 7fd0bbe9b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0b40cce90 0x7fd0b415d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.192+0000 7fd0bbe9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b40ce580 0x7fd0b4156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.192+0000 7fd0bbe9b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0b415f540 0x7fd0b40b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0bbe9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b40b6750 con 0x7fd0b40ce580 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0bbe9b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b40b65d0 con 0x7fd0b415f540 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0bbe9b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b40b68d0 con 0x7fd0b40cce90 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b40ce580 0x7fd0b4156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b40ce580 0x7fd0b4156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38828/0 (socket says 172.21.15.28:38828) 2024-09-10T20:46:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 -- 172.21.15.28:0/2366869636 learned_addr learned my addr 172.21.15.28:0/2366869636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9c37640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0b40cce90 0x7fd0b415d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0ba438640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0b415f540 0x7fd0b40b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 -- 172.21.15.28:0/2366869636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0b40cce90 msgr2=0x7fd0b415d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0b40cce90 0x7fd0b415d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 -- 172.21.15.28:0/2366869636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0b415f540 msgr2=0x7fd0b40b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd0b415f540 0x7fd0b40b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 -- 172.21.15.28:0/2366869636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b4156c80 con 0x7fd0b40ce580 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0b9436640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b40ce580 0x7fd0b4156740 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c920 tx=0x7fd0a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4007c70 con 0x7fd0b40ce580 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0a400f450 con 0x7fd0b40ce580 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a400f8a0 con 0x7fd0b40ce580 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.193+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b41f8270 con 0x7fd0b40ce580 2024-09-10T20:46:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.194+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b41f87f0 con 0x7fd0b40ce580 2024-09-10T20:46:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.194+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd0b4073020 con 0x7fd0b40ce580 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.195+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fd0a4018400 con 0x7fd0b40ce580 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0aa7fc640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd098078b40 0x7fd09807b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fd0a4099580 con 0x7fd0b40ce580 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0b9c37640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd098078b40 0x7fd09807b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0aa7fc640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7fd09807ea00 0x7fd098080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 --> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0980814f0 con 0x7fd09807ea00 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7fd0a409c1f0 con 0x7fd0b40ce580 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0ba438640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7fd09807ea00 0x7fd098080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0b9c37640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd098078b40 0x7fd09807b000 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0033000 tx=0x7fd0b0004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.196+0000 7fd0ba438640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7fd09807ea00 0x7fd098080e20 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.197+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== osd.0 v2:172.21.15.28:6808/2488809808 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd0980814f0 con 0x7fd09807ea00 2024-09-10T20:46:31.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 --> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd0b40cf500 con 0x7fd09807ea00 2024-09-10T20:46:31.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0aa7fc640 1 -- 172.21.15.28:0/2366869636 <== osd.0 v2:172.21.15.28:6808/2488809808 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd0b40cf500 con 0x7fd09807ea00 2024-09-10T20:46:31.214 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:31.214 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:31.214 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7fd09807ea00 msgr2=0x7fd098080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0bbe9b640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6808/2488809808,v1:172.21.15.28:6809/2488809808] conn(0x7fd09807ea00 0x7fd098080e20 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd098078b40 msgr2=0x7fd09807b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0bbe9b640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fd098078b40 0x7fd09807b000 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0033000 tx=0x7fd0b0004730 comp rx=0 tx=0).stop 2024-09-10T20:46:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b40ce580 msgr2=0x7fd0b4156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0bbe9b640 1 --2- 172.21.15.28:0/2366869636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b40ce580 0x7fd0b4156740 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c920 tx=0x7fd0a400cde0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.210+0000 7fd0ba438640 1 -- 172.21.15.28:0/2366869636 reap_dead start 2024-09-10T20:46:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.211+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 shutdown_connections 2024-09-10T20:46:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.211+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 >> 172.21.15.28:0/2366869636 conn(0x7fd0b40d1e40 msgr2=0x7fd0b40d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.211+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 shutdown_connections 2024-09-10T20:46:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.211+0000 7fd0bbe9b640 1 -- 172.21.15.28:0/2366869636 wait complete. 2024-09-10T20:46:31.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-10T20:46:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.329+0000 7f1e60f75640 1 -- 172.21.15.28:0/2953368407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e48003fd0 msgr2=0x7f1e48004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.329+0000 7f1e60f75640 1 --2- 172.21.15.28:0/2953368407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e48003fd0 0x7f1e48004800 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009b80 tx=0x7f1e48005830 comp rx=0 tx=0).stop 2024-09-10T20:46:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.329+0000 7f1e5ad9d640 1 -- 172.21.15.28:0/2953368407 reap_dead start 2024-09-10T20:46:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.329+0000 7f1e60f75640 1 -- 172.21.15.28:0/2953368407 shutdown_connections 2024-09-10T20:46:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.329+0000 7f1e60f75640 1 -- 172.21.15.28:0/2953368407 >> 172.21.15.28:0/2953368407 conn(0x7f1e5c0d3fb0 msgr2=0x7f1e5c0d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.329+0000 7f1e60f75640 1 -- 172.21.15.28:0/2953368407 shutdown_connections 2024-09-10T20:46:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.329+0000 7f1e60f75640 1 -- 172.21.15.28:0/2953368407 wait complete. 2024-09-10T20:46:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.330+0000 7f1e60f75640 1 Processor -- start 2024-09-10T20:46:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.330+0000 7f1e60f75640 1 -- start start 2024-09-10T20:46:31.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e60f75640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e48003fd0 0x7f1e5c1f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e60f75640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e5c1f50b0 0x7f1e5c15a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e60f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c09ddb0 0x7f1e5c09e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e60f75640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c15e3d0 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e60f75640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c15e250 con 0x7f1e48003fd0 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e60f75640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f1e5c161b50 con 0x7f1e5c1f50b0 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e5ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c09ddb0 0x7f1e5c09e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e5ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c09ddb0 0x7f1e5c09e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38842/0 (socket says 172.21.15.28:38842) 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e5ad9d640 1 -- 172.21.15.28:0/902390851 learned_addr learned my addr 172.21.15.28:0/902390851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e5a59c640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e48003fd0 0x7f1e5c1f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.331+0000 7f1e59d9b640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e5c1f50b0 0x7f1e5c15a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e5ad9d640 1 -- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e5c1f50b0 msgr2=0x7f1e5c15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e5ad9d640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e5c1f50b0 0x7f1e5c15a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e5ad9d640 1 -- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e48003fd0 msgr2=0x7f1e5c1f86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e5ad9d640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e48003fd0 0x7f1e5c1f86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e5ad9d640 1 -- 172.21.15.28:0/902390851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c1f7c20 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e59d9b640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e5c1f50b0 0x7f1e5c15a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e5ad9d640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c09ddb0 0x7f1e5c09e230 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0d04a0 tx=0x7f1e54011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e52ffd640 1 -- 172.21.15.28:0/902390851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5401e070 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e52ffd640 1 -- 172.21.15.28:0/902390851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e5400fcf0 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c0cbd50 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e52ffd640 1 -- 172.21.15.28:0/902390851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54019690 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.332+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c1f7ec0 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.333+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c0cf780 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.334+0000 7f1e52ffd640 1 -- 172.21.15.28:0/902390851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f1e540223b0 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.334+0000 7f1e52ffd640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e3c078b70 0x7f1e3c07b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.335+0000 7f1e5a59c640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e3c078b70 0x7f1e3c07b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.335+0000 7f1e52ffd640 1 -- 172.21.15.28:0/902390851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f1e54099620 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.335+0000 7f1e5a59c640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e3c078b70 0x7f1e3c07b030 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0cb990 tx=0x7f1e4c0018b0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.339+0000 7f1e52ffd640 1 -- 172.21.15.28:0/902390851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1e54063970 con 0x7f1e5c09ddb0 2024-09-10T20:46:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.521+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f1e5c0ce590 con 0x7f1e3c078b70 2024-09-10T20:46:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.522+0000 7f1e52ffd640 1 -- 172.21.15.28:0/902390851 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1e5c0ce590 con 0x7f1e3c078b70 2024-09-10T20:46:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e3c078b70 msgr2=0x7f1e3c07b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e3c078b70 0x7f1e3c07b030 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0cb990 tx=0x7f1e4c0018b0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c09ddb0 msgr2=0x7f1e5c09e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c09ddb0 0x7f1e5c09e230 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0d04a0 tx=0x7f1e54011b10 comp rx=0 tx=0).stop 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 shutdown_connections 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1e3c078b70 0x7f1e3c07b030 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c09ddb0 0x7f1e5c09e230 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1e5c1f50b0 0x7f1e5c15a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 --2- 172.21.15.28:0/902390851 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1e48003fd0 0x7f1e5c1f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 >> 172.21.15.28:0/902390851 conn(0x7f1e5c0d3fb0 msgr2=0x7f1e5c0b7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 shutdown_connections 2024-09-10T20:46:31.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.524+0000 7f1e60f75640 1 -- 172.21.15.28:0/902390851 wait complete. 2024-09-10T20:46:31.538 INFO:tasks.ceph:Scrubbing osd.1 2024-09-10T20:46:31.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 -- 172.21.15.28:0/1562478707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc7c0702e0 msgr2=0x7fcc7c0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 --2- 172.21.15.28:0/1562478707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc7c0702e0 0x7fcc7c0cda70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c06f7b0 tx=0x7fcc7402aa30 comp rx=0 tx=0).stop 2024-09-10T20:46:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 -- 172.21.15.28:0/1562478707 shutdown_connections 2024-09-10T20:46:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 --2- 172.21.15.28:0/1562478707 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7c0cabc0 0x7fcc7c06fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 --2- 172.21.15.28:0/1562478707 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc7c0702e0 0x7fcc7c0cda70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 -- 172.21.15.28:0/1562478707 >> 172.21.15.28:0/1562478707 conn(0x7fcc7c0cfc80 msgr2=0x7fcc7c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 -- 172.21.15.28:0/1562478707 shutdown_connections 2024-09-10T20:46:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.645+0000 7fcc81f5d640 1 -- 172.21.15.28:0/1562478707 wait complete. 2024-09-10T20:46:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.647+0000 7fcc81f5d640 1 Processor -- start 2024-09-10T20:46:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.647+0000 7fcc81f5d640 1 -- start start 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc81f5d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc7c0cabc0 0x7fcc7c153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc81f5d640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7c15a350 0x7fcc7c149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc81f5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c0c5600 0x7fcc7c15c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc81f5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c06e510 con 0x7fcc7c0c5600 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc81f5d640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c06e390 con 0x7fcc7c0cabc0 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc81f5d640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7c06e690 con 0x7fcc7c15a350 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c0c5600 0x7fcc7c15c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c0c5600 0x7fcc7c15c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38858/0 (socket says 172.21.15.28:38858) 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 -- 172.21.15.28:0/627026553 learned_addr learned my addr 172.21.15.28:0/627026553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7affd640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7c15a350 0x7fcc7c149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7b7fe640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc7c0cabc0 0x7fcc7c153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 -- 172.21.15.28:0/627026553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7c15a350 msgr2=0x7fcc7c149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcc7c15a350 0x7fcc7c149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 -- 172.21.15.28:0/627026553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc7c0cabc0 msgr2=0x7fcc7c153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcc7c0cabc0 0x7fcc7c153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 -- 172.21.15.28:0/627026553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c14b0e0 con 0x7fcc7c0c5600 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.648+0000 7fcc7bfff640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c0c5600 0x7fcc7c15c7d0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009d90 tx=0x7fcc74003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.649+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74047070 con 0x7fcc7c0c5600 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.649+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7c14b940 con 0x7fcc7c0c5600 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.649+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc7402f650 con 0x7fcc7c0c5600 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.649+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74033050 con 0x7fcc7c0c5600 2024-09-10T20:46:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.649+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7c14b660 con 0x7fcc7c0c5600 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.649+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc7c15ac50 con 0x7fcc7c0c5600 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.650+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fcc740044b0 con 0x7fcc7c0c5600 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc63fff640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcc68078b40 0x7fcc6807b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fcc740b9cd0 con 0x7fcc7c0c5600 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc7b7fe640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcc68078b40 0x7fcc6807b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc63fff640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7fcc6807ea00 0x7fcc68080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 --> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc680814f0 con 0x7fcc6807ea00 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7fcc740bca50 con 0x7fcc7c0c5600 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc7affd640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7fcc6807ea00 0x7fcc68080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc7b7fe640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcc68078b40 0x7fcc6807b000 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0c36a0 tx=0x7fcc6c009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.651+0000 7fcc7affd640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7fcc6807ea00 0x7fcc68080e20 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.652+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== osd.1 v2:172.21.15.28:6800/1047588896 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc680814f0 con 0x7fcc6807ea00 2024-09-10T20:46:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 --> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcc7c0ce7d0 con 0x7fcc6807ea00 2024-09-10T20:46:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc63fff640 1 -- 172.21.15.28:0/627026553 <== osd.1 v2:172.21.15.28:6800/1047588896 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcc7c0ce7d0 con 0x7fcc6807ea00 2024-09-10T20:46:31.669 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:31.669 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:31.669 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7fcc6807ea00 msgr2=0x7fcc68080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6800/1047588896,v1:172.21.15.28:6801/1047588896] conn(0x7fcc6807ea00 0x7fcc68080e20 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcc68078b40 msgr2=0x7fcc6807b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fcc68078b40 0x7fcc6807b000 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0c36a0 tx=0x7fcc6c009040 comp rx=0 tx=0).stop 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c0c5600 msgr2=0x7fcc7c15c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 --2- 172.21.15.28:0/627026553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c0c5600 0x7fcc7c15c7d0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009d90 tx=0x7fcc74003ea0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc7bfff640 1 -- 172.21.15.28:0/627026553 reap_dead start 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 shutdown_connections 2024-09-10T20:46:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 >> 172.21.15.28:0/627026553 conn(0x7fcc7c0cfc80 msgr2=0x7fcc7c0af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 shutdown_connections 2024-09-10T20:46:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.665+0000 7fcc81f5d640 1 -- 172.21.15.28:0/627026553 wait complete. 2024-09-10T20:46:31.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-10T20:46:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.783+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3936366401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80051c0 msgr2=0x7f7cb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.783+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3936366401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80051c0 0x7f7cb800b9c0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800c640 tx=0x7f7cb8005830 comp rx=0 tx=0).stop 2024-09-10T20:46:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.783+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3936366401 shutdown_connections 2024-09-10T20:46:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.783+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3936366401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80051c0 0x7f7cb800b9c0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.783+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3936366401 >> 172.21.15.28:0/3936366401 conn(0x7f7cd006cf00 msgr2=0x7f7cd006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.783+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3936366401 shutdown_connections 2024-09-10T20:46:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.783+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3936366401 wait complete. 2024-09-10T20:46:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.784+0000 7f7cd5ba3640 1 Processor -- start 2024-09-10T20:46:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.784+0000 7f7cd5ba3640 1 -- start start 2024-09-10T20:46:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cd5ba3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cb80051c0 0x7f7cd0154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cd5ba3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd0148d30 0x7f7cd00b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cd5ba3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd01519c0 0x7f7cd0151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cd5ba3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd00b4780 con 0x7f7cd0148d30 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cd5ba3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd00b4600 con 0x7f7cd01519c0 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cd5ba3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd00b4900 con 0x7f7cb80051c0 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd0148d30 0x7f7cd00b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd0148d30 0x7f7cd00b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38878/0 (socket says 172.21.15.28:38878) 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 -- 172.21.15.28:0/3565823136 learned_addr learned my addr 172.21.15.28:0/3565823136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7ccf7fe640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cb80051c0 0x7f7cd0154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7ccffff640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd01519c0 0x7f7cd0151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 -- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cb80051c0 msgr2=0x7f7cd0154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cb80051c0 0x7f7cd0154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 -- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd01519c0 msgr2=0x7f7cd0151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd01519c0 0x7f7cd0151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.785+0000 7f7cceffd640 1 -- 172.21.15.28:0/3565823136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd0154600 con 0x7f7cd0148d30 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.786+0000 7f7cceffd640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd0148d30 0x7f7cd00b13e0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4004ed0 tx=0x7f7cc4001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.786+0000 7f7cb7fff640 1 -- 172.21.15.28:0/3565823136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4015020 con 0x7f7cd0148d30 2024-09-10T20:46:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.786+0000 7f7cb7fff640 1 -- 172.21.15.28:0/3565823136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cc4011960 con 0x7f7cd0148d30 2024-09-10T20:46:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.786+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd0154040 con 0x7f7cd0148d30 2024-09-10T20:46:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.786+0000 7f7cb7fff640 1 -- 172.21.15.28:0/3565823136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4011d90 con 0x7f7cd0148d30 2024-09-10T20:46:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.786+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd00b1920 con 0x7f7cd0148d30 2024-09-10T20:46:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.787+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd00b4e10 con 0x7f7cd0148d30 2024-09-10T20:46:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.787+0000 7f7cb7fff640 1 -- 172.21.15.28:0/3565823136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f7cc401a400 con 0x7f7cd0148d30 2024-09-10T20:46:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.788+0000 7f7cb7fff640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cbc078b70 0x7f7cbc07b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.788+0000 7f7cb7fff640 1 -- 172.21.15.28:0/3565823136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f7cc4099540 con 0x7f7cd0148d30 2024-09-10T20:46:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.788+0000 7f7ccf7fe640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cbc078b70 0x7f7cbc07b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.788+0000 7f7ccf7fe640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cbc078b70 0x7f7cbc07b030 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009bd0 tx=0x7f7cc0005f90 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.792+0000 7f7cb7fff640 1 -- 172.21.15.28:0/3565823136 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7cc401f020 con 0x7f7cd0148d30 2024-09-10T20:46:31.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.976+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f7cd0154fc0 con 0x7f7cbc078b70 2024-09-10T20:46:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.976+0000 7f7cb7fff640 1 -- 172.21.15.28:0/3565823136 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7cd0154fc0 con 0x7f7cbc078b70 2024-09-10T20:46:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cbc078b70 msgr2=0x7f7cbc07b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cbc078b70 0x7f7cbc07b030 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009bd0 tx=0x7f7cc0005f90 comp rx=0 tx=0).stop 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd0148d30 msgr2=0x7f7cd00b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd0148d30 0x7f7cd00b13e0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4004ed0 tx=0x7f7cc4001630 comp rx=0 tx=0).stop 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 shutdown_connections 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7cbc078b70 0x7f7cbc07b030 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7cd01519c0 0x7f7cd0151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd0148d30 0x7f7cd00b13e0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 --2- 172.21.15.28:0/3565823136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7cb80051c0 0x7f7cd0154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 >> 172.21.15.28:0/3565823136 conn(0x7f7cd006cf00 msgr2=0x7f7cd015b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 shutdown_connections 2024-09-10T20:46:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:31.979+0000 7f7cd5ba3640 1 -- 172.21.15.28:0/3565823136 wait complete. 2024-09-10T20:46:31.994 INFO:tasks.ceph:Scrubbing osd.2 2024-09-10T20:46:31.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.099+0000 7fe71092b640 1 -- 172.21.15.28:0/3598316023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c151cf0 msgr2=0x7fe70c15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.099+0000 7fe71092b640 1 --2- 172.21.15.28:0/3598316023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c151cf0 0x7fe70c15bd70 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009830 tx=0x7fe6fc02a510 comp rx=0 tx=0).stop 2024-09-10T20:46:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.099+0000 7fe71092b640 1 -- 172.21.15.28:0/3598316023 shutdown_connections 2024-09-10T20:46:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.099+0000 7fe71092b640 1 --2- 172.21.15.28:0/3598316023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c151cf0 0x7fe70c15bd70 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.099+0000 7fe71092b640 1 -- 172.21.15.28:0/3598316023 >> 172.21.15.28:0/3598316023 conn(0x7fe70c0cfc80 msgr2=0x7fe70c0d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.099+0000 7fe71092b640 1 -- 172.21.15.28:0/3598316023 shutdown_connections 2024-09-10T20:46:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.099+0000 7fe71092b640 1 -- 172.21.15.28:0/3598316023 wait complete. 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.100+0000 7fe71092b640 1 Processor -- start 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe71092b640 1 -- start start 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe71092b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c070440 0x7fe70c14d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe71092b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe70c072db0 0x7fe70c1f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe71092b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe70c151cf0 0x7fe70c0c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe71092b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c0cb140 con 0x7fe70c070440 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe71092b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c0cafc0 con 0x7fe70c151cf0 2024-09-10T20:46:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe71092b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fe70c0cb2c0 con 0x7fe70c072db0 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c070440 0x7fe70c14d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c070440 0x7fe70c14d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38904/0 (socket says 172.21.15.28:38904) 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.101+0000 7fe70a59c640 1 -- 172.21.15.28:0/874573351 learned_addr learned my addr 172.21.15.28:0/874573351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe70a59c640 1 -- 172.21.15.28:0/874573351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe70c072db0 msgr2=0x7fe70c1f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe70a59c640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe70c072db0 0x7fe70c1f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe70a59c640 1 -- 172.21.15.28:0/874573351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe70c151cf0 msgr2=0x7fe70c0c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe70a59c640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe70c151cf0 0x7fe70c0c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe70a59c640 1 -- 172.21.15.28:0/874573351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70c14ff30 con 0x7fe70c070440 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe70a59c640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c070440 0x7fe70c14d100 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc000c00 tx=0x7fe6fc033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc031d60 con 0x7fe70c070440 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6fc0330d0 con 0x7fe70c070440 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70c150210 con 0x7fe70c070440 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc033520 con 0x7fe70c070440 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.102+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe70c152490 con 0x7fe70c070440 2024-09-10T20:46:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.103+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe70c1f2920 con 0x7fe70c070440 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fe6fc04b030 con 0x7fe70c070440 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe6faffd640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6ec078b40 0x7fe6ec07b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fe6fc0ba440 con 0x7fe70c070440 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe709d9b640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6ec078b40 0x7fe6ec07b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe6faffd640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7fe6ec07ea00 0x7fe6ec080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 --> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6fc001d30 con 0x7fe6ec07ea00 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7fe6fc0bd1c0 con 0x7fe70c070440 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.104+0000 7fe70ad9d640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7fe6ec07ea00 0x7fe6ec080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.105+0000 7fe709d9b640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6ec078b40 0x7fe6ec07b000 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe70c0ca030 tx=0x7fe700007a30 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.105+0000 7fe70ad9d640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7fe6ec07ea00 0x7fe6ec080e20 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.105+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== osd.2 v2:172.21.15.28:6816/4001706199 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe6fc001d30 con 0x7fe6ec07ea00 2024-09-10T20:46:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.118+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 --> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe70c14d980 con 0x7fe6ec07ea00 2024-09-10T20:46:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.118+0000 7fe6faffd640 1 -- 172.21.15.28:0/874573351 <== osd.2 v2:172.21.15.28:6816/4001706199 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe70c14d980 con 0x7fe6ec07ea00 2024-09-10T20:46:32.122 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:32.122 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:32.122 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.118+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7fe6ec07ea00 msgr2=0x7fe6ec080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.118+0000 7fe71092b640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6816/4001706199,v1:172.21.15.28:6817/4001706199] conn(0x7fe6ec07ea00 0x7fe6ec080e20 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6ec078b40 msgr2=0x7fe6ec07b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fe6ec078b40 0x7fe6ec07b000 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe70c0ca030 tx=0x7fe700007a30 comp rx=0 tx=0).stop 2024-09-10T20:46:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c070440 msgr2=0x7fe70c14d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 --2- 172.21.15.28:0/874573351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe70c070440 0x7fe70c14d100 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc000c00 tx=0x7fe6fc033d60 comp rx=0 tx=0).stop 2024-09-10T20:46:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe70ad9d640 1 -- 172.21.15.28:0/874573351 reap_dead start 2024-09-10T20:46:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 shutdown_connections 2024-09-10T20:46:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 >> 172.21.15.28:0/874573351 conn(0x7fe70c0cfc80 msgr2=0x7fe70c0d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 shutdown_connections 2024-09-10T20:46:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.119+0000 7fe71092b640 1 -- 172.21.15.28:0/874573351 wait complete. 2024-09-10T20:46:32.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-10T20:46:32.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.235+0000 7f26c8f64640 1 -- 172.21.15.28:0/3940651553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ac003f70 msgr2=0x7f26ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.235+0000 7f26c8f64640 1 --2- 172.21.15.28:0/3940651553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ac003f70 0x7f26ac004800 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c260 tx=0x7f26ac005830 comp rx=0 tx=0).stop 2024-09-10T20:46:32.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.235+0000 7f26c2d9d640 1 -- 172.21.15.28:0/3940651553 reap_dead start 2024-09-10T20:46:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.235+0000 7f26c8f64640 1 -- 172.21.15.28:0/3940651553 shutdown_connections 2024-09-10T20:46:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.235+0000 7f26c8f64640 1 -- 172.21.15.28:0/3940651553 >> 172.21.15.28:0/3940651553 conn(0x7f26c40d1e40 msgr2=0x7f26c40d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.235+0000 7f26c8f64640 1 -- 172.21.15.28:0/3940651553 shutdown_connections 2024-09-10T20:46:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.235+0000 7f26c8f64640 1 -- 172.21.15.28:0/3940651553 wait complete. 2024-09-10T20:46:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 Processor -- start 2024-09-10T20:46:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 -- start start 2024-09-10T20:46:32.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26ac003f70 0x7f26c415f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26c414a360 0x7f26c415daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26c40cb0e0 0x7f26c40cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26c40cd5c0 con 0x7f26c414a360 2024-09-10T20:46:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f26c40d0d50 con 0x7f26c40cb0e0 2024-09-10T20:46:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.237+0000 7f26c8f64640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f26c40b5d20 con 0x7f26ac003f70 2024-09-10T20:46:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26c414a360 0x7f26c415daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26c414a360 0x7f26c415daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38930/0 (socket says 172.21.15.28:38930) 2024-09-10T20:46:32.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 -- 172.21.15.28:0/683072626 learned_addr learned my addr 172.21.15.28:0/683072626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:32.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 -- 172.21.15.28:0/683072626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26ac003f70 msgr2=0x7f26c415f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26ac003f70 0x7f26c415f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 -- 172.21.15.28:0/683072626 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26c40cb0e0 msgr2=0x7f26c40cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26c40cb0e0 0x7f26c40cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 -- 172.21.15.28:0/683072626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c415ca50 con 0x7f26c414a360 2024-09-10T20:46:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c1d9b640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26c414a360 0x7f26c415daf0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f26b802b0d0 tx=0x7f26b80011c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26b2ffd640 1 -- 172.21.15.28:0/683072626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b8031040 con 0x7f26c414a360 2024-09-10T20:46:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26b2ffd640 1 -- 172.21.15.28:0/683072626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26b80035f0 con 0x7f26c414a360 2024-09-10T20:46:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c415cd30 con 0x7f26c414a360 2024-09-10T20:46:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26b2ffd640 1 -- 172.21.15.28:0/683072626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b8003a20 con 0x7f26c414a360 2024-09-10T20:46:32.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.238+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c415d2f0 con 0x7f26c414a360 2024-09-10T20:46:32.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.239+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c40b5610 con 0x7f26c414a360 2024-09-10T20:46:32.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.240+0000 7f26b2ffd640 1 -- 172.21.15.28:0/683072626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f26b8048030 con 0x7f26c414a360 2024-09-10T20:46:32.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.240+0000 7f26b2ffd640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f26a4078b40 0x7f26a407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.240+0000 7f26b2ffd640 1 -- 172.21.15.28:0/683072626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f26b80b8400 con 0x7f26c414a360 2024-09-10T20:46:32.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.240+0000 7f26c259c640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f26a4078b40 0x7f26a407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.241+0000 7f26c259c640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f26a4078b40 0x7f26a407b000 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f26b40067f0 tx=0x7f26b4006780 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.245+0000 7f26b2ffd640 1 -- 172.21.15.28:0/683072626 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f26b80829b0 con 0x7f26c414a360 2024-09-10T20:46:32.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.430+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f26c406cf90 con 0x7f26a4078b40 2024-09-10T20:46:32.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.430+0000 7f26b2ffd640 1 -- 172.21.15.28:0/683072626 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f26c406cf90 con 0x7f26a4078b40 2024-09-10T20:46:32.434 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f26a4078b40 msgr2=0x7f26a407b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f26a4078b40 0x7f26a407b000 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f26b40067f0 tx=0x7f26b4006780 comp rx=0 tx=0).stop 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26c414a360 msgr2=0x7f26c415daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26c414a360 0x7f26c415daf0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f26b802b0d0 tx=0x7f26b80011c0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 shutdown_connections 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f26a4078b40 0x7f26a407b000 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f26c40cb0e0 0x7f26c40cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26c414a360 0x7f26c415daf0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 --2- 172.21.15.28:0/683072626 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f26ac003f70 0x7f26c415f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 >> 172.21.15.28:0/683072626 conn(0x7f26c40d1e40 msgr2=0x7f26c40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 shutdown_connections 2024-09-10T20:46:32.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.433+0000 7f26c8f64640 1 -- 172.21.15.28:0/683072626 wait complete. 2024-09-10T20:46:32.448 INFO:tasks.ceph:Scrubbing osd.3 2024-09-10T20:46:32.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:32.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 -- 172.21.15.28:0/3219129027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 msgr2=0x7f18f80d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 --2- 172.21.15.28:0/3219129027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 0x7f18f80d4a80 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f18f808d390 tx=0x7f18ec02a5f0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 -- 172.21.15.28:0/3219129027 shutdown_connections 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 --2- 172.21.15.28:0/3219129027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18f81623e0 0x7f18f8161680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 --2- 172.21.15.28:0/3219129027 >> v2:172.21.15.159:3300/0 conn(0x7f18f80b4c20 0x7f18f8161140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 --2- 172.21.15.28:0/3219129027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 0x7f18f80d4a80 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 -- 172.21.15.28:0/3219129027 >> 172.21.15.28:0/3219129027 conn(0x7f18f80d3f90 msgr2=0x7f18f80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 -- 172.21.15.28:0/3219129027 shutdown_connections 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.553+0000 7f18fedf8640 1 -- 172.21.15.28:0/3219129027 wait complete. 2024-09-10T20:46:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.554+0000 7f18fedf8640 1 Processor -- start 2024-09-10T20:46:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.555+0000 7f18fedf8640 1 -- start start 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.555+0000 7f18fedf8640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18f80b4c20 0x7f18f815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.555+0000 7f18fedf8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 0x7f18f81606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.555+0000 7f18fedf8640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18f81623e0 0x7f18f80cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.555+0000 7f18fedf8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18f80d0ae0 con 0x7f18f8162000 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.555+0000 7f18fedf8640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f18f80d0960 con 0x7f18f80b4c20 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.555+0000 7f18fedf8640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f18f80d0c60 con 0x7f18f81623e0 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 0x7f18f81606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 0x7f18f81606d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38948/0 (socket says 172.21.15.28:38948) 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 -- 172.21.15.28:0/418560580 learned_addr learned my addr 172.21.15.28:0/418560580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18fcb94640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18f80b4c20 0x7f18f815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18fd395640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18f81623e0 0x7f18f80cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 -- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18f81623e0 msgr2=0x7f18f80cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18f81623e0 0x7f18f80cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 -- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18f80b4c20 msgr2=0x7f18f815dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18f80b4c20 0x7f18f815dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 -- 172.21.15.28:0/418560580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f80ccb60 con 0x7f18f8162000 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18fcb94640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18f80b4c20 0x7f18f815dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18fd395640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18f81623e0 0x7f18f80cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f7fff640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 0x7f18f81606d0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f18e400b4d0 tx=0x7f18e400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e4014070 con 0x7f18f8162000 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18e4010960 con 0x7f18f8162000 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f80cce40 con 0x7f18f8162000 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e4010db0 con 0x7f18f8162000 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.556+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f80cc860 con 0x7f18f8162000 2024-09-10T20:46:32.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.557+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f18f8160c10 con 0x7f18f8162000 2024-09-10T20:46:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.558+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f18e4023440 con 0x7f18f8162000 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18f57fa640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18dc078b40 0x7f18dc07b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18fcb94640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18dc078b40 0x7f18dc07b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f18e4098d90 con 0x7f18f8162000 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18f57fa640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7f18dc07ea00 0x7f18dc080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 --> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18e4017e50 con 0x7f18dc07ea00 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7f18e409b670 con 0x7f18f8162000 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18fd395640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7f18dc07ea00 0x7f18dc080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18fcb94640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18dc078b40 0x7f18dc07b000 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f18ec033000 tx=0x7f18ec004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.559+0000 7f18fd395640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7f18dc07ea00 0x7f18dc080e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.560+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== osd.3 v2:172.21.15.28:6824/1557654590 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f18e4017e50 con 0x7f18dc07ea00 2024-09-10T20:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 --> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f18f81ef350 con 0x7f18dc07ea00 2024-09-10T20:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18f57fa640 1 -- 172.21.15.28:0/418560580 <== osd.3 v2:172.21.15.28:6824/1557654590 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f18f81ef350 con 0x7f18dc07ea00 2024-09-10T20:46:32.577 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:32.578 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:32.578 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7f18dc07ea00 msgr2=0x7f18dc080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6824/1557654590,v1:172.21.15.28:6825/1557654590] conn(0x7f18dc07ea00 0x7f18dc080e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18dc078b40 msgr2=0x7f18dc07b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f18dc078b40 0x7f18dc07b000 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f18ec033000 tx=0x7f18ec004730 comp rx=0 tx=0).stop 2024-09-10T20:46:32.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 msgr2=0x7f18f81606d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 --2- 172.21.15.28:0/418560580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f8162000 0x7f18f81606d0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f18e400b4d0 tx=0x7f18e400b990 comp rx=0 tx=0).stop 2024-09-10T20:46:32.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fd395640 1 -- 172.21.15.28:0/418560580 reap_dead start 2024-09-10T20:46:32.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 shutdown_connections 2024-09-10T20:46:32.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 >> 172.21.15.28:0/418560580 conn(0x7f18f80d3f90 msgr2=0x7f18f8160f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 shutdown_connections 2024-09-10T20:46:32.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.573+0000 7f18fedf8640 1 -- 172.21.15.28:0/418560580 wait complete. 2024-09-10T20:46:32.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-10T20:46:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.691+0000 7ff0216bd640 1 -- 172.21.15.28:0/2038734268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0080051c0 msgr2=0x7ff00800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.691+0000 7ff0216bd640 1 --2- 172.21.15.28:0/2038734268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0080051c0 0x7ff00800b9c0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ff014001dc0 tx=0x7ff008005830 comp rx=0 tx=0).stop 2024-09-10T20:46:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.691+0000 7ff0216bd640 1 -- 172.21.15.28:0/2038734268 shutdown_connections 2024-09-10T20:46:32.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.691+0000 7ff0216bd640 1 --2- 172.21.15.28:0/2038734268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0080051c0 0x7ff00800b9c0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.691+0000 7ff0216bd640 1 -- 172.21.15.28:0/2038734268 >> 172.21.15.28:0/2038734268 conn(0x7ff01c0b1c40 msgr2=0x7ff01c0b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.692+0000 7ff0216bd640 1 -- 172.21.15.28:0/2038734268 shutdown_connections 2024-09-10T20:46:32.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.692+0000 7ff0216bd640 1 -- 172.21.15.28:0/2038734268 wait complete. 2024-09-10T20:46:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 Processor -- start 2024-09-10T20:46:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 -- start start 2024-09-10T20:46:32.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0080051c0 0x7ff01c151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff01c14ab50 0x7ff01c15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff01c1f4990 0x7ff01c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c070840 con 0x7ff01c1f4990 2024-09-10T20:46:32.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c0706c0 con 0x7ff01c14ab50 2024-09-10T20:46:32.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.693+0000 7ff0216bd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7ff01c0709c0 con 0x7ff0080051c0 2024-09-10T20:46:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff01c1f4990 0x7ff01c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff01c1f4990 0x7ff01c0cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38964/0 (socket says 172.21.15.28:38964) 2024-09-10T20:46:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 -- 172.21.15.28:0/1999274157 learned_addr learned my addr 172.21.15.28:0/1999274157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff013fff640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff01c14ab50 0x7ff01c15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01affd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0080051c0 0x7ff01c151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 -- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0080051c0 msgr2=0x7ff01c151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0080051c0 0x7ff01c151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 -- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff01c14ab50 msgr2=0x7ff01c15b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff01c14ab50 0x7ff01c15b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 -- 172.21.15.28:0/1999274157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c14afd0 con 0x7ff01c1f4990 2024-09-10T20:46:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01affd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0080051c0 0x7ff01c151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff01b7fe640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff01c1f4990 0x7ff01c0cf730 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7ff014001eb0 tx=0x7ff014001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff0137fe640 1 -- 172.21.15.28:0/1999274157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014003d80 con 0x7ff01c1f4990 2024-09-10T20:46:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff0137fe640 1 -- 172.21.15.28:0/1999274157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff014032470 con 0x7ff01c1f4990 2024-09-10T20:46:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff0137fe640 1 -- 172.21.15.28:0/1999274157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140328c0 con 0x7ff01c1f4990 2024-09-10T20:46:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01c151aa0 con 0x7ff01c1f4990 2024-09-10T20:46:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.694+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01c151d60 con 0x7ff01c1f4990 2024-09-10T20:46:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.695+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01c073e40 con 0x7ff01c1f4990 2024-09-10T20:46:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.696+0000 7ff0137fe640 1 -- 172.21.15.28:0/1999274157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7ff014046030 con 0x7ff01c1f4990 2024-09-10T20:46:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.697+0000 7ff0137fe640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feffc078b40 0x7feffc07b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.697+0000 7ff01affd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feffc078b40 0x7feffc07b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.697+0000 7ff0137fe640 1 -- 172.21.15.28:0/1999274157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7ff0140b8530 con 0x7ff01c1f4990 2024-09-10T20:46:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.697+0000 7ff01affd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feffc078b40 0x7feffc07b000 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff00c001050 tx=0x7ff00c0067a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.701+0000 7ff0137fe640 1 -- 172.21.15.28:0/1999274157 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff01c14afd0 con 0x7ff01c1f4990 2024-09-10T20:46:32.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:32.873 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.886+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7ff01c158c60 con 0x7feffc078b40 2024-09-10T20:46:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.887+0000 7ff0137fe640 1 -- 172.21.15.28:0/1999274157 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff01c158c60 con 0x7feffc078b40 2024-09-10T20:46:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2024-09-10T20:46:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feffc078b40 msgr2=0x7feffc07b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feffc078b40 0x7feffc07b000 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff00c001050 tx=0x7ff00c0067a0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff01c1f4990 msgr2=0x7ff01c0cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff01c1f4990 0x7ff01c0cf730 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7ff014001eb0 tx=0x7ff014001a00 comp rx=0 tx=0).stop 2024-09-10T20:46:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 shutdown_connections 2024-09-10T20:46:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7feffc078b40 0x7feffc07b000 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff01c1f4990 0x7ff01c0cf730 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff01c14ab50 0x7ff01c15b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 --2- 172.21.15.28:0/1999274157 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0080051c0 0x7ff01c151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 >> 172.21.15.28:0/1999274157 conn(0x7ff01c0b1c40 msgr2=0x7ff01c0b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 shutdown_connections 2024-09-10T20:46:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:32.889+0000 7ff0216bd640 1 -- 172.21.15.28:0/1999274157 wait complete. 2024-09-10T20:46:32.904 INFO:tasks.ceph:Scrubbing osd.4 2024-09-10T20:46:32.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 -- 172.21.15.28:0/2156318079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c00b5a80 msgr2=0x7f12c0073700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 --2- 172.21.15.28:0/2156318079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c00b5a80 0x7f12c0073700 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f12b0005fd0 tx=0x7f12b002a5d0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 -- 172.21.15.28:0/2156318079 shutdown_connections 2024-09-10T20:46:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 --2- 172.21.15.28:0/2156318079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12c0073c40 0x7f12c00add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 --2- 172.21.15.28:0/2156318079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c00b5a80 0x7f12c0073700 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 -- 172.21.15.28:0/2156318079 >> 172.21.15.28:0/2156318079 conn(0x7f12c00b2920 msgr2=0x7f12c00b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 -- 172.21.15.28:0/2156318079 shutdown_connections 2024-09-10T20:46:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.066+0000 7f12c8152640 1 -- 172.21.15.28:0/2156318079 wait complete. 2024-09-10T20:46:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.067+0000 7f12c8152640 1 Processor -- start 2024-09-10T20:46:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.067+0000 7f12c8152640 1 -- start start 2024-09-10T20:46:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c8152640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12c0073c40 0x7f12c0149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c8152640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12c00b5a80 0x7f12c01f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c8152640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c015ab00 0x7f12c01594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c8152640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12c015aee0 con 0x7f12c015ab00 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c8152640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f12c00b4630 con 0x7f12c0073c40 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c8152640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f12c015b060 con 0x7f12c00b5a80 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c015ab00 0x7f12c01594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c015ab00 0x7f12c01594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38986/0 (socket says 172.21.15.28:38986) 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 -- 172.21.15.28:0/3161105265 learned_addr learned my addr 172.21.15.28:0/3161105265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 -- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12c00b5a80 msgr2=0x7f12c01f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c5eee640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12c0073c40 0x7f12c0149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c56ed640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12c00b5a80 0x7f12c01f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f12c00b5a80 0x7f12c01f1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 -- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12c0073c40 msgr2=0x7f12c0149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f12c0073c40 0x7f12c0149100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 -- 172.21.15.28:0/3161105265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c0149790 con 0x7f12c015ab00 2024-09-10T20:46:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12c66ef640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c015ab00 0x7f12c01594f0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f12b800cad0 tx=0x7f12b800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.068+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b800ae20 con 0x7f12c015ab00 2024-09-10T20:46:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.069+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12b8004b20 con 0x7f12c015ab00 2024-09-10T20:46:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.069+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8005020 con 0x7f12c015ab00 2024-09-10T20:46:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.069+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c0149a70 con 0x7f12c015ab00 2024-09-10T20:46:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.069+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c01f4a20 con 0x7f12c015ab00 2024-09-10T20:46:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.069+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f12c00ae510 con 0x7f12c015ab00 2024-09-10T20:46:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.070+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f12b80051c0 con 0x7f12c015ab00 2024-09-10T20:46:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12b67fc640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12a4078b40 0x7f12a407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f12b80994c0 con 0x7f12c015ab00 2024-09-10T20:46:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12c5eee640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12a4078b40 0x7f12a407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12b67fc640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f12a407ea00 0x7f12a4080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 --> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12a40814f0 con 0x7f12a407ea00 2024-09-10T20:46:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7f12b809bd20 con 0x7f12c015ab00 2024-09-10T20:46:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12c5eee640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12a4078b40 0x7f12a407b000 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12bc000f60 tx=0x7f12bc009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.071+0000 7f12c56ed640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f12a407ea00 0x7f12a4080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.072+0000 7f12c56ed640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f12a407ea00 0x7f12a4080e20 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.072+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== osd.4 v2:172.21.15.159:6802/1548693517 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f12a40814f0 con 0x7f12a407ea00 2024-09-10T20:46:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.085+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 --> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f12c009eea0 con 0x7f12a407ea00 2024-09-10T20:46:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.085+0000 7f12b67fc640 1 -- 172.21.15.28:0/3161105265 <== osd.4 v2:172.21.15.159:6802/1548693517 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f12c009eea0 con 0x7f12a407ea00 2024-09-10T20:46:33.090 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:33.090 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:33.090 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f12a407ea00 msgr2=0x7f12a4080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.159:6802/1548693517,v1:172.21.15.159:6803/1548693517] conn(0x7f12a407ea00 0x7f12a4080e20 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12a4078b40 msgr2=0x7f12a407b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f12a4078b40 0x7f12a407b000 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12bc000f60 tx=0x7f12bc009040 comp rx=0 tx=0).stop 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c015ab00 msgr2=0x7f12c01594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 --2- 172.21.15.28:0/3161105265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12c015ab00 0x7f12c01594f0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f12b800cad0 tx=0x7f12b800cf90 comp rx=0 tx=0).stop 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c66ef640 1 -- 172.21.15.28:0/3161105265 reap_dead start 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 shutdown_connections 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 >> 172.21.15.28:0/3161105265 conn(0x7f12c00b2920 msgr2=0x7f12c015b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 shutdown_connections 2024-09-10T20:46:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.086+0000 7f12c8152640 1 -- 172.21.15.28:0/3161105265 wait complete. 2024-09-10T20:46:33.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-10T20:46:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 -- 172.21.15.28:0/3259698166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce58161b50 msgr2=0x7fce58161fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/3259698166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce58161b50 0x7fce58161fc0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fce5808c020 tx=0x7fce5002aa30 comp rx=0 tx=0).stop 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 -- 172.21.15.28:0/3259698166 shutdown_connections 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/3259698166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce58161b50 0x7fce58161fc0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/3259698166 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce580cf780 0x7fce580cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/3259698166 >> v2:172.21.15.159:3300/0 conn(0x7fce5815de70 0x7fce580d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 -- 172.21.15.28:0/3259698166 >> 172.21.15.28:0/3259698166 conn(0x7fce580d3fb0 msgr2=0x7fce580d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 -- 172.21.15.28:0/3259698166 shutdown_connections 2024-09-10T20:46:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.201+0000 7fce5c8a2640 1 -- 172.21.15.28:0/3259698166 wait complete. 2024-09-10T20:46:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.203+0000 7fce5c8a2640 1 Processor -- start 2024-09-10T20:46:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.203+0000 7fce5c8a2640 1 -- start start 2024-09-10T20:46:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.203+0000 7fce5c8a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce580cf780 0x7fce58158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.203+0000 7fce5c8a2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce5815de70 0x7fce5814f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5c8a2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce58161b50 0x7fce5815ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5c8a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce580b6e00 con 0x7fce580cf780 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5c8a2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fce5815e460 con 0x7fce5815de70 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5c8a2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fce580b6f80 con 0x7fce58161b50 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce580cf780 0x7fce58158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce580cf780 0x7fce58158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39004/0 (socket says 172.21.15.28:39004) 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 -- 172.21.15.28:0/1384931696 learned_addr learned my addr 172.21.15.28:0/1384931696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce56d9d640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce58161b50 0x7fce5815ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce4ffff640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce5815de70 0x7fce5814f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 -- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce58161b50 msgr2=0x7fce5815ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce58161b50 0x7fce5815ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 -- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce5815de70 msgr2=0x7fce5814f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce5815de70 0x7fce5814f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 -- 172.21.15.28:0/1384931696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce58161780 con 0x7fce580cf780 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce4ffff640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce5815de70 0x7fce5814f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5659c640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce580cf780 0x7fce58158310 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fce4800b4a0 tx=0x7fce4800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce4effd640 1 -- 172.21.15.28:0/1384931696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48014070 con 0x7fce580cf780 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce4effd640 1 -- 172.21.15.28:0/1384931696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce48010700 con 0x7fce580cf780 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce58159130 con 0x7fce580cf780 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce4effd640 1 -- 172.21.15.28:0/1384931696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48010b50 con 0x7fce580cf780 2024-09-10T20:46:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.204+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce581614a0 con 0x7fce580cf780 2024-09-10T20:46:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.205+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce580d2b90 con 0x7fce580cf780 2024-09-10T20:46:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.206+0000 7fce4effd640 1 -- 172.21.15.28:0/1384931696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fce48017360 con 0x7fce580cf780 2024-09-10T20:46:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.207+0000 7fce4effd640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fce38078b70 0x7fce3807b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.207+0000 7fce4ffff640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fce38078b70 0x7fce3807b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.207+0000 7fce4effd640 1 -- 172.21.15.28:0/1384931696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fce48098680 con 0x7fce580cf780 2024-09-10T20:46:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.207+0000 7fce4ffff640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fce38078b70 0x7fce3807b030 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fce580b86f0 tx=0x7fce40008040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.211+0000 7fce4effd640 1 -- 172.21.15.28:0/1384931696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fce480629d0 con 0x7fce580cf780 2024-09-10T20:46:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.397+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fce580b9f30 con 0x7fce38078b70 2024-09-10T20:46:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.397+0000 7fce4effd640 1 -- 172.21.15.28:0/1384931696 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fce580b9f30 con 0x7fce38078b70 2024-09-10T20:46:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.399+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fce38078b70 msgr2=0x7fce3807b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.399+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fce38078b70 0x7fce3807b030 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fce580b86f0 tx=0x7fce40008040 comp rx=0 tx=0).stop 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.399+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce580cf780 msgr2=0x7fce58158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.399+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce580cf780 0x7fce58158310 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fce4800b4a0 tx=0x7fce4800b960 comp rx=0 tx=0).stop 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 shutdown_connections 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fce38078b70 0x7fce3807b030 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fce58161b50 0x7fce5815ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fce5815de70 0x7fce5814f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 --2- 172.21.15.28:0/1384931696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce580cf780 0x7fce58158310 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 >> 172.21.15.28:0/1384931696 conn(0x7fce580d3fb0 msgr2=0x7fce580cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 shutdown_connections 2024-09-10T20:46:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.400+0000 7fce5c8a2640 1 -- 172.21.15.28:0/1384931696 wait complete. 2024-09-10T20:46:33.414 INFO:tasks.ceph:Scrubbing osd.5 2024-09-10T20:46:33.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.520+0000 7f195e6bd640 1 -- 172.21.15.28:0/2964172424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 msgr2=0x7f19580b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.520+0000 7f195e6bd640 1 --2- 172.21.15.28:0/2964172424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 0x7f19580b5a80 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f1948005fd0 tx=0x7f194802a670 comp rx=0 tx=0).stop 2024-09-10T20:46:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.520+0000 7f195e6bd640 1 -- 172.21.15.28:0/2964172424 shutdown_connections 2024-09-10T20:46:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.520+0000 7f195e6bd640 1 --2- 172.21.15.28:0/2964172424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 0x7f19580b5a80 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.520+0000 7f195e6bd640 1 -- 172.21.15.28:0/2964172424 >> 172.21.15.28:0/2964172424 conn(0x7f19580b2920 msgr2=0x7f19580b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.520+0000 7f195e6bd640 1 -- 172.21.15.28:0/2964172424 shutdown_connections 2024-09-10T20:46:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.520+0000 7f195e6bd640 1 -- 172.21.15.28:0/2964172424 wait complete. 2024-09-10T20:46:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.522+0000 7f195e6bd640 1 Processor -- start 2024-09-10T20:46:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.522+0000 7f195e6bd640 1 -- start start 2024-09-10T20:46:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.522+0000 7f195e6bd640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1958070e50 0x7f195814e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195e6bd640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1958073e10 0x7f195815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195e6bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 0x7f195809d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195e6bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19580ae600 con 0x7f195815ab00 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195e6bd640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f19580ae480 con 0x7f1958073e10 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195e6bd640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f19580ca3b0 con 0x7f1958070e50 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 0x7f195809d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 0x7f195809d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39010/0 (socket says 172.21.15.28:39010) 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 -- 172.21.15.28:0/2456877654 learned_addr learned my addr 172.21.15.28:0/2456877654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f19577fe640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1958073e10 0x7f195815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f1957fff640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1958070e50 0x7f195814e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 -- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1958070e50 msgr2=0x7f195814e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1958070e50 0x7f195814e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 -- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1958073e10 msgr2=0x7f195815bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1958073e10 0x7f195815bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 -- 172.21.15.28:0/2456877654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195814eef0 con 0x7f195815ab00 2024-09-10T20:46:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195cc5a640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 0x7f195809d420 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f195000c7a0 tx=0x7f195000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950003b70 con 0x7f195815ab00 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1950003d10 con 0x7f195815ab00 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19580cc430 con 0x7f195815ab00 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195000aab0 con 0x7f195815ab00 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.523+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19580cc9f0 con 0x7f195815ab00 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.524+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f195806f480 con 0x7f195815ab00 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.525+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f195000ac50 con 0x7f195815ab00 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f1954ff9640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1944078b40 0x7f194407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f1950099070 con 0x7f195815ab00 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f1957fff640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1944078b40 0x7f194407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f1954ff9640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7f194407ea00 0x7f1944080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 --> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1950014e50 con 0x7f194407ea00 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7f195009b950 con 0x7f195815ab00 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f19577fe640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7f194407ea00 0x7f1944080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.526+0000 7f1957fff640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1944078b40 0x7f194407b000 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1948033000 tx=0x7f1948004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.527+0000 7f19577fe640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7f194407ea00 0x7f1944080e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.527+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== osd.5 v2:172.21.15.159:6826/2363244923 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1950014e50 con 0x7f194407ea00 2024-09-10T20:46:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.540+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 --> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1958071480 con 0x7f194407ea00 2024-09-10T20:46:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.540+0000 7f1954ff9640 1 -- 172.21.15.28:0/2456877654 <== osd.5 v2:172.21.15.159:6826/2363244923 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1958071480 con 0x7f194407ea00 2024-09-10T20:46:33.545 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:33.545 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:33.545 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.540+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7f194407ea00 msgr2=0x7f1944080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.540+0000 7f195e6bd640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.159:6826/2363244923,v1:172.21.15.159:6827/2363244923] conn(0x7f194407ea00 0x7f1944080e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1944078b40 msgr2=0x7f194407b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f1944078b40 0x7f194407b000 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1948033000 tx=0x7f1948004730 comp rx=0 tx=0).stop 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 msgr2=0x7f195809d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 --2- 172.21.15.28:0/2456877654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195815ab00 0x7f195809d420 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f195000c7a0 tx=0x7f195000cc60 comp rx=0 tx=0).stop 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195cc5a640 1 -- 172.21.15.28:0/2456877654 reap_dead start 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 shutdown_connections 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 >> 172.21.15.28:0/2456877654 conn(0x7f19580b2920 msgr2=0x7f19580add70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 shutdown_connections 2024-09-10T20:46:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.541+0000 7f195e6bd640 1 -- 172.21.15.28:0/2456877654 wait complete. 2024-09-10T20:46:33.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-10T20:46:33.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 -- 172.21.15.28:0/1394404589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc15c050 msgr2=0x7fa8fc0cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 --2- 172.21.15.28:0/1394404589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc15c050 0x7fa8fc0cfa10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009830 tx=0x7fa8f802a510 comp rx=0 tx=0).stop 2024-09-10T20:46:33.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 -- 172.21.15.28:0/1394404589 shutdown_connections 2024-09-10T20:46:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 --2- 172.21.15.28:0/1394404589 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0d2960 0x7fa8fc0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 --2- 172.21.15.28:0/1394404589 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc15c050 0x7fa8fc0cfa10 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 -- 172.21.15.28:0/1394404589 >> 172.21.15.28:0/1394404589 conn(0x7fa8fc0d1e40 msgr2=0x7fa8fc0d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 -- 172.21.15.28:0/1394404589 shutdown_connections 2024-09-10T20:46:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.657+0000 7fa903bc2640 1 -- 172.21.15.28:0/1394404589 wait complete. 2024-09-10T20:46:33.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.658+0000 7fa903bc2640 1 Processor -- start 2024-09-10T20:46:33.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.658+0000 7fa903bc2640 1 -- start start 2024-09-10T20:46:33.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa903bc2640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0cd9b0 0x7fa8fc0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa903bc2640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc0d2960 0x7fa8fc0c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa903bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8fc15c050 0x7fa8fc0c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa903bc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc0b1af0 con 0x7fa8fc15c050 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa903bc2640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc0cf110 con 0x7fa8fc0d2960 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa903bc2640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fa8fc0b1c70 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90195e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0cd9b0 0x7fa8fc0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90115d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc0d2960 0x7fa8fc0c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90215f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8fc15c050 0x7fa8fc0c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90195e640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0cd9b0 0x7fa8fc0ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.28:37518/0 (socket says 172.21.15.28:37518) 2024-09-10T20:46:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90115d640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc0d2960 0x7fa8fc0c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.28:52930/0 (socket says 172.21.15.28:52930) 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90195e640 1 -- 172.21.15.28:0/2700761181 learned_addr learned my addr 172.21.15.28:0/2700761181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90195e640 1 -- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc0d2960 msgr2=0x7fa8fc0c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90195e640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc0d2960 0x7fa8fc0c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.659+0000 7fa90195e640 1 -- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8fc15c050 msgr2=0x7fa8fc0c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa90195e640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8fc15c050 0x7fa8fc0c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa90195e640 1 -- 172.21.15.28:0/2700761181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc0c6d00 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa90215f640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8fc15c050 0x7fa8fc0c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa90115d640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc0d2960 0x7fa8fc0c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa90195e640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0cd9b0 0x7fa8fc0ca600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009950 tx=0x7fa8f8033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa8f27fc640 1 -- 172.21.15.28:0/2700761181 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8031d60 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa8f27fc640 1 -- 172.21.15.28:0/2700761181 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8f8033e10 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8fc0c9560 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa8f27fc640 1 -- 172.21.15.28:0/2700761181 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8005260 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.660+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8fc0c9b20 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.661+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8fc0724d0 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.662+0000 7fa8f27fc640 1 -- 172.21.15.28:0/2700761181 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fa8f8005400 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.662+0000 7fa8f27fc640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa8e0078b70 0x7fa8e007b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.662+0000 7fa8f27fc640 1 -- 172.21.15.28:0/2700761181 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fa8f80b9920 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.662+0000 7fa90115d640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa8e0078b70 0x7fa8e007b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.663+0000 7fa90115d640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa8e0078b70 0x7fa8e007b030 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0096c0 tx=0x7fa8ec009260 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.667+0000 7fa8f27fc640 1 -- 172.21.15.28:0/2700761181 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa8fc0724d0 con 0x7fa8fc0cd9b0 2024-09-10T20:46:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.855+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fa8fc076360 con 0x7fa8e0078b70 2024-09-10T20:46:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.855+0000 7fa8f27fc640 1 -- 172.21.15.28:0/2700761181 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa8fc076360 con 0x7fa8e0078b70 2024-09-10T20:46:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2024-09-10T20:46:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa8e0078b70 msgr2=0x7fa8e007b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa8e0078b70 0x7fa8e007b030 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0096c0 tx=0x7fa8ec009260 comp rx=0 tx=0).stop 2024-09-10T20:46:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0cd9b0 msgr2=0x7fa8fc0ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0cd9b0 0x7fa8fc0ca600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009950 tx=0x7fa8f8033c20 comp rx=0 tx=0).stop 2024-09-10T20:46:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 shutdown_connections 2024-09-10T20:46:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fa8e0078b70 0x7fa8e007b030 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8fc15c050 0x7fa8fc0c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa8fc0d2960 0x7fa8fc0c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 --2- 172.21.15.28:0/2700761181 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa8fc0cd9b0 0x7fa8fc0ca600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 >> 172.21.15.28:0/2700761181 conn(0x7fa8fc0d1e40 msgr2=0x7fa8fc071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 shutdown_connections 2024-09-10T20:46:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.857+0000 7fa903bc2640 1 -- 172.21.15.28:0/2700761181 wait complete. 2024-09-10T20:46:33.871 INFO:tasks.ceph:Scrubbing osd.6 2024-09-10T20:46:33.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 -- 172.21.15.28:0/2024032545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0d28c0 msgr2=0x7fae2c0ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 --2- 172.21.15.28:0/2024032545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0d28c0 0x7fae2c0ceb90 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fae2c069e30 tx=0x7fae2002a5f0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 -- 172.21.15.28:0/2024032545 shutdown_connections 2024-09-10T20:46:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 --2- 172.21.15.28:0/2024032545 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae2c0cc840 0x7fae2c0ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 --2- 172.21.15.28:0/2024032545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0d28c0 0x7fae2c0ceb90 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 --2- 172.21.15.28:0/2024032545 >> v2:172.21.15.159:3300/0 conn(0x7fae2c0ce7b0 0x7fae2c0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 -- 172.21.15.28:0/2024032545 >> 172.21.15.28:0/2024032545 conn(0x7fae2c0d1da0 msgr2=0x7fae2c0d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 -- 172.21.15.28:0/2024032545 shutdown_connections 2024-09-10T20:46:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.978+0000 7fae33994640 1 -- 172.21.15.28:0/2024032545 wait complete. 2024-09-10T20:46:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.979+0000 7fae33994640 1 Processor -- start 2024-09-10T20:46:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.979+0000 7fae33994640 1 -- start start 2024-09-10T20:46:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae33994640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0cc840 0x7fae2c158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae33994640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae2c0ce7b0 0x7fae2c14b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae33994640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae2c0d28c0 0x7fae2c15ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae33994640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0cd310 con 0x7fae2c0cc840 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae33994640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0cd190 con 0x7fae2c0d28c0 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae33994640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7fae2c0cd490 con 0x7fae2c0ce7b0 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae31730640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0cc840 0x7fae2c158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae31730640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0cc840 0x7fae2c158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39046/0 (socket says 172.21.15.28:39046) 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae31730640 1 -- 172.21.15.28:0/2118062158 learned_addr learned my addr 172.21.15.28:0/2118062158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae30f2f640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae2c0ce7b0 0x7fae2c14b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.980+0000 7fae31f31640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae2c0d28c0 0x7fae2c15ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae31730640 1 -- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae2c0ce7b0 msgr2=0x7fae2c14b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae31730640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fae2c0ce7b0 0x7fae2c14b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae31730640 1 -- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae2c0d28c0 msgr2=0x7fae2c15ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae31730640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fae2c0d28c0 0x7fae2c15ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae31730640 1 -- 172.21.15.28:0/2118062158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c14bd30 con 0x7fae2c0cc840 2024-09-10T20:46:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae31730640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0cc840 0x7fae2c158980 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00ebf0 tx=0x7fae1c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c010630 con 0x7fae2c0cc840 2024-09-10T20:46:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae1c0108c0 con 0x7fae2c0cc840 2024-09-10T20:46:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c010d10 con 0x7fae2c0cc840 2024-09-10T20:46:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2c1578e0 con 0x7fae2c0cc840 2024-09-10T20:46:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.981+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2c158180 con 0x7fae2c0cc840 2024-09-10T20:46:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.982+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae2c0cf970 con 0x7fae2c0cc840 2024-09-10T20:46:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.983+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7fae1c018420 con 0x7fae2c0cc840 2024-09-10T20:46:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.983+0000 7fae29ffb640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae18078b40 0x7fae1807b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7fae1c099fe0 con 0x7fae2c0cc840 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae30f2f640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae18078b40 0x7fae1807b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae29ffb640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7fae1807ea00 0x7fae18080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 --> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae1c01bec0 con 0x7fae1807ea00 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7fae1c09cc50 con 0x7fae2c0cc840 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae31f31640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7fae1807ea00 0x7fae18080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae30f2f640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae18078b40 0x7fae1807b000 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fae20033000 tx=0x7fae20004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.984+0000 7fae31f31640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7fae1807ea00 0x7fae18080e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.985+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== osd.6 v2:172.21.15.159:6810/3584510546 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae1c01bec0 con 0x7fae1807ea00 2024-09-10T20:46:34.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 --> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fae2c0ce590 con 0x7fae1807ea00 2024-09-10T20:46:34.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae29ffb640 1 -- 172.21.15.28:0/2118062158 <== osd.6 v2:172.21.15.159:6810/3584510546 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fae2c0ce590 con 0x7fae1807ea00 2024-09-10T20:46:34.002 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:34.002 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:34.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7fae1807ea00 msgr2=0x7fae18080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae33994640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.159:6810/3584510546,v1:172.21.15.159:6811/3584510546] conn(0x7fae1807ea00 0x7fae18080e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae18078b40 msgr2=0x7fae1807b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae33994640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7fae18078b40 0x7fae1807b000 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fae20033000 tx=0x7fae20004730 comp rx=0 tx=0).stop 2024-09-10T20:46:34.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0cc840 msgr2=0x7fae2c158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae33994640 1 --2- 172.21.15.28:0/2118062158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae2c0cc840 0x7fae2c158980 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00ebf0 tx=0x7fae1c00c6a0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.998+0000 7fae31f31640 1 -- 172.21.15.28:0/2118062158 reap_dead start 2024-09-10T20:46:34.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.999+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 shutdown_connections 2024-09-10T20:46:34.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.999+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 >> 172.21.15.28:0/2118062158 conn(0x7fae2c0d1da0 msgr2=0x7fae2c0d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.999+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 shutdown_connections 2024-09-10T20:46:34.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:33.999+0000 7fae33994640 1 -- 172.21.15.28:0/2118062158 wait complete. 2024-09-10T20:46:34.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 -- 172.21.15.28:0/1214257738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 msgr2=0x7f80080b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 --2- 172.21.15.28:0/1214257738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f80080b2ba0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009830 tx=0x7f7ff802a480 comp rx=0 tx=0).stop 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 -- 172.21.15.28:0/1214257738 shutdown_connections 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 --2- 172.21.15.28:0/1214257738 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80080b4d10 0x7f80080ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 --2- 172.21.15.28:0/1214257738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f80080b2ba0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 -- 172.21.15.28:0/1214257738 >> 172.21.15.28:0/1214257738 conn(0x7f800806cf00 msgr2=0x7f800806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 -- 172.21.15.28:0/1214257738 shutdown_connections 2024-09-10T20:46:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.114+0000 7f800d517640 1 -- 172.21.15.28:0/1214257738 wait complete. 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.116+0000 7f800d517640 1 Processor -- start 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.116+0000 7f800d517640 1 -- start start 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.116+0000 7f800d517640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80080b4d10 0x7f80081e4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f800d517640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f8008159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f800d517640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f800815df70 0x7f80081e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f800d517640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f800815a6f0 con 0x7f800815a190 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f800d517640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f800815a570 con 0x7f800815df70 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f800d517640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f800815a870 con 0x7f80080b4d10 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f8008159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f8008159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39066/0 (socket says 172.21.15.28:39066) 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 -- 172.21.15.28:0/1130506377 learned_addr learned my addr 172.21.15.28:0/1130506377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f8006ffd640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80080b4d10 0x7f80081e4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80077fe640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f800815df70 0x7f80081e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 -- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80080b4d10 msgr2=0x7f80081e4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80080b4d10 0x7f80081e4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 -- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f800815df70 msgr2=0x7f80081e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f800815df70 0x7f80081e59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 -- 172.21.15.28:0/1130506377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80081e3ea0 con 0x7f800815a190 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f80067fc640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f8008159220 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00c370 tx=0x7f7ffc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f7fef7fe640 1 -- 172.21.15.28:0/1130506377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc011790 con 0x7f800815a190 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f7fef7fe640 1 -- 172.21.15.28:0/1130506377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ffc017070 con 0x7f800815a190 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f7fef7fe640 1 -- 172.21.15.28:0/1130506377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc011c80 con 0x7f800815a190 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80081e4180 con 0x7f800815a190 2024-09-10T20:46:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.117+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80081506e0 con 0x7f800815a190 2024-09-10T20:46:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.119+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80080ce030 con 0x7f800815a190 2024-09-10T20:46:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.119+0000 7f7fef7fe640 1 -- 172.21.15.28:0/1130506377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f7ffc01c460 con 0x7f800815a190 2024-09-10T20:46:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.120+0000 7f7fef7fe640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ff4078b40 0x7f7ff407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.120+0000 7f7fef7fe640 1 -- 172.21.15.28:0/1130506377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f7ffc0985c0 con 0x7f800815a190 2024-09-10T20:46:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.120+0000 7f8006ffd640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ff4078b40 0x7f7ff407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.120+0000 7f8006ffd640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ff4078b40 0x7f7ff407b000 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8034000 tx=0x7f7ff8004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.124+0000 7f7fef7fe640 1 -- 172.21.15.28:0/1130506377 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7ffc062240 con 0x7f800815a190 2024-09-10T20:46:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.309+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f800809de90 con 0x7f7ff4078b40 2024-09-10T20:46:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.309+0000 7f7fef7fe640 1 -- 172.21.15.28:0/1130506377 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f800809de90 con 0x7f7ff4078b40 2024-09-10T20:46:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 6 to deep-scrub 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ff4078b40 msgr2=0x7f7ff407b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ff4078b40 0x7f7ff407b000 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8034000 tx=0x7f7ff8004730 comp rx=0 tx=0).stop 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 msgr2=0x7f8008159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f8008159220 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00c370 tx=0x7f7ffc00c830 comp rx=0 tx=0).stop 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 shutdown_connections 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f7ff4078b40 0x7f7ff407b000 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f800815df70 0x7f80081e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800815a190 0x7f8008159220 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 --2- 172.21.15.28:0/1130506377 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80080b4d10 0x7f80081e4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 >> 172.21.15.28:0/1130506377 conn(0x7f800806cf00 msgr2=0x7f800806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 shutdown_connections 2024-09-10T20:46:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.312+0000 7f800d517640 1 -- 172.21.15.28:0/1130506377 wait complete. 2024-09-10T20:46:34.327 INFO:tasks.ceph:Scrubbing osd.7 2024-09-10T20:46:34.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:46:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 -- 172.21.15.28:0/3748418854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 msgr2=0x7f02c80b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 --2- 172.21.15.28:0/3748418854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 0x7f02c80b6050 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f02bc005fd0 tx=0x7f02bc02a540 comp rx=0 tx=0).stop 2024-09-10T20:46:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 -- 172.21.15.28:0/3748418854 shutdown_connections 2024-09-10T20:46:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 --2- 172.21.15.28:0/3748418854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 0x7f02c80b6050 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 --2- 172.21.15.28:0/3748418854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c806e5e0 0x7f02c806e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 -- 172.21.15.28:0/3748418854 >> 172.21.15.28:0/3748418854 conn(0x7f02c80d1da0 msgr2=0x7f02c80d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 -- 172.21.15.28:0/3748418854 shutdown_connections 2024-09-10T20:46:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.432+0000 7f02cf3d3640 1 -- 172.21.15.28:0/3748418854 wait complete. 2024-09-10T20:46:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.433+0000 7f02cf3d3640 1 Processor -- start 2024-09-10T20:46:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.433+0000 7f02cf3d3640 1 -- start start 2024-09-10T20:46:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cf3d3640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c806e5e0 0x7f02c81f8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cf3d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 0x7f02c80ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cf3d3640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c80b6590 0x7f02c809f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cf3d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02c806dd30 con 0x7f02c80b2b40 2024-09-10T20:46:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cf3d3640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f02c8162ba0 con 0x7f02c80b6590 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cf3d3640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f02c80b7150 con 0x7f02c806e5e0 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cc96e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 0x7f02c80ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cc96e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 0x7f02c80ae9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39088/0 (socket says 172.21.15.28:39088) 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cc96e640 1 -- 172.21.15.28:0/1260462681 learned_addr learned my addr 172.21.15.28:0/1260462681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cd16f640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c806e5e0 0x7f02c81f8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.434+0000 7f02cd970640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c80b6590 0x7f02c809f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cc96e640 1 -- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c806e5e0 msgr2=0x7f02c81f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cc96e640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f02c806e5e0 0x7f02c81f8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cc96e640 1 -- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c80b6590 msgr2=0x7f02c809f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cc96e640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f02c80b6590 0x7f02c809f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cc96e640 1 -- 172.21.15.28:0/1260462681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c809f900 con 0x7f02c80b2b40 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cc96e640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 0x7f02c80ae9b0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009850 tx=0x7f02bc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc03f070 con 0x7f02c80b2b40 2024-09-10T20:46:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02bc0047a0 con 0x7f02c80b2b40 2024-09-10T20:46:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c81f4ca0 con 0x7f02c80b2b40 2024-09-10T20:46:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc035030 con 0x7f02c80b2b40 2024-09-10T20:46:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.435+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c81f5260 con 0x7f02c80b2b40 2024-09-10T20:46:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.436+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f02c80d1070 con 0x7f02c80b2b40 2024-09-10T20:46:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.437+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f02bc004b90 con 0x7f02c80b2b40 2024-09-10T20:46:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02c5ffb640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f02b4078b40 0x7f02b407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02cd16f640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f02b4078b40 0x7f02b407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f02bc0b9b20 con 0x7f02c80b2b40 2024-09-10T20:46:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02c5ffb640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7f02b407ea00 0x7f02b4080e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 --> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02b40814f0 con 0x7f02b407ea00 2024-09-10T20:46:34.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=167) ==== 24+0+0 (secure 0 0 0) 0x7f02bc0c3050 con 0x7f02c80b2b40 2024-09-10T20:46:34.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02cd16f640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f02b4078b40 0x7f02b407b000 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f02c80b5040 tx=0x7f02b80094a0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.438+0000 7f02cd970640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7f02b407ea00 0x7f02b4080e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.439+0000 7f02cd970640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7f02b407ea00 0x7f02b4080e20 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.439+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== osd.7 v2:172.21.15.159:6818/88942910 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f02b40814f0 con 0x7f02b407ea00 2024-09-10T20:46:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.452+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 --> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f02c8070140 con 0x7f02b407ea00 2024-09-10T20:46:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.452+0000 7f02c5ffb640 1 -- 172.21.15.28:0/1260462681 <== osd.7 v2:172.21.15.159:6818/88942910 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f02c8070140 con 0x7f02b407ea00 2024-09-10T20:46:34.456 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T20:46:34.457 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:46:34.457 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.452+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7f02b407ea00 msgr2=0x7f02b4080e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.452+0000 7f02cf3d3640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.159:6818/88942910,v1:172.21.15.159:6819/88942910] conn(0x7f02b407ea00 0x7f02b4080e20 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f02b4078b40 msgr2=0x7f02b407b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f02b4078b40 0x7f02b407b000 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f02c80b5040 tx=0x7f02b80094a0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 msgr2=0x7f02c80ae9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 --2- 172.21.15.28:0/1260462681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02c80b2b40 0x7f02c80ae9b0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009850 tx=0x7f02bc0038a0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cd970640 1 -- 172.21.15.28:0/1260462681 reap_dead start 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 shutdown_connections 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 >> 172.21.15.28:0/1260462681 conn(0x7f02c80d1da0 msgr2=0x7f02c815bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 shutdown_connections 2024-09-10T20:46:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.453+0000 7f02cf3d3640 1 -- 172.21.15.28:0/1260462681 wait complete. 2024-09-10T20:46:34.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-10T20:46:34.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 -- 172.21.15.28:0/475149957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 msgr2=0x7f35380cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 --2- 172.21.15.28:0/475149957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380cfa10 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f35380cc420 tx=0x7f352802a530 comp rx=0 tx=0).stop 2024-09-10T20:46:34.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 -- 172.21.15.28:0/475149957 shutdown_connections 2024-09-10T20:46:34.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 --2- 172.21.15.28:0/475149957 >> v2:172.21.15.159:3300/0 conn(0x7f35380cef60 0x7f35380b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 --2- 172.21.15.28:0/475149957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380cfa10 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 -- 172.21.15.28:0/475149957 >> 172.21.15.28:0/475149957 conn(0x7f35380d1e40 msgr2=0x7f35380d2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 -- 172.21.15.28:0/475149957 shutdown_connections 2024-09-10T20:46:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.569+0000 7f353ead4640 1 -- 172.21.15.28:0/475149957 wait complete. 2024-09-10T20:46:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.570+0000 7f353ead4640 1 Processor -- start 2024-09-10T20:46:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.570+0000 7f353ead4640 1 -- start start 2024-09-10T20:46:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353ead4640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35380cef60 0x7f353809f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353ead4640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35380d2960 0x7f35380c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353ead4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353ead4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35380b1920 con 0x7f353815c050 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353ead4640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f35380cdce0 con 0x7f35380d2960 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353ead4640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f35380b1aa0 con 0x7f35380cef60 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353d071640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353d071640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39104/0 (socket says 172.21.15.28:39104) 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f3537fff640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35380d2960 0x7f35380c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353d071640 1 -- 172.21.15.28:0/3900494420 learned_addr learned my addr 172.21.15.28:0/3900494420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.571+0000 7f353c870640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35380cef60 0x7f353809f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353d071640 1 -- 172.21.15.28:0/3900494420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35380cef60 msgr2=0x7f353809f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353d071640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35380cef60 0x7f353809f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353d071640 1 -- 172.21.15.28:0/3900494420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35380d2960 msgr2=0x7f35380c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353d071640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35380d2960 0x7f35380c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353d071640 1 -- 172.21.15.28:0/3900494420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35381f2a40 con 0x7f353815c050 2024-09-10T20:46:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353d071640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380c74c0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f353000d720 tx=0x7f353000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f35357fa640 1 -- 172.21.15.28:0/3900494420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530017070 con 0x7f353815c050 2024-09-10T20:46:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f35357fa640 1 -- 172.21.15.28:0/3900494420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3530009ce0 con 0x7f353815c050 2024-09-10T20:46:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f35357fa640 1 -- 172.21.15.28:0/3900494420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530005280 con 0x7f353815c050 2024-09-10T20:46:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35381f2d20 con 0x7f353815c050 2024-09-10T20:46:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.572+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381f32e0 con 0x7f353815c050 2024-09-10T20:46:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.573+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35380b1f00 con 0x7f353815c050 2024-09-10T20:46:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.574+0000 7f35357fa640 1 -- 172.21.15.28:0/3900494420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f3530005420 con 0x7f353815c050 2024-09-10T20:46:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.574+0000 7f35357fa640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3518078b40 0x7f351807b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.574+0000 7f353c870640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3518078b40 0x7f351807b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.574+0000 7f35357fa640 1 -- 172.21.15.28:0/3900494420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f3530098a60 con 0x7f353815c050 2024-09-10T20:46:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.575+0000 7f353c870640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3518078b40 0x7f351807b000 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3528034000 tx=0x7f3528004730 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.578+0000 7f35357fa640 1 -- 172.21.15.28:0/3900494420 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35300626e0 con 0x7f353815c050 2024-09-10T20:46:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.763+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f3538076360 con 0x7f3518078b40 2024-09-10T20:46:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.763+0000 7f35357fa640 1 -- 172.21.15.28:0/3900494420 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3538076360 con 0x7f3518078b40 2024-09-10T20:46:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 7 to deep-scrub 2024-09-10T20:46:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3518078b40 msgr2=0x7f351807b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3518078b40 0x7f351807b000 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3528034000 tx=0x7f3528004730 comp rx=0 tx=0).stop 2024-09-10T20:46:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 msgr2=0x7f35380c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380c74c0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f353000d720 tx=0x7f353000dbe0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 shutdown_connections 2024-09-10T20:46:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f3518078b40 0x7f351807b000 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353815c050 0x7f35380c74c0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35380d2960 0x7f35380c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 --2- 172.21.15.28:0/3900494420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35380cef60 0x7f353809f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 >> 172.21.15.28:0/3900494420 conn(0x7f35380d1e40 msgr2=0x7f353815b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 shutdown_connections 2024-09-10T20:46:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.765+0000 7f353ead4640 1 -- 172.21.15.28:0/3900494420 wait complete. 2024-09-10T20:46:34.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:46:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.881+0000 7f8774833640 1 -- 172.21.15.28:0/2683935755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8770151cc0 msgr2=0x7f87700cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.881+0000 7f8774833640 1 --2- 172.21.15.28:0/2683935755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8770151cc0 0x7f87700cf0f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8768009be0 tx=0x7f876802a920 comp rx=0 tx=0).stop 2024-09-10T20:46:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.881+0000 7f8774833640 1 -- 172.21.15.28:0/2683935755 shutdown_connections 2024-09-10T20:46:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.881+0000 7f8774833640 1 --2- 172.21.15.28:0/2683935755 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8770151cc0 0x7f87700cf0f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.881+0000 7f8774833640 1 -- 172.21.15.28:0/2683935755 >> 172.21.15.28:0/2683935755 conn(0x7f87700d0420 msgr2=0x7f87700d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.881+0000 7f8774833640 1 -- 172.21.15.28:0/2683935755 shutdown_connections 2024-09-10T20:46:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.881+0000 7f8774833640 1 -- 172.21.15.28:0/2683935755 wait complete. 2024-09-10T20:46:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.882+0000 7f8774833640 1 Processor -- start 2024-09-10T20:46:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.882+0000 7f8774833640 1 -- start start 2024-09-10T20:46:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f8774833640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87700738b0 0x7f87701f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f8774833640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8770151cc0 0x7f8770146130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f8774833640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f877015eef0 0x7f87700b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f8774833640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f877015f620 con 0x7f87700738b0 2024-09-10T20:46:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f8774833640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f8770073f00 con 0x7f8770151cc0 2024-09-10T20:46:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f8774833640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f87700cb920 con 0x7f877015eef0 2024-09-10T20:46:34.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f876e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87700738b0 0x7f87701f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f876e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87700738b0 0x7f87701f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39114/0 (socket says 172.21.15.28:39114) 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f876e59c640 1 -- 172.21.15.28:0/3699373513 learned_addr learned my addr 172.21.15.28:0/3699373513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f876ed9d640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f877015eef0 0x7f87700b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.883+0000 7f876dd9b640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8770151cc0 0x7f8770146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f876e59c640 1 -- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f877015eef0 msgr2=0x7f87700b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f876e59c640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f877015eef0 0x7f87700b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f876e59c640 1 -- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8770151cc0 msgr2=0x7f8770146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f876e59c640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8770151cc0 0x7f8770146130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f876e59c640 1 -- 172.21.15.28:0/3699373513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8770146670 con 0x7f87700738b0 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f876e59c640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87700738b0 0x7f87701f6870 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8768009be0 tx=0x7f8768003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f8766ffd640 1 -- 172.21.15.28:0/3699373513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768004240 con 0x7f87700738b0 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f8766ffd640 1 -- 172.21.15.28:0/3699373513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8768003e50 con 0x7f87700738b0 2024-09-10T20:46:34.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87701f28f0 con 0x7f87700738b0 2024-09-10T20:46:34.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f8766ffd640 1 -- 172.21.15.28:0/3699373513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768004680 con 0x7f87700738b0 2024-09-10T20:46:34.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.884+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87701f2eb0 con 0x7f87700738b0 2024-09-10T20:46:34.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.885+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8770068d10 con 0x7f87700738b0 2024-09-10T20:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.886+0000 7f8766ffd640 1 -- 172.21.15.28:0/3699373513 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f876803a450 con 0x7f87700738b0 2024-09-10T20:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.886+0000 7f8766ffd640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8750078b40 0x7f875007b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.887+0000 7f876dd9b640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8750078b40 0x7f875007b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.887+0000 7f8766ffd640 1 -- 172.21.15.28:0/3699373513 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f87680ba300 con 0x7f87700738b0 2024-09-10T20:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.887+0000 7f876dd9b640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8750078b40 0x7f875007b000 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8770149d50 tx=0x7f8758009040 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:34.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:34.891+0000 7f8766ffd640 1 -- 172.21.15.28:0/3699373513 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8770146670 con 0x7f87700738b0 2024-09-10T20:46:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.074+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 --> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f87700cf0f0 con 0x7f8750078b40 2024-09-10T20:46:35.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.074+0000 7f8766ffd640 1 -- 172.21.15.28:0/3699373513 <== mgr.4156 v2:172.21.15.28:6836/4163551983 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13373 (secure 0 0 0) 0x7f87700cf0f0 con 0x7f8750078b40 2024-09-10T20:46:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:35.079 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8750078b40 msgr2=0x7f875007b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8750078b40 0x7f875007b000 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8770149d50 tx=0x7f8758009040 comp rx=0 tx=0).stop 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87700738b0 msgr2=0x7f87701f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87700738b0 0x7f87701f6870 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8768009be0 tx=0x7f8768003ba0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 shutdown_connections 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f8750078b40 0x7f875007b000 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f877015eef0 0x7f87700b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8770151cc0 0x7f8770146130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 --2- 172.21.15.28:0/3699373513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87700738b0 0x7f87701f6870 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 >> 172.21.15.28:0/3699373513 conn(0x7f87700d0420 msgr2=0x7f87700d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:35.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 shutdown_connections 2024-09-10T20:46:35.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.076+0000 7f8774833640 1 -- 172.21.15.28:0/3699373513 wait complete. 2024-09-10T20:46:35.091 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-10T20:46:34.344235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":283784,"kb_used_data":17640,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754690936,"statfs":{"total":773094113280,"available":772803518464,"internally_reserved":0,"allocated":18063360,"data_stored":8642352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-4,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.449935"},"pg_stats":[{"pgid":"1.0","version":"167'81","reported_seq":420,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-10T20:46:32.502001+0000","last_change":"2024-09-10T20:46:32.502001+0000","last_active":"2024-09-10T20:46:32.502001+0000","last_peered":"2024-09-10T20:46:32.502001+0000","last_clean":"2024-09-10T20:46:32.502001+0000","last_became_active":"2024-09-10T20:38:01.346244+0000","last_became_peered":"2024-09-10T20:38:01.346244+0000","last_unstale":"2024-09-10T20:46:32.502001+0000","last_undegraded":"2024-09-10T20:46:32.502001+0000","last_fullsized":"2024-09-10T20:46:32.502001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'81","last_scrub_stamp":"2024-09-10T20:46:32.501918+0000","last_deep_scrub":"167'81","last_deep_scrub_stamp":"2024-09-10T20:46:32.501918+0000","last_clean_scrub_stamp":"2024-09-10T20:46:32.501918+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:55.040208+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33252,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":2109440,"data_stored":932702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33248,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338592,"statfs":{"total":96636764160,"available":96602718208,"internally_reserved":0,"allocated":2109440,"data_stored":932702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542249,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2703360,"data_stored":1523070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33252,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":2109440,"data_stored":932702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37408,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334432,"statfs":{"total":96636764160,"available":96598458368,"internally_reserved":0,"allocated":2109440,"data_stored":932702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37408,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334432,"statfs":{"total":96636764160,"available":96598458368,"internally_reserved":0,"allocated":2109440,"data_stored":932702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37984,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333856,"statfs":{"total":96636764160,"available":96597868544,"internally_reserved":0,"allocated":2703360,"data_stored":1523070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37408,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334432,"statfs":{"total":96636764160,"available":96598458368,"internally_reserved":0,"allocated":2109440,"data_stored":932702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:46:35.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-10T20:46:35.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 -- 172.21.15.28:0/1390796781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 msgr2=0x7f6060163ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 --2- 172.21.15.28:0/1390796781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f6060163ed0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f60600b2900 tx=0x7f605002aa30 comp rx=0 tx=0).stop 2024-09-10T20:46:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 -- 172.21.15.28:0/1390796781 shutdown_connections 2024-09-10T20:46:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 --2- 172.21.15.28:0/1390796781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f6060163ed0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 --2- 172.21.15.28:0/1390796781 >> v2:172.21.15.159:3300/0 conn(0x7f60600b2b70 0x7f6060074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 -- 172.21.15.28:0/1390796781 >> 172.21.15.28:0/1390796781 conn(0x7f606006d110 msgr2=0x7f60600b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 -- 172.21.15.28:0/1390796781 shutdown_connections 2024-09-10T20:46:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.195+0000 7f6068416640 1 -- 172.21.15.28:0/1390796781 wait complete. 2024-09-10T20:46:35.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.196+0000 7f6068416640 1 Processor -- start 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.196+0000 7f6068416640 1 -- start start 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f6068416640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f60600aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f6068416640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60600b2b70 0x7f60601f5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f6068416640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60600b2f50 0x7f606014d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f6068416640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6060164590 con 0x7f606006f8a0 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f6068416640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f6060164410 con 0x7f60600b2f50 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f6068416640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f6060164710 con 0x7f60600b2b70 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60661b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f60600aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60661b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f60600aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39138/0 (socket says 172.21.15.28:39138) 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60661b2640 1 -- 172.21.15.28:0/1328227867 learned_addr learned my addr 172.21.15.28:0/1328227867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60659b1640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60600b2b70 0x7f60601f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60669b3640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60600b2f50 0x7f606014d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60661b2640 1 -- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60600b2b70 msgr2=0x7f60601f5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60661b2640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60600b2b70 0x7f60601f5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60661b2640 1 -- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60600b2f50 msgr2=0x7f606014d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.197+0000 7f60661b2640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60600b2f50 0x7f606014d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.198+0000 7f60661b2640 1 -- 172.21.15.28:0/1328227867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60600d0450 con 0x7f606006f8a0 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.198+0000 7f60661b2640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f60600aede0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f605800d7a0 tx=0x7f605800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.198+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058017070 con 0x7f606006f8a0 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.198+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6058009d20 con 0x7f606006f8a0 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.198+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6060154bc0 con 0x7f606006f8a0 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.198+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058005280 con 0x7f606006f8a0 2024-09-10T20:46:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.198+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6060155180 con 0x7f606006f8a0 2024-09-10T20:46:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.199+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6060074340 con 0x7f606006f8a0 2024-09-10T20:46:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.200+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f6058005420 con 0x7f606006f8a0 2024-09-10T20:46:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.200+0000 7f605effd640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f604c078b40 0x7f604c07b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.201+0000 7f60659b1640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f604c078b40 0x7f604c07b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.201+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f6058098a70 con 0x7f606006f8a0 2024-09-10T20:46:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.201+0000 7f60659b1640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f604c078b40 0x7f604c07b000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f60600b3510 tx=0x7f6054007e30 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.205+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6058062770 con 0x7f606006f8a0 2024-09-10T20:46:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.390+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f60600d0800 con 0x7f606006f8a0 2024-09-10T20:46:35.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.393+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v50)=0 v50) ==== 123+0+0 (secure 0 0 0) 0x7f60580671a0 con 0x7f606006f8a0 2024-09-10T20:46:35.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.394+0000 7f605effd640 1 -- 172.21.15.28:0/1328227867 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f6058062280 con 0x7f606006f8a0 2024-09-10T20:46:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f604c078b40 msgr2=0x7f604c07b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f604c078b40 0x7f604c07b000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f60600b3510 tx=0x7f6054007e30 comp rx=0 tx=0).stop 2024-09-10T20:46:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 msgr2=0x7f60600aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f60600aede0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f605800d7a0 tx=0x7f605800dc60 comp rx=0 tx=0).stop 2024-09-10T20:46:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 shutdown_connections 2024-09-10T20:46:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f604c078b40 0x7f604c07b000 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f60600b2f50 0x7f606014d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f60600b2b70 0x7f60601f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 --2- 172.21.15.28:0/1328227867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f606006f8a0 0x7f60600aede0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 >> 172.21.15.28:0/1328227867 conn(0x7f606006d110 msgr2=0x7f606014b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 shutdown_connections 2024-09-10T20:46:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.396+0000 7f6068416640 1 -- 172.21.15.28:0/1328227867 wait complete. 2024-09-10T20:46:35.412 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-10T20:46:35.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-10T20:46:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.513+0000 7f85d9f2b640 1 -- 172.21.15.28:0/2997045335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 msgr2=0x7f85d4161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.513+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/2997045335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d4161b50 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009830 tx=0x7f85c802a510 comp rx=0 tx=0).stop 2024-09-10T20:46:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.513+0000 7f85d9f2b640 1 -- 172.21.15.28:0/2997045335 shutdown_connections 2024-09-10T20:46:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.513+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/2997045335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d4161b50 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.513+0000 7f85d9f2b640 1 -- 172.21.15.28:0/2997045335 >> 172.21.15.28:0/2997045335 conn(0x7f85d40d3fb0 msgr2=0x7f85d40d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.514+0000 7f85d9f2b640 1 -- 172.21.15.28:0/2997045335 shutdown_connections 2024-09-10T20:46:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.514+0000 7f85d9f2b640 1 -- 172.21.15.28:0/2997045335 wait complete. 2024-09-10T20:46:35.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.515+0000 7f85d9f2b640 1 Processor -- start 2024-09-10T20:46:35.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.515+0000 7f85d9f2b640 1 -- start start 2024-09-10T20:46:35.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d9f2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d41612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d9f2b640 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85d415de70 0x7f85d414e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d9f2b640 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85d4162090 0x7f85d41f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d9f2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85d415e3d0 con 0x7f85d40d2b90 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d9f2b640 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 -- 0x7f85d415e250 con 0x7f85d4162090 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d9f2b640 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 -- 0x7f85d40b7510 con 0x7f85d415de70 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d41612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d41612a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39154/0 (socket says 172.21.15.28:39154) 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d37fe640 1 -- 172.21.15.28:0/3595183444 learned_addr learned my addr 172.21.15.28:0/3595183444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d3fff640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85d4162090 0x7f85d41f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.516+0000 7f85d2ffd640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85d415de70 0x7f85d414e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d37fe640 1 -- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85d415de70 msgr2=0x7f85d414e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d37fe640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85d415de70 0x7f85d414e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d37fe640 1 -- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85d4162090 msgr2=0x7f85d41f4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d37fe640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85d4162090 0x7f85d41f4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d37fe640 1 -- 172.21.15.28:0/3595183444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d4157f80 con 0x7f85d40d2b90 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d37fe640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d41612a0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f85c400c430 tx=0x7f85c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85bbfff640 1 -- 172.21.15.28:0/3595183444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4011790 con 0x7f85d40d2b90 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85bbfff640 1 -- 172.21.15.28:0/3595183444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f85c4017070 con 0x7f85d40d2b90 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d41f54d0 con 0x7f85d40d2b90 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85bbfff640 1 -- 172.21.15.28:0/3595183444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4011c30 con 0x7f85d40d2b90 2024-09-10T20:46:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.517+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d41f5a90 con 0x7f85d40d2b90 2024-09-10T20:46:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.518+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d4161b50 con 0x7f85d40d2b90 2024-09-10T20:46:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.519+0000 7f85bbfff640 1 -- 172.21.15.28:0/3595183444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 29) ==== 101672+0+0 (secure 0 0 0) 0x7f85c401c460 con 0x7f85d40d2b90 2024-09-10T20:46:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.519+0000 7f85bbfff640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f85c0078b40 0x7f85c007b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:46:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.519+0000 7f85d2ffd640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f85c0078b40 0x7f85c007b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:46:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.519+0000 7f85bbfff640 1 -- 172.21.15.28:0/3595183444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4971+0+0 (secure 0 0 0) 0x7f85c4098860 con 0x7f85d40d2b90 2024-09-10T20:46:35.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.520+0000 7f85d2ffd640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f85c0078b40 0x7f85c007b000 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f85c8034000 tx=0x7f85c80046c0 comp rx=0 tx=0).ready entity=mgr.4156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:46:35.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.523+0000 7f85bbfff640 1 -- 172.21.15.28:0/3595183444 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f85c4062b10 con 0x7f85d40d2b90 2024-09-10T20:46:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.744+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85d40ca6b0 con 0x7f85d40d2b90 2024-09-10T20:46:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.745+0000 7f85bbfff640 1 -- 172.21.15.28:0/3595183444 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+3919 (secure 0 0 0) 0x7f85c406b230 con 0x7f85d40d2b90 2024-09-10T20:46:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T20:46:35.749 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":119,"btime":"2024-09-10T20:46:24:886704+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/4088436065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4088436065},{"type":"v1","addr":"172.21.15.28:6834","nonce":4088436065}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6801/2443719102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":2443719102},{"type":"v1","addr":"172.21.15.159:6801","nonce":2443719102}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":7250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/3153361013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":3153361013},{"type":"v1","addr":"172.21.15.159:6835","nonce":3153361013}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111},{"gid":9357,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/662833795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":662833795},{"type":"v1","addr":"172.21.15.28:6835","nonce":662833795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[]} 2024-09-10T20:46:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 119 2024-09-10T20:46:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f85c0078b40 msgr2=0x7f85c007b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f85c0078b40 0x7f85c007b000 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f85c8034000 tx=0x7f85c80046c0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 msgr2=0x7f85d41612a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d41612a0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f85c400c430 tx=0x7f85c400c8f0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 shutdown_connections 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:6836/4163551983,v1:172.21.15.28:6837/4163551983] conn(0x7f85c0078b40 0x7f85c007b000 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f85d4162090 0x7f85d41f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f85d415de70 0x7f85d414e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 --2- 172.21.15.28:0/3595183444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85d40d2b90 0x7f85d41612a0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 >> 172.21.15.28:0/3595183444 conn(0x7f85d40d3fb0 msgr2=0x7f85d40b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 shutdown_connections 2024-09-10T20:46:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-10T20:46:35.747+0000 7f85d9f2b640 1 -- 172.21.15.28:0/3595183444 wait complete. 2024-09-10T20:46:35.762 DEBUG:tasks.cephfs.filesystem:already dead... 2024-09-10T20:46:35.762 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-10T20:46:35.762 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-10T20:46:35.763 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:35.771 INFO:tasks.ceph.mds.a:Stopped 2024-09-10T20:46:35.772 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-10T20:46:35.772 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:35.821 INFO:tasks.ceph.mds.c:Stopped 2024-09-10T20:46:35.821 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-10T20:46:35.821 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:35.832 INFO:tasks.ceph.mds.b:Stopped 2024-09-10T20:46:35.832 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-10T20:46:35.832 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:35.883 INFO:tasks.ceph.mds.d:Stopped 2024-09-10T20:46:35.883 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-10T20:46:35.883 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-10T20:46:35.884 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:35.912 INFO:tasks.ceph.osd.0:Stopped 2024-09-10T20:46:35.912 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-10T20:46:35.912 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:35.979 INFO:tasks.ceph.osd.1:Stopped 2024-09-10T20:46:35.980 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-10T20:46:35.980 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.047 INFO:tasks.ceph.osd.2:Stopped 2024-09-10T20:46:36.048 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-10T20:46:36.048 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.124 INFO:tasks.ceph.osd.3:Stopped 2024-09-10T20:46:36.124 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-10T20:46:36.125 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.159 INFO:tasks.ceph.osd.4:Stopped 2024-09-10T20:46:36.159 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-10T20:46:36.159 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.235 INFO:tasks.ceph.osd.5:Stopped 2024-09-10T20:46:36.235 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-10T20:46:36.235 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.305 INFO:tasks.ceph.osd.6:Stopped 2024-09-10T20:46:36.305 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-10T20:46:36.305 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.383 INFO:tasks.ceph.osd.7:Stopped 2024-09-10T20:46:36.384 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-10T20:46:36.384 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-10T20:46:36.384 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.417 INFO:tasks.ceph.mgr.y:Stopped 2024-09-10T20:46:36.417 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-10T20:46:36.417 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.452 INFO:tasks.ceph.mgr.x:Stopped 2024-09-10T20:46:36.453 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-10T20:46:36.453 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T20:46:36.453 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.470 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T20:46:36.470 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T20:46:36.470 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.505 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T20:46:36.506 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T20:46:36.506 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:46:36.558 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T20:46:36.558 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-10T20:46:36.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-09-10T20:46:36.603 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:46:36.603 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-10T20:46:36.785 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:46:36.786 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-10T20:46:36.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:46:36.850 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-10T20:46:36.900 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:46:36.900 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-10T20:46:36.942 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:46:36.942 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-10T20:46:37.100 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:46:37.100 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-10T20:46:37.226 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:46:37.226 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-10T20:46:37.282 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:46:37.282 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-10T20:46:37.340 INFO:tasks.ceph:Archiving mon data... 2024-09-10T20:46:37.341 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/data/mon.a.tgz 2024-09-10T20:46:37.341 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-10T20:46:37.357 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.LxxoO3gUPt 2024-09-10T20:46:37.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LxxoO3gUPt 2024-09-10T20:46:37.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.LxxoO3gUPt 2024-09-10T20:46:37.700 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.LxxoO3gUPt is 1MB 2024-09-10T20:46:37.811 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.LxxoO3gUPt 2024-09-10T20:46:37.827 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/data/mon.b.tgz 2024-09-10T20:46:37.827 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-09-10T20:46:37.845 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.WFtnFtIj77 2024-09-10T20:46:37.846 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WFtnFtIj77 2024-09-10T20:46:38.107 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.WFtnFtIj77 2024-09-10T20:46:38.191 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.WFtnFtIj77 is 1MB 2024-09-10T20:46:38.303 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.WFtnFtIj77 2024-09-10T20:46:38.320 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/data/mon.c.tgz 2024-09-10T20:46:38.320 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-09-10T20:46:38.374 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.xx1nTwFSui 2024-09-10T20:46:38.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xx1nTwFSui 2024-09-10T20:46:38.618 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.xx1nTwFSui 2024-09-10T20:46:38.701 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.xx1nTwFSui is 1MB 2024-09-10T20:46:38.810 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.xx1nTwFSui 2024-09-10T20:46:38.825 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-10T20:46:38.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T20:46:38.828 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T20:46:38.914 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T20:46:38.945 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T20:46:38.945 INFO:tasks.ceph:Archiving crash dumps... 2024-09-10T20:46:38.948 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/remote/smithi028/crash 2024-09-10T20:46:38.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T20:46:38.983 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/remote/smithi159/crash 2024-09-10T20:46:38.984 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T20:46:39.009 INFO:tasks.ceph:Shutting down logrotate 2024-09-10T20:46:39.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:39.027 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:39.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:39.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-10T20:46:39.163 INFO:tasks.ceph:Compressing logs... 2024-09-10T20:46:39.163 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T20:46:39.181 DEBUG:teuthology.orchestra.run.smithi159:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T20:46:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-10T20:46:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44158.log 2024-09-10T20:46:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70317.log 2024-09-10T20:46:39.256 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42224.log 2024-09-10T20:46:39.256 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47362.log 2024-09-10T20:46:39.256 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.42224.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-10T20:46:39.256 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42224.log.gz 2024-09-10T20:46:39.256 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46829.log 2024-09-10T20:46:39.256 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.48348.log 2024-09-10T20:46:39.256 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43218.log 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.47362.log: /var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-client.1.46829.log: /var/log/ceph/ceph-client.1.48348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47358.log 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47362.log.gz 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46829.log.gz 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.48348.log.gz 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.43218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43218.log.gz 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42219.log 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.47358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46332.log 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47358.log.gz 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.42219.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.1.42219.log.gz 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr: -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.46332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45367.log 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46332.log.gz 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.48344.log 2024-09-10T20:46:39.257 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.45367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44823.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.45367.log.gz 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.48344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.48344.log.gz 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.44823.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44823.log.gz 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45363.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43741.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.45363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36378.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.45363.log.gz 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.43741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44283.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43741.log.gz 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36458.log 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36378.log.gz 2024-09-10T20:46:39.258 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.44283.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44283.log.gz 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45849.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36458.log.gz 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44819.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.45849.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.1.45849.log.gz 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr: --verbose -- /var/log/ceph/ceph-client.1.44287.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.44819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44819.log.gz 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43737.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.44287.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.1.44287.log.gz 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr: --verbose -- /var/log/ceph/ceph-client.1.46825.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.43737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43737.log.gzgzip 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.46825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47862.log 2024-09-10T20:46:39.259 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46825.log.gz 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.47862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47862.log.gz 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45845.log 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr: 93.8%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46328.log 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.45845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.45845.log.gz 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.46328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46328.log.gz 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mds.d.log: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T20:46:39.260 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36338.log 2024-09-10T20:46:39.262 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-10T20:46:39.262 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36338.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.36338.log.gz 2024-09-10T20:46:39.276 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43214.log 2024-09-10T20:46:39.290 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36298.log 2024-09-10T20:46:39.290 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.43214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43214.log.gz 2024-09-10T20:46:39.308 INFO:teuthology.orchestra.run.smithi159.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-10T20:46:39.308 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36254.log 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36298.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36298.log.gz 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42619.log 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36254.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36254.log.gz 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36498.log 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.3.42619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.42619.log.gz 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47866.log 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36498.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36498.log.gz 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42623.log 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.1.47866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47866.log.gz 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36207.log 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.3.42623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.42623.log.gz 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36418.log 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36207.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36207.log.gz 2024-09-10T20:46:39.309 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.36418.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36418.log.gz 2024-09-10T20:46:39.379 INFO:teuthology.orchestra.run.smithi159.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-10T20:46:39.397 INFO:teuthology.orchestra.run.smithi159.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-10T20:46:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60256.log 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70317.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.44158.log.gz 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36858.log 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70317.log.gz 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74070.log 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60256.log: /var/log/ceph/ceph-client.admin.36858.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60256.log.gz 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81588.log 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr: 75.8% -- replaced with /var/log/ceph/ceph-client.admin.36858.log.gz 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.74070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74070.log.gz 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66373.log 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.81588.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58746.log 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66373.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.81588.log.gz -5 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.71583.log 2024-09-10T20:46:39.525 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66373.log.gz 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72613.log 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58746.log.gz 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45988.log 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.71583.log.gz 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51015.log 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72613.log.gz 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55869.log 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45988.log.gz 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55911.log 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51015.log.gz 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36802.log 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55869.log.gz 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73546.log 2024-09-10T20:46:39.526 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55911.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70527.log 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr: 77.5% -- replaced with /var/log/ceph/ceph-client.admin.36802.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76149.log 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73546.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47485.log 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70527.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65738.log 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76149.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80575.log 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47485.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78950.log 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65738.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61334.log 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.80575.log.gz 2024-09-10T20:46:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53810.log 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78950.log.gz 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48762.log 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61334.log.gz 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75525.log 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53810.log.gz 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45778.log 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48762.log.gz 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46638.log 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75525.log.gz 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76598.log 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45778.log.gz 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79160.log 2024-09-10T20:46:39.528 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46638.log.gz 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73420.log 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76598.log.gz 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72403.log 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79160.log.gz 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76363.log 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73420.log.gz 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79833.log 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72403.log.gz 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.76363.log: --verbose -- /var/log/ceph/ceph-client.0.67742.log 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79833.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76363.log.gz 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66144.log 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79833.log.gz 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.67742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69196.log 2024-09-10T20:46:39.529 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67742.log.gz 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81024.log 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66144.log.gz 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81318.log 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69196.log.gz 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47390.log 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81318.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81024.log.gz 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69154.log 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.81318.log.gz 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56380.log 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47390.log.gz 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70017.log 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69154.log.gz 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58533.log 2024-09-10T20:46:39.530 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56380.log.gz 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59884.log 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70017.log.gz 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55743.log 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58533.log.gz 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78497.log 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59884.log.gz 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79502.log 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55743.log.gz 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51904.log 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78497.log.gz 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.79502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61800.log 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79502.log.gz 2024-09-10T20:46:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78820.log 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51904.log.gz 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66779.log 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61800.log.gz 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76910.log 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66779.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78820.log.gz 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57741.log 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66779.log.gz 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50745.log 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76910.log.gz 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69070.log 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57741.log: /var/log/ceph/ceph-client.admin.50745.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57741.log.gzgzip 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47346.log 2024-09-10T20:46:39.532 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50745.log.gz 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74702.log 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69070.log.gz 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47346.log.gz 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62322.log 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74702.log.gz 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62979.log 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44160.log 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62322.log.gz 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36522.log 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.62979.log.gz 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67950.log 2024-09-10T20:46:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44160.log: /var/log/ceph/ceph-client.admin.36522.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44160.log.gz 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51397.log 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr: 77.4% -- replaced with /var/log/ceph/ceph-client.admin.36522.log.gz 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58475.log 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67950.log.gz 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61555.log 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51397.log.gz 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52990.log 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61555.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58475.log.gz 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52888.log 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.61555.log.gz 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77841.log 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52990.log.gz 2024-09-10T20:46:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42380.log 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52888.log.gz 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74236.log 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77841.log.gz 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79580.log 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42380.log.gz 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75273.log 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74236.log.gz 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66289.log 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79580.log.gz 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60214.log 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.75273.log.gz 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51215.log 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60214.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66289.log.gz 2024-09-10T20:46:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55262.log 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60214.log.gz 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42546.log 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51215.log.gz 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73336.log 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55262.log.gz 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60435.log 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42546.log.gz 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81546.log 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73336.log.gz 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67746.log 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.60435.log.gz 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65456.log 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81546.log.gz 2024-09-10T20:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.67746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51625.log 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67746.log.gz 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45820.log 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65456.log.gz 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51625.log.gz 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77182.log 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45820.log.gz 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71296.log 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77182.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46722.log 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77182.log.gz 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46261.log 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.71296.log.gz 2024-09-10T20:46:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42802.log 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46722.log.gz 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68986.log 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46261.log.gz 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43822.log 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42802.log.gz 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73504.log 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68986.log.gz 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52028.log 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43822.log.gz 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69467.log 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73504.log.gz 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81504.log 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52028.log.gz 2024-09-10T20:46:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69425.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69467.log.gz 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47569.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81504.log.gz 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65269.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69425.log.gz 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36606.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47569.log.gz 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65269.log: /var/log/ceph/ceph-client.admin.36606.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65269.log.gz 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43823.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.36606.log.gz 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65330.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75189.log 2024-09-10T20:46:39.539 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43823.log.gz 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43180.log 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65330.log.gz 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61376.log 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75189.log: /var/log/ceph/ceph-client.admin.43180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66883.log 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.75189.log.gz 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43180.log.gz 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63719.log 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61376.log.gz 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46183.log 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66883.log: /var/log/ceph/ceph-client.admin.63719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50451.log 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66883.log.gz 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63719.log.gz 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74515.log 2024-09-10T20:46:39.540 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46183.log.gz 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65625.log 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50451.log: /var/log/ceph/ceph-client.admin.74515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46806.log 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50451.log.gz 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.74515.log.gz 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76952.log 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.65625.log.gz 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75357.log 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46806.log: /var/log/ceph/ceph-client.admin.76952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74194.log 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46806.log.gz 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76952.log.gz 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47909.log 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75357.log.gz 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78578.log 2024-09-10T20:46:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76790.log 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47909.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74194.log.gz 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44058.log 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78578.log.gz 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.47909.log.gz 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.76790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67554.log 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.76790.log.gz 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51563.log 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44058.log: /var/log/ceph/ceph-client.admin.67554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43138.log 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44058.log.gz 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67554.log.gz 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60519.log 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51563.log.gz 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72655.log 2024-09-10T20:46:39.542 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43138.log: /var/log/ceph/ceph-client.admin.60519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78280.log 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43138.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.60519.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49589.log 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72655.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67320.log 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78280.log: /var/log/ceph/ceph-client.admin.49589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69341.log 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78280.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67320.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49589.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52170.log 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67320.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77715.log 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69341.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43480.log 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52170.log.gz 2024-09-10T20:46:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48347.log 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77715.log.gz 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65517.log 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43480.log.gz 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59478.log 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48347.log: /var/log/ceph/ceph-client.admin.65517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69635.log 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48347.log.gz 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.65517.log.gz 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65583.log 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.59478.log.gz 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49845.log 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69635.log: /var/log/ceph/ceph-client.admin.65583.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69635.log.gz 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80940.log 2024-09-10T20:46:39.544 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65583.log.gz 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59436.log 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49845.log: /var/log/ceph/ceph-client.admin.80940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62636.log 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49845.log.gz 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80940.log.gz 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57621.log 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59436.log.gz 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56210.log 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.62636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62636.log.gz 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56517.log 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56210.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57621.log.gz 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49202.log 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56210.log.gz 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63075.log 2024-09-10T20:46:39.545 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56517.log.gz 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76107.log 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49202.log.gz 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73714.log 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63075.log.gz 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78070.log 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76107.log: /var/log/ceph/ceph-client.admin.73714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70359.log 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76107.log.gz 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73714.log.gz 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50018.log 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78070.log.gz 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71964.log 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70359.log: /var/log/ceph/ceph-client.admin.50018.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70359.log.gz 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53337.log 2024-09-10T20:46:39.546 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50018.log.gz 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75784.log 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71964.log: /var/log/ceph/ceph-client.admin.53337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58872.log 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr: 84.2% -- replaced with /var/log/ceph/ceph-client.admin.71964.log.gz 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53337.log.gz 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69593.log 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75784.log.gz 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49108.log 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58872.log.gz 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70827.log 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69593.log: /var/log/ceph/ceph-client.admin.49108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79718.log 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69593.log.gz 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49108.log.gz 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.70827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50409.log 2024-09-10T20:46:39.547 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.70827.log.gz 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77673.log 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79718.log: /var/log/ceph/ceph-client.admin.50409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59343.log 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79718.log.gz 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.50409.log.gz 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53116.log 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77673.log.gz 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56336.log 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55785.log 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53116.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59343.log.gz 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56336.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53116.log.gz 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56773.log 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56336.log.gz 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42083.log 2024-09-10T20:46:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55785.log: /var/log/ceph/ceph-client.admin.56773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65372.log 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55785.log.gz 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.56773.log.gz 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44159.log 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42083.log.gz 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76321.log 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65372.log: /var/log/ceph/ceph-client.admin.44159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46928.log 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.65372.log.gz 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44159.log.gz 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81066.log 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76321.log.gz 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46428.log 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46928.log: /var/log/ceph/ceph-client.admin.81066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70823.log 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46928.log.gz 2024-09-10T20:46:39.549 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81066.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57949.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46428.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53722.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.70823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.70823.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62196.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57949.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47615.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53722.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51501.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49929.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47615.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62196.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51501.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47615.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66458.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51501.log.gz 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76237.log 2024-09-10T20:46:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49929.log: /var/log/ceph/ceph-client.admin.66458.log: gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.49929.log.gz --verbose 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.72487.log 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66458.log.gz 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49764.log 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76237.log.gz 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69677.log 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72487.log: /var/log/ceph/ceph-client.admin.49764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77757.log 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72487.log.gz 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49764.log.gz 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49505.log 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69677.log.gz 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45599.log 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77757.log: /var/log/ceph/ceph-client.admin.49505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54146.log 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77757.log.gz 2024-09-10T20:46:39.551 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49505.log.gz 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.2.45599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49684.log 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45599.log.gz 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57463.log 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54146.log: /var/log/ceph/ceph-client.admin.49684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60072.log 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49684.log.gz 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48607.log 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54146.log.gz 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.52701.log 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57463.log.gz 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.60072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.60072.log.gz 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51141.log 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48607.log.gz 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63033.log 2024-09-10T20:46:39.552 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52701.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58830.log 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51141.log: /var/log/ceph/ceph-client.admin.63033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69238.log 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51141.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58830.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63033.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44566.log 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58830.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45946.log 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69238.log: /var/log/ceph/ceph-client.admin.44566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58433.log 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69238.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44566.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46030.log 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45946.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49066.log 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58433.log: /var/log/ceph/ceph-client.admin.46030.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58433.log.gz 2024-09-10T20:46:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50111.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46030.log.gz 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69551.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49066.log: /var/log/ceph/ceph-client.admin.50111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51439.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49066.log.gz 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50111.log.gz 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58662.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69551.log.gz 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42284.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51439.log.gz 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58296.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58662.log: /var/log/ceph/ceph-client.admin.42284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76489.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58662.log.gz 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42284.log.gz 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36466.log 2024-09-10T20:46:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77382.log 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58296.log.gz/var/log/ceph/ceph-client.admin.76489.log: 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36466.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76489.log.gz 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66679.log 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr: 76.8% -- replaced with /var/log/ceph/ceph-client.admin.36466.log.gz 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65414.log 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77382.log: /var/log/ceph/ceph-client.admin.66679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68405.log 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.77382.log.gz 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66679.log.gz 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56605.log 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.65414.log.gz 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36345.log 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68405.log.gz 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45128.log 2024-09-10T20:46:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56605.log: /var/log/ceph/ceph-client.admin.36345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80517.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56605.log.gz 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr: 79.4% -- replaced with /var/log/ceph/ceph-client.admin.36345.log.gz 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.45128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45128.log.gz 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66500.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62112.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80517.log: /var/log/ceph/ceph-client.admin.66500.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80517.log.gz 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62238.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66500.log.gz 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81630.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62112.log: /var/log/ceph/ceph-client.admin.62238.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62112.log.gz 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81108.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.62238.log.gz 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67488.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81630.log: /var/log/ceph/ceph-client.admin.81108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79034.log 2024-09-10T20:46:39.556 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81630.log.gz 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81108.log.gz 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73756.log 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50283.log 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.67488.log.gz 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79034.log.gz 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48898.log 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3%/var/log/ceph/ceph-client.admin.50283.log: -- replaced with /var/log/ceph/ceph-client.admin.73756.log.gz 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42590.log 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50283.log.gz 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76405.log 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50153.log 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48898.log: /var/log/ceph/ceph-client.admin.42590.log: /var/log/ceph/ceph-client.admin.76405.log: 87.4% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42590.log.gz -- replaced with /var/log/ceph/ceph-client.admin.48898.log.gz 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61758.log 2024-09-10T20:46:39.558 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76405.log.gz 2024-09-10T20:46:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44524.log 2024-09-10T20:46:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50153.log: /var/log/ceph/ceph-client.admin.61758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43054.log 2024-09-10T20:46:39.558 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50153.log.gz 2024-09-10T20:46:39.558 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61758.log.gz 2024-09-10T20:46:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63117.log 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44524.log.gz 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46512.log 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43054.log.gz 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70485.log 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63117.log: /var/log/ceph/ceph-client.admin.46512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76994.log 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.63117.log.gz 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70485.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46512.log.gz 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73588.log 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70485.log.gz 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43824.log 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76994.log: /var/log/ceph/ceph-client.admin.73588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58254.log 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76994.log.gz 2024-09-10T20:46:39.560 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73588.log.gz 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61940.log 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50367.log 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43824.log.gz 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58254.log: /var/log/ceph/ceph-client.admin.61940.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58254.log.gz 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81880.log 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61940.log.gz 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73205.log 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50367.log: /var/log/ceph/ceph-client.admin.81880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79887.log 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50367.log.gz 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81880.log.gz 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53764.log 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73205.log.gz 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71800.log 2024-09-10T20:46:39.561 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.79887.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46554.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53764.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36690.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.71800.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43222.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77941.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36690.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46554.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36690.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71841.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77941.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43222.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56128.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77941.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59089.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71841.log: /var/log/ceph/ceph-client.admin.56128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47783.log 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.71841.log.gz 2024-09-10T20:46:39.562 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56128.log.gz 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50937.log 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59089.log.gz 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47741.log 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47783.log: /var/log/ceph/ceph-client.0.50937.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47783.log.gz 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.0.50937.log.gz -5 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44059.log 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55572.log 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47741.log: /var/log/ceph/ceph-client.admin.44059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56983.log 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.47741.log.gz 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44059.log.gz 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61124.log 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55572.log: /var/log/ceph/ceph-client.admin.56983.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55572.log.gz 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52617.log 2024-09-10T20:46:39.563 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56983.log.gz 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77883.log 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61124.log: /var/log/ceph/ceph-client.admin.52617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73864.log 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61124.log.gz 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52617.log.gz 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80746.log 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77883.log.gz 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68321.log 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73864.log: /var/log/ceph/ceph-client.admin.80746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43306.log 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73864.log.gz 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.80746.log.gz 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68321.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.68321.log.gz 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75063.log 2024-09-10T20:46:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43306.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55701.log 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43306.log.gz 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73462.log 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75063.log.gz 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67446.log 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55701.log.gz 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57699.log 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73462.log.gz 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78769.log 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67446.log: /var/log/ceph/ceph-client.admin.57699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36438.log 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.67446.log.gz 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57699.log.gz 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69852.log 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78769.log.gz 2024-09-10T20:46:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79498.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36438.log: 76.8% -- replaced with /var/log/ceph/ceph-client.admin.36438.log.gz/var/log/ceph/ceph-client.admin.69852.log: 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52948.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69852.log.gz 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.79498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70275.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79498.log.gz 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr: 89.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52948.log: /var/log/ceph/ceph-client.admin.70275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52294.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52948.log.gz 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.70275.log.gz 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70191.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75483.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52294.log: /var/log/ceph/ceph-client.admin.70191.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.53032.log 2024-09-10T20:46:39.566 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52294.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70191.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.75483.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.71542.log 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53032.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75483.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75704.log 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53032.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69383.log 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.71542.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77799.log 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75704.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78904.log 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69383.log.gz 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71031.log 2024-09-10T20:46:39.567 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77799.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54887.log 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78904.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54524.log 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71031.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65874.log 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54887.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57547.log 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54524.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75949.log 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65874.log: /var/log/ceph/ceph-client.admin.57547.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65874.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47136.log 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57547.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53936.log 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75949.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75949.log.gz 2024-09-10T20:46:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58704.log 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50637.log 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53936.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47136.log.gz 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58704.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53936.log.gz 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61635.log 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58704.log.gz 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57421.log 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61635.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50637.log.gz 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50069.log 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61635.log.gz 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36662.log 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57421.log: /var/log/ceph/ceph-client.admin.50069.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57421.log.gz 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50069.log.gz 2024-09-10T20:46:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77631.log 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr: 78.6% -- replaced with /var/log/ceph/ceph-client.admin.36662.log.gz 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72177.log 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60853.log 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77631.log.gz 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59734.log 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72177.log: /var/log/ceph/ceph-client.admin.60853.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72177.log.gz 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54766.log 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60853.log.gz 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51293.log 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59734.log.gz 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47867.log 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54766.log.gz 2024-09-10T20:46:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61889.log 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51293.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51293.log.gz 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72781.log 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77573.log 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47867.log.gz 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61889.log: /var/log/ceph/ceph-client.admin.72781.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61889.log.gz 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30301.log 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72781.log.gz 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50199.log 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77573.log: /var/log/ceph/ceph.tmp-client.admin.30301.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30301.log.gz 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67068.log 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77573.log.gz 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59650.log 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.50199.log.gz 2024-09-10T20:46:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71624.log 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67068.log.gz 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81714.log 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59650.log: /var/log/ceph/ceph-client.admin.71624.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.59650.log.gz -5 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78662.log 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.71624.log.gz 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58391.log 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81714.log.gz 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59171.log 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47262.log 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58391.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78662.log.gz 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59171.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58391.log.gz 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46596.log 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.59171.log.gz -- 2024-09-10T20:46:39.572 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.74576.log/var/log/ceph/ceph-client.admin.47262.log: 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47262.log.gz 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55659.log 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46596.log.gz 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80620.log 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74576.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74576.log.gz 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81276.log 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55659.log.gz 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76786.log 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80620.log: /var/log/ceph/ceph-client.admin.81276.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80620.log.gz 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48546.log 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.81276.log.gz 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56475.log 2024-09-10T20:46:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.76786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.76786.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56475.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48546.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71091.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56475.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59776.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69933.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.71091.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68737.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59776.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61081.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69933.log: /var/log/ceph/ceph-client.admin.68737.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69933.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45736.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.68737.log.gz 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67194.log 2024-09-10T20:46:39.574 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61081.log.gz 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68527.log 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45736.log.gz 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63243.log 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67194.log: /var/log/ceph/ceph-client.admin.68527.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67194.log.gz 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59255.log 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68527.log.gz 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79310.log 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63243.log: /var/log/ceph/ceph-client.admin.59255.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63243.log.gz 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70233.log 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59255.log.gz 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49253.log 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7%/var/log/ceph/ceph-client.admin.70233.log: -- replaced with /var/log/ceph/ceph-client.admin.79310.log.gz 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81756.log 2024-09-10T20:46:39.575 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70233.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54929.log 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81756.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49253.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49337.log 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81756.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54971.log 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54929.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60479.log 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49337.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65215.log 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54971.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54020.log 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.60479.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81234.log 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65215.log.gz 2024-09-10T20:46:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63591.log 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54020.log.gz 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53158.log 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81234.log.gz 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50595.log 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53158.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.63591.log.gz 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75315.log 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53158.log.gz 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59524.log 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50595.log.gz 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66542.log 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75315.log.gz 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73247.log 2024-09-10T20:46:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59524.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59524.log.gz 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67278.log 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66542.log.gz 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73630.log 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73247.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73247.log.gz 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57907.log 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72319.log 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67278.log.gz 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73630.log: /var/log/ceph/ceph-client.admin.57907.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.73630.log.gz 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55013.log 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57907.log.gz 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47527.log 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72319.log: /var/log/ceph/ceph-client.admin.55013.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72319.log.gz 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-10T20:46:39.578 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55013.log.gz 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75105.log 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47527.log: /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79244.log 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47527.log.gz 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49150.log 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75105.log.gz 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69975.log 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54062.log 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49150.log.gz 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61250.log 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79244.log.gz 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69975.log.gz/var/log/ceph/ceph-client.admin.54062.log: 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60711.log 2024-09-10T20:46:39.579 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54062.log.gz 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59299.log 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61250.log.gz 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53242.log 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60711.log.gz 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51966.log 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59299.log.gz 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56815.log 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53242.log: /var/log/ceph/ceph-client.admin.51966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48667.log 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53242.log.gz 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51966.log.gz 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77495.log 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56815.log.gz -5 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49024.log 2024-09-10T20:46:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48667.log: /var/log/ceph/ceph-client.admin.77495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70569.log 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48667.log.gz 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77495.log.gz 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72277.log 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49024.log.gz 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61292.log 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70569.log: /var/log/ceph/ceph-client.admin.72277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55430.log 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72277.log.gz 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58914.log 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.70569.log.gz 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61292.log.gz 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49463.log 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55430.log.gz 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71882.log 2024-09-10T20:46:39.581 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.58914.log.gz 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61166.log 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49463.log: /var/log/ceph/ceph-client.admin.71882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57325.log 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49463.log.gz 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.71882.log.gz 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55346.log 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0%/var/log/ceph/ceph-client.0.57325.log: -- replaced with /var/log/ceph/ceph-client.admin.61166.log.gz 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53894.log 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.57325.log.gz 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47657.log 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55346.log.gz 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47434.log 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53894.log.gz 2024-09-10T20:46:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49421.log 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47657.log.gz 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61021.log 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47434.log.gz 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37329.log 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49421.log.gz 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74358.log 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61021.log.gz 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37329.log.gz 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69028.log 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43817.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74358.log.gz 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49547.log 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2024-09-10T20:46:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66247.log 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69028.log.gz 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71923.log 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49547.log.gz 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59818.log 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66247.log.gz 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43487.log 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.71923.log.gz 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79118.log 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81462.log 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.59818.log.gz 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43487.log.gz 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75441.log 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.79118.log.gz 2024-09-10T20:46:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81360.log 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81462.log.gz 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75441.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75441.log.gz 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47699.log 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81360.log: /var/log/ceph/ceph-client.admin.43483.log: gzip -5 87.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.81360.log.gz 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.55304.log 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80263.log 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47699.log.gz 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49974.log 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55304.log.gz 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71337.log 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80263.log: /var/log/ceph/ceph-client.admin.49974.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80263.log.gz 2024-09-10T20:46:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62070.log 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49974.log.gz 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45133.log 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.71337.log.gz 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73798.log 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.62070.log.gz 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.45133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62640.log 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45133.log.gz 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71759.log 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.62640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52418.log 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62640.log.gz 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71759.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.73798.log.gz 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36578.log 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.71759.log.gz 2024-09-10T20:46:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52418.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52418.log.gz 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53978.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr: 78.2% -- replaced with /var/log/ceph/ceph-client.admin.36578.log.gz 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71255.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73292.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53978.log.gz/var/log/ceph/ceph-client.admin.71255.log: 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46848.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.71255.log.gz 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58338.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73292.log.gz 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48504.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46848.log.gz 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54212.log 2024-09-10T20:46:39.587 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58338.log.gz 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71173.log 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.48504.log.gz 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54404.log 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54212.log: /var/log/ceph/ceph-client.admin.71173.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54212.log.gz 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65172.log 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.71173.log.gz/var/log/ceph/ceph-client.0.54404.log: 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54404.log.gz 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52533.log 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43812.log 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr: 86.5%/var/log/ceph/ceph-client.admin.52533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74412.log 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65172.log.gz 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52533.log.gz 2024-09-10T20:46:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57845.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.43812.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60769.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74412.log: /var/log/ceph/ceph-client.admin.57845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66969.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74412.log.gz 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71419.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57845.log.gz 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66969.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60769.log.gz 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70443.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71419.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66969.log.gz 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72529.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.71419.log.gz 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49295.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42844.log 2024-09-10T20:46:39.589 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.70443.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49295.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75021.log 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72529.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49295.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57803.log 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42844.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75865.log 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75021.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48940.log 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57803.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68928.log 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75865.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48940.log: gzip -5 87.2% --verbose -- /var/log/ceph/ceph-client.admin.44060.log 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48940.log.gz 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45862.log 2024-09-10T20:46:39.590 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68928.log.gz 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42760.log 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45862.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44060.log.gz 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80205.log 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45862.log.gz 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55126.log 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42760.log.gz 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72361.log 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6%/var/log/ceph/ceph-client.admin.55126.log: -- replaced with /var/log/ceph/ceph-client.admin.80205.log.gz 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46680.log 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55126.log.gz 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44156.log 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72361.log.gz/var/log/ceph/ceph-client.admin.46680.log: 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68695.log 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46680.log.gz 2024-09-10T20:46:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75567.log 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36402.log 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44156.log.gz 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68695.log: /var/log/ceph/ceph-client.admin.75567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81798.log 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68695.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75567.log.gz 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80475.log 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36402.log: /var/log/ceph/ceph-client.admin.81798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43486.log 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: 78.8% -- replaced with /var/log/ceph/ceph-client.admin.36402.log.gz 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81798.log.gz/var/log/ceph/ceph-client.admin.80475.log: 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47220.log 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80475.log.gz 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66416.log 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43486.log.gz 2024-09-10T20:46:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73002.log 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47220.log.gz 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54566.log 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66416.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71677.log 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73002.log: /var/log/ceph/ceph-client.admin.54566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63761.log 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73002.log.gz 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54566.log.gz 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52785.log 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71677.log: /var/log/ceph/ceph-client.admin.63761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65916.log 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: 84.3% 86.9% -- replaced with /var/log/ceph/ceph-client.admin.63761.log.gz 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68363.log 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71677.log.gz 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52785.log.gz 2024-09-10T20:46:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50241.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65916.log.gz 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68363.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68363.log.gz 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62028.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50241.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78196.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50241.log.gz 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47975.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62028.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77986.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78196.log: /var/log/ceph/ceph-client.admin.47975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37418.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78196.log.gz 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47975.log.gz/var/log/ceph/ceph-client.admin.77986.log: 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47304.log 2024-09-10T20:46:39.594 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77986.log.gz 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.37418.log: --verbose -- /var/log/ceph/ceph-client.admin.80032.log 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47304.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37418.log.gz 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66584.log 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47304.log.gz 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48167.log 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80032.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80032.log.gz -5 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74979.log 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66584.log: gzip/var/log/ceph/ceph-client.0.48167.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55388.log 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48167.log.gz 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74979.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66584.log.gz 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74979.log.gz 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46470.log 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63837.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55388.log.gz 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46470.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46470.log.gz 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.65976.log 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63837.log: gzip -5 --verbose -- 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63837.log.gz 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.68447.log 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66018.log 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65976.log.gz 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67236.log 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.68447.log.gz 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66331.log 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66018.log.gz 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45904.log 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74642.log 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5%/var/log/ceph/ceph-client.admin.66331.log: -- replaced with /var/log/ceph/ceph-client.admin.67236.log.gz 2024-09-10T20:46:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45904.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66331.log.gz 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45904.log.gz 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78620.log 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70149.log 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74642.log.gz/var/log/ceph/ceph-client.admin.78620.log: 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73672.log 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78620.log.gz 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.70149.log: -- /var/log/ceph/ceph-client.admin.75147.log 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73672.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70149.log.gz 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47825.log 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73672.log.gz 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53498.log 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55514.log 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75147.log.gz 2024-09-10T20:46:39.597 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47825.log: gzip/var/log/ceph/ceph-client.admin.53498.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.69772.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53498.log.gz 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55514.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47825.log.gz 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55514.log.gz 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54482.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62719.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69772.log.gz 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54400.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54482.log.gz 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.62719.log: -- /var/log/ceph/ceph-client.admin.60340.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.54400.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62719.log.gz 0.0% 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.0.54400.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63159.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51687.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60340.log: /var/log/ceph/ceph-client.admin.63159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73378.log 2024-09-10T20:46:39.598 INFO:teuthology.orchestra.run.smithi028.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.60340.log.gz 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63159.log.gz/var/log/ceph/ceph-client.admin.51687.log: 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51687.log.gz 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.42928.log 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73378.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73378.log.gz 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.81838.log 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42928.log: gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42928.log.gz 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-mgr.y.log 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69509.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81838.log.gz 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43479.log 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80982.log 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69509.log.gz 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42634.log 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43479.log.gz 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52659.log 2024-09-10T20:46:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42634.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.80982.log.gz 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42634.log.gz 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47178.log 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75994.log 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52659.log: /var/log/ceph/ceph-client.admin.47178.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52659.log.gz 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47178.log.gz 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43264.log 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70947.log 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75994.log: /var/log/ceph/ceph-client.admin.43264.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75994.log.gz 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43264.log.gz 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74074.log 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72823.log 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.74074.log: /var/log/ceph/ceph-client.admin.70947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74074.log.gz 2024-09-10T20:46:39.600 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70947.log.gz 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76447.log 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73163.log 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72823.log.gz 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36746.log 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73163.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76447.log.gz 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73163.log.gz 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80433.log 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72865.log 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36746.log: /var/log/ceph/ceph-client.admin.80433.log: 77.0% -- replaced with /var/log/ceph/ceph-client.admin.36746.log.gz 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48163.log 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80433.log.gz 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78417.log 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72865.log.gz 2024-09-10T20:46:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.48163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48163.log.gz 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63658.log 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43475.log 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78417.log.gz 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36634.log 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43475.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63658.log.gz 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43475.log.gz 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70905.log 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71132.log 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36634.log.gz 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67866.log 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70905.log.gz 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67404.log 2024-09-10T20:46:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67866.log: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.71132.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67866.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69112.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67404.log: /var/log/ceph/ceph-client.admin.69112.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67404.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69112.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80127.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71501.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.80127.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80127.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51335.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71501.log: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.71501.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56294.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51335.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51335.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67152.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56294.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56294.log.gz 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43096.log 2024-09-10T20:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67152.log: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67152.log.gz 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60076.log 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71718.log 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43096.log: 87.2%/var/log/ceph/ceph-client.0.60076.log: -- replaced with /var/log/ceph/ceph-client.admin.43096.log.gz 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.60076.log.gz 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66925.log 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67362.log 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.71718.log.gz 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58788.log 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67362.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66925.log.gz 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67362.log.gz 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36550.log 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60979.log 2024-09-10T20:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58788.log: /var/log/ceph/ceph-client.admin.36550.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58788.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36550.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75231.log 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81672.log 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60979.log: /var/log/ceph/ceph-client.admin.75231.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60979.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75231.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79676.log 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60895.log 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81672.log: /var/log/ceph/ceph-client.admin.79676.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81672.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79676.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58578.log 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56559.log 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58578.log: /var/log/ceph/ceph-client.admin.60895.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58578.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60895.log.gz 2024-09-10T20:46:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45603.log 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43488.log 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56559.log.gz 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.2.45603.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70635.log 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.2.45603.log.gz 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43488.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43488.log.gz 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43816.log 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72739.log 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70635.log.gz 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62448.log 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72739.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43816.log.gz 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72739.log.gz 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44440.log 2024-09-10T20:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.62448.log: /var/log/ceph/ceph-client.admin.80305.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62448.log.gz 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53417.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80305.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44440.log.gz 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80305.log.gz 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59692.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80704.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53417.log.gz 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53582.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80704.log: 86.6% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.80704.log.gz 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59692.log.gz 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61208.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62154.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53582.log.gz 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59009.log 2024-09-10T20:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62154.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61208.log.gz 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62154.log.gz 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60154.log 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72907.log 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59009.log: /var/log/ceph/ceph-client.admin.60154.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59009.log.gz 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36494.log 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60154.log.gz 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72907.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72907.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62280.log 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55204.log 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36494.log: /var/log/ceph/ceph-client.admin.62280.log: 77.1% -- replaced with /var/log/ceph/ceph-client.admin.36494.log.gz 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62280.log.gz 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51842.log 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58620.log 2024-09-10T20:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55204.log.gz 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43012.log 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58620.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51842.log.gz 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58620.log.gz 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67110.log 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74879.log 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43012.log: /var/log/ceph/ceph-client.admin.67110.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43012.log.gz 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51099.log 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67110.log.gz 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74879.log: /var/log/ceph/ceph-client.admin.51099.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74879.log.gz 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67908.log 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.51099.log.gz 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67013.log 2024-09-10T20:46:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67908.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67908.log.gz 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79948.log 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56689.log 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67013.log: /var/log/ceph/ceph-client.admin.79948.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67013.log.gz 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79948.log.gz 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74278.log 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47093.log 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56689.log.gz 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60633.log 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47093.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74278.log.gz 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47093.log.gz 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75399.log 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66202.log 2024-09-10T20:46:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60633.log: /var/log/ceph/ceph-client.admin.75399.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60633.log.gz 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75399.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74937.log 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66841.log 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74937.log: /var/log/ceph/ceph-client.admin.66202.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74937.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66202.log.gz 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70107.log 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66841.log: /var/log/ceph/ceph-client.admin.70107.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66841.log.gz 2024-09-10T20:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70107.log.gz 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69290.log 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59608.log 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69290.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69290.log.gz 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52356.log 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54104.log 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59608.log: 87.5%/var/log/ceph/ceph-client.admin.52356.log: -- replaced with /var/log/ceph/ceph-client.admin.59608.log.gz 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52356.log.gz 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53671.log 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55472.log 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54104.log.gz 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74152.log 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55472.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53671.log.gz 2024-09-10T20:46:39.612 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55472.log.gz 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75907.log 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74152.log: /var/log/ceph/ceph-client.admin.44608.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74152.log.gz 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78862.log 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78112.log 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75907.log: /var/log/ceph/ceph-client.admin.78862.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75907.log.gz 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56252.log 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78862.log.gz 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80349.log 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78112.log: /var/log/ceph/ceph-client.admin.56252.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78112.log.gz 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56252.log.gz 2024-09-10T20:46:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51771.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70401.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80349.log: /var/log/ceph/ceph-client.admin.51771.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80349.log.gz 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51771.log.gz 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56857.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66060.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70401.log: /var/log/ceph/ceph-client.admin.56857.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70401.log.gz 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56857.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53074.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44356.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66060.log: /var/log/ceph/ceph-client.admin.53074.log: 87.2% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66060.log.gz 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53074.log.gz 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59394.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78992.log 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44356.log: /var/log/ceph/ceph-client.admin.59394.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44356.log.gz 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59394.log.gz 2024-09-10T20:46:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80662.log 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58042.log 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78992.log: /var/log/ceph/ceph-client.admin.80662.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78992.log.gz 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37128.log 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80662.log.gz 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56647.log 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58042.log: /var/log/ceph/ceph-client.admin.37128.log: 86.8% 86.5% -- replaced with /var/log/ceph/ceph-client.admin.58042.log.gz 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37128.log.gz 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53200.log 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78028.log 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56647.log: /var/log/ceph/ceph-client.admin.53200.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56647.log.gz 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53200.log.gz 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62382.log 2024-09-10T20:46:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52743.log 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78028.log: /var/log/ceph/ceph-client.admin.62382.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.78028.log.gz 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78707.log 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9%/var/log/ceph/ceph-client.admin.52743.log: -- replaced with /var/log/ceph/ceph-client.admin.62382.log.gz 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52743.log.gz 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71460.log 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68850.log 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78707.log.gz 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42718.log 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.71460.log.gz 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49379.log 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42718.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68850.log.gz 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42718.log.gz 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57025.log 2024-09-10T20:46:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.49379.log: -- /var/log/ceph/ceph-client.admin.79202.log 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49379.log.gz 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61460.log 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79202.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57025.log.gz 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79202.log.gz 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45292.log 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50933.log 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61460.log.gz 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54608.log 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.50933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50933.log.gz 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45292.log.gz 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76868.log 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61982.log 2024-09-10T20:46:39.617 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54608.log.gz 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48840.log 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61982.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76868.log.gz 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.61982.log.gz 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76279.log 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62761.log 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48840.log: /var/log/ceph/ceph-client.admin.76279.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48840.log.gz 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76279.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68267.log 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72697.log 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68267.log: /var/log/ceph/ceph-client.admin.62761.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62761.log.gz 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.68267.log.gz 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79633.log 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56048.log 2024-09-10T20:46:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72697.log: /var/log/ceph/ceph-client.admin.79633.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.72697.log.gz 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79633.log.gz 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58212.log 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77316.log 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56048.log: /var/log/ceph/ceph-client.admin.58212.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56048.log.gz 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58212.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60298.log 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79076.log 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60298.log: /var/log/ceph/ceph-client.admin.77316.log: 87.1% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60298.log.gz 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77316.log.gz 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71214.log 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57321.log 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79076.log.gz 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71378.log 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.57321.log: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.71214.log.gz 2024-09-10T20:46:39.619 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.57321.log.gz 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75609.log 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44157.log 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.71378.log.gz 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42970.log 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44157.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75609.log.gz 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44157.log.gz 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59566.log 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61845.log 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42970.log: /var/log/ceph/ceph-client.admin.59566.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42970.log.gz 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59566.log.gz 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78238.log 2024-09-10T20:46:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53540.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61845.log: /var/log/ceph/ceph-client.admin.78238.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61845.log.gz 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67824.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78238.log.gz 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56424.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67824.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53540.log.gz 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67824.log.gz 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62803.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.56424.log: -- /var/log/ceph/ceph-client.admin.73082.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56424.log.gz 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61418.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73082.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62803.log.gz 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73082.log.gz 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77236.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47051.log 2024-09-10T20:46:39.621 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61418.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55953.log 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47051.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77236.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47051.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80391.log 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55953.log: /var/log/ceph/ceph-client.admin.60811.log 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55953.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55617.log 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60811.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80391.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60811.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56941.log 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52090.log 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55617.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81402.log 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52090.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56941.log.gz 2024-09-10T20:46:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52090.log.gz 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81402.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81402.log.gz 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54705.log 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57133.log 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50553.log: /var/log/ceph/ceph-client.admin.54705.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54705.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52232.log 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46351.log 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57133.log: /var/log/ceph/ceph-client.admin.52232.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57133.log.gz 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52575.log 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52232.log.gz 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60380.log 2024-09-10T20:46:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52575.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46351.log.gz 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52575.log.gz 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74798.log 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57505.log 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60380.log: /var/log/ceph/ceph-client.admin.74798.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.60380.log.gz 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81192.log 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74798.log.gz 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68593.log 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57505.log.gz 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81192.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47009.log 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81192.log.gz 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60937.log 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.68593.log.gz 2024-09-10T20:46:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50493.log 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60937.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60937.log.gz 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47009.log.gz 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72099.log 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48245.log 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.50493.log.gz 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48982.log 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48245.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72099.log.gz 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48245.log.gz 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81150.log 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56731.log 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48982.log.gz 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42886.log 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56731.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81150.log.gz 2024-09-10T20:46:39.625 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56731.log.gz 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53852.log 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42886.log: /var/log/ceph/ceph-client.admin.49887.log 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42886.log.gz 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55827.log 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49887.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53852.log.gz 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49887.log.gz 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48389.log 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51057.log 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55827.log: /var/log/ceph/ceph-client.admin.48389.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55827.log.gz 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70989.log 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48389.log.gz 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68635.log 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51057.log: /var/log/ceph/ceph-client.admin.70989.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70989.log.gz 2024-09-10T20:46:39.626 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51057.log.gz 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43820.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43814.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.68635.log.gz 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43820.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43814.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43820.log.gz 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43814.log.gz 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61716.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52842.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.61716.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61716.log.gz 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76038.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52842.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52842.log.gz 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76191.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76038.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76038.log.gz 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66102.log 2024-09-10T20:46:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76191.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76191.log.gz 2024-09-10T20:46:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74454.log 2024-09-10T20:46:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66102.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66102.log.gz 2024-09-10T20:46:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-10T20:46:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74454.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74454.log.gz 2024-09-10T20:46:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46764.log 2024-09-10T20:46:39.629 INFO:teuthology.orchestra.run.smithi159.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-10T20:46:39.629 INFO:teuthology.orchestra.run.smithi159.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-10T20:46:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36718.log 2024-09-10T20:46:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46764.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46764.log.gz 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57067.log 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36718.log.gz 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56899.log 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57067.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57067.log.gz 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50325.log 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56899.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56899.log.gz 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36830.log 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50325.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50325.log.gz 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59213.log 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36830.log: 78.6% -- replaced with /var/log/ceph/ceph-client.admin.36830.log.gz 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63201.log 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59213.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59213.log.gz 2024-09-10T20:46:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63549.log 2024-09-10T20:46:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63201.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63201.log.gz 2024-09-10T20:46:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72005.log 2024-09-10T20:46:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63549.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63549.log.gz 2024-09-10T20:46:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44788.log 2024-09-10T20:46:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72005.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.72005.log.gz 2024-09-10T20:46:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78322.log 2024-09-10T20:46:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44788.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44788.log.gz 2024-09-10T20:46:39.659 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65816.log 2024-09-10T20:46:39.660 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78322.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78322.log.gz 2024-09-10T20:46:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48305.log 2024-09-10T20:46:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65816.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65816.log.gz 2024-09-10T20:46:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72445.log 2024-09-10T20:46:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48305.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48305.log.gz 2024-09-10T20:46:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53627.log 2024-09-10T20:46:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72445.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72445.log.gz 2024-09-10T20:46:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72571.log 2024-09-10T20:46:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53627.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53627.log.gz 2024-09-10T20:46:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46072.log 2024-09-10T20:46:39.717 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72571.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72571.log.gz 2024-09-10T20:46:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78154.log 2024-09-10T20:46:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54808.log 2024-09-10T20:46:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78154.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78154.log.gz 2024-09-10T20:46:39.739 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46072.log.gz 2024-09-10T20:46:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2024-09-10T20:46:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54808.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.54808.log.gz 2024-09-10T20:46:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76532.log 2024-09-10T20:46:39.767 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44398.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2024-09-10T20:46:39.776 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72235.log 2024-09-10T20:46:39.785 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42676.log 2024-09-10T20:46:39.785 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72235.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72235.log.gz 2024-09-10T20:46:39.789 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.76532.log.gz 2024-09-10T20:46:39.793 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62845.log 2024-09-10T20:46:39.807 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42676.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42676.log.gz 2024-09-10T20:46:39.812 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70063.log 2024-09-10T20:46:39.813 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62845.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62845.log.gz 2024-09-10T20:46:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70063.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.70063.log.gz 2024-09-10T20:46:39.875 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-10T20:46:39.897 INFO:teuthology.orchestra.run.smithi028.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-10T20:46:39.900 INFO:teuthology.orchestra.run.smithi159.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-10T20:46:39.927 INFO:teuthology.orchestra.run.smithi159.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-10T20:46:39.964 INFO:teuthology.orchestra.run.smithi159.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-10T20:46:40.007 INFO:teuthology.orchestra.run.smithi159.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-10T20:46:40.009 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-09-10T20:46:40.009 INFO:teuthology.orchestra.run.smithi159.stderr:real 0m0.792s 2024-09-10T20:46:40.009 INFO:teuthology.orchestra.run.smithi159.stderr:user 0m3.725s 2024-09-10T20:46:40.009 INFO:teuthology.orchestra.run.smithi159.stderr:sys 0m0.163s 2024-09-10T20:46:40.028 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-10T20:46:40.068 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-10T20:46:40.220 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-10T20:46:40.237 INFO:teuthology.orchestra.run.smithi028.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-10T20:46:40.315 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-10T20:46:40.422 INFO:teuthology.orchestra.run.smithi028.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-10T20:46:40.424 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T20:46:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m1.230s 2024-09-10T20:46:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m6.243s 2024-09-10T20:46:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.672s 2024-09-10T20:46:40.425 INFO:tasks.ceph:Archiving logs... 2024-09-10T20:46:40.425 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/remote/smithi028/log 2024-09-10T20:46:40.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T20:46:41.723 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/remote/smithi159/log 2024-09-10T20:46:41.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T20:46:42.264 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T20:46:42.273 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T20:46:42.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T20:46:42.277 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T20:46:42.383 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T20:46:42.393 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T20:46:42.394 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:46:42.396 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:46:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:46:42.413 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:46:42.418 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:46:42.418 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-09-10T20:46:42.418 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 -458us[ -600us] +/- 52ms 2024-09-10T20:46:42.418 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 +2133us[+1991us] +/- 68ms 2024-09-10T20:46:42.418 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -1363us[-1363us] +/- 62ms 2024-09-10T20:46:42.418 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:46:42.418 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:46:42.419 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-10T20:46:42.419 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 -524us[ -613us] +/- 52ms 2024-09-10T20:46:42.419 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 +2068us[+2068us] +/- 68ms 2024-09-10T20:46:42.419 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -1441us[-1441us] +/- 62ms 2024-09-10T20:46:42.419 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:46:42.419 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T20:46:42.428 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T20:46:42.428 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T20:46:42.454 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T20:46:42.556 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T20:46:42.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:46:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:46:42.660 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:46:42.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:46:42.689 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T20:46:42.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T20:46:42.738 INFO:teuthology.task.internal:Duration was 946.338754 seconds 2024-09-10T20:46:42.738 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T20:46:42.771 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T20:46:42.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T20:46:42.774 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T20:46:42.814 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:46:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:46:43.150 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T20:46:43.150 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T20:46:43.150 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T20:46:43.172 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-09-10T20:46:43.172 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T20:46:43.202 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T20:46:43.202 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T20:46:43.215 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T20:46:43.269 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T20:46:43.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T20:46:43.283 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T20:46:43.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T20:46:43.431 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T20:46:43.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T20:46:43.457 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T20:46:43.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T20:46:43.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:46:43.500 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:46:43.522 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-09-10T20:46:43.545 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-09-10T20:46:43.558 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:46:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:46:43.592 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:46:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:46:43.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T20:46:43.623 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T20:46:43.624 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/remote/smithi028 2024-09-10T20:46:43.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T20:46:43.669 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899622/remote/smithi159 2024-09-10T20:46:43.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T20:46:43.703 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T20:46:43.703 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T20:46:43.706 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T20:46:43.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T20:46:43.764 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T20:46:43.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T20:46:43.779 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T20:46:43.780 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T20:46:43.782 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T20:46:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 20:46 /home/ubuntu/cephtest 2024-09-10T20:46:43.808 INFO:teuthology.orchestra.run.smithi159.stdout: 265861 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 20:46 /home/ubuntu/cephtest 2024-09-10T20:46:43.810 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T20:46:43.819 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T20:46:43.883 INFO:teuthology.run:Summary data: description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} duration: 946.3387544155121 flavor: default owner: scheduled_hyelloji@teuthology success: true 2024-09-10T20:46:43.883 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:46:43.976 INFO:teuthology.run:pass